What to do when it goes wrong.

This past weekend Coldfusion Muse (A.K.A. Mark Kruger) at CF Webtools emailed me to get my help with a situation that a client brought to us. A ColdFusion 8.0.1 Standard server was not working after an attempted patch upgrade. After researching the issues a bit and reading tech notes at Adobe's website I came up with a plan to get the server back online. Here was the full situation I found myself in:

  • A 32 bit Linux server with CF8.0.1 Standard
  • ColdFusion server was running but would not get data from a DB
  • No previous knowledge of the server or anything installed on it
  • No documentation or notes on what if any patches had been applied
  • No information on what items had or had not been working prior to the attempted patch
  • The client tried to apply a patch and then tried to 'roll back' the patch

This is sort of “situation normal” for me when I get called to fix an emergency issue. So, what do YOU do when you get confronted with this? These steps should help you get your ColdFusion server patched and back online after a failed patch attempt. (We hope!)

First of all I was confronted with an error I had not seen before “coldfusion/runtime/QueryTableWrapper”. No fear, Google to the rescue. Odds are you are not the first person to ever see a particular error message and I know I have not seen them all, so Google it, or Bing it or Yodel on Yahoo. Usually you will find results and sometimes those results will yield something useful. In this case something useful did come up. Apparently when you try to apply the “Security Hotfix | ColdFusion 8, 8.0.1, 9, 9.0.1” patch it needs this file “cfmx_bootstrap.jar” from HS4 or you get that error.

So now I have a very possible reason and the client just emailed me back and confirmed that yes, they had tried to apply that Security Hotfix. Does this mean I can just copy over that correct version of that file and say 'there ya go, have a nice day!' or is there something else I should look at? Maybe, but how do I know? Someone is paying for me to fix the problem and I have to know it's fixed or I may have to go back and fix it right later for free.

A patch attempt failed, I do not know at what point if failed. If this one file was not patched, I don't know what other files are patched, not patched etc. I need to leave this server in a “known state”. Having most things unknown, I decided the best way to get to a known state was to roll everything forward, not backwards, because I don't know the previous state of the server. Also, as I looked around at the files on the server, I saw nothing to indicate any other patches had been attempted. Nothing in the {ColdFusion-Home}/libs/updates folder.

Recently there had been a few good posts on patching ColdFusion server to the latest patch level. Charlie Arehart has an excellent post with details on all the ColdFusion patches. I used this for reference while fixing this server. Adobe posts Tech Notes for each Hotfix or Security update. Each post contains detailed instructions on how to install the patch in question. It is absolutely CRITICAL to follow ALL the steps. Be warned, file paths and folder names are different between Standard and the Enterprise installation. It's possible someone can get confused. The Adobe instructions contain the correct paths and filenames. Be careful when going through them.

Back to the server. I did not have a known state and I decided to install all the Adobe Hotfixes, Updates and Security patches so I could bring the server to a known state. The first thing I wanted to do was not do more damage. So I ran a backup of the /opt/coldfusion8 folder. This way I have a full reference point if I need.

When I referenced Charlie's post, I saw a long list of updates for ColdFusion 8.0.1. I went to each Adobe Tech Note and read in detail the instructions for installing each. After doing so I found that HF1 gets completely uninstalled and replaced by HF2. So no need to install HF1. HF3 completely uninstalls HF2 so HF2 is not needed. However, there are new steps for HF3. Several manual steps beside just uploading the new .jar file via the CFAdmin. It is critical that you do these steps exactly. There are several jar files to upload to the server and copied in place. HF4 also has a lot of manual steps and HS4 has several additional security fixes that need to be installed. Apply them, all of them.

Then install the Security Update. Do every step here as well, even though you just did some of those in the previous HS4 updates. There are a few changes and the Tech Note says to re-apply these fixes even if you already did them before. As I reread the instructions for the Security Hotfix there is no mention of it requiring HF4. But I think it is required. I think this is what tripped up this client. As I tried to get more information from the tech that tried to do the patch I found out he was just given a link to the security update page and told to do that update. It looks like he did everything he was supposed to do. What the Tech Note for the Security Update does not mention is that it requires the updated “cfmx_bootstrap.jar” file from HF4. To test this theory I applied the latest Security Update to my Dev Edition of ColdFusion 8.0.1 and ran into the same issue and the exact same error message. I had to apply HS4 to make the Security Hotfix work. So this is what most likely tripped up the client and may trip you up as well. HS4 is needed in order to apply the latest Security Hotfix.

Lessons:

  1. Always keep your ColdFusion install updated
  2. If you didn't, when do you update it, start at the beginning. Don't just install the latest update because as we saw in this case the newest update requires an older update and as best as I can tell this is not documented.
  3. Find a 'Known state'. Even if you don't know the state of the installation, you can create a reference point (backup of the install folder) and start from there.
  4. If you don't know the previous good state to roll back to, then go forward and create a known good state.
  5. Documentation is critical. When I finished I gave them an email detailing what I had updated. A server should be documented so that the next person that has to work on it knows the state of the server. 

I hope this helps you get your ColdFusion server patched or re-patched and back to a good known state.

 

This entry was posted on March 7, 2011 at 3:57 PM and has received 2729 views. There are currently 2 comments. Print this entry.