ColdFusion Debugging on Production, That wasn't a Good Idea

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases Iwe saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

Waiting on VERY Large Data Transfers

What do you do while waiting for very large data transfers to complete? By large I'm talking upwards of Terabytes. My mind wanders and suddenly the theme for Rawhide gets stuck in my mind! But it's different. The music is similar, but the words have changed. It's almost like Dust Puppy was singing the song.

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes

Keep Flowin', Flowin', Flowin'
Though the datastreams are swollen
Keep them bytes Flowin', Raw Bytes
Through switches and wires and routers
Hell bent for bandwidth

Wishin' my smartphone was by my side
All the things I'm missin'
Good cat videos, Netflix and tweetin'
Are waiting at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Keep Flowin', Flowin', Flowin'
Though they're disapprovin'
Keep them bytes Flowin', Raw Bytes
Don't try to understand 'em
Just grep 'em, zip, and save 'em

Soon we'll be streamin' high and wide
My smartphone calculatin'
My true love will be tweetin'
Be waitin' at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes
Raw Bytes

The things we do while migrating data into the cloud.

It's "Retired" Jim!

In another chapter of "The Cloud Never Crashes", I woke up Sunday to one of my AWS instances that was 'crashed' with a notice of "Amazon EC2 Instance scheduled for retirement". Retirement? What does that mean? I went to check my email and realized that the "retired" instance was the email server. Doh! It took me a little while to figure out what they meant. It means this "An instance is scheduled to be retired when AWS detects irreparable failure of the underlying hardware hosting the instance." This serves as a good reminder that the cloud is really someone else's server.

In theory this is an easy fix. The instructions at Amazon claims that stopping and restarting the instance will launch it on new hardware. In practice I could not get the instance to stop. This is where having physical hardware and a power cord to pull would have been nice. Failing to get the instance to stop I could not detach the EBS root volume. Even force detaching the EBS root volume didn't work. This is where daily snapshots of EBS volumes comes in handy. I was able to launch a new EC2 instance and then convert the last snapshot to an EBS volume and attach that to the new EC2 instance. Then I moved the elastic IP from the "Retired" instance to the new instance and hit "start'. Full recovery!

Now I'm left with a hanging EC2 instance that is still "Stopping" and an EBS volume that I cannot use, detach, delete etc. I tried reissuing stop commands a couple times. Eventually I noticed a "Force Stop" option. I do not remember seeing this on earlier attempts. I do not know if this shows up after the first failed stopped attempt or after several. I'm not sure, but I think that sends a trained monkey into the datacenter to pull the power cord. In any case it worked. This let me detach my EBS volume. From there was was able to stop the new instance, detach the EBS volume and attach my original EBS root volume. Now I have full recovery and I was able to clean up the loose ends.

Amazon Web Service has given us a new euphemism. Retired means It's Dead Jim!

CF Webtools is an Amazon Web Services Partner. Our Operations Group can build, manage, and maintain your AWS services. We also handle migration of physical servers into AWS Cloud services. If you are looking for professional AWS management our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion Bloggers Tweets

When I brought ColdFusionBloggers.og back to life, several of us were discussing the service on the CFML Slack channel. One of the many things that was mentioned was that some blogs these days are static and don't have a 'ping' service to ping an aggregator. That led to the venerable Sean Corfield mentioning that he does not read a blog post if he does not see it on his Twitter feed. I suspect others get their updates this was as well. I'm still 'old school' I guess because I still use and online news reader. I don't have a solution yet for those static blogs, but I'm percolating thoughts on that. However, during the process of resurrecting ColdFusionBloggers.og I noticed that at one time there was a Twitter account for @CFBloggers. I checked with Ray Camden and luckily he was able to remember the credentials for this account and passed those on to me.

The current iteration of ColdFusionBloggers.og is coded in Node.js. I've been learning what I can about Node as fast as I can. This past weekend started the process with Twitter to get my API tokens. While waiting on those I learned what I needed to get Node to send a Tweet. Today I received the Twitter API access tokens and this evening I was able to debug my hacked up Node code. The first tweet was sent! WooHoo! I think this is all working.

The short story is all new blog posts aggregated by ColdFusionBloggers.og will now be tweeted in real time. This being the first blog post to tweet. Cheers!

I hope this is the first of many changes for ColdFusionBloggers.og. Contact me to have your ColdFusion related blog added to the feed.

ColdFusion Bloggers is Alive

A short note to alert everyone that ColdFusionBloggers.org is back online and back to aggregating your blog posts.

The great Raymond Camden created this awesome resource and technically it will always be his. I happen to be the current caretaker if you will of this valuable service. If anyone was wondering why Ray decided to step away from ColdFusion Bloggers, see his blog post here. I reached out to Ray and he granted me the 'keys' so to speak. Thank you Ray! I've moved hosting to my AWS account for hosting and tonight I was able to work through the hiccups of migrating older code to a newer environment on newer versions of the backend software. I just completed adding SSL via Lets Encrypt and verified the ping service works. ColdFusion Bloggers is back in action!

Spread the word! Add your blog!

Living Longer Than Dad

For the past two years I've been contemplating turning 50. It's not that 50 is scary or some huge significant number to me like it is for some. To the contrary I've been looking forward to reaching that significant number, because being 50 is something that for a long time I thought I'd never be. My father died of a massive heart attack at the age of 48. I was 28 at the time.

Over the next 20 years I've been married, divorced, changed careers, changed jobs numerous times, become reasonably successful in my new career, basically all the things people do. During that time it was always in my head that Dad didn't make it past the age of 48. Somewhere in there it set into my mind that I wouldn't make it past 48 myself. I know it's irrational. It's nonsensical. Its crazy talk. There no scientific way to determine how long you will live based on how long your father lived. Yet there it was, embedded in my mind, that my time was over by the age of 48. So much so that I was living my life not expecting to live longer.

Two years ago, when my age hit that number, I made a trip up north into the woods near the edge of the BWCA where we spread Dad's ashes. It had been twenty years since I was there last and it had been twenty years since I had a chance to sit with Dad, talk, have a drink, listen to his words. I went with two things, a flask of his favorite scotch and a cigar. After hiking in for about a mile I found the granite outcropping in the river where we spread his ashes. For the next hour or so I sat there smoking a cigar, taking a sip of scotch from the flask and pouring a shot from the flask into the river. After all, it's been twenty years since he had a drink. I sat here listening for his words. Trying to find peace of mind.

Two years later and two years older than my father I'm still listening for his words and trying to find my way in this world. Trying to find some peace in my mind. I don't know where life goes from here or how long life will last. I heard something a while back that inspired me to reflect a little and thus write my thoughts down today. "Be grateful for the extra innings."

USPS Shipping API Ending TLS 1.1 and TLS 1.0 Support, is your ColdFusion Server Ready?

At CF Webtools we recently went through a round of server upgrades to handle the Authorize.net ending support for older TLS versions. Now USPS, United State Postal Service, is doing the same thing with their Shipping APIs. This is going to be happening for all API's and most likely all this year as PCI requirements for ending support for TLS 1.1 and older at the end of June 2018. This is according to the PCI Security Standards Council.

USPS will be turning off support for TLS 1.1 and older for testing. In advance of the changes to production, TLS version 1.0 and 1.1 support will be discontinued in the lower Web Tools environments and available for testing on 5/22/18: https://stg-secure.shippingapis.com/shippingapi.dll): 06/11/18.

This message explains some security improvements planned for our services. Effective 06/22/18, Web Tools will discontinue support of Transport Layer Security (TLS) version 1.0 and 1.1 for securing connections to our HTTPS APIs through the following URL: https://stg-secure.shippingapis.com/shippingapi.dll. This includes, but is not limited to, all shipping label and package pickup APIs. After this change, integrations leveraging TLS version 1.0 and 1.1 will fail when attempting to access the APIs.

You are receiving this message because the Web Tools UserID associated with your email address has made HTTPS requests over the past year. It is possible that no changes are necessary to retain Web Tools services and benefit from the improvements. Please review the entire message carefully and share with your web developer, software vendor, or IT service provider to determine if your use of the Web Tools APIs will be affected. If you have already updated your security certificates please disregard this message. If you are not sure if any changes are necessary, please ask your IT service provider.

In advance of the changes to production, TLS version 1.0 and 1.1 support will be discontinued in the lower Web Tools environments and available for testing on 5/22/18: https://stg-secure.shippingapis.com/shippingapi.dll): 06/11/18.

Further background: Security research published in recent years demonstrated that TLS version 1.0 and 1.1 contained weaknesses that limited its ability to protect and secure communications. These weaknesses have been addressed in the TLS 1.2 version. Major browser software vendors have been supporting TLS 1.2 for some time. Consistent with our priority to protect USPS Web Tools customers, Web Tools will only support versions of the more modern TLS 1.2 as of the effective date noted above.

Contact us at WebTools@usps.gov with any questions or concerns.

This means that if you are using older methods to make calls to USPS that are not capable of making TLS 1.2 connections then you will NOT be able to process Shipping API transactions.

This affects ALL ColdFusion versions 9.0.2 and older! This also affects ColdFusion 10 Update 17 and older. If your server is running any of these older versions of ColdFusion and your server is processing Shipping API transactions with USPS then this advisory applies to your server.

Mitigation

Getting compliant depends on age of your server operating system. There are three main ways to get your server to handle TLS 1.2.

  1. If you're running on Windows Server 2008 Standard (not R2) or older then the only solution is to migrate to a newer server. This can be challenging and time consuming. It's best to start planning now if a plan isn't already in place and being acted upon.
  2. If your server is running ColdFusion 10 and newer on Windows Server 2008 R2 or newer then the solution is most likely very simple. In most cases you'll need to install the ColdFusion patches and upgrade to Java 1.8.0_nn.
  3. There is a solution for the in between systems running ColdFusion 9 and older on Windows 2008 R2. This does require using a third party extension to ColdFusion and some refactoring of your code to call the API.
  4. There are sure to be outlier cases that will require either migration or patching depending on the exact combination of operating system, ColdFusion version and Java version.

CF Webtools has been successfully mitigating this issue for clients servers for the past couple years and we are very experienced in resolving these security related issues. In a previous blog post I tested which TLS levels were supported by various ColdFusion versions on various Java versions and produced an easy to read chart.

If your ColdFusion server is affected by this or if you do not know if your ColdFusion server is affected by this then please contact us (much) sooner than later. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

Get Your Beta Here! ColdFusion 2018 Beta!

Adobe has announced the Public Beta of Adobe ColdFusion 2018 is now available. This release brings an all new Performance Monitoring Toolset that is available with both the Standard and Enterprise versions (So I've been told). There's plenty of language improvements and updates and a new Public Beta of ColdFusion Builder 2018. Hurry up while supplies last!

There a large number of changes including an all new ColdFusion Administrator. Here's a partial list of new things according to Adobe:

  1. ColdFusion (2018 release) has a new User Interface. The new interface is based on a tiled interface. We have also enriched the search experience on the Administrator portal.
  2. We have removed Server Monitor. We have introduced a tool called Performance Monitoring Toolset, which is more intuitive, includes more features, and provides better visibility of your application's performance.
  3. We have made significant improvements to the core language features. Here is a brief list of the changes:
    1. Introduced NULL support
    2. Introduced closures in tags
    3. Introduced Asynchronous programming using Future
    4. Enhanced Object-Oriented Programming with the following:
      1. Abstract components and methods
      2. Final component, method, and variable
      3. Default functions in interfaces
      4. Covariance
    5. Semi-colons are now optional in a cfscript code
    6. Introduced named parameters in functions
    7. Introduced slicing in arrays
    8. New operator support using name-spaces for java, webservices, dotnet com, corba, and cfc
    9. Introduced support for typed arrays
    10. Introduced string literals and support for numeric member functions
    11. Introduced negative indices support for arrays
    12. New functions- ArrayFirst, Arraylast, QueryDeleteColumn, and QueryDeleteRow
  4. Enhanced CLI and introduced REPL.
  5. Introduced REST Playground application for testing your REST APIs.
  6. Added support for REST PATCH verb.
  7. Filter fields from JSON request.
  8. Enhance performance through Caching with the newly added engines:
    1. Memcached
    2. JCS
    3. Redis
    4. Using a custom cache plugin
  9. New Admin APIs to support the caching engines
  10. Hibernate upgraded to ver 5.2
  11. New configuration settings in wsconfig tool
  12. Updates to ColdFusion Builder.

This is a huge update! Get it while it's hot!

ColdFusion Security updates for ColdFusion 2016 and ColdFusion 11

Adobe released important security updates and big fixes today, update 6 and update 14 for ColdFusion 2016 and ColdFusion 11 respectively.

These updates resolve an important insecure library loading vulnerability (CVE-2018-4938), an important cross-site scripting vulnerability that could lead to code injection (CVE-2018-4940) and an important cross-site scripting vulnerability that could lead to information disclosure (CVE-2018-4941). These updates also include a mitigation for a critical unsafe Java deserialization vulnerability (CVE-2018-4939) and a mitigation for a critical unsafe XML parsing vulnerability (CVE-2018-4942).

There is a bug of great importance to many that has finally been fixed. I've blogged about this before and I was able to create a work around to resolve this issue until it was fixed by Adobe. The SFTP/FTPS bug would not allow connections to secure FTP servers that utilized newer SSL protocols. When using CFFTP to connect to some S-FTP server, during connection, you can see an error message. This has been a growing issue as more and more companies replace plain text FTP servers with SFTP or FTPS servers that utilize stronger protocols.

For ColdFusion 2016 this update upgrades Tomcat to version 8.5.28 and OpenSSL to version 1.0.2n.

For ColdFusion 11 this update upgrades Tomcat to version 7.0.85 and OpenSSL to version 1.0.2n.

The security updates referenced in the above Tech Notes require JDK 8u121 or higher (for ColdFusion 2016) and JDK 7u131 or JDK 8u121 (for ColdFusion 11).

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

*Note: ColdFusion 11 when it was first released came with a version of Java 1.7.0_nn. Adobe later re-released ColdFusion 11 with Java 1.8.0_25. If you have ColdFusion 11 still running on Java 1.7 I highly recommend that Java be upgraded to Java 1.8. Oracle is no longer supporting Java 1.7 and 1.7 is long past it's end of life. Even though the Adobe instructions for this current security update states that you can run Java 1.7.0_131, I highly recommend upgrading to Java 1.8. Personally I will not install Java 1.7 on a clients servers and sign off on it being 'secure'.

ColdFusion SFTP and FTPS Secure Connection Failure

I have seen a lot more people asking questions about making SFTP or FTPS secure connections from ColdFusion using the <CFFTP> tag. They are trying to figure out why they cannot make a connection. Often the error is "Algorithm negotiation fail" or "Connection Error". People are posting their questions on many support forums including Adobes forums and their new ColdFusion Community Portal. This is a problem people are experiencing in ColdFusion 10 and ColdFusion 11.

In the last few years we've seen a huge shift in SSL/TLS security including the removal of older less secure protocols and forcing secure connections to use the newer stronger protocols with stronger TLS certificates and stronger encryption cyphers. As such older systems need to be upgraded to handle the newer security protocols. More recently plain old unsecure FTP portals have been the focus of change to SFTP or FTPS.

At CF Webtools we've run into this same problem several times with multiple clients. It was so much of a problem that I needed to spend some dedicated time to see how we could resolve this issue.

The first thing I discovered is that this issue is a known "bug" that has been reported to Adobe. It's been a long known issue and somehow the fix which is in ColdFusion 2016 has not been included in an update for earlier ColdFusion versions. However, Adobe has affirmed to me that this fix is scheduled for an upcoming update.

Because it was fixed in ColdFusion 2016 I was able to inspect the included jar files to see if the one that handles CFFTP or secure communications was newer than the one(s) in ColdFusion 11. What I found is that jsch-0.1.44m.jar had been replaced by jsch-0.1.52m.jar. The JSCH jar library is the library that handles Java Secure Channel communications. "JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs."

After seeing this was upgraded I had an ah-ha moment and figured it was worth a try to copy this newer version into my ColdFusion 11 test server and see what happened. The new version is in ./ColdFusion2016/cfusion/lib folder. You can download the free ColdFusion 2016 Developer Edition and install it anywhere so you can get access to the updated jar file. Once you have the new jar file copy it into ColdFusion 11. The proper way to do this is to remove or rename the old jar file version in your ColdFusion11/cfusion (or instance name)/lib folder then copy the new jar file version into the same folder. Then start or restart ColdFusion 11. That's it. You're done. The bug is fixed and you're good to go with SFTP or FTPS using <CFFTP> in ColdFusion 11.

This is not an approved fix from Adobe. I do not know if there is some unknown issue that could be created by doing this. However, I do know that everyone I've talked to that has tried this has had their secure FTP issues resolved. Additionally I have not tried this 'fix' in ColdFusion 10. However, if you are running into this issue with ColdFusion 10 it's worth the minimal effort to give it a try.

If you need someone to make this change on your ColdFusion server then contact us, we can help. CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

More Entries