ColdFusion SFTP and FTPS Secure Connection Failure

I have seen a lot more people asking questions about making SFTP or FTPS secure connections from ColdFusion using the <CFFTP> tag. They are trying to figure out why they cannot make a connection. Often the error is "Algorithm negotiation fail" or "Connection Error". People are posting their questions on many support forums including Adobes forums and their new ColdFusion Community Portal. This is a problem people are experiencing in ColdFusion 10 and ColdFusion 11.

In the last few years we've seen a huge shift in SSL/TLS security including the removal of older less secure protocols and forcing secure connections to use the newer stronger protocols with stronger TLS certificates and stronger encryption cyphers. As such older systems need to be upgraded to handle the newer security protocols. More recently plain old unsecure FTP portals have been the focus of change to SFTP or FTPS.

At CF Webtools we've run into this same problem several times with multiple clients. It was so much of a problem that I needed to spend some dedicated time to see how we could resolve this issue.

The first thing I discovered is that this issue is a known "bug" that has been reported to Adobe. It's been a long known issue and somehow the fix which is in ColdFusion 2016 has not been included in an update for earlier ColdFusion versions. However, Adobe has affirmed to me that this fix is scheduled for an upcoming update.

Because it was fixed in ColdFusion 2016 I was able to inspect the included jar files to see if the one that handles CFFTP or secure communications was newer than the one(s) in ColdFusion 11. What I found is that jsch-0.1.44m.jar had been replaced by jsch-0.1.52m.jar. The JSCH jar library is the library that handles Java Secure Channel communications. "JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs."

After seeing this was upgraded I had an ah-ha moment and figured it was worth a try to copy this newer version into my ColdFusion 11 test server and see what happened. The new version is in ./ColdFusion2016/cfusion/lib folder. You can download the free ColdFusion 2016 Developer Edition and install it anywhere so you can get access to the updated jar file. Once you have the new jar file copy it into ColdFusion 11. The proper way to do this is to remove or rename the old jar file version in your ColdFusion11/cfusion (or instance name)/lib folder then copy the new jar file version into the same folder. Then start or restart ColdFusion 11. That's it. You're done. The bug is fixed and you're good to go with SFTP or FTPS using <CFFTP> in ColdFusion 11.

This is not an approved fix from Adobe. I do not know if there is some unknown issue that could be created by doing this. However, I do know that everyone I've talked to that has tried this has had their secure FTP issues resolved. Additionally I have not tried this 'fix' in ColdFusion 10. However, if you are running into this issue with ColdFusion 10 it's worth the minimal effort to give it a try.

If you need someone to make this change on your ColdFusion server then contact us, we can help. CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

Connect ColdFusion JDBC to Sybase SQL Anywhere

This is something that might not come up often, but every once in a while we have to connect to a Sybase database. This is a built in feature in the Enterprise version of ColdFusion. However, if you have the Standard version of ColdFusion you have to manually add the JDBC jar file and build the connection string by hand. This is easy to do once you have the correct information and correct format of the connection string. Finding that correct information was nearly impossible and required a lot of trial and error.

Here's the case we had to resolve at CF Webtools. One of our clients has been using ColdFusion and Sybase for ages. For the record this is Sybase SQL Anywhere 16. For those that are not aware SAP owns Sybase thus the official name is SAP SQL Anywhere 16. For the longest time they were using ODBC connectors and older versions of ColdFusion on older Windows servers. More recently they have upgraded to ColdFusion 11 on newer Windows servers and were still trying to make the connections to Sybase via ODBC. This is a large multi-tenant operation in which there are hundreds of databases on the Sybase servers. Yes, plural servers. There are two servers that are replicated and handle failover. This means the ColdFusion Datasource connection also needs to handle failover. With ODBC failover is handled by Microsoft ODBC settings. With JDBC we had to setup failover in the JDBC connection string.

[More]

TLS1.2 for ColdFusion 9 and Older

The upcoming Authorize.NET switch to using TLS 1.2 only has a lot of people scrambling to get their servers updated. This has been a long planned transition at Authorize.NET and at many/most/all other payment processing companies. The inevitable facts are that TLS 1.0 and TLS 1.1 are outdated and they are going away. At CF Webtools we have been preparing for this inevitable day for the past few years.

ColdFusion 9.0.n is not tested to work on Java 1.8 and I have had cases were certain features of ColdFusion 9 did not work with Java 1.8. I have not tried any older versions of ColdFusion on Java 1.8 and I'm not going to. Adobe has not certified any versions of ColdFusion older than version 10 Update 14 (or ColdFusion 11 Update 2 and older). All of that being said, there is a workaround that uses a 3rd party commercial solution to make TLS 1.2 connections from ColdFusion 9. It works well, but I do not recommend that as a long term solution. The preferred long term solution is upgrading the server(s) and ColdFusion version to currently supported versions. This way there will be security updates to help protect against new threats. The commercial third-party CFX tag will require recoding the CFHTTP calls for the new CFX tag. The tag is CFX_HTTP5 and it is available here.

Follow the installation instructions that comes with the download and then you will have to recode your CFHTTP calls similar to the examples below. The code examples are for the older Authorize.NET Advanced Integration Method (AIM) API calls that you are most likely using in your older ColdFusion CFHTTP calls.

view plain print about
1<cfset authURL = "https://test.authorize.net/gateway/transact.dll" />
2 <cfif AuthNetMode eq "live">
3 <cfset authURL = "https://secure.authorize.net/gateway/transact.dll" />
4 </cfif>
5
6<!--- CFHTTP Call - Your code might look something like this --->
7<cfhttp url="#authURL#" method="post" result="cfhttp">
8 <cfhttpparam type="FORMFIELD" name="x_Login" value="#AuthLogin#">
9 <cfhttpparam type="FORMFIELD" name="x_Password" value="#AuthPassword#">
10 <cfhttpparam type="FORMFIELD" name="x_merchant_email" value="#AuthEmail#">
11 <cfhttpparam type="FORMFIELD" name="x_delim_data" value="true">
12 <cfhttpparam type="FORMFIELD" name="x_test_request" value="#x_test_request#">
13
14 <!--- we're using AUTH_ONLY so the card isn't charged until the order is processed --->
15 <cfhttpparam type="FORMFIELD" name="x_type" value="AUTH_ONLY">
16 <cfhttpparam type="FORMFIELD" name="x_method" value="cc">
17
18 <cfhttpparam type="FORMFIELD" name="x_amount" value="#orderTotal#">
19 <cfhttpparam type="FORMFIELD" name="x_card_num" value="#cardNumber#">
20 <cfhttpparam type="FORMFIELD" name="x_exp_date" value="#cardExpiration#">
21 <cfif isDefined("cardSecurityCode") and cardSecurityCode eq "">
22 <cfhttpparam type="FORMFIELD" name="x_card_code" value="#cardSecurityCode#">
23 </cfif>
24
25 <!--- If you want an email to go to the customer via authorize.net
26change this to true. Make sure authorize.net is configured properly. --->

27 <cfhttpparam type="FORMFIELD" name="x_email_customer" value="#x_email_customer#">
28
29 <cfhttpparam type="FORMFIELD" name="x_first_name" value="#billingFirstName#">
30 <cfhttpparam type="FORMFIELD" name="x_last_name" value="#billingLastName#">
31 <cfhttpparam type="FORMFIELD" name="x_company" value="#billingCompany#">
32 <cfhttpparam type="FORMFIELD" name="x_address" value="#billingAddress#">
33 <cfhttpparam type="FORMFIELD" name="x_city" value="#billingCity#">
34 <cfhttpparam type="FORMFIELD" name="x_state" value="#billingState#">
35 <cfhttpparam type="FORMFIELD" name="x_zip" value="#billingZip#">
36 <cfhttpparam type="FORMFIELD" name="x_country" value="#billingCountry#">
37
38 <cfhttpparam type="FORMFIELD" name="x_customer_ip" value="#cgi.remote_address#">
39 <cfhttpparam type="FORMFIELD" name="x_Email" value="#billingEmail#">
40 <cfhttpparam type="FORMFIELD" name="x_Phone" value="#billingPhone#">
41
42 <cfhttpparam type="FORMFIELD" name="x_ship_to_first_name" value="#shippingFirstName#">
43 <cfhttpparam type="FORMFIELD" name="x_ship_to_last_name" value="#shippingLastName#">
44 <cfhttpparam type="FORMFIELD" name="x_ship_to_company" value="#shippingCompany#">
45 <cfhttpparam type="FORMFIELD" name="x_ship_to_address" value="#shippingAddress#">
46 <cfhttpparam type="FORMFIELD" name="x_ship_to_city" value="#shippingCity#">
47 <cfhttpparam type="FORMFIELD" name="x_ship_to_state" value="#shippingState#">
48 <cfhttpparam type="FORMFIELD" name="x_ship_to_zip" value="#shippingZip#">
49 <cfhttpparam type="FORMFIELD" name="x_ship_to_country" value="#shippingCountry#">
50 <cfhttpparam type="FORMFIELD" name="x_Description" value="#description#">
51 <cfhttpparam type="FORMFIELD" name="x_invoice_num" value="#invoicenum#">
52 </cfhttp>
53
54 <cfset response = cfhttp.fileContent>

To refactor your code you will want to do something like this.

view plain print about
1<cfset authURL = "https://test.authorize.net/gateway/transact.dll" />
2 <cfif AuthNetMode eq "live">
3 <cfset authURL = "https://secure.authorize.net/gateway/transact.dll" />
4 </cfif>
5<!--- CFX_HTTP5 Call - You'll want to refactor your code in this fashion --->
6
7<cfset httpBody = "x_Login=#AuthLogin#&
8 x_Password=#AuthPassword#&
9 x_merchant_email=#AuthEmail#&
10 x_delim_data=true&
11 x_test_request=#x_test_request#&
12 x_type=AUTH_ONLY&
13 x_method=cc&
14 x_amount=#orderTotal#&
15 x_card_num=#cardNumber#&
16 x_exp_date=#cardExpiration#&
17 x_first_name=#billingFirstName#&
18 x_last_name=#billingLastName#&
19 x_company=#billingCompany#&
20 x_address=#billingAddress#&
21 x_city=#billingCity#&
22 x_state=#billingState#&
23 x_zip=#billingZip#&
24 x_country=#billingCountry#&
25 x_customer_ip=#cgi.remote_address#&
26 x_Email=#billingEmail#&
27 x_Phone=#billingPhone#&
28 x_ship_to_first_name=#shippingFirstName#&
29 x_ship_to_last_name=#shippingLastName#&
30 x_ship_to_company=#shippingCompany#&
31 x_ship_to_address=#shippingAddress#&
32 x_ship_to_city=#shippingCity#&
33 x_ship_to_state=#shippingState#&
34 x_ship_to_zip=#shippingZip#&
35 x_ship_to_country=#shippingCountry#&
36 x_Description=#description#&
37 x_invoice_num=#invoicenum#"
>

38
39 <!--- If you want an email to go to the customer via authorize.net
40change this to true. Make sure authorize.net is configured properly. --->

41 <cfset httpBody = httpBody & "&x_email_customer=#x_email_customer#">
42
43 <cfif isDefined("cardSecurityCode") and cardSecurityCode eq "">
44 <cfset httpBody = httpBody & "&x_card_code=#cardSecurityCode#">
45 </cfif>
46
47 <cfset cfxhttp = {}>
48 <cfset headers = "Content-Type: application/x-www-form-urlencoded">
49 <cfx_http5 url="#authURL#" method="post" out="cfxhttp.body" outqhead="cfxhttp.QHEAD" outhead="cfxhttp.RHEAD" ssl="5" body="#httpBody#" header="#headers#">
50 </cfx_http5>
51
52 <cfset response = cfxhttp.body>

The code is a minor change and relatively easy to do. I've tested this method in a production environment and it works fine. I do not recommend this as a long term solution. The preferred long term solution is upgrading the server(s) and ColdFusion version to currently supported versions. This way there will be security updates to help protect against new threats. If you are on ColdFusion 10 or 11 then the best option is to install the ColdFusion patches and upgrade the Java version to 1.8 then you will be good to go. If you need an experience ColdFusion developer to make these changes then please do contact us, we will be happy to assist.

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

Authorize.NET Temporarily Ending TLS 1.1 and TLS 1.0 Support, is your ColdFusion Server Ready?

At CF Webtools we have been preparing for this inevitable day for the past few years. We've been upgrading our clients servers and services to handle TLS 1.2 calls to Authorize.Net and other third party processors for a while now. Recently Authorize.Net announced a "Temporary Disablement of TLS 1.0/1.1" for "a few hours on January 30, 2018 and then again on February 8, 2018." This is in preparation for the final disablement of TLS1.0/1.1 on February 28, 2018.

As you may be aware, new PCI DSS requirements state that all payment systems must disable earlier versions of TLS protocols. These older protocols, TLS 1.0 and TLS 1.1, are highly vulnerable to security breaches and will be disabled by Authorize.Net on February 28, 2018.

To help you identify if you're using one of the older TLS protocols, Authorize.Net will temporarily disable those connections for a few hours on January 30, 2018 and then again on February 8, 2018.

Based on the API connection you are using, on either one of these two days you will not be able to process transactions for a short period of time. If you don't know which API you're using, your solution provider or development partner might be a good resource to help identify it. This disablement will occur on one of the following dates and time:

  • Akamai-enabled API connections will occur on January 30, 2018 between 9:00 AM and 1:00 PM Pacific time.
  • All other API connections will occur on February 8, 2018 between 11:00 AM and 1:00 PM Pacific time.
Merchants using TLS 1.2 by these dates will not be affected by the temporary disablement. We strongly recommend that connections still using TLS 1.0 or TLS 1.1 be updated as soon as possible to the stronger TLS 1.2 protocol.

This means that if you are using older methods to make calls to Authorize.Net that are not capable of making TLS 1.2 connections then you will NOT be able to process credit card transactions.

This affects ALL ColdFusion versions 9.0.2 and older! This also affects ColdFusion 10 Update 17 and older. If your server is running any of these older versions of ColdFusion and your server is processing credit cards with Authorize.Net then this advisory applies to your server.

CF Webtools has been successfully mitigating this issue for clients servers for the past couple years and we are very experienced in resolving these security related issues. In a previous blog post I tested which TLS levels were supported by various ColdFusion versions on various Java versions and produced an easy to read chart.

If your ColdFusion server is affected by this or if you do not know if your ColdFusion server is affected by this then please contact us (much) sooner than later. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion MailSpoolService Performance

In my last article about the Adobe ColdFusion MailSpoolService I mentioned that I was going to try to get specifics on expected performance in the Standard Edition vs Enterprise edition of the MailSpoolService. Adobe has not respond to my requests with actual data. While attending the ColdFusion Summit 2017 I tried to get a clear answer from any of the Adobe ColdFusion engineering team members that were at the conference. They didn't know the answer. Because I didn't get the response I wanted from Adobe I decided to start testing.

My first test was to setup a Windows VM with ColdFusion 11 installed with a standard license. I also created a simple CFML page that uses CFMAIL to send an email with a CFLOOP to send that same email a lot of times. To make this a more realistic test I made up a new disposable email address on our mail server at CF Webtools and sent the emails from my email server on AWS. This means that the ColdFusion MailSpoolService has to actually communicate with a mail server. SMTP connections can at times take time. The emails I generated have several paragraphs of Lorem Ipsum text to simulate actual email sizes. My first test was to verify one email did indeed get sent. It did. The next test was to send 1000 emails while timing with my iPhone's stop watch. We also have ColdFusion 11 Enterprise which meant I was able to test the performance against the Enterprise Edition. Lastly, I was asked to test on the Developer Edition because it is often stated that the Developer Edition is essentially Enterprise Edition with a two connection limit. I ran this test a couple times each from ColdFusion 11 Standard, ColdFusion 11 Developer, and ColdFusion 11 Enterprise servers.

Standard Edition
It took approximately 23 minutes to process 1000 emails in the mail spool. This comes down to about 44/45 emails per minute. Which works out to about 11/12 emails per 15 second pooling interval or 2600 email an hour. Which is a little more that 60,000 emails per day processing 24 hours straight without any connection issues. That's not too shabby for being the single threaded version of the MailSpoolService.

Developer Edition
After running the same tests a couple times in Developer Edition I got the exact same results as I did for Standard Edition.

Enterprise Edition
This is where you can say "You get what you pay for!". Before I go into the numbers let me also remind everyone that the Enterprise Edition of the MailSpoolService is multi-threaded and you can specify the number of threads. I think the default is 10 threads. This setting is in the Mail section of the ColdFusion Administrator Enterprise Edition ONLY in the sub section "Mail Spool Settings". There is nothing that indicates that there is a maximum number of threads. My tests are with 10 threads.

I had to run this test several more times just to make sure I saw what I saw. All 1000 emails were sent in a single polling of the mailSpoolService. That's 1000 emails sent in under 15 seconds. I ramped it up a bit and sent 5000 emails. This time it took two polling intervals and sent 5000 emails in about 30 seconds. To get absurd I increased the test to 10,000 emails and the Enterprise Edition cleared those out in less than 60 seconds. This means it took 4 polling intervals to process 10,000 emails which comes out to 2,500 every 15 seconds with 10 MailSpoolThreads. I wanted to verify this exactly so I decreased the polling interval from 15 seconds to 30 seconds. I wanted to fill the mail spool completely beforehand and see how many emails were processed on each polling interval. What I saw is that I'm not nearly at the limit of what the Enterprise Edition MailSpoolService can handle. By slowing down the polling interval my CFML script was able to put all 10,000 emails into the mail spool folder before the MailSpoolService started processing. Then it happened, all 10,000 emails were process in one single polling interval of less than 15 seconds time. I'm not sure were the limit is, but it's fairly clear that the Enterprise edition can send more emails than most of us will ever need. Even if you're running a bulk mail service.

Summary
My results are not scientific. However, I do believe they are closer to what real people will see on real servers based on my experience with hundreds of servers. It would be really nice if Adobe would respond with some real numbers so we could help clients decide if this feature is worth buying Enterprise Edition instead of Standard Edition. However, based on my testing, if sending emails is your high priority and the amount of emails is going to be over 50,000 emails per day then you might want to weigh the cost of an Enterprise license.

Note:
The reason I was testing on ColdFusion 11 is this is the version that several different clients have that are having issues with the MailSpoolService. I think I know that for one client they really are trying to send near or over 50,000 emails per day and this is why they thought there was an issue with the MailSpoolService.

On my Way to CFSummit 2017

For the first time ever I'm headed to CF Summit. This should be fun and exciting! I'm waiting on Uber to show up and then I'm gone. I just have to get through TSA without a full cavity search. I'll be landing in Vegas around 4pm Vegas time. Let the party begin!

So far I'm thinking these sessions. All subject to change.

9:00 AM - 10:15 AM Day 1 General Session
10:30 AM - 11:30 AM send.Better() - Giving Email a REST
11:45 AM - 12:45 PM Solving problems in ways never before possible, with FusionReactor 7 and FR CLOUD
1:45 PM - 2:45 PM Dockerizing a ColdFusion Enterprise Application, a Case Study
3:00 PM - 4:00 PM Level Up Your Web Apps with Amazon Web Services
4:15 PM - 5:15 PM Power of Simplicity in FW/1 Framework

9:00 AM - 10:00 AM Day 2 General Session: How APIs Accelerate Digital Transformation
10:15 AM - 11:15 AM Application Performance Monitoring Suite in ColdFusion Aether.
11:30 AM - 12:30 PM Securing Mature CFML Codebases
2:45 PM - 3:45 PM Language improvements in ColdFusion Aether
4:00 PM - 5:00 PM CFConfig - A New Way to Manage Your ColdFusion Engine Config
5:15 PM - 5:30 PM Closing Session and Raffle Drawing

Cryptojacking: Hacking for Bitcoins

This is a brief follow up to my previous article on Hacking for Bitcoins in which I detailed how servers were being hijacked with cryptocurrency miners and using your servers CPU power to mine for Bitcoins or other blockchain cryptocurrencies. This is an updated twist on that hack. I saw this Ars Technica article today and it points out that the newer twist is to inject code into your websites code and then process cryptocurrency mining on your website user's computers. This distributes the CPU processing by thousands instead of just taking over a few of your servers.

To do this, hackers are using Coinhive.com which offers an easy-to-use programming interface that lets you setup your own website to process cryptocurrency on your visitors computers. There isn't a requirement to give notice to users that you are going to do this. What hackers are doing is using vulnerabilities in your server(s) and/or website(s) to inject this code in your website. It is estimated that there are about 2,500 websites that are currently compromised and using their users to process cryptocurrency. The fine article at Ars Technica indicates that it appears most are connected to two Coinhive.com accounts. This might mean that the hackers can easily be traced and stopped. But others will surely follow in their path.

How do I know?

When Cryptojacking occurs, a direct side effect is that the website user CPU's are maxed out and system heat starts to increase. This is a tell tale sign that the website you are using is either using your computer for their gain or has been compromised and a hacker is using your computer for their gain. (It could also be one of those annoying Flash based ads that we all hate.) But check the site source code to see if there is anything linking to Coinhive or similar. Ars Technica also reported "Most of the affected sites concealed the connection to Coinhive by adding a link to the domain siteverification.online or one masquerading as a Sucuri firewall."

This is a growing problem and recently Malwarebytes reported that on average it performs about 8 million blocks per day to unauthorized mining pages. People who want to avoid these Cryptojacking scams can use Malwarebytes or another antivirus program that blocks abusive pages

From our point of view at CF Webtools, this is a good reminder to make sure your ColdFusion servers are secure, updated and patched. It's also a good reason as to why your website code (all code really) should be in a secured version control system. That way if something like this did happen to your website code you can replace it from a known clean copy instead of digging through the code looking for the injected code. Additionally, CF Webtools offers PenTesting to check your website code for vulnerabilities. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion MailSpoolService Manual Restart Issue

I've seen a few different ColdFusion 11 Standard servers that have been sending duplicate emails. We've had several clients at CF Webtools reporting this issue and over time I've had to research this to try to determine how this is happening. During my investigations, I've been able to see this behavior happen on each of the servers in question. The obvious response that I've see from Adobe and others is that code must be creating to emails in error. However, I've been able to prove beyond all doubt that this is not the case. ColdFusion creates a file that it places in the /Mail/Spool/ folder that is named something like Mail4087177804601873442.cfmail. The MailSpoolService runs on a time interval defined in the ColdFusion Administrator. Typically this interval is set to run every 15 seconds. According to Adobe, the Standard edition of ColdFusion MailSpoolService is supposed to be single threaded as opposed to the multi-threaded version that is in the Enterprise version.

To test the sending of duplicated emails I've done two primary tests. One test was to create a very simple CFML script that generate one single email and run it once. Before running the script, I have the Mail/Spool folder open so I can verify only one email file was generated. I also slow the MailSpoolService Interval to 60 seconds to provide time to verify the file(s) created. The result of this test on the various servers where we seen this issue is that I can replicate the duplicate email issue after certain conditions. This is verified by receiving to identical emails a couple seconds apart and the mailsent.log in ColdFusion logs that shows it sent the same email twice. The second test I did was to take a copy of the generated email file and save it outside of the /Mail/Spool folder. Then copy this file once into the /Mail/Spool folder to ensure that only one copy of the mail file was gin the /Mail/Spool folder. In these test I had the exact same results. The email was sent twice, I received it twice and the mailsent.log file showed it was sent twice. There is zero doubt that there is a flaw in how the mailSpoolService works in ColdFusion Standard edition.

To date I've never been able to reproduce this issue on Enterprise or Developer Editions of ColdFusion 11. Nor have I been able to reproduce this on ColdFusion 10 Standard Edition.

view plain print about
1"Information","scheduler-2","10/25/17","14:21:16",,"Mail: 'Multipart Test' From:'john@example.com' To:'jim@example.com' was successfully sent using mail.example.com"
2"Information","scheduler-5","10/25/17","14:21:19",,"Mail: 'Multipart Test' From:'john@example.com' To:'jim@example.com' was successfully sent using mail.example.com"

What is the MailSpoolService and how are people accessing it? The MailSpoolService is part of the ColdFusion server ServiceFactory written in Java. As I previously noted it is the service in ColdFusion that polls the /Mail/Spool folder for email files generated by the ColdFusion CFMAIL tag or the cfmail() function in CFScript. In the ColdFusion administrator you can specify a few settings about the behavior of the MailSpoolService. There are several ColdFusion dedicated blogs that have over the years posted how to access this service via Java code in your CFML files. The following code lets you access the MailSpoolService and from there you can supposedly do a few things. Things such as see if the mail spool is enabled, check to see if the mail spool is disk or memory based, and so on. If you dump the object you can see all the potentially usable methods.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset MailSpoolService = sFactory.mailSpoolService>
3Mail Spool Enabled: <cfoutput>#MailSpoolService.isSpoolEnable()#</cfoutput>
4
5Mail Spool Location: <cfif MailSpoolService.isSpoolToMemory()>Memory<cfelse>Disk</cfif>
6
7<cfdump var="#MailSpoolService#">

There are even methods to stop and start the MailSpoolService.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset sFactory.mailSpoolService.stop()>
3<cfset sFactory.mailSpoolService.start()>

It is the stopping and starting of the MailSpoolService that is causing the issues here. It seems that, and Adobe has confirmed, the stop feature doesn't actually stop anything. If you output the status of the MailSpoolService after stopping it, it reports back that it is indeed still enabled. When using the code to stop and start the mailSpool service, the MailSpoolService never stops, but a second instance gets started. This is where the problems start. Now there are two instances of the MailSpoolService running, neither can be stopped, and both are polling the /Mail/Spool/ folder for emails to send. When doing this both instances end up reading the same email files and and both instances sending the same emails. During this process of both instances running there are occasions when one cannot find or delete the mail file and then logs an error message in the mail.log that it could not find the particular mail file. This is another clue that there is more than one instance of the MailSpoolService running.

view plain print about
1"Error","scheduler-1","10/30/17","22:38:15",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail4139740176796644009.cfmail (The system cannot find the file specified)"
2"Error","scheduler-5","10/30/17","22:40:56",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail7339303340500448710.cfmail (The system cannot find the file specified)"
3"Error","scheduler-4","10/30/17","22:45:17",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail8468648677617392229.cfmail (The system cannot find the file specified)"

Why are people manually stopping and starting the MailSpoolService? As I noted before in Standard edition the mail spool is single threaded. At times when large amounts of emails are generated with CFML code, say emails for a mailing list, it appears that the MailSpoolService becomes overwhelmed or even stops processing emails. Creative developers figured out how to access the MailSpoolService, which is not a documented API, and discovered the stop and start features. After running stop and start they noticed the emails were being processed again. It is highly likely that the stop feature worked in older versions of ColdFusion as many of the blog posts are from the era of ColdFusion 7, 8, and 9. It's most likely the behavior changed with ColdFusion 10 as that was a major rewrite of ColdFusion. The Service Factory is not a documented for public use API and thus subject to change at anytime.

The bigger issue remaining is why does the MailSpoolService slow way down or even stop completely when under load in ColdFusion Standard version. To me this appears to be a bug. I've been privately speaking with Adobe on this issue and we are actively trying to figure out what is happening. I've requested full documentation on exactly how the MailSpoolService is supposed to function in Standard Edition including how many emails are processed every time the MailSpoolService polls the /Mail/Spool folder. In older versions of ColdFusion (9 and older) I could actually see that up to 100 emails would be processed at a time. From ColdFusion 10 and on it appears that fewer are processed per polling of the spool. I am waiting on Adobe to verify this question in particular. While on the phone with one individual from Adobe, I was told that in Standard Edition, it processed only one email per polling interval. This means that at best Standard Edition is capable of only 4 emails per minute. From my own testing on Standard Edition I know this isn't the case and this is why I've requested clarification and documentation from Adobe. According to ColdFusion 11 Standard itself it reports there are 10 threads for the mail spool. To get this information I ran the following code.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset MailSpoolService = sFactory.mailSpoolService>
3
4<cfoutput>
5Schedule: #MailSpoolService.getSchedule()#
6
7Max Delivery Threads: #MailSpoolService.getMaxDeliveryThreads()#
8
9Spool Messages Limit: #MailSpoolService.getSpoolMessagesLimit()#
10
11Maintain Connections: #MailSpoolService.isMaintainConnections()#
12
13</cfoutput>
Schedule: 15000
Max Delivery Threads: 10
Spool Messages Limit: 50000
Maintain Connections: NO
I'm not sure what to make of the inconsistent information between Adobe and what ColdFusion itself is reporting. I'm still working with Adobe on this and I hope to have solid answers as to how the mailSpool truly works.

In summary, DON'T DO IT! Do not use the code above that is found on numerous ColdFusion dedicated tech blogs (just Google "MailSpoolService" to see them) to manually stop and start the ColdFusion MailSpoolService. For now the best thing to do when the MailSpoolService stops processing email in ColdFusion 11 Standard Edition is to restart ColdFusion.

Need help upgrading your VM or patching your server (or anything else)? Need help troubleshooting a perplexing problem? Our operations group is standing by 24/7 (Wait what? Mark, you said I get to sleep!) - give us a call at 402-408-3733, or send a note to operations at CF Webtools

Hacking for Bitcoins

Are your free CPU cycles making others rich? There's a chance they are and it's at your expense. A recent article at Vice.com states that "At Least 1.65 Million Computers Are Mining Cryptocurrency for Hackers So Far This Year". If this is to be believed then it's possible a server you are running has been compromised and is actually mining cyrptocurrency for the hackers. This type of breach is called Cryptojacking and it's costing you and/or your company money.

Cyrptocurrency is an anonymous, digital currency that is supposed to be untraceable. It's used on the internet to purchase more and more products and services. One of the most common forms of cryptocurrency is Bitcoin. This is from the Wikipedia entry on Bitcoin.

Bitcoin is a worldwide cryptocurrency and digital payment system called the first decentralized digital currency, since the system works without a central repository or single administrator. It was invented by an unknown programmer, or a group of programmers, under the name Satoshi Nakamoto and released as open-source software in 2009. The system is peer-to-peer, and transactions take place between users directly, without an intermediary. These transactions are verified by network nodes and recorded in a public distributed ledger called a blockchain.

Besides being created as a reward for mining, bitcoin can be exchanged for other currencies, products, and services. As of February 2015, over 100,000 merchants and vendors accepted bitcoin as payment. Bitcoin can also be held as an investment. According to research produced by Cambridge University in 2017, there are 2.9 to 5.8 million unique users using a cryptocurrency wallet, most of them using bitcoin. ...

Bitcoin Mining is a record-keeping service that runs on peoples computers, servers, or specialized Mining Devices, that are setup by individuals to help process Bitcoin transactions. As a reward for doing this you are given newly created bitcoins and transaction fees. ie. You can make money by mining for Bitcoin.

This reward is enough that hackers have taken it to the next level and started hacking servers around the world so they can install mining software and use YOUR computers and servers to make money for themselves.

Case Study

CF Webtools has seen this type of hack in the real world. We recently had a company come to us seeking our services for both Server Administration and ColdFusion programming. Part of taking this new company on as a client we performed a security review on all of their servers. They also had existing issues that we needed to look at in particular. Their web server was rebooting multiple times per day at what seemed like random intervals.

Upon review we found the web server was always running at 100% CPU usage with no services claiming to be using that much CPU power. Certainly not ColdFusion or IIS. We decided to install Malware Bytes and scanned for malware. It didn't take long to find that indeed there was malware running on the server. What we found surprised us only because we had not seen this in action before. It was a cryptocurrency miner and it was so intensive that it would crash the server. All attempts to remove the malware failed. It would end up back on the server in a short period of time. The fact is this server was compromised. To resolve the issue we sent one of our decommissioned, but powerful servers, preinstalled with a clean OS to their data center. Then our Operations Manager went on the road to install the new server as well as a physical firewall. We essentially rearchitected their entire server setup. Meanwhile, Malware bytes did it's best to keep the malware at bay while I recreated their web server on the new server. It was a busy week (or more), but we were able to clean the code on the clients server and put that on the new server. We also had to research and rebuild all the web application dependancies from scratch. When it was all said and done we replaced the compromised server with the new one and put all their servers behind a Cisco ASA.

This case of Hacking for Bitcoins proved costly in the end to the company who's systems were compromised all while providing a free profit to the hacker(s).

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

UPDATE: I saw this Ars Technica article today and it is directly related to this. Cryptojacking is bigger than most people think.

I'm Still Alive!

Whatever happened to that "Wil" guy that always wrote about ColdFusion stuff?

I'm still here! I've really neglected writing for the past couple years. My last real post (other than yesterdays) was in November of 2015. My excuses are I wasn't 'feeling it', I was too busy working, I didn't care about writing. Whatever. I think some people figured I dropped from the face of the earth. I sort of did in some ways. In other ways I've been doing the same things I've always been doing. WORKING and WORKING.

I'm still with CF Webtools, seven years now, and I'm primarily still doing ColdFusion stuff. I say stuff because I almost never do any actual coding. I'm more or less "The Wolf" aka Winston Wolf and "I Solve Problems". At CF Webtools, we are called in to solve problems for companies of all types. What kinds of problems? All problems whether it's coding, database/performance, web server/performance, hackers (oh my), bugs, etc. We solve problems so you can go about the business of your business. I guess I was so bogged down in the front lines that I forgot to write about a few of the battles and share what I've learned.

What's New?

Well, for the most part I'm doing AWS work. I'm midway through some AWS certification courses and I've been building out 'virtual data centers' for clients. Don't get me wrong, I'm still working with ColdFusion servers, but I'm mostly migrating them from physical iron, or other hosting providers, over to AWS. Some of these migrations are mixed environments and include things like PHP and .NET. They may also include full database server migrations into AWS RDS, DNS migrations to Route53, CloudWatch monitoring and more. For the most part AWS is fun and I like working on the platform. CF Webtools will build and/or manage your AWS solutions. Drop us an email.

The 2017 Solar Eclipse

[More]

More Entries