On my Way to CFSummit 2017

For the first time ever I'm headed to CF Summit. This should be fun and exciting! I'm waiting on Uber to show up and then I'm gone. I just have to get through TSA without a full cavity search. I'll be landing in Vegas around 4pm Vegas time. Let the party begin!

So far I'm thinking these sessions. All subject to change.

9:00 AM - 10:15 AM Day 1 General Session
10:30 AM - 11:30 AM send.Better() - Giving Email a REST
11:45 AM - 12:45 PM Solving problems in ways never before possible, with FusionReactor 7 and FR CLOUD
1:45 PM - 2:45 PM Dockerizing a ColdFusion Enterprise Application, a Case Study
3:00 PM - 4:00 PM Level Up Your Web Apps with Amazon Web Services
4:15 PM - 5:15 PM Power of Simplicity in FW/1 Framework

9:00 AM - 10:00 AM Day 2 General Session: How APIs Accelerate Digital Transformation
10:15 AM - 11:15 AM Application Performance Monitoring Suite in ColdFusion Aether.
11:30 AM - 12:30 PM Securing Mature CFML Codebases
2:45 PM - 3:45 PM Language improvements in ColdFusion Aether
4:00 PM - 5:00 PM CFConfig - A New Way to Manage Your ColdFusion Engine Config
5:15 PM - 5:30 PM Closing Session and Raffle Drawing

Cryptojacking: Hacking for Bitcoins

This is a brief follow up to my previous article on Hacking for Bitcoins in which I detailed how servers were being hijacked with cryptocurrency miners and using your servers CPU power to mine for Bitcoins or other blockchain cryptocurrencies. This is an updated twist on that hack. I saw this Ars Technica article today and it points out that the newer twist is to inject code into your websites code and then process cryptocurrency mining on your website user's computers. This distributes the CPU processing by thousands instead of just taking over a few of your servers.

To do this, hackers are using Coinhive.com which offers an easy-to-use programming interface that lets you setup your own website to process cryptocurrency on your visitors computers. There isn't a requirement to give notice to users that you are going to do this. What hackers are doing is using vulnerabilities in your server(s) and/or website(s) to inject this code in your website. It is estimated that there are about 2,500 websites that are currently compromised and using their users to process cryptocurrency. The fine article at Ars Technica indicates that it appears most are connected to two Coinhive.com accounts. This might mean that the hackers can easily be traced and stopped. But others will surely follow in their path.

How do I know?

When Cryptojacking occurs, a direct side effect is that the website user CPU's are maxed out and system heat starts to increase. This is a tell tale sign that the website you are using is either using your computer for their gain or has been compromised and a hacker is using your computer for their gain. (It could also be one of those annoying Flash based ads that we all hate.) But check the site source code to see if there is anything linking to Coinhive or similar. Ars Technica also reported "Most of the affected sites concealed the connection to Coinhive by adding a link to the domain siteverification.online or one masquerading as a Sucuri firewall."

This is a growing problem and recently Malwarebytes reported that on average it performs about 8 million blocks per day to unauthorized mining pages. People who want to avoid these Cryptojacking scams can use Malwarebytes or another antivirus program that blocks abusive pages

From our point of view at CF Webtools, this is a good reminder to make sure your ColdFusion servers are secure, updated and patched. It's also a good reason as to why your website code (all code really) should be in a secured version control system. That way if something like this did happen to your website code you can replace it from a known clean copy instead of digging through the code looking for the injected code. Additionally, CF Webtools offers PenTesting to check your website code for vulnerabilities. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion MailSpoolService Manual Restart Issue

I've seen a few different ColdFusion 11 Standard servers that have been sending duplicate emails. We've had several clients at CF Webtools reporting this issue and over time I've had to research this to try to determine how this is happening. During my investigations, I've been able to see this behavior happen on each of the servers in question. The obvious response that I've see from Adobe and others is that code must be creating to emails in error. However, I've been able to prove beyond all doubt that this is not the case. ColdFusion creates a file that it places in the /Mail/Spool/ folder that is named something like Mail4087177804601873442.cfmail. The MailSpoolService runs on a time interval defined in the ColdFusion Administrator. Typically this interval is set to run every 15 seconds. According to Adobe, the Standard edition of ColdFusion MailSpoolService is supposed to be single threaded as opposed to the multi-threaded version that is in the Enterprise version.

To test the sending of duplicated emails I've done two primary tests. One test was to create a very simple CFML script that generate one single email and run it once. Before running the script, I have the Mail/Spool folder open so I can verify only one email file was generated. I also slow the MailSpoolService Interval to 60 seconds to provide time to verify the file(s) created. The result of this test on the various servers where we seen this issue is that I can replicate the duplicate email issue after certain conditions. This is verified by receiving to identical emails a couple seconds apart and the mailsent.log in ColdFusion logs that shows it sent the same email twice. The second test I did was to take a copy of the generated email file and save it outside of the /Mail/Spool folder. Then copy this file once into the /Mail/Spool folder to ensure that only one copy of the mail file was gin the /Mail/Spool folder. In these test I had the exact same results. The email was sent twice, I received it twice and the mailsent.log file showed it was sent twice. There is zero doubt that there is a flaw in how the mailSpoolService works in ColdFusion Standard edition.

To date I've never been able to reproduce this issue on Enterprise or Developer Editions of ColdFusion 11. Nor have I been able to reproduce this on ColdFusion 10 Standard Edition.

view plain print about
1"Information","scheduler-2","10/25/17","14:21:16",,"Mail: 'Multipart Test' From:'john@example.com' To:'jim@example.com' was successfully sent using mail.example.com"
2"Information","scheduler-5","10/25/17","14:21:19",,"Mail: 'Multipart Test' From:'john@example.com' To:'jim@example.com' was successfully sent using mail.example.com"

What is the MailSpoolService and how are people accessing it? The MailSpoolService is part of the ColdFusion server ServiceFactory written in Java. As I previously noted it is the service in ColdFusion that polls the /Mail/Spool folder for email files generated by the ColdFusion CFMAIL tag or the cfmail() function in CFScript. In the ColdFusion administrator you can specify a few settings about the behavior of the MailSpoolService. There are several ColdFusion dedicated blogs that have over the years posted how to access this service via Java code in your CFML files. The following code lets you access the MailSpoolService and from there you can supposedly do a few things. Things such as see if the mail spool is enabled, check to see if the mail spool is disk or memory based, and so on. If you dump the object you can see all the potentially usable methods.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset MailSpoolService = sFactory.mailSpoolService>
3Mail Spool Enabled: <cfoutput>#MailSpoolService.isSpoolEnable()#</cfoutput>
4
5Mail Spool Location: <cfif MailSpoolService.isSpoolToMemory()>Memory<cfelse>Disk</cfif>
6
7<cfdump var="#MailSpoolService#">

There are even methods to stop and start the MailSpoolService.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset sFactory.mailSpoolService.stop()>
3<cfset sFactory.mailSpoolService.start()>

It is the stopping and starting of the MailSpoolService that is causing the issues here. It seems that, and Adobe has confirmed, the stop feature doesn't actually stop anything. If you output the status of the MailSpoolService after stopping it, it reports back that it is indeed still enabled. When using the code to stop and start the mailSpool service, the MailSpoolService never stops, but a second instance gets started. This is where the problems start. Now there are two instances of the MailSpoolService running, neither can be stopped, and both are polling the /Mail/Spool/ folder for emails to send. When doing this both instances end up reading the same email files and and both instances sending the same emails. During this process of both instances running there are occasions when one cannot find or delete the mail file and then logs an error message in the mail.log that it could not find the particular mail file. This is another clue that there is more than one instance of the MailSpoolService running.

view plain print about
1"Error","scheduler-1","10/30/17","22:38:15",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail4139740176796644009.cfmail (The system cannot find the file specified)"
2"Error","scheduler-5","10/30/17","22:40:56",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail7339303340500448710.cfmail (The system cannot find the file specified)"
3"Error","scheduler-4","10/30/17","22:45:17",,"C:\ColdFusion11\cfusion\Mail\Spool\Mail8468648677617392229.cfmail (The system cannot find the file specified)"

Why are people manually stopping and starting the MailSpoolService? As I noted before in Standard edition the mail spool is single threaded. At times when large amounts of emails are generated with CFML code, say emails for a mailing list, it appears that the MailSpoolService becomes overwhelmed or even stops processing emails. Creative developers figured out how to access the MailSpoolService, which is not a documented API, and discovered the stop and start features. After running stop and start they noticed the emails were being processed again. It is highly likely that the stop feature worked in older versions of ColdFusion as many of the blog posts are from the era of ColdFusion 7, 8, and 9. It's most likely the behavior changed with ColdFusion 10 as that was a major rewrite of ColdFusion. The Service Factory is not a documented for public use API and thus subject to change at anytime.

The bigger issue remaining is why does the MailSpoolService slow way down or even stop completely when under load in ColdFusion Standard version. To me this appears to be a bug. I've been privately speaking with Adobe on this issue and we are actively trying to figure out what is happening. I've requested full documentation on exactly how the MailSpoolService is supposed to function in Standard Edition including how many emails are processed every time the MailSpoolService polls the /Mail/Spool folder. In older versions of ColdFusion (9 and older) I could actually see that up to 100 emails would be processed at a time. From ColdFusion 10 and on it appears that fewer are processed per polling of the spool. I am waiting on Adobe to verify this question in particular. While on the phone with one individual from Adobe, I was told that in Standard Edition, it processed only one email per polling interval. This means that at best Standard Edition is capable of only 4 emails per minute. From my own testing on Standard Edition I know this isn't the case and this is why I've requested clarification and documentation from Adobe. According to ColdFusion 11 Standard itself it reports there are 10 threads for the mail spool. To get this information I ran the following code.

view plain print about
1<cfset sFactory = CreateObject("java","coldfusion.server.ServiceFactory")>
2<cfset MailSpoolService = sFactory.mailSpoolService>
3
4<cfoutput>
5Schedule: #MailSpoolService.getSchedule()#
6
7Max Delivery Threads: #MailSpoolService.getMaxDeliveryThreads()#
8
9Spool Messages Limit: #MailSpoolService.getSpoolMessagesLimit()#
10
11Maintain Connections: #MailSpoolService.isMaintainConnections()#
12
13</cfoutput>
Schedule: 15000
Max Delivery Threads: 10
Spool Messages Limit: 50000
Maintain Connections: NO
I'm not sure what to make of the inconsistent information between Adobe and what ColdFusion itself is reporting. I'm still working with Adobe on this and I hope to have solid answers as to how the mailSpool truly works.

In summary, DON'T DO IT! Do not use the code above that is found on numerous ColdFusion dedicated tech blogs (just Google "MailSpoolService" to see them) to manually stop and start the ColdFusion MailSpoolService. For now the best thing to do when the MailSpoolService stops processing email in ColdFusion 11 Standard Edition is to restart ColdFusion.

Need help upgrading your VM or patching your server (or anything else)? Need help troubleshooting a perplexing problem? Our operations group is standing by 24/7 (Wait what? Mark, you said I get to sleep!) - give us a call at 402-408-3733, or send a note to operations at CF Webtools