Using CDN for Entire Website and Country Blocking - Part 3

This is Part 3 in a short series of articles about blocking entire countries from a website. Parts one and two cover CloudFlare and CloudFront.

CF Webtools has been asked numerous times to block an entire country or countries by many clients. The issue is that there's a lot of hacker activity from certain identified countries and the client(s) does not do any business with those countries. Typically it's entire server hacking attempts, but more recently it's to use the client's shopping cart to "test" stolen credit cards. This is a very serious problem and as such clients are asking us to help them prevent this from happening. One potential solution is to block the IP addresses that these attacks are coming from. I refer to this as the Whack-A-Mole method because it's just like that arcade game. As soon as you block one IP they switch to another IP address.

We need a better solution. I looked into what we could do and how reasonable and feasible the various options are in terms of technology and cost. In my previous two articles I wrote about using CloudFlare and AWS CloudFront. In this article I'm writing about using a slightly better hammer in the Whack-A-Mole method to block entire countries. This is one of the simplest but also least effective methods.

The option many of us have traditionally done is blocking problematic IP's on a case by case basis and in extreme cases blocking entire IP ranges. I've often referred to this as the Whack-A-Mole method. It's reactive and not proactive. A real hacker would not use their own personal IP and there is no guarantee that the IP will always remain with an unscrupulous user. Normally I do not block an IP because bad stuff happened from that IP once. However, I have noticed the same IP or IP ranges launching attacks on multiple unrelated, hosted at different locations, and different client's servers. That's when I start pounding the IP with the ol' Ban Hammer! Also, blocking and entire country with this method would mean being able to know all the possible IP addresses or address blocks assigned to a particular country. This is knowable!

I did some research on this and found a few very helpful resources. Resources like this http://ipdeny.com/ipblocks/ and this https://www.sitepoint.com/how-to-block-entire-countries-from-accessing-website/. These sites keep an updated list of IP addresses assigned to every country in the world. These are made available in the form of individual text files per country. And in the case of the SitePoint page, you can download a pre-scripted config file for many versions of web servers and firewalls. Hammer Time!

In the case of the country our client wants to block there are over 130 IP entries. These are in the form of CIDR IP ranges. This is the good news. The harder part here is that means there would have to be 130 plus entries manually added into IIS or a firewall. And this is for a smaller country. Larger countries, including countries that are known for hacking, have many thousands of CIDR IP ranges. But at least I can download the scripts for Apache and IIS from the SitePoint page and paste them into the respective config files.

What are the downsides to this method? First off I do not know if there would be any performance hit to IIS or Apache if we were to start entering thousands of IP restrictions. I do know that AWS restricts Network ACL's to an absolute max of 40 rules in their VPC's due to "performance issues" if more were added. We're still whacking at moles. IP assignments for countries can change thus you would need to update your static list of IP bans in your web server.

This is an example of how Apache 2.4 is configured.

view plain print about
1<RequireAll>
2 Require all granted
3 Require not ip 5.11.40.0/21
4 Require not ip 5.34.160.0/21
5 Require not ip 5.43.192.0/19
6 Require not ip 5.102.96.0/19
7.....
8 Require not ip 217.78.48.0/20
9</RequireAll>

This is an example of how the IIS XML web.config is configured. The CIRD notation needs to be converted to IP and network mask format.

view plain print about
1<?xml version="1.0"?>
2<configuration>
3<system.webServer>
4<security>
5<ipSecurity allowUnlisted="true">
6<clear/>
7<add ipAddress="5.11.40.0" subnetMask="255.255.248.0"/>
8<add ipAddress="5.34.160.0" subnetMask="255.255.248.0"/>
9<add ipAddress="5.43.192.0" subnetMask="255.255.224.0"/>
10<add ipAddress="5.102.96.0" subnetMask="255.255.224.0"/>
11.....
12<add ipAddress="217.78.48.0" subnetMask="255.255.240.0"/>
13</ipSecurity>
14</security>
15<modules runAllManagedModulesForAllRequests="true"/>
16</system.webServer>
17</configuration>

In conclusion each option; CloudFlare, CloudFront, and IP Banning, each have their benefits and costs. CloudFront was the easiest of the three to setup and if the downsides of the IP address masking isn't an issue then it is likely the most viable solution. The AWS CloudFront solution may be best if you are already on AWS and you have an understanding of AWS Solutions Architecting. Both CDN options have country restrictions (and rate limiting) that will help in preventing potential credit card scammers from misusing your shopping carts. IP Banning is simplistic, it has no additional dollar costs. But it may be a performance hit to your web server if you have a very large number of IP restrictions. You may also have to update the IP lists if IP assignments to a country change. It's also worth noting that all methods can be bypassed via proxies.

CF Webtools is an Amazon Web Services Partner. Our Operations Group can build, manage, and maintain your AWS services. We also handle migration of physical servers into AWS Cloud services. If you are looking for professional AWS management our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

Using CDN for Entire Website and Country Blocking - Part 2

This is Part 2 in a short series of articles about blocking entire countries from a website. See Part 1.

CF Webtools has been asked numerous times to block an entire country or countries by many clients. The issue is that there's a lot of hacker activity from certain identified countries and the client(s) does not do any business with those countries. Typically it's entire server hacking attempts, but more recently it's to use the client's shopping cart to "test" stolen credit cards. This is a very serious problem and as such clients are asking us to help them prevent this from happening. One potential solution is to block the IP addresses that these attacks are coming from. I refer to this as the Whack-A-Mole method because it's just like that arcade game. As soon as you block one IP they switch to another IP address.

We need a better solution. I looked into what we could do and how reasonable and feasible the various options are in terms of technology and cost. In this article I'm writing about using Amazon Web Services CloudFront to block entire countries.

Amazon AWS CloudFront
AWS CloudFront does offer country blocking. I thought this would be an easy setup, but it isn't. When I tried to setup AWS CloudFront to 'front' an entire website I found there are many pieces that needed to be in place in order for CloudFront to handle the entire website.

Route 53 is needed or any other DNS that allows an ALIAS record for the Zone Apex record. This is because the Zone Apex record (root record) will be set to the URL provided by CloudFront and not an IP address.

Elastic Load Balancing is needed. The CloudFront origin (EC2 server) needs to be behind an TCP Elastic Load balancer. If there is only one site then the ELB target can be the instance itself. If the EC2 instance hosts multiple different sites, then we need to add multiple internal IP addresses to the instance and configure the origin site to be on it's own IP. Then the ELB should be configured to that internal IP address and not instance. If you are passing host headers in the CloudFront 'Behavior' section then you can have a single IP on the web server with multiple sites per usual for virtual name hosting. You have to setup the TCP ELB as TCP port 80 passthrough in order to pass the original IP addresses to the web server.

AWS Certificate Manager is needed to create a new free SSL for the domain name being setup in CloudFront. (I say it's needed because all sites should be using TLS protocols these days.) I found a wild card certificate works well.

Then lastly AWS CloudFront itself can be setup. The settings are a bit tricky. The Origin will be the ELB which will then pass requests to the EC2 instance. If you want or need forms to be posted to the website then you need to select "GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE" option for Allowed HTTP Methods. If you need to allow logins then you have to choose "All" for Forward Cookies.

There are costs to each part. Route 53 charges by zone and number of requests. Elastic Load Balancing charges by the hour and by data transfer amounts. Then Cloud Front charges by data transfer amount.

There are downsides to this method as well. In addition to the AWS method being harder and more complex to setup there are more costs involved. I can pass the original requesting IP address through to the web server, it still comes through in the X-Forwarded-For custom header. In Apache it's easy to globally capture and place this value into log files or the CGI scope. IIS does not allow this to be done at a global level meaning each IIS site must be configured for the custom headers. Additionally, you may need to custom code the web application to read X-Forwarded-For no matter which web server you are using.

After you have all of that setup, configured, and working you can now start blocking countries. This is done in the AWS CloudFront Restrictions section. You can add a Geo-Restriction blacklist or whitelist by country.

Part 3 will cover using IIS and Apache and a slightly better hammer in the Whack-A-Mole method.

CF Webtools is an Amazon Web Services Partner. Our Operations Group can build, manage, and maintain your AWS services. We also handle migration of physical servers into AWS Cloud services. If you are looking for professional AWS management our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

Using CDN for Entire Website and Country Blocking - Part 1

CF Webtools has been asked numerous times to block an entire country or countries by many clients. The issue is that there's a lot of hacker activity from certain identified countries and the client(s) does not do any business with those countries. Typically it's entire server hacking attempts, but more recently it's to use the client's shopping cart to "test" stolen credit cards. This is a very serious problem and as such clients are asking us to help them prevent this from happening. One potential solution is to block the IP addresses that these attacks are coming from. I refer to this as the Whack-A-Mole method because it's just like that arcade game. As soon as you block one IP they switch to another IP address.

We need a better solution. I looked into what we could do and how reasonable and feasible the various options are in terms of technology and cost. In this article I'm writing about using CloudFlare CDN to block entire countries.

CloudFlare
I was not familiar with CloudFlare other than it's a CDN. They do offer advanced services for a price. There is a free tier that has CDN capability and limited Firewall features. The firewall features include the ability to setup 5 firewall rules.

To test the features and capabilities of CloudFlare I created a free account for myself and setup my blog to use CloudFlare. My blogs uptime is not critical like the client's business is and it gets real traffic thus it can be used to test various features.

Using the free firewall features I can block multiple countries in a single firewall rule. The rules allow for chaining filters with AND OR statements. See the example below.

I don't know yet if there is a limit to the number of conditions I can add to a single rule. However, at the moment it seems to be sufficient.

The negative side effect that I can see so far is that all the IP addresses that get logged on the origin web server are from CloudFlare. This defeats many clients needs/desires to have a valid IP address of their valid customers. Cloudflare does offer the option to pass through the original HTTP headers, but that is under their top Enterprise plan. They do not provide a cost for this. You need to request an estimate.

CloudFlare does pass through custom headers that has the original IP and other custom headers. However, these are not standard and web servers need to be configured to first read the custom header fields and then the application code needs to be updated to use the custom headers fields. It's far easier to do this in Apache than it is in IIS. IIS does not allow this to be done at a global level meaning each IIS site must be configured for the custom headers. Additionally, you may need to custom code the web application to read X-Forwarded-For no matter which web server you are using.

Another issue is that CloudFlare requires you move your DNS to them. Depending on the client, gaining access to their DNS and registrar can be challenging.

Part 2 will cover using AWS CloudFront to achieve the same results.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

ColdFusion Exploit in the Wild

On September 11th of 2018 Adobe released a critical security patch to patch a very dangerous flaw (CVE-2018-15961) that could allow an attacker to upload a file that can be used to exploit and take control of the server. Adobe updated their security note to alert everyone that there are active exploits in the wild.

"UPDATE: As of September 28, Adobe is aware of a report that CVE-2018-15961 is being actively exploited in the wild. The updates for ColdFusion 2018 and ColdFusion 2016 announced in this bulletin have been elevated to Priority 1. Adobe recommends customers update to the latest version as soon as possible." - Adobe

Today it is being reported by multiple news outlets including ZDNet that the exploit is in the wild and being used by a nation-state cyber-espionage group.

"A nation-state cyber-espionage group is actively hacking into Adobe ColdFusion servers and planting backdoors for future operations, Volexity researchers have told ZDNet.

The attacks have been taking place since late September and have targeted ColdFusion servers that were not updated with security patches that Adobe released two weeks before, on September 11." - ZDNet

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to "operations at cfwebtools.com".

ColdFusion Debugging on Production, That wasn't a Good Idea

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases Iwe saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

Waiting on VERY Large Data Transfers

What do you do while waiting for very large data transfers to complete? By large I'm talking upwards of Terabytes. My mind wanders and suddenly the theme for Rawhide gets stuck in my mind! But it's different. The music is similar, but the words have changed. It's almost like Dust Puppy was singing the song.

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes

Keep Flowin', Flowin', Flowin'
Though the datastreams are swollen
Keep them bytes Flowin', Raw Bytes
Through switches and wires and routers
Hell bent for bandwidth

Wishin' my smartphone was by my side
All the things I'm missin'
Good cat videos, Netflix and tweetin'
Are waiting at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Keep Flowin', Flowin', Flowin'
Though they're disapprovin'
Keep them bytes Flowin', Raw Bytes
Don't try to understand 'em
Just grep 'em, zip, and save 'em

Soon we'll be streamin' high and wide
My smartphone calculatin'
My true love will be tweetin'
Be waitin' at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes
Raw Bytes

The things we do while migrating data into the cloud.

It's "Retired" Jim!

In another chapter of "The Cloud Never Crashes", I woke up Sunday to one of my AWS instances that was 'crashed' with a notice of "Amazon EC2 Instance scheduled for retirement". Retirement? What does that mean? I went to check my email and realized that the "retired" instance was the email server. Doh! It took me a little while to figure out what they meant. It means this "An instance is scheduled to be retired when AWS detects irreparable failure of the underlying hardware hosting the instance." This serves as a good reminder that the cloud is really someone else's server.

In theory this is an easy fix. The instructions at Amazon claims that stopping and restarting the instance will launch it on new hardware. In practice I could not get the instance to stop. This is where having physical hardware and a power cord to pull would have been nice. Failing to get the instance to stop I could not detach the EBS root volume. Even force detaching the EBS root volume didn't work. This is where daily snapshots of EBS volumes comes in handy. I was able to launch a new EC2 instance and then convert the last snapshot to an EBS volume and attach that to the new EC2 instance. Then I moved the elastic IP from the "Retired" instance to the new instance and hit "start'. Full recovery!

Now I'm left with a hanging EC2 instance that is still "Stopping" and an EBS volume that I cannot use, detach, delete etc. I tried reissuing stop commands a couple times. Eventually I noticed a "Force Stop" option. I do not remember seeing this on earlier attempts. I do not know if this shows up after the first failed stopped attempt or after several. I'm not sure, but I think that sends a trained monkey into the datacenter to pull the power cord. In any case it worked. This let me detach my EBS volume. From there was was able to stop the new instance, detach the EBS volume and attach my original EBS root volume. Now I have full recovery and I was able to clean up the loose ends.

Amazon Web Service has given us a new euphemism. Retired means It's Dead Jim!

CF Webtools is an Amazon Web Services Partner. Our Operations Group can build, manage, and maintain your AWS services. We also handle migration of physical servers into AWS Cloud services. If you are looking for professional AWS management our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion Bloggers Tweets

When I brought ColdFusionBloggers.og back to life, several of us were discussing the service on the CFML Slack channel. One of the many things that was mentioned was that some blogs these days are static and don't have a 'ping' service to ping an aggregator. That led to the venerable Sean Corfield mentioning that he does not read a blog post if he does not see it on his Twitter feed. I suspect others get their updates this was as well. I'm still 'old school' I guess because I still use and online news reader. I don't have a solution yet for those static blogs, but I'm percolating thoughts on that. However, during the process of resurrecting ColdFusionBloggers.og I noticed that at one time there was a Twitter account for @CFBloggers. I checked with Ray Camden and luckily he was able to remember the credentials for this account and passed those on to me.

The current iteration of ColdFusionBloggers.og is coded in Node.js. I've been learning what I can about Node as fast as I can. This past weekend started the process with Twitter to get my API tokens. While waiting on those I learned what I needed to get Node to send a Tweet. Today I received the Twitter API access tokens and this evening I was able to debug my hacked up Node code. The first tweet was sent! WooHoo! I think this is all working.

The short story is all new blog posts aggregated by ColdFusionBloggers.og will now be tweeted in real time. This being the first blog post to tweet. Cheers!

I hope this is the first of many changes for ColdFusionBloggers.og. Contact me to have your ColdFusion related blog added to the feed.

ColdFusion Bloggers is Alive

A short note to alert everyone that ColdFusionBloggers.org is back online and back to aggregating your blog posts.

The great Raymond Camden created this awesome resource and technically it will always be his. I happen to be the current caretaker if you will of this valuable service. If anyone was wondering why Ray decided to step away from ColdFusion Bloggers, see his blog post here. I reached out to Ray and he granted me the 'keys' so to speak. Thank you Ray! I've moved hosting to my AWS account for hosting and tonight I was able to work through the hiccups of migrating older code to a newer environment on newer versions of the backend software. I just completed adding SSL via Lets Encrypt and verified the ping service works. ColdFusion Bloggers is back in action!

Spread the word! Add your blog!

Living Longer Than Dad

For the past two years I've been contemplating turning 50. It's not that 50 is scary or some huge significant number to me like it is for some. To the contrary I've been looking forward to reaching that significant number, because being 50 is something that for a long time I thought I'd never be. My father died of a massive heart attack at the age of 48. I was 28 at the time.

Over the next 20 years I've been married, divorced, changed careers, changed jobs numerous times, become reasonably successful in my new career, basically all the things people do. During that time it was always in my head that Dad didn't make it past the age of 48. Somewhere in there it set into my mind that I wouldn't make it past 48 myself. I know it's irrational. It's nonsensical. Its crazy talk. There no scientific way to determine how long you will live based on how long your father lived. Yet there it was, embedded in my mind, that my time was over by the age of 48. So much so that I was living my life not expecting to live longer.

Two years ago, when my age hit that number, I made a trip up north into the woods near the edge of the BWCA where we spread Dad's ashes. It had been twenty years since I was there last and it had been twenty years since I had a chance to sit with Dad, talk, have a drink, listen to his words. I went with two things, a flask of his favorite scotch and a cigar. After hiking in for about a mile I found the granite outcropping in the river where we spread his ashes. For the next hour or so I sat there smoking a cigar, taking a sip of scotch from the flask and pouring a shot from the flask into the river. After all, it's been twenty years since he had a drink. I sat here listening for his words. Trying to find peace of mind.

Two years later and two years older than my father I'm still listening for his words and trying to find my way in this world. Trying to find some peace in my mind. I don't know where life goes from here or how long life will last. I heard something a while back that inspired me to reflect a little and thus write my thoughts down today. "Be grateful for the extra innings."

More Entries