Waiting on VERY Large Data Transfers

What do you do while waiting for very large data transfers to complete? By large I'm talking upwards of Terabytes. My mind wanders and suddenly the theme for Rawhide gets stuck in my mind! But it's different. The music is similar, but the words have changed. It's almost like Dust Puppy was singing the song.

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes

Keep Flowin', Flowin', Flowin'
Though the datastreams are swollen
Keep them bytes Flowin', Raw Bytes
Through switches and wires and routers
Hell bent for bandwidth

Wishin' my smartphone was by my side
All the things I'm missin'
Good cat videos, Netflix and tweetin'
Are waiting at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Keep Flowin', Flowin', Flowin'
Though they're disapprovin'
Keep them bytes Flowin', Raw Bytes
Don't try to understand 'em
Just grep 'em, zip, and save 'em

Soon we'll be streamin' high and wide
My smartphone calculatin'
My true love will be tweetin'
Be waitin' at the end of my transfer

Move 'em on, zip 'em up
zip 'em up, move 'em on
Move 'em on, zip 'em up, Raw Bytes
Cut 'em out, pipe 'em in
Pipe 'em in, cut 'em out
Cut 'em out, Pipe 'em in, Raw Bytes

Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Flowin' Flowin' Flowin'
Raw Bytes
Raw Bytes

The things we do while migrating data into the cloud.

It's "Retired" Jim!

In another chapter of "The Cloud Never Crashes", I woke up Sunday to one of my AWS instances that was 'crashed' with a notice of "Amazon EC2 Instance scheduled for retirement". Retirement? What does that mean? I went to check my email and realized that the "retired" instance was the email server. Doh! It took me a little while to figure out what they meant. It means this "An instance is scheduled to be retired when AWS detects irreparable failure of the underlying hardware hosting the instance." This serves as a good reminder that the cloud is really someone else's server.

In theory this is an easy fix. The instructions at Amazon claims that stopping and restarting the instance will launch it on new hardware. In practice I could not get the instance to stop. This is where having physical hardware and a power cord to pull would have been nice. Failing to get the instance to stop I could not detach the EBS root volume. Even force detaching the EBS root volume didn't work. This is where daily snapshots of EBS volumes comes in handy. I was able to launch a new EC2 instance and then convert the last snapshot to an EBS volume and attach that to the new EC2 instance. Then I moved the elastic IP from the "Retired" instance to the new instance and hit "start'. Full recovery!

Now I'm left with a hanging EC2 instance that is still "Stopping" and an EBS volume that I cannot use, detach, delete etc. I tried reissuing stop commands a couple times. Eventually I noticed a "Force Stop" option. I do not remember seeing this on earlier attempts. I do not know if this shows up after the first failed stopped attempt or after several. I'm not sure, but I think that sends a trained monkey into the datacenter to pull the power cord. In any case it worked. This let me detach my EBS volume. From there was was able to stop the new instance, detach the EBS volume and attach my original EBS root volume. Now I have full recovery and I was able to clean up the loose ends.

Amazon Web Service has given us a new euphemism. Retired means It's Dead Jim!

CF Webtools is an Amazon Web Services Partner. Our Operations Group can build, manage, and maintain your AWS services. We also handle migration of physical servers into AWS Cloud services. If you are looking for professional AWS management our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.