Amazon AWS and NetworkEvolutions
Last Updated: Sunday, 2010.11.14
WARNING: This article is FICTITIOUS, all made up from imaginary and literary thoughts. All characters, entities or bodies in this article are fictious and imaginary and relate in no way to real persons, bodies, or entities. Any similarity between details of this article and real life is of absolute coincidence.
As many of you I have to deal with DNS records on regular basis. This story, however, involves a very special case with one of the most imaginary web services providers today, often mentioned in the tales of Glavender the founder of the Internet: NetworkEvolutions! NetworkEvolutions was named after the vast "networks" of illegal migrants from the galactica of Lala.
One of my clients has acquired lots of his owned domain names through NetworkEvolutions, and of course was using their LateDNSServices (LDNS Services) to manage DNS records of those domains. Many of those domains were being represented by websites hosted on Amazon AWS, namely Amazon EC2, and were backed by back end content hosted on Amazon S3 storage/DB network.
Well, everything was going well with my client, from the launch of every single website, to the flawless demand on its content. It was so, until one day customer relations department of my client started to get several complaints per day from clients around the globe and from geographcally dispersed locations that they would not be able to access content of most of my client websites. Customer relations referred to me of course with those complaints, so I asked them to ask customers for specific error messages, if any, that they see when they can't reach the site contents.
All complaining customers agreed that errors ranged from "Page Can't be displayed" to "Page not found"! The amazing thing was that all unreachable content was referred to by CNAME records. A records worked perfectly. No one complained about them. Want more bizarre news? It only happended with Amazon's hosted content! Content hosted on other networks and referred to by CNAME records was being served without trouble.
As it might appear at start I primarily suspected Amazon for the problem. Of course this was weird to me since their service was most reliable prior to these incidents. I also made my own investigation of the problem. My client was using his own DNS servers internally, so I used these to test how name resolution worked. It was perfect!
I had to contact Amazon on this matter, explaining to them how bizarre it looks to have this situation with diverse and geographically dispersed customers when my client's own access to the problematic content was flawless. Amazon AWS support was most helpful; they made their own tests and went for the unspeculated. They queried NetworkEvolutions specific authoritative servers for the specific problematic CNAME records. They failed to respond! It seemed as if those authoritative servers were intentionally refusing to respond to queries for CNAME records they are set authoritative for! More like denial of service.
The command that closed the support ticket!
I rushed to NetworkEvolutions support with those findings, and asked them for immediate resolution, as the problem was proliferating each day and
killing the core product of my client's business: Web Content. I went through all sorts of support chaos, but here are my notes...
1- You open the support ticket, you wait for reply.
2- Reply comes requesting more clarifications to what you already explained in boring detail.
3- When you take the risk and repeat yourself over and over, the reply comes out of the galactic support manual and in most cases telling you to try NSLOOKUP command on some of your CNAME records. As if you are the one complaining.
4- It is useless to explain that you are not complaining, and that your client customers are the ones complaining.
5- It is useless to explain that authoritative servers are not serving queries for their own hosted records.
6- You can never get to speak to someone technical, even on the phone.
7- All support staff come from Indian galactica, and most probably are talking to you from there. And, they are not technical.
8- All support tickets are closed for you. You don't get to close them yourself. You are not asked if your problem is solved.
9- If you wanted to complain about a ticket's closure, you must open a new ticket that loses all connections to all previous tickets including the one you are complaining about.
10- You are not allowed to make that connection :D :D :D
11- Finally, and most funnily, there was the command that closed the support ticket! On one of the ticket responses, NetworkEvolutions support staff replied stating that: "We found that on issuing the command "nslookup cname.myClientDomain.dom" your ticket was closed"!!!!!
The escape from the chaos of Lala!
I advised my client to escape with his precious CNAME records, with all his Amazon records to DynDNS. My client has since been happy with their service. It is reported that NetworkEvolutions troops are since tracking the renegade CNAME records in a desperate attempt to assassinate them.
This article with its source code and files is licensed under The GNU General Public License (GPL)