Our experience with Earthlink ('Earthlink sucks')

Our experience with Earthlink ('Earthlink sucks')

The following is a short chronicle of our experiences hosting a dedicated server at the Earthlink Cloud facility in New York state.

Before this we had two servers based in Amsterdam with a local IT firm. One had suffered a (RAID1) drive failure and the other needed a 'hands on' upgrade. It was time to move on.

After some searching we found a promising company in the US providing dedicated server hosting with explicit support for Debian GNU/Linux and gave them a call.

Unbeknownst to us at the time they had recently been subsumed into the Earthlink network.

Unable to Linux

The first task was for them to install a basic Debian server with SSH so we could log in remotely and start work on configuring the web server and transferring websites.

What a child could manage in less than an hour ended up taking them three and a half weeks. It appears they had/have no-one with Debian experience and no sign of a competent Linux/UNIX sysadmin.

But the server was finally put online, we received a credit note for time lost, and we pressed forward.

Completely NAT'ted

The Earthlink Cloud hosting centre uses NAT to provide a secure environment. This basically means a gateway between the Internet IP addresses and their own internal network of IP addresses.

This is good for security, but makes some sysadmin tasks difficult as the server itself is unable to see the Internet IP addresses and must instead work with only local addresses.

We eventually came up with a one-line firewall trick enabling the server to get around this, but not before Earthlink threatened to charge hundreds of dollars for just daring to ask them about a solution.

Turned off our server

In early 2012, just as we were getting our websites onto the new server, it suddenly disappeared. Some late night phone calls revealed a disturbing sequence of events:

• Earthlink received an automated email from an email filter provider suggesting (incorrectly) that our server had sent spam to one of their customers;
• Without contacting us, Earthlink then broke into our server using the local console (disabling the firewall and changing the root password in the process) and started poking around;
• After trying to read as text a log file that is actually in binary (lots of strange characters) they concluded the server was 'hacked' and shut it down. Seriously!

467 vi /etc/ssh/sshd_config
468 service sshd restart
469 service ssh restart
470 exit
471 vi /etc/ssh/sshd_config
472 service ssh restart
473 netstat -an | less
474 history | less
475 iptables -L
476 cat /etc/hosts.allow
477 iptables -L | less
478 vi /etc/ssh/sshd_config
479 service ssh restart
480 passwd root
481 ls
482 pwd
483 iptables -L | less
484 cd /var
485 ls
486 cd /log
487 ls -l
488 cd log
489 ls
490 ls -ltr
491 ls -ltr
492 tail -f wtmp
493 clear
494 tail -f syslog
495 ls -l
496 tail -f wtmp
497 shutdown
498 shutdown -a

Fortunately we only had a few dozen websites running there at the time, but they were offline for a whole business day.

Afterwards we received a formal apology from Earthlink and a promise not to interfere with the server again without first contacting us. A promise they didn't manage to keep.

Broken payment system

In 2013 when we tried to pay the renewal invoice our credit cards were not accepted. Also when they tried to input our details manually over the phone. Not just us, apparently, but anyone with an address outside the US, even just over the border in Canada, was/is unable to pay by credit card without a US ZIP code.

Earthlink then proceeded to give us the wrong bank details for a wire transfer - over and over again for a couple of weeks - before finally getting it right. The late fee was eventually waived.

Broken support system

Their average response time for our (infrequent) support tickets was longer than one week - if we ever heard back from them at all. It didn't matter if it was a Technical or Billing enquiry. And invariably the first response was either incorrect or incomplete. Very frustrating.

Their latest approach to questions they can't answer is to offer a "remote hands" service costing in the order of US$200/hour.

Hacked into our server - again

In 2014 Earthlink broke into our server - again - to run tests for the Shellshock exploit vulnerability - without our knowledge or permission. At least they didn't break anything this time, but it wasn't appreciated. God knows how many other servers they violated at the same time.

Couldn't add an IP address

When we requested an extra IP address we were told that they were unable to provide one because 'the next adjacent number was being used'. Complete nonsense of course as an ip address can be taken from anywhere in the network - the entire point of NAT is that you don't/can't run out of addresses.

At this point we set up a new Linode server to host our SSL websites. With Linode you can request a new address as soon as you have a new SSL certificate ready to install.

Forget about IPv6

Our new host Linode is fully IPv6 capable. Every server is allocated a unique IPv6 address and you can receive on request (for free) your own block of 18,446,744,073,709,551,616 IPv6 addresses to allocate as you wish. This is the future.

Because the entire Earthlink network is behind a NAT firewall, any attempt to implement IPv6 would result in the whole network being exposed to the Internet, so it's not likely they'll be implementing it any time soon.

Still can't accept payments

Would you believe that in 2014 their online payment system still wouldn't accept non-US credit cards, and worse, that they tried to give us the exact same incorrect wire transfer details as a year earlier?

Extortion

In December 2014 we finally left Earthlink, after moving all remaining websites to another new cloud server over a period of 2-3 months. We 'toasted' the server shortly after Christmas and let them know we were gone.

In the meantime we continue to receive invoices that don't reflect our wire transfer payments, the switch from annual to monthly payments, or the removal of several IP addresses from our account.

And they expect us to pay an extra 30 days for a server that is literally completely empty.

So all up it was a sad and sorry experience, but educational. We now have most/all of our servers running in the cloud with Linode, where resources are unlimited and support requests are responded to in a matter of minutes with no complications.