AT&T Blockheads

Yesterday about 2pm Wednesday 13 March, part of the Internet mysteriously stopped working. Connections through U-verse to a machine at HostMonster in Utah simply didn't connect.

Naturally this happened when integrating and testing a demo slated for 9am Thursday.

Connect drops at first were intermittent, which adds extra fun to setting up POP3 services.  Eventually, the link went down hard.

Computers running over a VPN connection, however, continued to talk with the target host just fine.  What was going on?

S- in AT&T's India service center had no idea. After 20 minutes of Quality Chat Time, he passed along the number to AT&T ConnectTech (866-393-7577, 8am-11pm Central Time).

The U-verse Residential Gateway wasn't the problem. Resetting it -- and later rebooting it -- fixed nothing.

Pinging about a bit, it was clear that hosts one IP address lower (*.*.*.175) and one IP address higher (*.*.*.177) were responsive.  Hmmm, what does traceroute say?

Using the vanilla U-verse connection, AT&T appears to have blocked the host somewhere in their network:

...
6 * * *
7 12.83.38.129 (12.83.38.129) 42.938 ms 44.590 ms 30.206 ms
8 ded3-g4-3-0.sfldmi.ameritech.net (151.164.40.106) 83.389 ms 79.699 ms 82.885 ms
9 * * *
10 * * *
11 * * *
...

traceroute reports that other HostMonster machines are reachable, and the program finds the vanished host through the VPN as well. A chat session with the good folks in Provo, Utah, confirms:

(7:46:43am)Christopher H.: Okay. We are aware of a block from AT&T to our servers. We have requested them to unblock us and we are currently waiting for their reply.

Fixed. In three minutes the host became reachable intermittently and fully within an hour or so as the blocks within AT&T's network cleared.

Yet, what about other U-verse customers?

To recap: AT&T blocked a host and didn't tell anyone.  It wasn't on their service website and the call center had no idea.  When alerted to the block, the cloud service/hosting provider  knew immediately what to do.  Thank goodness for competent people here in the U.S. of A.

So without any consumers knowing or being able to find out, an entire suite of sites disappeared for all AT&T customers.  Vanished.  Gone.  Businesses, private parties, charities -- the one machine hosts dozens and dozens of 'em  --  ceased to exist for a day as far as any U-verse user could tell.  Those people have no recourse to recover opportunity losses, missed ad clicks, lost Girl Scout Cookie sales, etc.

How is that fair?  And when would service have returned if one particular customer hadn't dogged them for a solution?

To be extra clear, the software being demo'd is a triage tracking system.  Fire fighters and rescue workers use it as a centralized coordination point to manage mass casualty incidents (MCIs) like bus rollovers, train and airliner crashes, or building collapses in earthquakes.

You may hurt people, AT&T.  Let's hope not.