AT&T Unblocked

My friend Mark Gibbs took pity -- he's had his Forbes nodes blocked too.  (If you don't read Gibbs's stuff, you should.  He's funny and sharp and has been in the industry a very long time.  Find him at Network World, Forbes, or Twitter.)   Mark hooked me up with an internal AT&T contact.

Mark's contact forward the inquiry to a Mr. G. who works as Associate Director for Customer Appeals, AT&T Office of the President.  After receiving a detailed synopsis, Mr. G. looked into it and replied the same working day (!):

Mr. Geiger:

What I have learned is whenever someone comes across a situation where their IP has been blocked then you should receive a message basically stating “The website you are trying to reach is hosted on a server that has been compromised and was involved in malicious activity. AT&T has blocked the IP address on our network”.  I am not sure what type of message you received, but the easiest way to address this in the future is by calling our U-verse Care center who is equipped to assist you with this matter.  Their number is 1-800-288-2020 and if needed they can engage extra support to handle this type of issue. We do also have directions on how to identify the vulnerability so you as the hosting provider can contact noc@att.net and specify the IP address being blocked for instructions on how to remove the vulnerability and get the block lifted.  Please let me know if this addresses your concern and should you have any additional questions please let me know.  Thank you!

Thank you too, Mr G, for a prompt response!  Points to the AT&T and the U-verse team.

Alas, no message as noted above appeared. What triggered the block and what the trigger thresholds are remain unclear.  Why is perfectly understandable: so baddies cannot circumvent them.  Perhaps it was a glitch in the attack detection software?  This block is the first seen in nearly five years of solid and reliable U-verse service and three years of using HostMonster.

The workaround is to have a small script monitor hosts and raise an alarm should they suddenly become unpingable via AT&T's network.

Many thank to Mark, too.  I owe you lunch.  We gotta go to that place I keep telling you about in Santa Paula.

AT&T Blockheads

Yesterday about 2pm Wednesday 13 March, part of the Internet mysteriously stopped working. Connections through U-verse to a machine at HostMonster in Utah simply didn't connect.

Naturally this happened when integrating and testing a demo slated for 9am Thursday.

Connect drops at first were intermittent, which adds extra fun to setting up POP3 services.  Eventually, the link went down hard.

Computers running over a VPN connection, however, continued to talk with the target host just fine.  What was going on?

S- in AT&T's India service center had no idea. After 20 minutes of Quality Chat Time, he passed along the number to AT&T ConnectTech (866-393-7577, 8am-11pm Central Time).

The U-verse Residential Gateway wasn't the problem. Resetting it -- and later rebooting it -- fixed nothing.

Pinging about a bit, it was clear that hosts one IP address lower (*.*.*.175) and one IP address higher (*.*.*.177) were responsive.  Hmmm, what does traceroute say?

Using the vanilla U-verse connection, AT&T appears to have blocked the host somewhere in their network:

...
6 * * *
7 12.83.38.129 (12.83.38.129) 42.938 ms 44.590 ms 30.206 ms
8 ded3-g4-3-0.sfldmi.ameritech.net (151.164.40.106) 83.389 ms 79.699 ms 82.885 ms
9 * * *
10 * * *
11 * * *
...

traceroute reports that other HostMonster machines are reachable, and the program finds the vanished host through the VPN as well. A chat session with the good folks in Provo, Utah, confirms:

(7:46:43am)Christopher H.: Okay. We are aware of a block from AT&T to our servers. We have requested them to unblock us and we are currently waiting for their reply.

Fixed. In three minutes the host became reachable intermittently and fully within an hour or so as the blocks within AT&T's network cleared.

Yet, what about other U-verse customers?

To recap: AT&T blocked a host and didn't tell anyone.  It wasn't on their service website and the call center had no idea.  When alerted to the block, the cloud service/hosting provider  knew immediately what to do.  Thank goodness for competent people here in the U.S. of A.

So without any consumers knowing or being able to find out, an entire suite of sites disappeared for all AT&T customers.  Vanished.  Gone.  Businesses, private parties, charities -- the one machine hosts dozens and dozens of 'em  --  ceased to exist for a day as far as any U-verse user could tell.  Those people have no recourse to recover opportunity losses, missed ad clicks, lost Girl Scout Cookie sales, etc.

How is that fair?  And when would service have returned if one particular customer hadn't dogged them for a solution?

To be extra clear, the software being demo'd is a triage tracking system.  Fire fighters and rescue workers use it as a centralized coordination point to manage mass casualty incidents (MCIs) like bus rollovers, train and airliner crashes, or building collapses in earthquakes.

You may hurt people, AT&T.  Let's hope not.