AT&T Unblocked

My friend Mark Gibbs took pity -- he's had his Forbes nodes blocked too.  (If you don't read Gibbs's stuff, you should.  He's funny and sharp and has been in the industry a very long time.  Find him at Network World, Forbes, or Twitter.)   Mark hooked me up with an internal AT&T contact.

Mark's contact forward the inquiry to a Mr. G. who works as Associate Director for Customer Appeals, AT&T Office of the President.  After receiving a detailed synopsis, Mr. G. looked into it and replied the same working day (!):

Mr. Geiger:

What I have learned is whenever someone comes across a situation where their IP has been blocked then you should receive a message basically stating “The website you are trying to reach is hosted on a server that has been compromised and was involved in malicious activity. AT&T has blocked the IP address on our network”.  I am not sure what type of message you received, but the easiest way to address this in the future is by calling our U-verse Care center who is equipped to assist you with this matter.  Their number is 1-800-288-2020 and if needed they can engage extra support to handle this type of issue. We do also have directions on how to identify the vulnerability so you as the hosting provider can contact noc@att.net and specify the IP address being blocked for instructions on how to remove the vulnerability and get the block lifted.  Please let me know if this addresses your concern and should you have any additional questions please let me know.  Thank you!

Thank you too, Mr G, for a prompt response!  Points to the AT&T and the U-verse team.

Alas, no message as noted above appeared. What triggered the block and what the trigger thresholds are remain unclear.  Why is perfectly understandable: so baddies cannot circumvent them.  Perhaps it was a glitch in the attack detection software?  This block is the first seen in nearly five years of solid and reliable U-verse service and three years of using HostMonster.

The workaround is to have a small script monitor hosts and raise an alarm should they suddenly become unpingable via AT&T's network.

Many thank to Mark, too.  I owe you lunch.  We gotta go to that place I keep telling you about in Santa Paula.

AT&T Blockheads

Yesterday about 2pm Wednesday 13 March, part of the Internet mysteriously stopped working. Connections through U-verse to a machine at HostMonster in Utah simply didn't connect.

Naturally this happened when integrating and testing a demo slated for 9am Thursday.

Connect drops at first were intermittent, which adds extra fun to setting up POP3 services.  Eventually, the link went down hard.

Computers running over a VPN connection, however, continued to talk with the target host just fine.  What was going on?

S- in AT&T's India service center had no idea. After 20 minutes of Quality Chat Time, he passed along the number to AT&T ConnectTech (866-393-7577, 8am-11pm Central Time).

The U-verse Residential Gateway wasn't the problem. Resetting it -- and later rebooting it -- fixed nothing.

Pinging about a bit, it was clear that hosts one IP address lower (*.*.*.175) and one IP address higher (*.*.*.177) were responsive.  Hmmm, what does traceroute say?

Using the vanilla U-verse connection, AT&T appears to have blocked the host somewhere in their network:

...
6 * * *
7 12.83.38.129 (12.83.38.129) 42.938 ms 44.590 ms 30.206 ms
8 ded3-g4-3-0.sfldmi.ameritech.net (151.164.40.106) 83.389 ms 79.699 ms 82.885 ms
9 * * *
10 * * *
11 * * *
...

traceroute reports that other HostMonster machines are reachable, and the program finds the vanished host through the VPN as well. A chat session with the good folks in Provo, Utah, confirms:

(7:46:43am)Christopher H.: Okay. We are aware of a block from AT&T to our servers. We have requested them to unblock us and we are currently waiting for their reply.

Fixed. In three minutes the host became reachable intermittently and fully within an hour or so as the blocks within AT&T's network cleared.

Yet, what about other U-verse customers?

To recap: AT&T blocked a host and didn't tell anyone.  It wasn't on their service website and the call center had no idea.  When alerted to the block, the cloud service/hosting provider  knew immediately what to do.  Thank goodness for competent people here in the U.S. of A.

So without any consumers knowing or being able to find out, an entire suite of sites disappeared for all AT&T customers.  Vanished.  Gone.  Businesses, private parties, charities -- the one machine hosts dozens and dozens of 'em  --  ceased to exist for a day as far as any U-verse user could tell.  Those people have no recourse to recover opportunity losses, missed ad clicks, lost Girl Scout Cookie sales, etc.

How is that fair?  And when would service have returned if one particular customer hadn't dogged them for a solution?

To be extra clear, the software being demo'd is a triage tracking system.  Fire fighters and rescue workers use it as a centralized coordination point to manage mass casualty incidents (MCIs) like bus rollovers, train and airliner crashes, or building collapses in earthquakes.

You may hurt people, AT&T.  Let's hope not.

AT&T U-verse Suffers Major Outage

News reports indicate that AT&T has suffered an outage affecting 1% of customers in Florida and fourteen other states.

AT&T's website has only scant information.

Of course, it's impossible to see that page if you're trying to get to it via U-verse. Ah well.

It's working fine here in eastern Ventura County, California.

Be interesting to know what was the cause of such a wide-spread failure.

News coverage:

•  U-verse outage affecting 1 percent of customers (23 Jan 2013) 
•  AT&T blames U-verse outage on 'issue with server' (22 Jan 2013)

Torrents: the new TiVo

Dear CBS, NBC, ABC, FOX, et alii:

Please put out your shows as torrents.

We know you have contractual obligations with cable providers, iTunes, NetFlix, Hulu, and others.  Hang on for a minute and hear us out.

Torrents are the new TiVo.  More importantly, torrents are the new TiVo with a near-ZERO distribution cost.  Once seeded, your shows reside on 1000s of your customers machines.  You need never build a delivery service nor ever buy bandwidth.  No Amazon S3 cloud which died repeatedly, taking out Netflix.  No global data center placement hassles or edge location services like Akamai. No software "apps"; they're built.  Shows flow from you to consumer, world-wide, for free.

Your Nielsen nerds can track who downloads and how many times so you know which programs and episodes start and stay popular.  We know you read torrent tracker data; put it to work.  Use it to sell commercials.

And we will watch the commercials.  

We promise.  Shows' writers pace scripts around commercials.  We need those advertisements to (a) give our minds a pause from the subtleties of "Real Beverly Hills Housewives" and (b) make sense of consumer society.  Without commercials, we lose track of when the latest Doritos flavor becomes available or which movies to enqueue or what latest neato iThingy to feel smug about iNstalling.

In truth, we need the commercial breaks to go get more Doritos.

Torrent your product.  We watch, you profit. Then you go make more episodes of "Two and a Half Men" or whatever else Chuck Lorre poops onto paper this week.  You know we will watch anything, anywhere, anytime.  It's doomed to happen.  Make it so.


Sincerely,

:Couch Nerds of America

Out of frying pans, into fires

Fox Business news reports that Verizon may be interested in purchasing Netflix.Verizon also announced plans to buy wireless spectrum from ComCast, TimeWarner, and Brighthouse.

CATV is no longer where it's at. Virgin Media admitted they earn more from their Internet services than from TV in an SEC filing. Naturally: they pay content licenses to supply that service.  Content cost is pushed onto the consumer, who's licensing directly from studios or indirectly from Netflix, Amazon, Vudu, or whomever.

More consumers are watching shows on gaming consoles and other appliances.  Sony is thinking about starting its own TV content service.  Given the break-ins on Sony's Playstation Network, they may wish to harden their systems first.

Interesting times.

What will AT&T be doing?  Probably sending yet another "Please buy our U-verse TV" letter.

Thank you, NBC

Thank you NBC for streaming USC v. Notre Dame today.

We will watch just about any grade video and even the same crummy ad that airs one hundred eighty-five times per game (can't NBC sell ad time to more than one company?) in order to see this game. That NBC streams High-Definition video is just flat excellent.

What's interesting: the costly $60 month television package from U-verse. AT&T is trying to be a cable television company, selling bundles of channels, at a flat rate. It's just like the record companies selling entire albums for $16-$20 when the customer really wanted one or two songs, then wondering why file sharing destroyed their market.

We don't really want a business relationship with a cable TV provider, or, for that matter, a TV channel. We want to watch The Game, see The Movie or The Show, or hear The Music. U-verse is a means to an end. The fewer companies between the consumer and the consumed, the better.

NBC gets it. At least for now.

Netflix Nixes Qwikster

What's going on, Netflix?  Three weeks almost to the day later, and perhaps after Reed Hastings smoked a different bowl, Netflix nixes Qwikster:

Weirdly, Netflix didn't begin its backtrack with "I messed up," the preferred language for an apologia at the company (a, b, c). And no long-winded, rambling explanation, either. The message is signed "The Netflix Team", although the official blog post is singly signed "Reed".

This move comes after a weekend of intermittent downtime:

The we-are-betting-the-company-on-it streaming service becomes unavailable at 9pm Saturday night?

At this point, investors should be wondering who on the Netflix board "messed up" by keeping these guys around.

What's going on, Netflix?

---

Update: a detail article from The Wall Street Journal at 1:40 EDT / 10:40 PDT.