Examples of ATT Worldnet outages.

Dec 1, 1999 (8 hours)
April 6, 2000 (over 24 hours)
Feb 17 - 19, 2002 (two days)
Feb 28 - Mar 6, 2002 (Address blocked by ATT)
August 2-6, 2002  WorldNet blocks incoming mail from many domains

-------- Original Message --------
Subject: Re: Strange problem, not sure where it should go...
Date: 1 Dec 1999 22:07:48 -0000
From: AT&T Gage Robinson
Organization: AT&T WorldNet Service - NCMT
Newsgroups: worldnet.help.service-issues.misc

Greg, the service problems we encountered today were due to a configuration
error with WorldCom's backbone routers that caused all requests to our DNS
servers to be re-routed.  Stuff never reached our DNS servers. This was a
Worldcom (MCI) misconfiguration much like the two episodes we had earlier in
this year.  Unfortunately, WorldNet cannot correct such mistakes as it affects
the entire internet backbone and must be done at the source of the trouble.
We expect an official statement soon.
>
> Jay O'Brien wrote:
>
> > AT&T Nick Trifonoff wrote:
> > >
> > > Jay,
> > >
> > /snip/
> >
> > > WHat we currently have:
> > >
> > > The service problems we encountered today have been resolved. The
> > > problem was due to a router configuration error with another ISP that
> > > interfered with our own DNS servers. That ISP was contacted and the
> > > problem resolved. An official statement is being prepared and will be
> > > shared with you shortly.
> >
> > WOW!  Of course, the next question, WHY DID IT TAKE SO LONG?
> >
> > Nick, I don't expect you to answer, but I think it is a good question,
> > and I hope the "official" announcement addresses the timeliness of the
> > fix.
> >
> > Jay O'Brien
>
> --
> Gage Robinson
> AT&T WorldNet Service Customer Care - NCMT
> NOTICE: Please be aware that our first priority is to respond to the
> Service Issues newsgroups. Please post your issues and concerns there
> as I may not be able to respond to your personal email.
 

-------- Original Message --------
Subject: Re: No service??
Date: 1 Dec 1999 22:49:37 -0000
From: Jay O'Brien <jayobrien@att.net>
Organization: AT&T WorldNet Services
Newsgroups: worldnet.help.service-issues.mail-server

AT&T Gage Robinson wrote:
>
> It took so long because we had to figure out where the problem was,
> and once we did the other ISP had to get its engineers to investigate
> and make the corrections.

Ok, but given this post you just made to another newsgroup, isn't it
reasonable to question what happened to the history of the previous
episode that should have enabled AT&T to fix this in less than 8 hours
THIS TIME? My question stands, waiting for the "official" announcement.
Why so long to fix it?

Jay O'Brien
 

-------- Original Message --------
Subject: Attention Worldnet Customers!
Date: 2 Dec 1999 00:58:01 -0000
From: newsadm@netnews.worldnet.att.net
Organization: AT&T WorldNet Service
Newsgroups: worldnet.announce

At approximately 8:15 AM [EST] on Wednesday, December 1, AT&T began
experiencing a series of service interruptions on its dial Internet Protocol
[IP] platform. The impact was nationwide in scope and affected AT&T's
Worldnet Service, Business IP Dial Service, and portions of its Virtual
Private Network service.  AT&T's IP backbone was not affected by this
incident, nor were there any service disruptions to customers using AT&T's
dedicated Internet access service.

Reports indicate that incorrect routing by another Internet Service Provider
[ISP] generated this problem. That routing error has been corrected, the
network has been stabilized, and service has been restored.

We apologize for any inconvenience these interruptions may have caused.
Thank you for using AT&T WorldNet Service.
 

-------- Original Message --------
Subject: Unsatisfactory wrapup announcement
Date: 2 Dec 1999 01:24:26 -0000
From: Jay O'Brien <jayobrien@att.net>
Organization: AT&T WorldNet Services
Newsgroups: worldnet.help.service-issues.misc,worldnet.help.service-issues.mail-server

This announcement is patronizing. It doesn't name the perpetrator (MCI)
who was identified on a newsgroup by an AT&T employee. It doesn't explain
why it took so long, when we were told this was what had happened before
(see below). It doesn't tell us what is being done to reduce the time
necessary to identify and repair future outages. It doesn't tell us what
AT&T is going to do to keep us as customers.

This is what we get for an eight hour outage?

I'm disappointed. And just last Saturday I posted a message praising AT&T.

Jay O'Brien
 
 

-------- Original Message --------
Subject: Information Regarding Network Problems on 12/1/99
Date: 2 Dec 1999 23:48:02 -0000
From: newsadm@netnews.worldnet.att.net
Organization: AT&T WorldNet Service
Newsgroups: worldnet.announce

An Apology... And an Explanation.

On December 1, 1999, there was an outage that affected AT&T WorldNet®
Service members.  First I would like to apologize for any inconvenience
this may have caused, and second, I'd like to explain what happened.

This problem was not caused by AT&T's network.  However, it did affect
our system, and our members.  Another Internet Service Provider made a
network change that caused the public Internet to have incorrect
information on how to reach AT&T WorldNet servers.  As a result AT&T
WorldNet customers could not browse the Internet and mail from other
Internet Service Providers to AT&T WorldNet members was delayed.  AT&T
engineers worked to isolate the problem and provided a work around to
restore service by 4:30 p.m.  In addition, AT&T notified the other
Internet Service Provider, who in turn corrected their configuration.
The smooth operation of the entire Internet depends on accurate
information and cooperation from all Internet Service Providers.  This
reflects the distributed nature of the Internet and its design
characteristics.

AT&T WorldNet Service continually works to improve its service.  In
fact, based on a recent study by Solomon Wolff Associates, AT&T WorldNet
Service was rated as having the highest level of customer satisfaction
among the five largest Internet Service Providers.

Again, you have our sincerest apologies for yesterday's outage.  Please
be assured that we are working to prevent this kind of an interruption
in the future.

Thank you for being an AT&T WorldNet Service customer.

Ed Chatlos AT&T WorldNet® Service Vice President & General Manager
 

================================================================
(Please do not reply via e-mail.  The address is automated and
unattended. Messages sent to it will not be seen by AT&T.)

-------- Original Message --------
Subject: Re: An Explanation?
Date: Thu, 06 Apr 2000 20:01:05 GMT
From: AT&T David Alexander
Reply-To: dhaatt@att.net
Organization: AT&T Worldnet Services
Newsgroups: worldnet.help.service-issues.mail-server

Jay,
That is the only information I have available at this time.

On Thu, 06 Apr 2000 19:54:52 GMT, Jay O'Brien <jayobrien@att.net>
wrote:

>That's it?  That's all we get as an explanation for a mail outage
>of over 24 hours? You think we don't know already that it "was a
>software issue with the email servers"?
>
>I agree that no mail was lost. However, it was delayed long enough
>that I had to send mail again. What I sent "later" arrived before
>what I sent "first". Again, I agree, mail wasn't lost; some was,
>however, delayed for many hours and arrived out of sequence.
>
>Ok, nobody is perfect, but why not explain what happened?
>
>
>AT&T David Alexander wrote:
>>
>> Jay,
>> Apparently it was a software issue with the email servers.
>> The following was posted on our SERVICE BULLETINS page at
>> http://care.worldnet.att.net/bulletins/index.html
>>
>> "AT&T WorldNet Service customers may have been experiencing delays in
>> receiving email over the last two days.  As of this morning, April
>> 5th, this software issue has been resolved by our network engineers.
>> Please be advised that no emails were lost during this time period.
>> We appreciate your patience and apologize for this inconvenience."
>>
>> On Thu, 06 Apr 2000 12:16:09 -0700, AT&T David Alexander
>> <dhaatt@att.net> wrote:
>>
>> >Jay,
>> >We do not have any further explanation at this time, however, I will
>> >check again.
>> >
>> >On Thu, 06 Apr 2000 19:07:15 GMT, Jay O'Brien <jayobrien@att.net>
>> >wrote:
>> >
>> >>Any idea when we will hear the "post mortem"?
>> >>
>> >>AT&T Gage Robinson wrote:
>> >>>
>> >>> Our folks are still monitoring the email, when we get a post
mortem on the email impairment we will know what occurred.
>> >>>
>> >>> Jay O'Brien wrote:
>> >>>
>> >>> > How and where will we get the explanation you have promised?
>> >>> >
>> >>> > This is the first att outage I've experienced where att acts
like the other ISPs, that is, provide no information to the customers
about what is going on.
>> >>> >
>> >>> > At least you aren't avoiding the problem by telling me to reload
Windows (pacbell's first response to any problem).
>> >>> >
>> >>> > Jay O'Brien
>> >>> >
>> >>> > AT&T John Oliver wrote:
>> >>> > >
>> >>> > > Mary,
>> >>> > >
>> >>> > > There will be an explanation provided once all the data from the outage
>> >>> > > has been compiled and the situation has been removed from monitoring
>> >>> > > status.
>> >>> > >
>> >>> > > Mary Samios wrote:
>> >>> > > >
>> >>> > > > Will an explanation be forthcoming regarding the multitude of e-mail
>> >>> > > > problems of late?  My own seems to be remedied now (finally received a
>> >>> > > > multitude of "test" messages to my various e-mail addresses!<G>), so this
>> >>> > > > is more a matter or curiosity, than anything else.
>> >>> > > >
>> >>> > > > TIA,
>> >>> > > >
>> >>> > > > M Samios
 
 

------- Original Message --------
Subject: Re: An Explanation?
Date: Thu, 06 Apr 2000 21:11:19 GMT
From: "AT&T Ed Kelley"
Organization: AT&T Worldnet
Newsgroups: worldnet.help.service-issues.mail-server

Jay

The details of the software we use and how it works are considered
proprietary. Full disclosure of the root of the problem would entail
disclosing some of that information. As such, the best we can say is that
the problem was due to a software configuration issue.

--
Ed Kelley
AT&T WorldNet Service Customer Care - Newsgroups Channel Manager