Ooops - 3

So we recently built a new site with online ordering for a company with 400+ franchises in multiple countries. It was a bit stressful for me taking the site live, and we pulled some long hours to work around a few oddities in the hosting environment. None of this was helped by the guy who is supposed to be in charge of the site (the client's head of IT). Then today, we get this awesome email from him.

I am the IT Admin for the <Client X> site and I do not have access to the File Manager portion of the <Hosting Environment> Admin site to archive the logs. Is this something that your admin has access to or could grant me access?

<Name Redacted>

XMission Outage
---------------

XMission experienced a serious outage while we were performing some
standard UPS maintenance today. The outage affected all services and
started at approximately 2:00 p.m. on Tuesday, November 11th. Network
services for many were partially restored by about 2:30 p.m. but some
other services required a lot of attention and took much longer.

Details
-------
About 40% of our data center, including our server room, suffered a power
outage when a technician flipped a mislabled breaker during some standard
maintenance on one of our 3 UPS units. Although the power outage was
momentary, servers and routers often respond very poorly to losing power
and sometimes take extensive work to come back up. Unfortunately, such was
the case today with many systems.

Seriously Affected Systems
--------------------------
** An important router, which some connections and servers rely on,
required extensive attention from our network administrators.

** DNS (Domain Name Service) was sporadic for some customers for over an
hour.

** Email services were down for over 5 hours.

** Web hosting suffered the longest outage because our NetApp storage
appliance which houses all customer files and web sites lost multiple
hard drives. As a result, we are currently restoring files to our new
NetApp 2020 from our November 9th backup, which will take many hours
yet to complete. We recently purchased this new NetApp and were merely
days away from getting it online.

Conclusion
----------
Today's outage was exascerbated by multiple systems responding poorly to
losing power. In spite of the holiday, our systems administrators were on
site within minutes and continue to work tirelessly to restore all
services. In the end, we should have performed this maintenance on a day
when our systems administrators were on site because problems can arise no
matter how carefully you proceed.

He Said It

MG:
so you're coming up for Rock Band and Racquetball, right?

AoD:
OH, YES

MG:
great

AoD:
That's like asking someone if they're up for a great meal, a nice hotel, and awesome sex. Clearly Ming's is the great meal, I'll leave you to deduce the other two relationships

My lords, my ladies (and everybody else here not sitting on a cushion! ) let me present a link to a comment AoD made this week.


Continue reading "From the New Game Show"