?

Log in

Power Loss' Journal
 
[Most Recent Entries] [Calendar View] [Friends]

Below are the 15 most recent journal entries recorded in Power Loss' LiveJournal:

Tuesday, July 24th, 2007
10:06 pm
[kshandra]
Artur Bergman for the win.
From this nicely comprehensive post about today's unpleasantness:

Since I am a cynical operation veteran, I need to link to this press release where 365 Main congratulates themselves on 2 years of 100% uptime.

Strangely, said link comes back 404, now....

Current Mood: amused
Saturday, November 4th, 2006
4:22 pm
[azurelunatic]
Looks like we had another.
But was it the sock-monkey this time?
Saturday, July 29th, 2006
11:46 pm
[azurelunatic]
Wednesday, May 3rd, 2006
12:22 am
[kshandra]
Here we go again....
Okay, so nobody pulled an actual plug this time.

At 6:44 am GMT on Wednesday, May 3rd, admin writes:

Since approximately 4:00 pm Pacific Daylight Time, Six Apart has been the victim of a sophisticated distributed denial of service attack. This has affected all of Six Apart's sites, causing intermittent and limited availability for TypePad, LiveJournal, sixapart.com, movabletype.org and movabletype.com. Our network operations staff is working around the clock with our Internet access providers to resolve the issue. We appreciate your patience and support, and will provide updates as we have them.

Update, 11:36 pm PDT: Access and availability to our sites has improved. We continue to work closely with our Internet access providers and will monitor the situation through the night.


lysana was quick on the draw(ing program):

Monday, February 21st, 2005
8:06 pm
[kjpepper]
Fuck, it's contagious!
currently on Wikipedia.org...

We're currently recovering servers from a power failure in our colocation facility. This means backing up 170gb of database on serveral servers and running recovery. Back soon...

sounds eerily familiar, hey?

(xposted to me journal)

Current Mood: well fuck.
Friday, January 21st, 2005
5:08 am
[azurelunatic]
Monday, January 17th, 2005
6:43 pm
[azurelunatic]
Paid users -- two weeks extra!
http://www.livejournal.com/community/paidmembers/15415.html

Go and get your two weeks' free paid time in apology for the troubles.
1:00 pm
[someway]
What I did:


And because everyone else is making icons to share:


Both icons made by me. Take and credit if you want.

~Sar
7:30 am
[bookofnights]

The Great LiveJournal
Outage of 2005


During the outage I hoped that pressing reload page would bring live journal back.


What did you do?


Brought to you by geek-foo


Sunday, January 16th, 2005
11:38 pm
[ljblackout]
7:20 pm
[sarekofvulcan]
Has everyone seen nimori's icons yet?
6:08 pm
[fairyfey]
Icons

Made a few "I survived" type icons. Check 'em out:

. . .Collapse )

Credit would be nice. ^^



Current Mood: amused
11:32 am
[rintheamazing]
It had to be done...
So, since nobody else has asked yet...

What did everyone do in those tragic LJ-deprived hours?

Current Mood: curious
3:13 pm
[vampwillow]
Singalonga
flick got bored waiting for the power to return and wrote ...

A long, long time ago...
I can still remember how
Our LJs used to make us smile.
And I knew if Brad had his chance,
That he could make those servers dance,
And maybe we'd be blogging for a while.

But InterNap made LJ shiver,
There were no pages to deliver,
No gossip on our browsers
About who lost their trousers...

I can't remember if I cried
When I saw my page request denied
But I had to face the world outside,
The day that LJ died.




Full version (and I do mean *full*) at http://www.livejournal.com/users/flickgc/389188.html
12:54 am
[azurelunatic]
Power Loss
This community was inspired by the brilliant http://www.livejournal.com/powerloss:
Our data center (Internap, the same one we've been at for many years) lost all its power, including redundant backup power, for some unknown reason. (unknown to us, at least) We're currently dealing with verifying the correct operation of our 100+ servers. Not fun. We're not happy about this. Sorry... :-/ More details later.

Update #1, 7:35 pm PST: we have power again, and we're working to assess the state of the databases. The worst thing we could do right now is rush the site up in an unreliable state. We're checking all the hardware and data, making sure everything's consistent. Where it's not, we'll be restoring from recent backups and replaying all the changes since that time, to get to the current point in time, but in good shape. We'll be providing more technical details later, for those curious, on the power failure (when we learn more), the database details, and the recovery process. For now, please be patient. We'll be working all weekend on this if we have to.

Update #2, 10:11 pm: So far so good. Things are checking out, but we're being paranoid. A few annoying issues, but nothing that's not fixable. We're going to be buying a bunch of rack-mount UPS units on Monday so this doesn't happen again. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize the first time wasn't a total freak coincidence. C'est la vie.

Update #3: 2:42 am: We're starting to get tired, but all the hard stuff is done at least. Unfortunately a couple machines had lying hardware that didn't commit to disk when asked, so InnoDB's durability wasn't so durable (though no fault of InnoDB). We restored those machines from a recent backup and are replaying the binlogs (database changes) from the point of backup to present. That will take a couple hours to run. We'll also be replacing that hardware very shortly, or at least seeing if we can find/fix the reason it misbehaved. The four of us have been at this almost 12 hours, so we're going to take a bit of a break while the binlogs replay... Again, our apologies for the downtime. This has definitely been an experience.

Update #4: 9:12 am: We're back at it. We'll have the site up soon in some sort of crippled state while the clusters with the oldest backups continue to catch up.

Update #5: 1:58 pm: approaching 24 hours of downtime... *sigh* We're still at it. We'll be doing a full write-up when we're done, including what we'll be changing to make sure verify/restore operations don't take so long if this is ever necessary again. The good news is the databases already migrated to InnoDB did fine. The bad news (obviously) is that our verify/restore plan isn't fast enough. And also that some of our machine's storage subsystems lie. Anyway, we're still at it... it's long because we're making sure to back up even the partially out of sync databases that we're restoring, just in case we encounter any problems down the road with the restored copy, we'll be able to merge them. And unfortunately backups and networks are too slow.

Update #6: We're up again, but only partially. Some database clusters are still reconstructing/syncing. See status.livejournal.com.

and especially also by robyn_ma's brilliant parody of same:
Our data center (Internap, which can suck our balls right about now) lost all its fucking power, including redundant backup power, for whatever goddamn pinhead reason. We're currently dealing with thousands of people umbilically tied to their LJ friends lists. Not fun. We're not happy about this. Sorry... :-/ More details later. Stop fucking calling us.

Update #1, 7:35 pm PST: we have power again, and we're using it to watch TV. We toyed with the idea of rushing the site up in an unreliable state, because having it work for half an hour and then crash on everyone would just be funny, but, naah. We'll be providing more technical details later, for those who masturbate over them, on the power failure (when we learn more), the database details, and the recovery process. For now, shut the fuck up. We'll be working all year on this if we want to. It's not like you can do anything about it anyway.

Update #2, 10:11 pm: So far so good. Things are checking out, but we're wearing tinfoil hats. A few annoying LJ users, but nothing that's not fixable. We're going to be buying a bunch of weed on Monday so that, if this happens again, we'll just be too baked to care. In the past we've always trusted Internap's insanely redundant power and UPS systems, but now that this has happened to us twice, we realize Internap sucks big koala dick.

Update #3: 2:42 am: OMFG WE ARE SO TIRED. *headdesk* WE KNOW THIS IS TAKING A LONG TIME. SHUT THE FUCK UP AND FIND SOMETHING ELSE TO DO. JESUS FUCKING CHRIST. *passes out*

Update #4: 9:12 am: Coffee is good.

Update #5: 1:58 pm: approaching 24 hours of downtime... *facepalm* We're still at it. You're still clicking on your friends-list bookmark and seeing this page. The world continues to spin.

Update #6: We're up again, but only partially. If you're in one of the fucked clusters, tough titty. At least you didn't die in a fucking tsunami or something. Fuck.
And then robyn_ma went on to say that a community dedicated to same would rule. So. Community. Yay.

Community rules? There really aren't many. If I think you're being an arse, I'll boot/ban you (but I shan't be nearly as sensitive on my definition of "arse" as I am elsecommunity). Posting should be made relatable in some way/shape/form to the topic (i.e., unless you can make a ratings comm pimping relevant to the topic of LJ losing power and going down, that post, and probably you, are out). Otherwise, have fun. Assuming anyone even drops in.
LJ power loss page   About LiveJournal.com