GlennLog

Turning technology from mumbo-jumbo into rich tasty gumbo

< Dr. Who and Harry Potter | Main | Quest for Qwest >

July 29, 2007

Outnumbered, Lynn Takes to Blogging; Power Outages

Like a rock against which the ocean slams for eons, finally wearing it down to a nubbin, pebbles, and sand, so, too, has Lynn finally taken up blogging after years of me doing so. Lynn is a marvelous writer, which I say not just because I'm married to her (although her email to me in our early dating demonstrated her considerable store of wit, charm, and intelligence). With three Fleishmans in the house, if Lynn is ever going to get a word in edgewise, she  needs a forum.

In other news, power was fluctuating all over last night. At about 11.30 pm, when I had just managed to drop off, the power went out, silencing the "rain music" we use to help Rex sleep, turning off the A/C (which was mostly working as a fan), and turning off my CPAP. I got up and looked around through various windows, and it was our usual (infrequent) outage in which I can see lights at the top of the hill (Eastlake neighborhood's ridge/South Capitol Hill) and across the water (Wallingford).

We worried Rex would wake up either from my walking around, the lack of white noise, or the lack of light. I  was also concerned Ben would wake up and it would be pitchblack in his room with his night light out and neighbors' lights out. It was a full moon, though, and there was plenty of light leaking through the blinds.

Lynn suggested I pull out a battery system I got for camping and emergencies, that can deliver at least one night's electricity to my CPAP (with the humidifier turned off). Of course, I had stored it nearly behind the baby's crib. I did manage to get it out without making a massive amount of noise, hooked it up, and went back to sleep. At about 3 am, power came back on, fans and white noise powered back on, and we all went back to sleep again. The baby slept til 6.30, which is pretty unbelievable these days (5 to 5.30 am have been his usual range).

When I went to check email in the morning, I discovered that my database server had crashed overnight--it's a pretty fast machine by standards three years ago that just handle database operations for my various sites and partners (isbn.nu, db.tidbits.com, and wifinetnews.com, to name a few). It's hosted by digital forest, which has a very good track record on both their physical infrastructure and recovery from problems. I checked logs, and couldn't find any clue as to why it restarted twice in the wee hours. It managed to mostly recover, meaning my sites weren't down for long. Digital forest, when queried, said there was an anomaly with the rack on which the server lives, and they're investigating.

Here's the funny part: After the power outages and server downtime in San Francisco this last week, I was bragging to all and sundry that I had over 550 days of continuous uptime on this database server. Ah, well. Never tempt the gods of small particles and poetic justice.

Update: Digital forest discovered that another customer had used a power strip into which my database server was plugged. This is a big no-no. D.f., like all intelligent co-location facilities ("co-lo's," colloquially) has staff let customers into the co-lo and has cameras monitoring the facility, too. But that doesn't mean that someone can't do something stupid. This other customer's server had problems and was power cycling. Power cycling can draw a lot of peak power at startup, as fans run at maximum, drives all fire up, and the processor runs at full blast. This tripped a circuit breaker (twice, apparently), and brought my server down with it. The problem has been rectified, although I don't know what will happen to that other customer. (The correct procedure is to ask the on-site technical staff before plugging stuff in.)

Modern servers pull hundreds of watts of power each, if not more. This requires careful management to distribute power and avoid overheating and overloading. Digital forest just posted these photos of a new, still-in-stealth customer that is running 528 watt per square foot. A standard rack is 42U, where U is a unit equal to 1.75 inches. Looking at the photos, it's likely that the rack is pulling several kilowatts (d.f. says they can feed 50 kW per rack). Tricky stuff. They note that the customer's previous host made them divide their machines across four racks. Because we're charged by the rack (or half-rack), that's a rather expensive difference. Digital forest surcharges $20 per month per outlet, which is how they make up the electrical charge, however.

D.f. doesn't list their rack prices, but I'm guessing that they charge about $1,200 per rack, which would mean $4,800 for four racks. I'm not sure what this heavy power user's previous co-lo charged them per rack, but with the number of computers in the rack in the photo and the power surcharge, that would be more like $2,000 instead of $5,000.

That's all to say that you don't mess with plugging stuff in in a well-run co-lo. The co-lo does it or tells you where to.

Posted by Glennf at July 29, 2007 8:56 AM

Trackback Pings

TrackBack URL for this entry:
https://db.isbn.nu/mt3/mt-tb.pl/4712

Comments

Post a comment

Name:

Email Address:

URL:
Remember Me? YesNo

Comments: