LexiConn's "office" after the hurricane (click image to enlarge)

Hurricane Irene’s direct hit of Connecticut made for an eventful few days. Even though it was downgraded to a tropical storm right before landfall, that didn’t seem to make much of a difference in terms of power outages and destruction.

It had been 26 years since a hurricane hit our state directly, so this was quite the rare event.

Here’s a quick summary of how it all unfolded for Team LexiConn…

Before Sunday

In the days leading up to the hurricane, we made sure all of our disaster recovery plans were in place. These included:

• Verifying all shared servers were remotely backed up to a datacenter in Washington DC.
  (We do this every night for every shared hosting client for disaster recovery purposes.)

• Verifying all local backups were working properly.

• Verifying datacenter preps were completed including topping off diesel fuel, a schedule for being manned during the event, and fuel deliveries were scheduled.

Sunday

The hurricane started hammering the coastline of the state just after midnight. Both datacenters (DC1 and DC2) lost street power by 6:00 AM. Diesel generators kicked in as expected and no major issues were encountered.

By noon on Sunday, they estimated almost three-quarter of a million people were without power in Connecticut. Major flooding was occurring in many towns near the coast and near rivers. All LexiConn employees had lost power at their homes.

Winds stayed strong throughout the day, still gusting over 35 mph Sunday evening. Travel was treacherous at best due to downed lines and trees, as well as falling branches.

Monday

A fuel delivery for DC1 was scheduled for the morning. Unfortunately, since the cities the datacenters are in were 100% without power, hospitals and clinics took priority. This delayed the fuel delivery by several hours. The main generator ran out of fuel, and many servers and network segments went offline due to no power in DC1. DC2 got its delivery on time.

By 10:30 AM, fuel was delivered to DC1, and most of the network and servers were restored quickly. A few isolated issues were encountered due to failed hardware that took longer to resolve.

Roads were impassable to/from the datacenter, so the people who stayed in the building on the weekend were the only ones available. Cell towers were down, power was out, phone lines didn’t work, so communication with the datacenter was near impossible. The DC techs did a great job though keeping things running and trying to fix all the issues.

For Team LexiConn, both our homes and our support office were without power. I was running my laptop off a power inverter in my car and hooked up my cell phone to use as an internet connection. It was slow and spotty, but it got the job done to update our status page. All of us texted back and forth to verify everyone was safe, and we planned to meet up in Colchester where our support office is located.

Starbucks to the rescue!

A small group of stores in town had power, and one of those was Starbucks. They also had wifi. The line was out the door and it was packed, but a few of us snagged a table, powered up, and were “online”. It was quite hot in there, and the connection was quite slow (those pesky kids and their YouTube watching), but we were able to work, answer emails, check servers, etc…

We then went down the road to the local hot dog stand, Harry’s Place, which also had wifi. Luckily it was a nice day, and you can see our “office” in the image at the top of the page.

Both datacenters had street power by late Monday. None of our homes had power though.

Tuesday

Tuesday was a much better day. Our support office regained power (including phones), and our backup DSL connection worked well enough for us to all work in a somewhat normal fashion. The datacenters were humming along on street power and normal manning levels.

…

As of right now, I’m the only LexiConn employee with power at my house (no phone or internet though). It’s a slow recovery for the state, but we’re getting by. We’re thankful all of our families are safe, and considering the state-wide damage, we are quite happy and pleased with how well the datacenters held up with minimal impact for most clients.

We appreciated your patience and kind words during this event. Hopefully power will be restored to all of our homes in the coming days, and we’ll return to our normal lives.