Being on the Oprah Winfrey Show is a big deal. Some would say it’s the ultimate advertising opportunity. Just being mentioned by Oprah can generate thousands of website visitors. How about if some your products are featured, and even modeled on the show? For one of our hosting clients, Cosabella, they found out what’s it like to be featured on Oprah.

“We’re going to be on Oprah in two weeks!”

That was the phone call we first fielded with Cosabella. They found out they were going to be featured on Oprah, and it could be as soon as two weeks. Needless to say, they were excited, and were scrambling to get ready. Oprah’s IT department gave them estimates of being ready for 250,000 unique visitors the day the show airs.

That type of volume would easily overload the single managed dedicated server Cosabella had with us…

The conversation started down the path of setting up a cloud possibly, or hardware load balancing with sticky SSL sessions (tech speak for making SSL work seamlessly across many servers). Maybe database replication for the ShopSite ecommerce database across multiple servers. With only 2 weeks to prepare, this type of complexity would not only be expensive, but both LexiConn and Cosabella would be rushing to get this working, and there would be limited testing. In addition, these types of complex systems do not do well when problems arise, as the solution becomes more complex.

K.I.S.S. – Keep it Simple “Silly”

We’ve had previous experience with clients being featured on Oprah. Two summers back we assisted a woman’s apparel store drive their sales through the roof when they were a top summer pick by Oprah. Based on this experience, we suggested to Cosabella to keep things simple. This would maximize the number of visitors they could handle, and make it easy to adjust things when traffic spiked.

Here’s what we recommended and what they agreed upon:

• 5 front-end webservers added
  These servers would be stripped down to be as fast as possible (Dual core, 2 GB RAM, SATA hard drive). Apache (the webserver software) would be limited to essential modules only. The website pages themselves would be replicated to these front-end servers. The shopping cart software would be separate on their main dedicated server. For the “geeks” reading this, we custom compiled Apache with just the core modules and got each child’s memory footprint under 800 KB per instance.

• Round-robin DNS for these 5 servers
  Round-robin DNS for static websites works great. It distributes the load evenly across the servers, and with a low TTL and portable IPs, it’s simple to redirect traffic as needed in case a server dies, etc… And it costs next to nothing to implement.

• No dynamic scripting on catalog pages
  By excluding PHP and MySQL on the catalog pages, the site would load quickly, and use very little resources. This is key for surviving the onslaught of thousands of visitors at once. Cosabella used JavaScript, Flash, and CSS only in their catalog pages

• Ecommerce application (ShopSite) on one server only
  By setting the shopping cart system to a sub-domain on a separate server, we kept things simple and costs lower. The cart would not receive the brunt of the hits, and one heavy duty server could handle the extra orders. ShopSite would scale well to keep up with orders on just one server.

Before the event kicked off

To make things easy for Cosabella to manage, they worked on their website and shopping cart on just their one managed server. It was “business as usual” in terms of updating the site, adding content, etc… We then setup an rsync replication system (fancy speak for copying the webpage files to the 5 front-end servers) that ran every 5 minutes (or upon demand) that synced any changes to the website.

We tweaked the webserver config files on each server to maximize uptime and speed. We set the MaxClients setting to 500 on each server initially. This was done to make sure each server would not become overloaded right away. We tested the KeepAlive and Timeout values until we found an  optimal range for serving Cosabella’s content quickly and closing up unused connections. We set the MaxRequestsPerChild to 50,000 to prevent children dying off quickly and causing more overhead (no worries about memory leaks since Apache was essentially “naked”).

Gentlemen, Start Your Engines!

As it turned out, the Oprah event got pushed back a few weeks, which was helpful to allow for more testing and time to get ready. On the day of the big event, we had many custom scripts in place to be able to monitor all servers at once, issue commands simultaneously to each server, and scripts to adjust the DNS if needed. Now it was time to see if all the preparation paid off!

The Oprah Show airs at 10 AM CST in Chicago. Around this time we saw a small spike in traffic (around 1000 unique sessions), and about 15 minutes afterwards, a small surge in the number of orders being placed. Overall though, it was not something that would overwhelm one dedicated server, but it allowed us to make sure all was working correctly.

Here’s a look at the unique visitors graph for the day:

Unique Visitors (click image to enlarge)

4:45 PM ET – Now We’re Cooking!

At this time, the products were being displayed on the Oprah show, and the number of visitors skyrocketed. We noticed all 5 front-end servers were approaching the 500 MaxClients, so we upped the limit to 768 and issued a reload. As the numbers continued to rise, we also had a recompile of Apache going to change the hardcoded MaxClients to 2048. As soon as the compile was done, we pushed this out to the servers. The 768 limit was being tested, so we increased MaxClients to 1024 on all servers. Each server’s load was quite low, so there were no issues with overloading the actual servers.

At the peak, we saw about 900 Apache instances per server, which was around 5,000 Apache children serving pages to visitors. Things got a little slow in the first minute when we were tweaking settings and pushing a new compiled Apache version out, but we quickly got a handle on things and the website was humming along.

7:45 PM ET – The sun sets in the west

The west coast airing caused another spike in traffic around 7:45 PM ET. This spike was only about 60% of the East Coast airing, so it was easily handled. And as we saw with the East Coast, orders started coming in about 15-20 minutes after the initial surge in traffic.

Show me the money!

About 15 minutes after the initial surge in traffic, orders started pouring into the site:

Orders placed (click image to enlarge)

The primary ShopSite ecommerce server handled this high order volume without any issues. The cart was snappy and error free for all customers.

The next day…

Visitors and orders on the second day after the show aired were about 50% of the first day. Everything went well, no issues with slow loading pages or errors. Just a “boat load” of orders and traffic for 24 hours after the East Coast airing.

Increase in daily totals for the 24 hours following the Oprah airing:

• 60,000 unique visitors

• 750,000 page views

• 7.4 million hits

• 70 GB of traffic

Cosabella saw an increase in order volume of around 2,000 percent due to their products being featured on Oprah! The whole event was beyond expectations for Cosabella, as they said actual order volume was 2.5 times what they anticipated! The website and setup we provided to Cosabella worked as expected, and kept things running smoothly, even at peak traffic.

Lessons learned

• Compile Apache beforehand
  Based on what each server was able to handle, setting the MaxClients compiled value to 2048 beforehand would have been fine, and eliminates any delay.

• Set MaxClients to 1024 beforehand
  If Apache is stripped down, and enough memory is installed, setting MaxClients to 1024 is a safe setting to maximize traffic to start with.

…

I’m glad we were able to help Cosabella get the most out of their Oprah appearance. It was an exciting time to watch the traffic in real-time, and have our team respond quickly to manage the event and keep their website up and running optimally throughout the day. We learned a few things along the way, and it confirmed our predictions as to what was required to not only survive Oprah, but get the most out of it.