Twitter Migrates Data Center to Permanent Nest – No More Fail Whale?

If you had some kind of twisted love for seeing the Fail Whale, you’ll probably be disappointed with this news: Twitter has completed its migration into a custom-built data center that should be able to meet the needs of its explosive growth and demands on its servers.

In a blog post Monday, the Twitter Engineering Team announced that they had successfully moved Twitter’s data center to a new, and more permanent, home.

Anyone who has been on Twitter for a few months or years will know that, while the service has a lot to offer in terms of socially-produced content, it wasn’t always the best at avoiding downtime. In fact, in Twitter’s early years, you almost expected to see the Fail Whale, signaling that Twitter was over capacity, half the time you signed in.

As Monday’s blog post explains, the final straw was Twitter’s 10 hours of downtime in June of 2010.

Twitter’s engineers moved all of the hardware and software to a new custom-built location that will hopefully be the final place for its servers. If you’ve got some computer engineering blood in you, here’s an excerpted description of how they did it:

“First, our engineers extended many of Twitter’s core systems to replicate Tweets to multiple data centers. Simultaneously, our operations engineers divided into new teams and built new processes and software to allow us to qualify, burn-in, deploy, tear-down and monitor the thousands of servers, routers, and switches that are required to build out and operate Twitter. With hardware at a second data center in place, we moved some of our non-runtime systems there – giving us headroom to stay ahead of tweet growth. This second data center also served as a staging laboratory for our replication and migration strategies. Simultaneously, we prepped a third larger data center as our final nesting ground.

Next, we set out rewiring the rocket mid-flight by writing Tweets to both our primary data center and the second data center. Once we proved our replication strategy worked, we built out the full Twitter stack, and copied all 20TB of Tweets, from @jack’s first to @honeybadger’s latest Tweet to the second data center. Once all the data was in place we began serving live traffic from the second data center for end-to-end testing and to continue to shed load from our primary data center. Confident that our strategy for replicating Twitter was solid, we moved on to the final leg of the migration, building out and moving all of Twitter from the first and second data centers to the final nesting grounds. This essentially required us to move much of Twitter two times.”

Assuming all went well, we can expect to see much less of that cute – but ill-boding – Fail Whale from here on in. Which is important for Twitter, as it hopes to focus more on its business plan now than reacting to bouts of downtime.