Leaking Roofs, Power Outages, Missing Networks And Sheer Bad Luck Force Twitter Data Center Move

Last July I reported about Twitter’s plans to move to a new data center after a sizeable bump in user numbers led to a series of problems within the platform.

This custom-built venue in the Salt Lake City area, in Utah, was created to keep pace with the influx of daily user numbers (which has only got bigger) and by all accounts has been going swimmingly.

Unfortunately for Twitter, that’s a literal statement, as Reuters reports:

For Twitter, which had been renting servers from hosting provider NTT America for the past several years, moving into its own data center marked a significant milestone.

Within a couple of months of Twitter’s July announcement though, it was quietly moving many of its servers out of Salt Lake City and instead shipping all new gear from Sacramento, where server capacity and space was initially very tight.

Twitter concluded that the Utah facility failed to meet its needs, these sources said. The center initially lacked key features such as a second fiber network connection, and less than half of the electricity was actually available.

The roof leaked water onto the top of Twitter’s server cabinets with every rain, forcing staffers to move equipment out of harm’s way, sources said.

All of this has meant Twitter has had to rush-move their data centre to a new location last month, rumoured to be some 600 miles away in Sacramento. At what cost we can only guess, but this was after signing a 4-year, $24 million contract with C7 Data Centers in Utah. It’s unclear how much, if any of those fees will be returned.

Good times.

Twitter VP of Engineering Michael Abbott says the company has done more to upgrade their servers in the last 6 months than in the previous four-and-a-half years, but given they’ve doubled their userbase in about a year that was probably to be expected.

“Twitter now has the team and infrastructure in place to capitalize on the tremendous interest in Twitter and continue our record growth,” wrote Abbot in an email, somewhat absurdly.

C7 wouldn’t confirm Twitter as a customer but decided to issue some damage control/finger pointing anyway.

“We experience less than 1 percent churn in our customer base,” Swenson said in an interview. “But sometimes we deal with companies that are not very sophisticated; often times a lot of customers that have never owned their own equipment may go through a learning curve,” said President Wes Swenson, adding that full power was available from day one and a ‘brand new roof’ was installed in October.

For the moment, Twitter is using the remaining servers in Utah for other functionality, which includes analytics. You know, that stuff that only about 1% of very special users are privy to.

(Is that a drip? Part of me kind of hopes so.)

Yep, we can blame all of this on bad luck, but Twitter sure seems to have a lot of that when it comes to server problems caused by managing users and their data. As opposed to say, oh, somebody like Facebook or Google. Put your trunks on folks – it’s a long way to land.