Twitter Faces Social Graph Scaling Problems

There has been a bunch of drama in the blogosphere surrounding a recent blog post by Alex Payne, a developer at Twitter. It was Alex’s response to a bunch of feedback that they’ve received since the increase in service outages on the site. The center of the discussion revolved around the following statement which VentureBeat translated as Robert Scoble being the source of the problem:

The events that hit our system the hardest are generally when “popular” users – that is, users with large numbers of followers and people they’re following – perform a number of actions in rapid succession. This usually results in a number of big queries that pile up in our database(s). Not running scripts to follow thousands of users at a time would be a help, but that’s behavior we have to limit on our side.

Scoble took offense the reference and posted his own furious response on FriendFeed as though it was a personal attack on him. Ultimately this is just some noisy blogger drama but there is an important issue here surrounding scaling social websites. Facebook is forced to limit friends to 5000 people due to technical issues.

While Mike Arrington suggested that the friend limit may soon be raised, there is nothing else to suggest that the friend limit will be increased. It appears that FriendFeed and MySpace are among the few companies that have solved the problem of scaling the social graph. While Robert Scoble can frequently be the source of scaling issues due to his ability to incite mass dialogue, it’s also a great problem to have a great way to resolve bottle-necks.

While Twitter continues to face serious problems, it’s good that it is happening early on with the early-adopters and not late in the game when it could cost them their user base.