It took Google years to come up with a nuanced algorithm to determine the most relevant search results for your queries (and many people think they’ve still got a ways to go). And it sounds like Twitter is in the same boat.
Twitter’s search engine has to process real-time trends that can surge and ebb in popularity within seconds, and its engineering team faces the challenge of making its search agile, flexible and relevant to your queries at any given moment in time.
In a post on the Twitter Engineering blog, research scientist, analytics Jimmy Lin explains how Twitter is approaching its real-time search engine.
He dubs the fast-paced change in popular topics on Twitter “churn”, saying that the most frequent terms used on the network can vary from one hour to another – indicating extremely high churn.
For a paper the team will be presenting at an upcoming conference, they examined all search queries from October 2011. They found that, on average, 17 percent of the top 1,000 search terms from one hour are no longer in the top 1,000 the next hour. As Lin explains it, “17% of the top 1000 query terms “churn over” on an hourly basis.”
This churn is more intense than anything Google is seeing, as Twitter’s real-time, social nature promotes the sharing of breaking news, fast. This rapid exchange of information means that search queries and results are changing at a faster pace than anything we’ve ever seen.
As an example of what they’re dealing with, Lin and his team graphed the search queries that people used on the day that Steve Jobs passed away. The search term “steve jobs” spiked up to nearly one in queries immediately following the news.
Making Twitter search more relevant is something the team is working hard on. And in order to succeed, they’re likely going to have to rework the currently accepted search algorithms to handle real-time data more effectively.
(Search bar image via Shutterstock)