Gnip Opens Up Entire Twitter Archive for First Time

Every tweet since 2006 now available to analytics firms

Twitter cofounder Jack Dorsey sent the first tweet on March 21, 2006. While that tweet is freely available online, marketers and social analytics firms have been hard pressed in finding many tweets sent since 2006. While the Twitter analytics ecosystem has evolved over the years, it has centered around real-time monitoring—until now. Social data provider Gnip has opened up the entire Twitter archive. Gnip COO Chris Moody said the company is the first of any, including Twitter, to make the full tweet archive available.

Now “we can actually go back and understand at any moment in time [dating back to 2006] the conscious thought of the world,” Moody said.

By opening up the Twitter archives, marketers can now put their real-time social insights into context, Moody said. For example, if a device manufacturer is launching a new product, it could rewind back to their last few product launches (assuming those occurred within the last seven years) and compare consumer sentiment. Internally Gnip analyzed tweets sent around SXSW and found that barbecue has trended down as a popular topic whereas tacos have trended up. Conclusion: Expect more branded taco trucks next March.

The possibilities extend beyond marketing insights. Financial services companies often include social data as an input when managing assets, Moody said, and could now test current social signals against historical precedents to improve their confidence in what they think the market is telling them through social media.

A key event in the archive’s development came in 2010 when Twitter partnered with Gnip to house every tweet ever sent within the Library of Congress. That gave Gnip licensed access to the full corpus of tweets, but with limited access for others. People could research tweets within the archive, but companies couldn’t build analytics platforms that dive into those tweets at scale. Now they can.

Twitter has evolved a lot since 2010—let alone 2006—so Gnip does have to take platform changes into account when letting companies access the archive. For example, its data mining needs to comply with terms of service changes, which protects deleted tweets from being made public. Therefore, Moody said, Gnip—one of three Twitter-certified data providers—has to prevent any deleted tweets from being included when companies query the archive. Twitter took a “very active role” in helping to define how the data should be used, Moody said.

Dozens of Gnip customers, such as social monitoring and analytics firm Union Metrics, have had access to the archive as part of a closed beta over the last several months. Now Gnip is opening up the archive to more customers seeking to build on top of it. While offering historical context to current social insights is an immediate use example, Moody sees opportunity in evolving predictive modeling. That could help retailers forecast how future seasonal lines may perform, or political scientists anticipate the outcomes of revolutions, he said.