VIDEOS: Facebook’s @Scale 2014, Data Track

Facebook held its @Scale 2014 conference in San Francisco Sept. 15, and engineering manager Sambavi Muthukrishnan shared videos from the event’s data track in a post on the Facebook engineering blog.

AtScale650Facebook held its @Scale 2014 conference in San Francisco Sept. 15, and engineering manager Sambavi Muthukrishnan shared videos from the event’s data track in a post on the Facebook engineering blog.

Muthukrishnan wrote in introducing the videos:

Building applications and services that scale to hundreds of millions or even billions of people around the globe often brings with it an unprecedented set of challenges in the domain of data-processing systems. More people and more engagement usually lead to more data, more updates and more queries. How growing companies handle this can make or break the experience people have with their products.

Building on that, the rich variety of consumer experiences today introduces new data models and new query and update patterns — and this in turn has an impact on both online data-processing systems and analytics systems. The growth of mobile clients and apps adds on yet another dimension: By providing stateful devices with variable connectivity, client interactions with the back-end data systems need to be designed and optimized differently than they would be for the Web.

The data track at the @Scale 2014 conference brought together the community that is tackling these problems, with talks from Box, Facebook, Netflix, Pinterest and YouTube. Check out the talks to hear about how these companies are pushing the boundaries in every layer of the stack across cache, database, logging and analytics systems.

And the videos are:

Growing Facebook on Mobile, a Real-Time Analytics Story: In this session, Facebook engineer Weizhe Shi started out by providing context on the analytics ecosystem at Facebook before this project began — the team had tools to rapidly iterate on their Web product but didn’t have that same ability for mobile, even though a large number of people had started accessing Facebook through mobile devices. By combining limited releases with a real-time analytics pipeline, he explained how they filled that gap. He detailed the challenges faced in collecting analytics data on mobile and the frameworks built to overcome them. Anshul Jaiswal followed up with the challenges faced in processing and analyzing this data. He presented the framework they built for rapid evolution of analytical tools to understand the health of mobile releases in real time. Will Wirth covered specific examples of how this framework was used to make Facebook’s mobile applications stable, performant and engaging.

Data-Platform Architecture, Evolution and Philosophy at Netflix: Virtually all of Netflix’s data infrastructure is in the AWS (Amazon Web Services) cloud, and S3 is the central data hub. Netflix’s Kurt Brown delved into the framework of Netflix’s data platform in S3 and how the various technologies work together. He also previewed many open-source tools, including Inviso, a job-search and visualization tool, and Kragle, a big-data application-programming interface. Justin Becker wrapped up the talk with a deep dive into Mantis, a reactive stream-processing system that provides real-time insights to ensure that Netflix works for every user every time across many different dimensions (e.g. device, location, bit rate).

Zen: Pinterest’s Graph Storage Service: The session opened with Raghavendra Prabhu, engineering manager at Pinterest, who emphasized the importance of building infrastructure abstractions to enable product engineers to iterate very quickly. He then introduced Zen, which provides a graph data model abstraction on top of a database to provide storage as a service for various products at Pinterest. Xun Liu closed the talk with an explanation of Zen internals and production learnings.

Building Scalable Caching Systems via mcrouter: Facebook engineer Rajesh Nishtala announced the open sourcing of mcrouter, a memcached protocol router for scaling cache deployments. It manages all traffic to, from and between thousands of cache servers in dozens of clusters in Facebook data centers. He provided an overview of Facebook’s caching infrastructure — which processes more than 4 billion operations per second — and deep-dived into a few specific scalability and reliability features of the technology. Ricky Ramirez of Reddit wrapped up the talk by previewing how mcrouter is used at Reddit, which automatically scales between 170 and 300 app servers every day. Check out the blog post for more details on mcrouter.