Head of user and product operations Seth Melnick and senior back-end engineer Somar Musa said in their blog post that the issue, which impacted 225,611 unique hashtags, was caused by inefficiencies in TikTok’s data stream flushing service design.
They explained that on May 28, TikTok began receiving user reports that view counts on many hashtags were displaying with zero views in the applications compose screen, where users prepare their posts before uploading them.
TikTok began investigating two potential causes: an issue in its code, or a data loading or lagging issue.
On the first front, the TikTok team found that the code was bug-free and provided eventual consistency.
So, the team turned to the second potential cause, and it found a lag in the data stream that counts the number of times a video is viewed.
Melnick and Musa explained, “When we say we had a lag in a data-stream, we mean some data was so backed up that it needed to be queued to be consumed called. Normally, we calculate the computing power needed so as to consume the queued data with little to no delay or lag time.”
They also outlined preventative measures TikTok is putting in place to avoid a reoccurrence: We’ve explored the design of our data stream flushing and found inefficiencies. We’re upgrading the design to eradicate this issue. Specifically, the new system will recognize identical attributes and not flush redundantly. So, if an attribute is the same among all countries, we will flush it only once instead of 150 times (per country). This will allow us to process faster than before while using far fewer computing resources, meaning that we shouldn’t see a lag issue like this again. Moreover, as an extra layer of precaution, we will be increasing our monitoring, which will notify us of delays in data stream processing early on and help us correct these issues as they arise.”