Case: Buffering Rate Increment in YouTube

Often times when KPIs don't perform as expected or when there's a drastic change in key metrics that track the performance, diving through the possible hypothesis and probable reasons helps us figure out the root cause of symptoms. This is a take on investigating through one such case, digging into problem and finding a solution.

Buffering rate of Youtube, a video sharing platform, has increased by 10%. This has been impacting it's user engagement, resulting in increased complaint feedbacks from users.

Background:

YouTube is an online video sharing platform owned by Google. It has far left behind it's competitor like DailyMotion and is the second most viewed site. It is operated as a separate entity by Google.

It's business model relied primarily on advertisements for generating revenue. Upon being acquired by Google, it opened up other channels for revenue generation, expanding it's business model to premium subscriptions, to allow users watch ad free content. In addition to this it offers exclusive content and paid contents like movie.

Beyond the core website, it's also accessible through mobile app, network television and has the ability to link with other platforms.

The first few things to check and clarify on here, is, what exactly is the Buffering rate or how team calculates this metric. Buffering rate is taken to be the average buffering duration divided by average view duration of video for user.

Secondly, it would be worthwhile to check here, if the increment observed is gradual and over what time frame. The increment has been observed over last 5 days. 

If only a specific OS or platform, demography or region is impacted, in which case problem is contained and easier to figure out. However, all users in India across different OS and platforms has been impacted.

Lastly, another check and clarification that helps is, which type of video has seen the increment, usual long form ones or the short reel videos majorly. The issue was found on both short and long form videos.

That paints the picture a bit more clear, Buffering rate of Youtube, a video sharing platform, has increased over last 5 days, across all users by 10% on both video formats of Youtube. 

This is a cause of concern, as it's user engagement is being impacted, with multiple complaint feedbacks registered from user, due to increased buffering. This is an important key metrics that Youtube tracks as performance indicator, as it directly impacts it's advertising revenue.

Now, that the problem is a bit more clear, diving through probable causes will let us be more closer to finding the root cause of symptoms.

I have categorised probable causes into three segments:

• One that is all internal factors dependent

• One that is impacted from external factors

• One that might have resulted from System related issues

External: Probable hypothesis here resulting from external factors, 

• Malware attack: A large scale virus attacks that are targeted towards certain services and websites, might cause the disruption and slowness, leading to increased buffering rate.

•  Internet Connectivity: It could even result from slow down in internet connectivity due to various urgent factors like natural causes.

•  Seasonality: There could be an impact of seasonality, being a holiday season people stay home and watch/surf more videos than regular, there's an increased load, that could lead to increased buffering rate. 

• A new free VPN service launch : VPN ends up changing the client server location, which in turn might impact the routing of video to local system, resulting in increased buffer. Checking through market news, no such major new ad/campaign was found by VPN service companies, other than usual.

However, none of these reasons were found out to be true on further digging.

System Related: The technical issues sometimes lie at the core. 

• Data pipeline or Server routing issue: There could be internal technical changes that alter the routing mechanism of videos search and load. Or a problematic data pipeline can also result in increased buffering rate.

• Issue with calculation metrics: The metrics in question, i.e. Buffering rate, if have had some change in method by which calculations are being made, would result in metric being inflated from previous.

Checking with IT however didn't reveal any data pipeline issue, nor was there any changes or issue with metric calculation of buffering rate.

Internal: More often potential causes are internal such as

• A new feature release: It might cannibalize some other feature, altering user behaviour and creating an impact. No new feature release was announced however.

• Version Upgrade: There could have been a version upgrade, which pushes multiple changes in the backend. 

• Recommendation algorithm change: A change in recommendation algorithm only on web browser windows, which might put certain videos up on list re-directing heavy traffic to it and increasing the buffer rate. 

• Change in UI: A change in UI (since it's impacting only the videos watched through web browser, UI significantly differs in app v/s web browser) too ends up altering user behaviour, which in turn might impact traffic on certain videos, the ones that lie on top of list by default.

• A sudden increase in the number of trending videos: If the number of trending videos increase, if results in increase in traffic on certain videos. However checking upon this metric # trending videos watched in a day, it didn't show any significant jump.

Investigating and checking through, UI or algorithm change was not confirmed but it was confirmed that there was a version upgrade across all platforms some days back, which included some policy implementation changes.

Diving through multiple probable hypothesis above, it was found out that there was a version upgrade across all platforms, including policy implementation changes.

Let's dig in further and analyse the version upgrade hypothesis. There were multiple changes pushed and it's not clear what could have led to increased buffer rate. In order to understand better, we need to figure out at what step in user experience are we exactly noticing the problem, which might help in pin pointing to related change if any that was pushed in version upgrade.

User Funnel Analysis:

Walking through user experience funnel and checking with user comments from feedbacks received, points that they have been experiencing a delay while the video loads initially. On checking through metrics such as average advertisement watch time per video, average long form video watch time, average watch time of users per day no significant changes were found. Further checking on the trending videos, if there has been any recent increment in the number of trending videos and their watch time doesn't point a huge change. Which points that buffering issue doesn't occur either due to heavy traffic or any buffer issue mid video. So this zeros on the step where video initially loads, which is contributing to the buffer time increment. This could be due to internal technical issue or any recent change. There could be some issue with the content delivery network, that is responsible for routing and loading video from nearest server. Or it could also be result of any changes that was included in version upgrade.

Upon checking through the list of changes pushed in version upgrade, it was found that there was an intentionally added a piece of code that resulted in five second delay for users watching videos online. YouTube had intentionally added new code to artificially increase wait time on web browsers like Mozilla in a bid to push people to install Google Chrome, which is owned by same parent company as YouTube. YouTube confirmed that "it inserted the code into its video sharing website not to force users to change web browsers - but as part of it's ongoing efforts to stamp out ad blockers across all platforms."

So, this proves our hypothesis of recent version upgrade that had led to increase in buffer rate, caused an increased initial load time for videos.

Given the aim with which this change was added, was to move users towards stamping out ad blockers, on platforms for YouTube. This is understandable, given the direct impact on ad revenue. So the approach towards Problem resolution here, is to not make changes in original intent/goal but rather to look for alternative smart ways in which users can be moved towards stamping out ad blockers for YouTube.

Some potential solutions can be through A/B testing different scenarios, that would help understand users behaviour towards the value they see in trade off of delayed content v/s removing ad blockers. 

A/B Testing

Metrics to Track

Both these scenarios can be A/B tested to see through potential impact and which variant ends up providing better results.

In scenario 1, Control Group 1, where Variant 1, was assigned to be tested, # of CTA button clicks is the metrics that would track the results, to show how many participants actually paid heed to the top notice banner and appeal, to remove ad blocker.

In scenario 2, Control Group 2, where Variant 2, was assigned to be tested, apart from tracking the # of CTA button clicks, it would also be worthwhile to track # of users who stayed on page even after 5 seconds, this potentially would help track % of users who are interested, who paid close attention to the message and are more likely to take action to remove ad blocker. It also helps the premium subscription revenue performance of YouTube.

Lastly, over a longer run, an effort can be made to observe and learn over time through user data, regarding the placement of advertisements, so that users have less apprehension about advertisements, when they consider the trade off of quality content v/s watching ads.