Interview Prep #1 — Facebook Event clicks increased by 10%, how would you investigate?
Product Sense questions are very open-ended and hard to prepare — that’s why I started this series of interview preps to learn from data scientists and product managers on how to form a structured, hypothesis-driven and exhaustive answer to these questions.
For Day 1 of this series, I will work on a Facebook interview question shared on Glassdoor:
My Answer:
Ask Clarifying Questions:
INTERVIEWEE: Before we dive into details of the analysis, I want to make sure my understanding of the problem is correct — Facebook searches return multiple types of results: Post, Link, Group, etc. For this problem, we noticed that number of users who click on Event on search results page increased by 10% week-over-week. Is that right?
INTERVIEWER: Correct.
INTERVIEWEE: What’s the time period that this increase happens? Is it a sudden increase we just observed this week, or it has been happening for a while?
INTERVIEWER: Assume it is a sudden increase. Clicks on Event were stable, and it just increased by 10% this week.
Explain my overall approach:
INTERVIEWEE: Got it. The way I want to tackle this problem is as follows: I will first try some segmentation(e.g. by language/region/platform) to narrow down the scope of the problem; next, I will dive deeper into the root causes of this sudden increase, and determine whether it is a good or bad thing for each case scenario. Does it sound like a good approach to you?
INTERVIEWER: Sounds great.
Possible Segmentations:
- Geography: Did the increase happen in any particular region?
It is possible that some regions are celebrating holiday that increases either # of events available, or people’s willingness to take part in events. We can look up if there is such a big event happening, as well as Event feature usage in that region(metrics: # of upcoming events, # of active users) that can justify the increase in clicks on search results.
- Platform: Did the increase happen on any specific platform?
Facebook app is available on multiple platforms. If the increase is any platform specific, it is very likely to be something we did to our app on that platform. We can check recent feature launch/bug fix/algorithm change that could be related to the increase.
INTERVIEWER: You can assume this increase is consistent across all region/platforms and get straight to the deep dive.
Construct a MECE framework:
INTERVIEWEE: For the next step, I will analyze both internal & external factors that could possibly cause the increase in clicks on Event, as well as which cases would be good/bad to us. The root cause is either internal or external, so with this framework, we can make sure we don’t miss any notable cause.
For internal factors, I’m refering to something we did to our product — for example, we changed our ETL pipeline that caused error in data collection process, or we launched a new feature/algorithm that changed user behavior.
For external factors, I’m refering to factors unrelavant to Facebook’s decisions — e.g. seasonality, competitors, special events, etc.
As the increase is not specific to any region/language/platform, I think it’s less likely to be external reasons that caused the increase — because Facebook is a global product, and external reasons like seasonality or holidays are very unlikely to impact all regions/platforms at the same time. Therefore, I will start with analyzing possible internal factors.
Walk through probable causes:
- Data Accuracy: I want to first make sure we collected the right data. To verify that, I will check metrics that should correspond to clicks on Event on search result page. For example, if more users are clicking on Event, # of active users of Event feature should increase too. If there is no notable change in these metrics, it could imply there is a data collection error and we should talk with the engineers.
- Feature Launch, UI/Algorithm Change: After confirming the data we collect is correct, I will check if our search team recently launched something that could suddenly change user behavior. If so, I will then look at the tests they did prior to launch and check if test result corresponds to what we just observed.
Decide it’s good or bad:
INTERVIEWER: You talked about different causes that could lead to this increase — how do you determine if it’s a good or bad thing?
INTERVIEWEE: The goal of Facebook search is to enable users to find more content they’re interested in. As a result, users will create more meaningful social interactions and be more engaged with Facebook. Therefore, what would worry me is increasing users click on Facebook Event cannibalizing clicks on other types of results on search results page, which could eventually lead to decreasing overall user engagement.
Therefore, if it’s external factors causing the increase, I would not worry too much, as users search for and use Event feature more because of holidays or special events is very unlikely to change user behavior permanently and negatively impact their overall engagement on Facebook.
On the other hand, if we proactively changed UI/ranking algorithm to motivate users to click more on Facebook Event, we need to determine if it hurt the overall health of Search feature— Did users click on other types of results less? Did users spend more time to find content they want? Did users abandon more searches? (Metrics: weighted CTR, average rank of results that users click, abandon rate, avg piece of info seen before exit).
We’d also want to know how UI/ranking algo change # of downstream actions after they click into a search result(Metrics: # of comment/like/share on posts, # of like/follow on pages, etc), as well as overall engagement of entire Facebook ecosystem(Daily Active Users, # of session per user per day, etc).
To summarize, if more users clicking on Facebook Event comes with worse search experience and lower user engagement on Facebook, I think it is a bad thing; otherwise, I believe it is a good thing.
Test new ranking algorithm:
INTERVIEWER: Good. Here’s a natural extension of the original problem — The Events team wants to up-rank Events such that they show up higher in Search. How would you determine this is a good thing or a bad thing?
INTERVIEWEE: I will launch a A/B Test on a subset of our users to determine.
INTERVIEWER: How would you design the metrics?
INTERVIEWEE: I will start with analyzing why the Events team wants to up-rank Events in the first place, then talk about metrics to measure success of new ranking algorithm. I believe that the goal of up-ranking is to have more users click on Event on search result page, which would lead to more active users on Event. Do you think it is reasonable to assume so?
INTERVIEWER: Makes sense. Go ahead.
Design test metrics:
INTERVIEWEE: So our business hypothesis is that we expect that if we up-rank Events, more users will click on Events. Therefore, primary metric of our experiment is # of users click on Event. In terms of secondary metrics, we can also track # of downstream actions on Event like ‘Interested’, ‘Going’, ‘Invite’ to understand how up-ranking impacts engagement with Event. If only more users are viewing Event but they’re not creating more meaningful social interactions, maybe it’s worth to launch the new algorithm.
We should also track guardrail metrics — we talked about how more users clicking on Event could be a bad thing for us, if it hurts overall search UX and user engagement. Therefore, we should watch for Weighted Click-Through-Rate, as well as DAU, #of session per user per day(echo back to overall goal of Facebook and Search).
INTERVIEWER: Sounds good. Looks like we’re almost out of time, so we will skip details of experiment design. Thanks for your response.
Ricky Zhang
M.S. Data Science @ USFCA. Graduating in August.
- Data Scientists & Product Managers: if you come across this blog, please spend a minute to leave a comment on how I could possibly improve my answer. If you have any other general interview tips, please feel free to share too. I’d sincerely appreciate it!
- Hiring Managers: if you have Data Scientist, Analytics / Product Analyst / Business Intelligence Analyst opening on your team, let’s connect on LinkedIn or send me an email at rzhang67@usfca.edu . I appreciate your help!
