Interview Prep #3 — Fractional Share Trading, Launch or Not?

Product Sense questions are very open-ended and hard to prepare — that’s why I started this series of interview preps to learn from data scientists and product managers on how to form a structured, hypothesis-driven and exhaustive answer to these questions.

For Day 3 of this series, I will work on a Robinhood Data Analyst interview question:

Robinhood is going to launch fractional share trading — how do you determine whether this is a good idea or not?

After gathering feedback from experienced data science professionals and analyzing stats of first blog, I decide to slightly adjust the structure of future blogs — instead of simulating a dialogue, I will share only a synopsis for better interview prep results and better reading experience.

My Answer:

Ask clarifying question:

Before we jump into experiment detail, I want to make sure my understanding of fractional share trading is clear: Without fractional share trading, users could only buy whole number of shares; with fractional share trading, investors can buy any slice of a share of stock, starting at $1. Am I understanding correctly?

Goals of the feature:

With a feature like this there could be multiple goals. We can start with imagining intuitively what the feature can do for users:

1. From an user/investor’s perspective, especially retail investors, they may not have the fund to purchase even 1 share of the expensive stocks like Amazon and Google. With fractional trading these investors can finally invest in these stocks, in whatever amount they like. This is aligned with Robinhood’s mission — democratizing finance for all.
2. The easiest way to improve a product is to incentivize what users are already doing today — fractional share trading can make the trading process more smooth. Without fractional share, imagine you want to invest $1k in a stock but the stock price is $300, you have to decide to put $900 in for 3 shares or mount up to $1200. With fractional share an investor can easily purchase any amount of any stock.

In conclusion, fractional share trading would make stock trading even more accessible to retail investors, and also make their user experience better. Therefore, with the launch of the new feature, we expect to incentivize trading activities.

Business hypotheses and metrics:

So our business hypothesis is that — we expect that with launch of fractional share trading, trading activities will increase. That means our Null Hypothesis (Ho) is that there is no change due to the new feature.

Now we’re clear with the goal of the feature and hypotheses, we need to define some concrete metrics to measure success.

The primary metrics I will use are average number of orders per user per day, as well as average number of traders. At the same time we also want to know average order size — we don’t want to see average order number goes up but size go down, which could eventually leads to less revenue.

Possible guardrail metric: page load time, uninstall rate, etc.

Experiment details:

We can then start to talk about experiment details:

1. Randomization unit: discuss possible network effect with the interviewer. if there is network effect, use network-cluster randomization; otherwise simply randomize by user.
2. Sample size & Duration: use Minimum Detectable Effect, Significance Level, Statistical Power and Variance of metric to calculate sample size and duration.
3. Seasonality: Is this a good time to run experiment? Seasonality might not be a quite valid concept for the stock market — will want to discuss more with interviewer for a better solution.
4. Novelty Effect: this new feature probably is prone to novelty effect as it’s such a revolutionary one. Might want to run experiments only on new users to counter the effect.
5. External Effect: are we running any marketing campaign which gets a lot of new users? Users might be exposed to multiple experiments at the same time?

Analyze Results:

Sanity check first! Is data collected correctly? Randomization done right? Filter consistent across groups? Invariants approximately the same?

Then check for statistical significance: t-test for means, z-test for proportions.

Make decision: is the statistical significant result also practically sig? In other words, from business perspective, is the gain bigger than the costs(human labor cost, opportunity cost, risks of bugs, risk of criticism from public) if we implement the feature?

When we have conflicting results — translate to business impact. Will need actual numbers to do so. However, launch or not eventually depends on business goal of the feature. e.g. If we only care about number of orders go up, then launch if conversion goes up. Also, take short/long-term tradeoff into account.

Ricky Zhang

M.S. Data Science @ USFCA. Graduating in August.

• Data Scientists & Product Managers: if you come across this blog, please spend a minute to leave a comment on how I could possibly improve my answer. If you have any other general interview tips, please feel free to share too. I’d sincerely appreciate it!
• Hiring Managers: if you have Data Scientist, Analytics / Product Analyst / Business Intelligence Analyst opening on your team, let’s connect on LinkedIn or send me an email at rzhang67@usfca.edu . I appreciate your help!