Within the Netflix product organization, the majority of our customer-facing decisions are based heavily on the results of A/B tests. To this end, over the last 10 years, the experimentation team has developed a rigorous practice around sampling, statistical testing, metric selection, and analysis. Now, as we attempt to build a similarly rigorous testing practice within our marketing organization, we face new challenges. A fundamental principle of A/B testing is user-level random assignment into distinct test and control groups. However, what if we want to test the impact of shutting off our TV advertising and shifting the budget to digital? What if we want to measure the impact of outdoor media vs. other types of local spend like radio or print? How can we answer these questions within the construct of an A/B test? We cannot. Instead, these questions are necessarily answered via quasi-experimentation in which different time periods, different regions, or a combination thereof serve as our “quasi” controls. The results of quasi-experimentation will likely never be as precise as achieved with A/B testing, but we aspire towards a directional read on causality. Utilizing a diverse set of methods such as k-sparse means for identifying clusters of similar cities to serve as test and control groups, state-space models to adjust post-experiment results for historical differences, and, of course, market expertise, we have been able to run successful quasi-experiments around the globe.
Director, Experimentation & Algorithms for Growth & Targeting
Kelly Uphoff is the Director of Experimentation and Algorithms for Growth and Targeting at Netflix. She leads a team of data scientists who build algorithms and design and analyze experiments in the areas of Original content promotion, signup flow optimization, messaging, fraud, customer service and marketing. She started at Netflix in 2011 as a statistician on the experimentation team. Prior to Netflix, she did predictive modeling at GEICO in Washington DC. From 2007-2008, Kelly served as a Peace Corps Volunteer in the Republic of Georgia, helping small-scale entrepreneurs learn business skills.