Netflix Product Innovation VP: ‘Every Subscriber Is A Different Channel, So We Have 53 Million Channels’
- View Original
- November 11th, 2014
Carlos Gomez Uribe, Vice President of Product Innovation at Netflix Netflix, spoke about his company’s deep culture of innovation at the Mita TechTalks , run by the venture fund/business accelerator MITA, in Punta Mita, Mexico.
In characterizing the complexity of innovation for Netflix, Gomez Uribe wanted to make sure the audience understood the order of magnitude of the issue. And it’s big. “Every subscriber is a different channel, so we have 53 million channels. And most of them are really different,” he explained to the audience. The individualized channel is evident just when the customer signs onto Netflix through the choice of categories and the order of the product within the categories. “Since the algorithms are contained in everything…the challenge is to combine all of them to work together,” to produce the optimal page of recommendations for each consumer.
Gomez Uribe describes his work as managing the Netflix “portfolio of innovation.” In an email interview following the talk, he discussed how projects move through the portfolio. “We probably have 60-80 active algorithm projects at any given time, and within each, many variations, e.g., using different parameter values,” he wrote. “Typically perhaps about two-thirds of these will make it into an actual A/B test” which are run with real members. The parameters used to judge the success of the innovation are pre-determined, having been developed and “tuned through offline experiments.”
Gomez Uribe wrote that it’s a good thing that most projects end up being tested offline to begin with. “Sometimes, regardless of parameter values and generalizations, we find the offline experiment results are so bad or that the cost of building the algorithm is so high that we decide to abandon the projects.”
Testing innovations is quick, but not instantaneous. “Quick projects might take 3 months to develop to the point of launching an A/B test, others as much as a year.”
Real success for Netflix is about innovations improving customer retention, so that the lifetime value of customers increases for the same acquisition cost. Given the base of 53 million customers, an improvement of just a few months has an enormous impact. Gomez Uribe says “We have seen that improvements to each and every one of our core algorithms has improved retention, including the page algorithm that constructs the page of recommendations, video ranking, video-video similarity, search, Top Picks, etc.”
And the success rate for innovations in the portfolio?
- “Probably about 10-20% of these projects succeed in the A/B test and make it to our default algorithm experience.”
- “Most test results are flat. Few are negative.”
What do failed tests look like?
- “Most of our misses yield flat results: the offline experiments look great and encouraging, but our members are indifferent to them.”
- “We’ve always been fascinated by the feedback loops in our system: people interact with and watch what we recommend, and we then use the resulting data to drive future recommendations. So we’ve explored a number of ways to have our algorithms take this feedback loop into account, to relatively small success so far.”