apply() - Towards a Unified Real-Time ML Data Pipeline, from Training to Serving

Speakers: Aakash Sabharwal and Sheila Hu, Etsy

On a global marketplace like Etsy where buyers come to buy unique, varied items from sellers from around the globe, the inventory of items is constantly changing. User preferences also change in real time as they discover the latest selection being offered on the site. In such a dynamic environment, Machine Learning models for different applications (including search, recommendations or computational advertisement), need to collect different real time data signals, process them and finally leverage them to make the most relevant predictions.

In this talk we will detail how we use real-time feature logging to capture in-session / trending activities, build a typed unified feature store for sharing features across models from different domains and serve feature data at scale with the eventual goal of powering reactive systems. We will finally show how such a real-time ML data pipeline can be leveraged to build Reactive systems (Bandits / Reinforcement / Online learning applications) that can use state of the art ML algorithms to learn from user actions in real time.