More and more companies are using deep-learning based models. However, periodically updating products with such models is challenging due to large datasets and long training procedures.

Face2Gene, FDNA's community-driven platform, is used to aid in the identification of rare diseases. With the constantly growing database, periodic system updates are essential. To enable that, we developed a unified system of data management and a scalable infrastructure to prepare the data and parallelly train and evaluate dozens of models at scale. The result is a production ready system.

In this talk we will describe our system in detail, discuss the design challenges, and share best practices learned.