One of the key aspects of a solid data solution is reliability, as customers entrust their most valuable assets in the system. When it comes to data platforms performing at 100K IO operations per second, failure is eminent. In this talk I will present the basic hypothesis – “shit will happen” and go over the strategies to recover and maintain service in highly distributed, high-performance, micro-service driven architectures.