The Data Lake runs 120K jobs a day all dynamically generated from metadata with the goal of ingesting data from source, validating it, and then performing milestoning. This talk will cover the use case, architecture, challenges, solutions and our experience with using Flink. I will focus on optimizations and tuning in a dynamic and variable workload. We will also cover metrics and performance over using our original heterogeneous stack versus Flink.

Background:

Our Data Lake provides a platform to manage data in a central location so that anyone in the firm can rapidly query, analyze or refine the data in a standard way. While there are many data lakes in industry, we see ours differently in several key distinctions:

1) Decoupling of producers and consumers – Traditionally in the industry state-of-the-art, the producers have to build dedicated pipelines for each producer-consumer pair into a data warehouse where then transformations can be made. We see this as two user flows with the Lake acting as the intermediary, thereby changing an m (number of datasets) * n (number of consumers) problem to m + n. In the previous model, the transformations performed on the original dataset are then isolated whereas we capture the output of the transformations and put it back into the lake. We call these refiners, consumers of existing data and producers of new transformed data.

2) Dynamic ingestion pipelines – In order to scale, we make it self-service for our clients to onboard and define a schedule. The Data Lake then dynamically kicks off the jobs necessary to extract, validate, and merge/commit their data.

3) Technology agnostic – We acknowledge that the technologies around us can change. Producers have different source types that we can extract from and consumers may want their data in a variety of endpoints. Both of these are simply connecters into and out of the core Data Lake. Our core pipeline technology can also change without having our clients rebuild.

4) Shared infrastructure management – We own the central cluster that serves both as data nodes and compute for ingestion and refiner processing. We also own shared database pairs where consumers query from rather than having to provision and manage their own.

5) Built in Milestoning/Replication – Reproducibility is critical for regulatory functions. We milestone the data by comparing the incoming data with the existing data. We mark the records with the batch ids for which they were live from and thru. We handle replication centrally both from a core Lake perspective as well as the consumer endpoints.

6) Centralized controls and entitlements -- We centralize security of the datasets by maintaining a mapping between users and the datasets and apply these on both the files in our cluster as well as the consumer endpoints.

In addition to this, because we have the metadata on the levels of transformations done, we can derive data provenance as well as increase the incentives to ensure high levels of data quality.

The Data lake infrastructure runs around a Hadoop core consisting of a pair of Hadoop clusters each sized at 20k cores, 100TB of memory and 13PB of storage for multi-data center redundancy. We originally built this using a heterogeneous stack involving a mixture of homegrown tech, Spark, MapReduce but have since created a homogenous stack using Flink resulting in a 3-5x reduction in wall clock latency while reducing CPU and Memory utilization at the same time. The Flink pipeline is dynamically constructed on a per job basis from metadata described what the lake needs to do (what is the data, where is it, how often to get it, who is using it and so on). We generate about 120K jobs from this daily. In addition to a drop in replacement of the existing processing architecture, Flink allows to decouple the interdependency of having to ingest raw data fully before transforming it. More importantly, Flink offers us the ability to have simultaneous streaming and batch based versions of existing pipelines without writing code, potentially chaining the n-levels of refining resulting in a reduction of overall latency thereby realizing a 100% Kappa architecture. Replication of data/metadata between lake clusters is also handled using Flink jobs.

Conference Program