Cloud Native Data Pipelines

GOTO Chicago 2017

Big Data companies (e.g. LinkedIn, Facebook, Google, and Twitter) have historically built custom data pipelines over bare metal in custom-designed data centers. In order to meet strict requirements on data security, fault-tolerance, cost control, job scalability, network topology, and compute and storage placement, they need to closely manage their core technology. In recent years, many companies with Big Data needs have started migrating to one of the public cloud vendors. How does the public cloud change the game? Specifically, how can companies effectively marry cloud best-practices with big data technology in order to leverage the benefits of both? Agari, a leading email security company, is applying big data best practices to both the security industry and to the cloud in order to secure the world against email-bourne threats. We do this by building both batch and stream processing predictive data pipelines in the AWS cloud. Come to this talk to learn about our architectural best practices and technologies.