The Biggest Data of All? A Brief Introduction to Log Analysis

GOTO Chicago 2017

Logs have always been around, and we tend to take them for granted, at least until something goes wrong. It is just when we need it the most, that we realize that log analysis can be one of the most complex and demanding aspects of modern system architecture; in many scenarios, the volume of log data can exceed all other data put together!

This talk makes the claim that we should be architecting our systems for log shipping and analysis, and demonstrates a variety of different ways to approach the problem. We'll explore the different technologies commonly used for logs, including open-source tools like Elasticsearch and OK Log, as well as proprietary systems such as Humio and Splunk. We'll examine how various tools perform under real-world pressure, and how those performance characteristics should inform resource planning and other considerations. Finally, we'll consider some real-world hard problems which can test the limits of any log analysis system, and apply what we've learned to plan out an efficient and pragmatic solution.