A Survey of the Open-Source Tracing Ecosystem

Transcription

1 A Survey of the Open-Source Tracing Ecosystem KubeCon + CloudNativeCon Europe May 2, 2018 Ben Sigelman OpenTracing co-creator, LightStep co-founder and CEO

2 Talk #goals - What s distributed tracing? How does it compare with other monitoring tech? What are the moving parts? Why are there so many tracing projects??? and what s the difference between them?

3 Part I Where Distributed Tracing fits into the Monitoring Ecosystem

4 All of Monitoring (more or less) Timeseries Statistics Measuring Symptoms Structured Events p99 latency alerts Exception Monitoring (Sentry) Error ratio alerts Counting relatively infrequent things like software releases (etc) (etc) Explaining Symptoms Most big dashboards Grepping logs Filter + GroupBy on metrics Rolling up data in ad hoc ways, a la Kibana or similar (i.e., Root-Cause Analysis) (etc) and Distributed Tracing!

5 Distributed Traces: Events, Organized monolith microservices - Every request touches many services - These touches generate event data - Must glue all of the per-request event data back together - Once you re gluing, you re tracing

6 Example Request with Trace Visualization Client Client Web server Auth Web server Billing Verify Process Auth Billing DB DB time Time context visualization

7 Tracing: the Mental Model Service A Service B Network Call Parent Span Span Context Child Span Child Span log log log log Child Span log log Trace: A recording of a transaction as it moves through a distributed system. Span: A named, timed operation representing a piece of the workflow. Spans have a Timestamp and a Duration, are annotated with Tags and Logs. Span Context: A set of Trace Identifiers injected into each request, which the next service will extract and use in order to propagate the trace.

8 Part II Tracing, Tracing, Tracing, and Tracing : One word, four needs

9 Tracing: It s About Analyzing Transactions

10 Tracing: It s About Recording Transactions (Small) Context Data Service or Sidecar Process Tracer Runtime / APM Agent (Big) Trace Data Tracing/APM Backend System

11 Tracing: It s About Federating Transactions (Small) Context Data Service or Sidecar Process Tracer Runtime / APM Agent Managed Cloud Service (Big) Trace Data Cloud-Provider Tracing Service Tracing/APM Backend System

12 Tracing: It s About Describing Transactions Service or Sidecar Process Service / Biz Logic OSS Library Tracer Runtime / APM Agent

13 Tracing is (presently) all four Analyzing Transactions Recording Transactions Federating Transactions Describing Transactions All have their place; hopefully they stay decoupled.

14 Part III Noteworthy tracing projects and how they fit together

15 Composable Parts in a Tracing Deployment (Small) Context Data Tracing System Authors Service or Sidecar Process Tool Users App Instrumentors Service / Biz Logic OSS Instrumentors OSS Library Data Recorders Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

16 Zipkin surface area (Small) Context Data Tracing System Authors Zipkin is a distributed tracing system.... It manages both the collection and lookup of this data. Service or Sidecar Process Tool Users App Instrumentors Service / Biz Logic OSS Instrumentors OSS Library Data Recorders Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

17 Jaeger: open source, end-to-end distributed tracing Jaeger surface area See also: SkyWalking, AppDash (Small) Context Data Tracing System Authors Jaeger is OpenTracing-native so needs no instrumentation APIs of its own Service or Sidecar Process Tool Users Service / Biz Logic OSS Library Data Recorders Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

18 AWS X-Ray surface area See also: StackDriver Trace (Small) Context Data Tracing System Authors AWS X-Ray helps developers analyze and debug production, distributed applications, such as those built using a microservices architecture. Service or Sidecar Process Tool Users App Instrumentors Service / Biz Logic OSS Instrumentors OSS Library Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

19 w3c Trace-Context surface area (Small) Context Data The mission of this group is to define the standard for distributed trace context propagation. Tracing System Authors Service or Sidecar Process Service / Biz Logic OSS Library Tracer Runtime / APM Agent (Big) Trace Data Tracing / APM Tool Application data Trace data Separate, stable interface Managed Cloud Service Cloud-Provider Tracing Service

20 w3c data interchange format surface area (Small) Context Data Service or Sidecar Process The data interchange format includes both the serialization protocol and the semantics of the data (how to model HTTP requests, SQL queries, etc) Service / Biz Logic OSS Library Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

21 OpenCensus surface area (Small) Context Data Tracing System Authors A single distribution of libraries for metrics and distributed tracing with minimal overhead that allows you to export data to multiple backends. Service or Sidecar Process App Instrumentors Service / Biz Logic OSS Instrumentors OSS Library Data Recorders Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

22 Vendor-neutral APIs and instrumentation for distributed tracing OpenTracing surface area Focused on reducing instrumentor toil and forward-facing risk. (Small) Context Data Service or Sidecar Process App Instrumentors Service / Biz Logic OSS Instrumentors OSS Library Tracer Runtime / APM Agent Tracing System Authors Application data Trace data Separate, stable interface Tracing / APM Tool (Big) Trace Data Managed Cloud Service Cloud-Provider Tracing Service

23 Part IV The Case for Narrow Interfaces

24 Developer adoption pain (for tracing) Tracer SDKs / clients Instrumenting service logic and third-party open-source code ( Describing Transactions ) Tracing backends and UIs

25 One example: OpenTracing scope: - 9 languages instrumented packages Tracer diversity: - Many Tracer implementations: OSS and commercial - Some are not just tracing a la Dapper (debuggers, metrics, logging) A small formal API means: - A single, decoupled value proposition - Smaller deps at compile-time - Fewer surprises downstream

26 Another example: w3c TraceContext Many players: 25+ Many other (tempting) adjacent problems to solve - The Span data format for vendor-to-vendor interop - OpenTracing-style APIs for propagation among other things - More complex sampling strategies, as well as baggage, etc The narrow scope: - Keeps the discussion focused - Encourages a best-of-breed approach - Reduces coupling (e.g., between instrumentation and headers)

27 Summing up - Tracing is table stakes now Tracing must involve many constituents Tracing, Tracing, Tracing, & Tracing: not the same thing :) Each project exists for good reasons: understand them Not a zero-sum game! Also: I m rarely in Europe and want to meet you! bhs@lightstep.com