OpenTelemetry & LGTM Platform

The problem

Telemetry ends up coupled to whichever vendor was chosen first — instrumentation is rewritten every time the backend changes, and metrics, logs and traces stay in three unconnected tools.

Constraints

• Instrumentation must outlive any single observability vendor
• Metrics, logs and traces need to correlate on the same request
• Collection overhead has to stay proportionate to the workload

Architecture

Key decisions

What was chosen, what it was chosen over, and why.

• Chose

  OpenTelemetry Collector as the single ingest point

  Instead of

  Vendor agents exporting straight from each app

  Applications emit OTLP and know nothing about the backend, so changing or adding one becomes an exporter config change instead of a re-instrumentation project. That is the whole reason to accept the extra hop.

• Chose

  Agent DaemonSet feeding a central gateway

  Instead of

  Agent-only collection

  Node-local agents enrich with Kubernetes metadata and buffer through brief network trouble. The gateway holds everything needing a global view — filtering, batching and sampling — in one place instead of on every node.

• Chose

  Tail-based sampling at the gateway

  Instead of

  Head-based sampling at the source

  Keeps 100% of errors and latency outliers while shedding healthy high-volume traces. It cannot work at the agent, which only ever sees part of a trace — which is what forces the gateway tier.