In a monolithic application, debugging is easy: you open a log file and read the stack trace. In a distributed microservices architecture, a user clicks 'Checkout', which hits the API Gateway, which calls the Order Service, which calls the Payment Service, which calls a third-party Stripe API. If the request fails or takes 5 seconds, how do you know which service caused the problem? The answer is Observability (Logs, Metrics, and Traces).

Module 1: The Three Pillars of Observability

The Pillars

  • 1. Logs (What happened?): Discrete text records of an event (e.g., 'User 123 logged in'). Must be in JSON format for querying.
  • 2. Metrics (Is there a problem?): Aggregated numerical data over time (e.g., CPU usage is at 95%, Error rate is 4%).
  • 3. Traces (Where is the problem?): The journey of a single request across network boundaries. It shows exactly how many milliseconds were spent in the Order Service vs the Payment Service.

Module 2: OpenTelemetry (OTel)

Historically, companies used proprietary agents (like Datadog or New Relic) to collect this data, causing vendor lock-in. OpenTelemetry (a CNCF project) is now the industry standard. It provides open-source SDKs to instrument your code, and an OTel Collector that can forward the data to ANY backend.

instrumentation.js (Node.js)javascript
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { HttpInstrumentation } = require('@opentelemetry/instrumentation-http');
const { ExpressInstrumentation } = require('@opentelemetry/instrumentation-express');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://otel-collector:4318/v1/traces' }),
  instrumentations: [
    // Auto-instruments all HTTP requests and Express routes
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
  ],
});

// Must run BEFORE requiring the rest of your app
sdk.start();

Module 3: Distributed Tracing in Action

How does tracing work across multiple servers? Through Context Propagation. When the API Gateway receives a request, it generates a unique TraceID. When it calls the Order Service, it injects this TraceID into the HTTP headers (e.g., traceparent). The Order Service extracts the header, does its work, and passes the same TraceID to the Payment Service.

When all services send their trace data to the Collector, the backend (like Jaeger or DataDog) stitches them together into a beautiful Waterfall chart.

Module 4: Structured Logging

Never use console.log("User logged in"). You cannot easily search millions of text strings. Use Structured Logging (JSON) and attach the TraceID to the log.

logger.js (Pino)javascript
const pino = require('pino');
const { trace } = require('@opentelemetry/api');

const logger = pino();

app.post('/login', (req, res) => {
  // Get the current active trace span
  const span = trace.getActiveSpan();
  const traceId = span ? span.spanContext().traceId : 'none';

  // Log structured JSON
  logger.info({
    userId: req.body.id,
    action: 'login',
    trace_id: traceId
  }, 'User authentication successful');
  
  res.send('OK');
});