Observe execution and connect telemetry

Use Pipelogiq's run history for workflow state and your observability backend for service traces, metrics, and operational alerts. The dashboard connects these views through IDs and link templates. It does not replace instrumentation inside your business handlers.

Start with the workflow

Open a pipeline and inspect its failed or slow stage. Record the pipeline ID, stage ID, handler, execution attempt, and trace ID when present. These identifiers make an infrastructure investigation actionable: a timeout trend alone does not tell you which business operation needs recovery.

Administrators can open Observability for execution summaries, trace-linked run data, computed duration/error insights, and integration configuration. Access to these settings is separate from the Viewer/Operator pipeline investigation pages.

Configure OpenTelemetry export

The API and orchestration worker initialize tracing at startup. Set these in the deployment environment, using a collector hostname reachable from the containers:

OTEL_EXPORTER_OTLP_ENDPOINT=pipelogiq-tempo:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=1
OTEL_SERVICE_NAME=pipelogiq-app
OTEL_SERVICE_NAME_WORKER=pipelogiq-worker

This is the bundled local Tempo configuration. For your own collector, replace the endpoint and use TLS settings appropriate to that collector. OTEL_EXPORTER_OTLP_INSECURE=true is the plaintext local connection setting, not a recommendation for traffic over an untrusted network. HTTP export is also supported; match the protocol and port to your collector.

Restart the services after changing exporter environment. Saving a dashboard integration is not the same as reconfiguring the process exporter.

In registry Compose, leave OTEL_EXPORTER_OTLP_ENDPOINT empty to disable export. If using the bundled backend, set the endpoint above and start the optional profile:

docker compose --env-file .env -f infra/compose/docker-compose.registry.yml --profile observability up -d

Registry Compose does not publish Grafana's port. Reach it through your configured proxy or private network. The source-build stack starts Tempo/Grafana by default and publishes Grafana at http://localhost:3100.

Connect a run to its trace

Pipeline creation accepts an explicit traceId; otherwise the server can extract it from a traceparent context item or generate one. Stage jobs carry trace identifiers and trace context for workers to continue. Use the SDK's tracing support where available and instrument your business handler so its external calls become part of the trace.

Configure a trace link template in the OpenTelemetry integration using ${traceId} where the backend expects the trace ID. The URL must be reachable from the operator's browser. A Docker-only hostname can work for exporter traffic while being unusable in a browser link.

If no trace appears, compare the pipeline's trace ID with the backend, confirm the exporter endpoint/protocol, inspect sampling, and check that your application worker exported its own spans. A stored trace ID alone does not prove a tracing backend received data.

Scrape Prometheus metrics

Metrics use dedicated listeners:

For a Prometheus server attached to the same network:

scrape_configs:
  - job_name: pipelogiq-api
    metrics_path: /metrics
    static_configs:
      - targets: ["pipelogiq-app:9091"]
  - job_name: pipelogiq-worker
    metrics_path: /metrics
    static_configs:
      - targets: ["pipelogiq-worker:9090"]

The standard Compose stack does not include a Prometheus server. Add scraping to your existing monitoring service. Metrics are not at http://localhost:3300/api/metrics.

Useful signals include stage_published_total, stage_result_processed_total, stage_result_failed_total, stage_active_timed_out_total, and HTTP pull/ack counters. Schedule metrics include schedule_runs_total, schedule_lag_seconds, and schedule_next_run_timestamp_seconds. Histograms exist for schedule lag and selected API operations; do not assume all metrics are counters or all carry application/handler labels.

Configure and test alerts

Implemented delivery channels are Telegram and generic webhook. Other channel names in configuration are not a guarantee of working delivery.

1. Open Observability, then the Alerts integration.
2. Configure a Telegram bot token/chat ID or a generic webhook endpoint.
3. Select the events you want, such as stage failure, manual rerun/skip, or worker lifecycle changes.
4. Save and use the connection/test action.
5. Confirm the test arrived at the destination, then verify a representative event in a non-production workflow.

For Telegram, create the bot in BotFather and start a conversation with it (or add it to the intended group) before testing. For a webhook, make its receiver accept the Pipelogiq JSON notification envelope. Deduplication may suppress repeated alerts during its configured window; current delivery is not an on-call escalation service or a durable notification queue.

Protect integration credentials

Set the same SECRETS_ENCRYPTION_KEY in API and orchestration worker before saving tokens. Generate one with openssl rand -base64 32. Without it, integration secrets are stored unencrypted and the service logs a warning. Keep the key separately with your recovery materials; a database restore without the matching key cannot decrypt those values.

Pipeline-sensitive context uses a different mechanism: it masks public run data while workers receive real values. It does not encrypt all pipeline payloads at rest. See access and deployment.