Observability

Runwick uses the standard .NET tools and exports nothing unless you ask:

• Traces: one ActivitySource named Runwick.
• Metrics: one Meter named Runwick, with fixed names, units and attributes.
• Logs: ILogger, with fixed event IDs.

Payloads, credentials, connection strings and exception messages stay out by default. Job IDs may appear in spans and logs, never in metrics.

Turning it on with OpenTelemetry

dotnet add package Runwick.OpenTelemetry
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

using OpenTelemetry.Metrics;
using OpenTelemetry.Trace;

builder.Services.AddOpenTelemetry()
    .WithTracing(t => t.AddRunwickInstrumentation().AddOtlpExporter())
    .WithMetrics(m => m.AddRunwickInstrumentation().AddOtlpExporter());

Runwick.OpenTelemetry adds no exporter; use the one you already have. On the metrics side it keeps only Runwick's own attributes and sets a cardinality limit per instrument.

Without OpenTelemetry, any listener works: subscribe to the Runwick source and meter by name (the constant is RunwickTelemetry.SourceName).

Options

builder.Services.AddRunwick(r => r
    .Application("my-app")
    .UsePostgres(connectionString)
    .Telemetry(t =>
    {
        t.CollectApplicationMetrics = true;    // this process samples queue depth for the whole application
        t.ApplicationSamplePeriod = TimeSpan.FromSeconds(15);
    }));

If you raise MaxSeriesPerInstrument, pass the same number to AddRunwickInstrumentation(o => ...) so the SDK limit matches.

Traces

Lease renewals are not spans; they repeat too often. They show up in metrics.

Metrics

Durations are in seconds unless noted.

In Prometheus the names become runwick_queue_due, runwick_execution_duration_seconds_bucket and so on.

Logs

Categories are Runwick.Worker, Runwick.Dispatch, Runwick.Scheduler, Runwick.Outbox, Runwick.Management, Runwick.Retention, Runwick.Store and Runwick.Telemetry. Every message has a fixed event ID, for example:

Warnings are limited to 10 a minute per event ID per process. The rest are counted and summarised once a minute (event 1603).

Reading the numbers honestly

• Local and application-wide numbers are different. Worker gauges describe one process and can be summed across processes. Queue depth and the oldest due job come only from processes with CollectApplicationMetrics = true. Usually that is one process, for example the standalone management host.
• Never sum collector series. Runwick does not elect a collector. If two processes collect, both report the same queue. Use max by (runwick_queue); the shipped dashboard and alerts do.
• Missing is not zero. When a collector's sample is older than three periods, its queue gauges disappear instead of reporting 0, and runwick.collector.snapshot.age keeps growing.
• "Due" is not "runnable". A due job may still wait for a concurrency slot, a rate limit, a paused queue or a worker that can run it.
• A handler that succeeded is not a stored success. If the result could not be confirmed, the span says runwick.outcome = succeeded with runwick.settlement.result = unresolved. A replayed command is not counted twice.
• Counts can be lost when a process crashes. Use them for operations, not for billing or audit.

Grafana dashboard and Prometheus alerts

Runwick.OpenTelemetry ships two files in its package content folder (they are also in the repository's ops folder):

• content/grafana/runwick-overview.json: a Grafana dashboard with dispatch, execution, leases, pools, queues, schedules, management, retention and telemetry-health panels.
• content/prometheus/runwick-alerts.yml: alert rules.

The telemetry follows the OpenTelemetry semantic conventions version pinned in RunwickTelemetry.SemanticConventionsVersion, and no other.