kyo-stats-otlp

kyo-stats-otlp wires kyo-core's existing Stat counters, gauges, histograms, and trace spans to any OpenTelemetry Protocol collector. There is no new API to learn at the call site: you keep writing Stat.initScope(...).initCounter(...) and Stat.initScope(...).traceSpan(...) in your application. Adding this module to the classpath and setting OTEL_EXPORTER_OTLP_ENDPOINT is the entire integration.

Discovery happens through META-INF/services on JVM and Scala Native, and through an explicit @JSExportTopLevel registration on Scala.js. When the endpoint variable is unset, the factories return empty and the rest of the runtime sees the standard no-op exporter, so the module is safe to ship in builds that may or may not emit telemetry. Trace spans are buffered in a bounded channel and flushed in batches on a fixed schedule or when the batch-size threshold is hit; metrics are scraped from the global StatsRegistry on a separate periodic loop. W3C Trace Context propagation is layered as HTTP client/server filters discovered through the same service-loader path.

The module is published for JVM, Scala.js, and Scala Native. The wire encoding is OTLP-over-HTTP/JSON, not protobuf.

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=my-service

// "kyo-stats-otlp" % "<version>"

That is all that is required. No code changes are needed: the service-loader mechanism picks up the exporter automatically, and the environment variable controls whether export is active.

Setup and activation

Add kyo-stats-otlp to the classpath of any module that already uses kyo-core stats. No code changes are required to start exporting.

Service-loader registration

On JVM and Scala Native, kyo-stats-registry discovers exporter implementations through META-INF/services/kyo.stats.internal.ExporterFactory. The HTTP runtime independently discovers META-INF/services/kyo.HttpFilter$Factory. Both files ship in this module's jar, so dropping the jar on the classpath is enough.

On Scala.js, META-INF/services does not work. The module's JS-only OTLPRegistration object uses @JSExportTopLevel("__kyo_otel_init") to register both factories at module load time:

// js/src/main/scala/kyo/stats/otlp/OTLPRegistration.scala
import kyo.HttpFilter
import kyo.stats.internal.ExporterFactory
import kyo.stats.internal.JSServiceLoaderRegistry
import scala.scalajs.js.annotation.JSExportTopLevel

object OTLPRegistration:
    @JSExportTopLevel("__kyo_otel_init")
    val init: Boolean =
        JSServiceLoaderRegistry.register(classOf[ExporterFactory], new OTLPExporterFactory())
        JSServiceLoaderRegistry.register(classOf[HttpFilter.Factory], new OTLPHttpFilterFactory())
        true
    end init
end OTLPRegistration

The on/off switch

Setting OTEL_EXPORTER_OTLP_ENDPOINT is what activates the module. With the variable unset, every factory returns empty, and the runtime falls back to the standard no-op exporter. The application keeps running normally without exporting anything.

# Minimum configuration for a working OTLP setup
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=my-service

Configuration via OTEL_* environment variables

OTLPConfig is a plain case class capturing every tunable. The standard path is OTLPConfig.loadIfEnabled(), which reads the OTEL_* environment variables and returns Absent when the endpoint variable is unset. For tests and embedded use cases, build the value directly.

Loading from the environment

import kyo.*
import kyo.AllowUnsafe.embrace.danger
import kyo.stats.otlp.OTLPConfig

val maybe: Maybe[OTLPConfig] = OTLPConfig.loadIfEnabled()

maybe match
    case Present(cfg) =>
        // export enabled, cfg.endpoint is the OTLP base URL
        println(s"exporting to ${cfg.endpoint} as ${cfg.serviceName}")
    case Absent =>
        // OTEL_EXPORTER_OTLP_ENDPOINT not set, export disabled
        println("OTLP export disabled")
end match

loadIfEnabled reads System.getenv directly. The value is snapshotted at the call site, so changing environment variables after JVM start has no effect. Both OTLPExporterFactory and OTLPHttpFilterFactory call this independently, which means the environment is read multiple times during startup but always at startup.

Environment-variable reference

The full env-var contract follows the OpenTelemetry conventions:

Headers and resource attributes are parsed as comma-separated key=value pairs:

export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer abc123,x-tenant=acme"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment=prod,service.version=1.4.2"

Direct construction

For tests, ad-hoc tools, or non-environment-driven deployments, construct OTLPConfig directly:

import kyo.*
import kyo.stats.otlp.OTLPConfig

val cfg = OTLPConfig(
    endpoint = "http://localhost:4318",
    metricsEndpoint = "http://localhost:4318/v1/metrics",
    tracesEndpoint = "http://localhost:4318/v1/traces",
    headers = Map("authorization" -> "Bearer abc123"),
    timeout = 10.seconds,
    compression = "none",
    serviceName = "http-server",
    resourceAttributes = Map("env" -> "prod", "version" -> "1.0"),
    bspScheduleDelay = 5.seconds,
    bspMaxQueueSize = 2048,
    bspMaxExportBatchSize = 512,
    bspExportTimeout = 30.seconds,
    metricExportInterval = 60.seconds,
    metricExportTimeout = 30.seconds
)

Exporting trace spans

When OTLP export is active, kyo-stats-registry resolves TraceExporter.get to an OTLPTraceExporter instance. Application code uses the same Stat.traceSpan API it would use with any other exporter:

import kyo.*

val scope = Stat.initScope("http", "server")
val handler: String < (Sync & Async) =
    scope.traceSpan("GET /hello") {
        "hi"
    }

Each call to scope.traceSpan produces an OTLPSpan value at the wire boundary when the span completes.

Bounded queue and batched flush

OTLPTraceExporter owns a bounded Channel for completed spans. When a span's end() is called, the span is offered to the channel. If the channel is full, the span is dropped silently and only counted in kyo.stats.otel.spans.dropped. A background fiber drains up to bspMaxExportBatchSize spans per export call.

The flush loop is driven by two producers feeding a single consumer through a capacity-1 trigger channel:

• A periodic producer offers to the trigger every bspScheduleDelay.
• The exporter offers to the same trigger whenever the queue size reaches bspMaxExportBatchSize.

The consumer takes from the trigger and runs flush, which drains the queue, builds an ExportTraceRequest, and calls OTLPClient.sendTraces. The capacity-1 trigger naturally coalesces concurrent offers while the consumer is busy.

Fiber lifecycle

On JVM and Scala Native, the background fibers start in the OTLPTraceExporter constructor via OTLPInitPlatform.triggerStart. They run under Scope.run { ... Async.never }, so they are intentionally never interrupted. There is no public shutdown call on the exporter.

Span kind and status

OTLPTraceExporter always sets kind = OTLPModel.SpanKindInternal on exported spans. The OTLP enum constants for SpanKindServer and SpanKindClient exist on OTLPModel, but there is no API to mark a span as server, client, producer, or consumer. Status maps from kyo-core's UnsafeTraceSpan.Status straightforwardly:

// kyo.stats.internal.UnsafeTraceSpan.Status ->  OTLPModel constant
//   Status.Unset                            ->  StatusUnset (0)
//   Status.Ok                               ->  StatusOk    (1)
//   Status.Error(message)                   ->  StatusError (2), message preserved

Span attributes from kyo-core's Attributes collection map to OTLP KeyValue entries, with list-typed attributes joined into a single comma-separated stringValue.

Exporting metrics

OTLPMetricsExporter.run starts a background loop that scrapes the global StatsRegistry on metricExportInterval and exports any metrics that have non-zero activity since the last export. Application code uses the same Stat.initScope(...).initCounter(...), initHistogram(...), and initGauge(...) APIs as without this module.

import kyo.*

val scope    = Stat.initScope("http", "server")
val requests = scope.initCounter("requests", "Total requests")
val latency  = scope.initHistogram("latency_ms", "Request latency")

val handle: String < (Sync & Async) =
    for
        _ <- requests.inc
        _ <- latency.observe(12.3)
    yield "hi"

Counters and counter-gauges

Counters and counter-gauges export as OTLPSum with isMonotonic = true and aggregationTemporality = OTLPModel.DeltaTemporality. Each export reports the change since the last export, not a running total.

Histograms

Histograms export as OTLPHistogram with explicit bucket boundaries from kyo-core's histogram summary, plus per-bucket counts, min, and max. Aggregation temporality is also DELTA.

Gauges

Gauges export as OTLPGauge containing a single NumberDataPoint with the current value. Unlike counters, the value is not a delta.

Zero-activity intervals

Metrics with delta == 0 since the last export are skipped entirely. An export cycle with no traffic produces an empty payload, not zero-valued data points. Histograms with count == 0 are also skipped. This keeps cardinality cost down on the collector side but means downstream dashboards see gaps, not zeros, for idle periods.

Weak-reference cleanup for collected metric instances runs inline during the export iteration. A second pass over the same Scala Native map would deadlock, so cleanup and read share one traversal.

Distributed tracing across HTTP

W3C Trace Context propagation is provided by two filters in OTLPTraceContextFilter. They are auto-installed via OTLPHttpFilterFactory when OTLP is active:

import kyo.*
import kyo.stats.otlp.OTLPTraceContextFilter

val clientFilter: HttpFilter.Passthrough[Nothing] = OTLPTraceContextFilter.client
val serverFilter: HttpFilter.Passthrough[Nothing] = OTLPTraceContextFilter.server

Client-side injection

OTLPTraceContextFilter.client reads TraceSpan.current on each outgoing request. If the current span implements UnsafeTraceSpan.Propagatable, the filter adds a traceparent: 00-<traceId>-<spanId>-01 header:

// Inside OTLPTraceContextFilter.client (simplified):
//   TraceSpan.current.map {
//       case Present(TraceSpan(span: UnsafeTraceSpan.Propagatable)) =>
//           next(request.addHeader("traceparent", s"00-${span.traceId}-${span.spanId}-01"))
//       case _ => next(request)
//   }

Spans produced by OTLPTraceExporter implement Propagatable, and spans parsed by the server filter implement Propagatable. Spans from any other TraceExporter that does not extend Propagatable are silently skipped for header injection.

Server-side extraction

OTLPTraceContextFilter.server reads the traceparent header on each incoming request. If present and well-shaped, it parses the trace ID and span ID and uses TraceSpan.let to set a synthetic remote span as the current trace context for the rest of the handler:

import kyo.*
import kyo.stats.otlp.OTLPTraceContextFilter

// Pseudo-flow inside the server filter:
//   request.headers.get("traceparent") match
//       case Present(value) => parseTraceparent(value) match
//           case Present((traceId, spanId)) =>
//               TraceSpan.let(TraceSpan(new RemoteSpanUnsafe(traceId, spanId))) {
//                   next(request)
//               }
//           case _ => next(request)
//       case _ => next(request)

Operational visibility

Every OTLP failure path increments a kyo-core counter under the kyo.stats.otel scope. These counters are themselves visible to OTLPMetricsExporter and exported as ordinary metrics, so you can alert on them through the same collector you ship to:

A non-zero spans.dropped rate is the signal to raise OTEL_BSP_MAX_QUEUE_SIZE or OTEL_BSP_MAX_EXPORT_BATCH_SIZE. Non-zero *.export.failures after the retry policy is exhausted usually means the collector is unreachable, mis-authenticated (check OTEL_EXPORTER_OTLP_HEADERS), or rejecting payloads (check rejection counters for partial-success detail).

The retry policy on OTLPClient is Schedule.exponentialBackoff(1.second, 2.0, 5.seconds).take(5), gated on status.code == 429 || status.isServerError. Other 4xx responses are not retried and increment the failure counter immediately.

OTLP wire model

The model in OTLPModel.scala mirrors the OpenTelemetry Protocol protobuf messages, serialized as JSON via derives Schema. It is exposed publicly so tests, adapters, and direct callers of OTLPClient.sendTraces / OTLPClient.sendMetrics can build payloads without going through TraceSpan or Stat.

Building values directly

Each case class is constructed positionally. Optional fields use Maybe[A] (Absent/Present(a)):

import kyo.*
import kyo.stats.otlp.*

val span = OTLPSpan(
    traceId = "0123456789abcdef0123456789abcdef",
    spanId = "0123456789abcdef",
    name = "test",
    startTimeUnixNano = "1000",
    endTimeUnixNano = "2000",
    attributes = Seq(KeyValue("http.method", AnyValue.string("GET"))),
    status = SpanStatus(code = OTLPModel.StatusOk)
)

val request = ExportTraceRequest(
    resourceSpans = Seq(ResourceSpans(
        resource = OTLPResource(attributes = Seq(KeyValue("service.name", AnyValue.string("my-service")))),
        scopeSpans = Seq(ScopeSpans(
            scope = InstrumentationScope("kyo-stats", version = "1.0.0"),
            spans = Seq(span)
        ))
    ))
)

AnyValue: the OTLP union type

OTLP's AnyValue is a tagged union (string, int, double, bool) that this module encodes as a struct with at most one field set. Use the smart constructors on the AnyValue companion to build values; constructing the case class directly with multiple fields set is meaningless on the wire:

import kyo.stats.otlp.AnyValue

val s = AnyValue.string("hello") // stringValue = Present("hello")
val i = AnyValue.int(42L)        // intValue    = Present("42")  -- ints are wire-encoded as strings
val d = AnyValue.double(3.14)    // doubleValue = Present(3.14)
val b = AnyValue.boolean(true)   // boolValue   = Present(true)

OTLPModel constants

OTLPModel carries the integer enum constants used in the JSON wire form:

import kyo.stats.otlp.OTLPModel

val internal = OTLPModel.SpanKindInternal // 1
val server   = OTLPModel.SpanKindServer   // 2
val client   = OTLPModel.SpanKindClient   // 3

val unset = OTLPModel.StatusUnset // 0
val ok    = OTLPModel.StatusOk    // 1
val err   = OTLPModel.StatusError // 2

val delta      = OTLPModel.DeltaTemporality      // 1
val cumulative = OTLPModel.CumulativeTemporality // 2

OTLPTraceExporter always emits SpanKindInternal; the other constants are provided for adapters and synthesized payloads.

Building histograms and sums

The metric variants are case classes that mirror the OTLP wire shape exactly:

import kyo.*
import kyo.stats.otlp.*

val sumPoint = NumberDataPoint(
    startTimeUnixNano = "1000",
    timeUnixNano = "2000",
    asInt = Present("17")
)

val sum = OTLPSum(
    dataPoints = Seq(sumPoint),
    aggregationTemporality = OTLPModel.DeltaTemporality,
    isMonotonic = true
)

val histPoint = HistogramDataPoint(
    startTimeUnixNano = "1000",
    timeUnixNano = "2000",
    count = "3",
    explicitBounds = Seq(1.0, 5.0, 10.0),
    bucketCounts = Seq("1", "1", "1", "0"),
    min = 0.5,
    max = 7.0
)

val hist = OTLPHistogram(
    dataPoints = Seq(histPoint),
    aggregationTemporality = OTLPModel.DeltaTemporality
)

val counter = Metric(name = "http.server.requests", description = "Total requests", unit = "1", sum = Present(sum))
val latency = Metric(name = "http.server.latency_ms", description = "Request latency", unit = "1", histogram = Present(hist))

Sending payloads directly

When you have an OTLPConfig and an ExportTraceRequest or ExportMetricsRequest, you can call OTLPClient.sendTraces / OTLPClient.sendMetrics to send a single payload. Both methods carry the same retry policy, partial-success handling, and failure counters as the background exporter:

import kyo.*
import kyo.stats.otlp.*

def sendOne(config: OTLPConfig, request: ExportTraceRequest): Unit < Async =
    OTLPClient.sendTraces(config, request)

These are the seams adapters and tests reach for when they want to drive the wire without spinning up the background exporter.