OpenTelemetry ingestion

Source module: app/connectors/otel.py; endpoint in app/api/routers/otel.py. Kind: internal. Display name: "Internal agents (OpenTelemetry)". Auth: otel (keyed push endpoint). Requires the otel_ingestion feature (Scale and Enterprise).

The span attribute convention is specified in docs/otel-convention.md; this page covers the connector's behaviour.

Purpose

Lets a customer's own agents assert outcomes (as claims with vendor = internal) and record actions (as facts) by exporting OTLP traces, so internal automation is held to the same verification as vendor agents and reported with an internal transfer price for definition-delta purposes.

Auth and scopes

Item	Value
Method	Per-source endpoint key otk_… embedded in the URL: POST {API_URL}/v1/otel/{endpoint_key}/v1/traces. No session.
Key storage	sha256(key) in otel_sources.endpoint_key_hash; last 4 characters for display. The key is shown once at creation (POST /v1/otel/sources) and cannot be retrieved later; create a new source to rotate.
Scopes	None on the customer side. CommitLayer never connects to the customer's tracing backend; the customer pushes.
Management routes	GET /v1/otel/sources (viewer), POST /v1/otel/sources {name, mapping?} (admin), PUT /v1/otel/sources/{id}/mapping (admin), DELETE /v1/otel/sources/{id} (admin; revokes the underlying connection)
Accepted content type	application/json (OTLP/HTTP JSON). Protobuf is rejected with 415.

Fields pulled and why

Only these parts of a span are read; everything else is dropped at ingestion:

Span field / attribute (default mapping)	Used for
name	Stored as claim attribute span_name
startTimeUnixNano, endTimeUnixNano	Fallback timestamp (end preferred) for claims and facts
traceId, spanId	raw_ref (otel:span:{trace}:{span} / otel:action:{trace}:{span})
agent.outcome.type (outcome_type_attr)	Marks the span as a claim; outcome type (optionally remapped through mapping.outcome_types)
agent.outcome.subject_id (subject_attr)	Claim subject id, verbatim
agent.outcome.claimed_at (claimed_at_attr)	Claim timestamp; defaults to span end
agent.outcome.unit_price (unit_price_attr)	Internal transfer price; default from mapping.outcome_types[type].unit_price or 0
agent.name	Claim attribute agent
agent.channel	Claim channel
agent.action.type (action_type_attr)	Marks the span as a fact; mapped to a fact type
agent.action.subject_id (action_subject_attr)	Fact subject id, verbatim
agent.action.from, agent.action.to	Status transition attributes

Resource attributes (e.g. service.name), events, links, status and all other span attributes are ignored. Message bodies are never accepted.

What is hashed / never stored

• Nothing is hashed: the convention carries system-of-record ids only, no personal identifiers. Any other attribute is discarded before persistence.
• The endpoint key is stored only as a SHA-256 hash.

Incremental behaviour and watermark

• Ingestion is synchronous: each POST parses resourceSpans[].scopeSpans[].spans[] (legacy instrumentationLibrarySpans accepted) and upserts immediately. The response reports {accepted, claims, facts}.
• No watermark. Re-sending the same span (same trace/span ids) updates the same claim/fact because ids derive from sha256(tenant_id | raw_ref).
• otel_sources tracks last_seen_at, spans_received, claims_created; the connection's health shows the last batch.
• A sync_connection request for an OTel connection returns not_syncable.

Rate-limit handling

None specific; the endpoint is a normal API route. Batch spans on the client side (standard OTLP exporters do). The reserved ingest_otel_batch job exists for future asynchronous replay.

Fact types produced

agent.action.type	fact_type
status_change	status_change (with from/to)
escalate, handoff	escalation
reply	agent_message
anything else	agent_action

Facts: source = otel, subject_kind = ticket, attributes = {action, from, to, actor: "agent"}.

Claims produced

vendor = internal, contract_id from mapping.outcome_types[type].contract_id or mapping.contract_id (default internal_agent_otel), subject_kind from the mapping (default ticket), outcome_type (remapped or verbatim), claimed_at, unit_price, channel, attributes = {span_name, agent}.

Subject id format

Verbatim from the attribute. Use the fact source's format so the claim can be matched: zendesk_ticket_{id}, intercom_conv_{id}, sf_case_{Id}, hubspot_ticket_{id}, or the ids used in your CSV facts.

Known limitations

• JSON only; OTLP protobuf and gRPC are not accepted. Configure the exporter for http/json.
• Spans with agent.action.type but no recognizable timestamp produce a fact with an empty occurred_at; ensure spans carry endTimeUnixNano.
• The default internal_agent_otel contract's vendor lens is a customer-defined rule (record_status_in or customer confirmation), not a vendor's published definition; there is no external citation.
• A deleted source revokes its connection but does not delete already-ingested claims and facts.
• Ingestion is gated by the tenant's entitlement at request time; a tenant that drops below Scale receives 402 on the endpoint.