# OpenTelemetry ingestion

Source module: `app/connectors/otel.py`; endpoint in `app/api/routers/otel.py`. Kind: **internal**. Display name: "Internal agents (OpenTelemetry)". Auth: **otel** (keyed push endpoint). Requires the `otel_ingestion` feature (Scale and Enterprise).

The span attribute convention is specified in [`docs/otel-convention.md`](/docs/otel-convention); this page covers the connector's behaviour.

## Purpose

Lets a customer's own agents assert outcomes (as **claims** with `vendor = internal`) and record actions (as **facts**) by exporting OTLP traces, so internal automation is held to the same verification as vendor agents and reported with an internal transfer price for definition-delta purposes.

## Auth and scopes

| Item | Value |
|---|---|
| Method | Per-source endpoint key `otk_…` embedded in the URL: `POST {API_URL}/v1/otel/{endpoint_key}/v1/traces`. No session. |
| Key storage | `sha256(key)` in `otel_sources.endpoint_key_hash`; last 4 characters for display. The key is shown once at creation (`POST /v1/otel/sources`) and cannot be retrieved later; create a new source to rotate. |
| Scopes | None on the customer side. CommitLayer never connects to the customer's tracing backend; the customer pushes. |
| Management routes | `GET /v1/otel/sources` (viewer), `POST /v1/otel/sources {name, mapping?}` (admin), `PUT /v1/otel/sources/{id}/mapping` (admin), `DELETE /v1/otel/sources/{id}` (admin; revokes the underlying connection) |
| Accepted content type | `application/json` (OTLP/HTTP JSON). Protobuf is rejected with `415`. |

## Fields pulled and why

Only these parts of a span are read; everything else is dropped at ingestion:

| Span field / attribute (default mapping) | Used for |
|---|---|
| `name` | Stored as claim attribute `span_name` |
| `startTimeUnixNano`, `endTimeUnixNano` | Fallback timestamp (`end` preferred) for claims and facts |
| `traceId`, `spanId` | `raw_ref` (`otel:span:{trace}:{span}` / `otel:action:{trace}:{span}`) |
| `agent.outcome.type` (`outcome_type_attr`) | Marks the span as a claim; outcome type (optionally remapped through `mapping.outcome_types`) |
| `agent.outcome.subject_id` (`subject_attr`) | Claim subject id, verbatim |
| `agent.outcome.claimed_at` (`claimed_at_attr`) | Claim timestamp; defaults to span end |
| `agent.outcome.unit_price` (`unit_price_attr`) | Internal transfer price; default from `mapping.outcome_types[type].unit_price` or 0 |
| `agent.name` | Claim attribute `agent` |
| `agent.channel` | Claim `channel` |
| `agent.action.type` (`action_type_attr`) | Marks the span as a fact; mapped to a fact type |
| `agent.action.subject_id` (`action_subject_attr`) | Fact subject id, verbatim |
| `agent.action.from`, `agent.action.to` | Status transition attributes |

Resource attributes (e.g. `service.name`), events, links, status and all other span attributes are ignored. Message bodies are never accepted.

## What is hashed / never stored

- Nothing is hashed: the convention carries system-of-record ids only, no personal identifiers. Any other attribute is discarded before persistence.
- The endpoint key is stored only as a SHA-256 hash.

## Incremental behaviour and watermark

- Ingestion is synchronous: each POST parses `resourceSpans[].scopeSpans[].spans[]` (legacy `instrumentationLibrarySpans` accepted) and upserts immediately. The response reports `{accepted, claims, facts}`.
- No watermark. Re-sending the same span (same trace/span ids) updates the same claim/fact because ids derive from `sha256(tenant_id | raw_ref)`.
- `otel_sources` tracks `last_seen_at`, `spans_received`, `claims_created`; the connection's `health` shows the last batch.
- A `sync_connection` request for an OTel connection returns `not_syncable`.

## Rate-limit handling

None specific; the endpoint is a normal API route. Batch spans on the client side (standard OTLP exporters do). The reserved `ingest_otel_batch` job exists for future asynchronous replay.

## Fact types produced

| `agent.action.type` | `fact_type` |
|---|---|
| `status_change` | `status_change` (with `from`/`to`) |
| `escalate`, `handoff` | `escalation` |
| `reply` | `agent_message` |
| anything else | `agent_action` |

Facts: `source = otel`, `subject_kind = ticket`, `attributes = {action, from, to, actor: "agent"}`.

## Claims produced

`vendor = internal`, `contract_id` from `mapping.outcome_types[type].contract_id` or `mapping.contract_id` (default `internal_agent_otel`), `subject_kind` from the mapping (default `ticket`), `outcome_type` (remapped or verbatim), `claimed_at`, `unit_price`, `channel`, `attributes = {span_name, agent}`.

## Subject id format

Verbatim from the attribute. Use the fact source's format so the claim can be matched: `zendesk_ticket_{id}`, `intercom_conv_{id}`, `sf_case_{Id}`, `hubspot_ticket_{id}`, or the ids used in your CSV facts.

## Known limitations

- JSON only; OTLP protobuf and gRPC are not accepted. Configure the exporter for `http/json`.
- Spans with `agent.action.type` but no recognizable timestamp produce a fact with an empty `occurred_at`; ensure spans carry `endTimeUnixNano`.
- The default `internal_agent_otel` contract's vendor lens is a customer-defined rule (`record_status_in` or customer confirmation), not a vendor's published definition; there is no external citation.
- A deleted source revokes its connection but does not delete already-ingested claims and facts.
- Ingestion is gated by the tenant's entitlement at request time; a tenant that drops below Scale receives `402` on the endpoint.