Telemetry Conventions
Every series the platform reports for your workload, and every series your code reports about its own operations, is named by this page. The default dashboards and alerts read these names alone, so a series named another way is stored and queried but drawn and alerted on by nothing. The telemetry package writes these names for you, and the plain OpenTelemetry SDK writing them by hand is equally supported.
Version
Section titled “Version”These conventions are version 1. Your spec names the version its code
reports by as observability.conventions, and your resource carries it as
vantacloud.conventions.version. A later version adds names and never
renames or removes one of an earlier version, so code written to 1 stays
conformant.
Resource attributes
Section titled “Resource attributes”| Attribute | Set by | Holds |
|---|---|---|
service.name | Your code, or the sidecar from observability.service | Your service’s name |
service.version | Your code | The version of your build, recommended |
service.instance.id | Your code, or the package from the pod’s name | The pod the record or series came from, one value per replica |
vantacloud.conventions.version | Your code | 1 |
vantacloud.tenant.id, vantacloud.workload.name, vantacloud.pod.id, deployment.environment.name | The sidecar | Where the record came from; a value your code sets is replaced |
Attributes
Section titled “Attributes”| Attribute | Values |
|---|---|
vantacloud.operation | The operation’s name: an HTTP route as its method and route template, such as GET /v1/orders/{id}, never the concrete path; any other operation as lowercase dotted words naming its subject and verb, such as orders.settle |
vantacloud.outcome.class | ok; client-error, refused for the caller’s fault, an HTTP 4xx among them; server-error, failed for the service’s fault, an HTTP 5xx among them; cancelled, abandoned by the caller before it ended |
error.kind | A stable kebab-case code naming the failure, such as not-found or payment-declined, on the terms Error Logging states |
vantacloud.capability | vault, store, db, models, accounts, vantanet or telemetry |
vantacloud.probe.name | The probe’s name from your spec |
vantacloud.probe.target | The display name of the target a probe measures: your workload’s name for your probes |
vantacloud.probe.bound_ms | A probe’s latency bound in milliseconds, as a decimal string such as 300 |
vantacloud.db.database | A database’s name |
vantacloud.db.pool | The name of a connection pool: for a connection the sidecar brokers, the database’s name; for a process’s own pool, the pool of the server it connects to, such as accounts |
vantacloud.db.schema | On PostgreSQL, the schema a statement names, or the first schema of the connection’s search path, or none where neither names one; on Neo4j, which has no schemas, none |
db.operation.name | A statement’s command: on PostgreSQL, its command, such as SELECT, UPDATE or CREATE TABLE; on Neo4j, as Neo4j statements states; OTHER for a statement whose command is none of the engine’s |
gen_ai.request.model | The model a request names |
vantacloud.inference.endpoint | The provider endpoint that served a request, as <provider>/<model> |
vantacloud.inference.outcome | A model request’s outcome code, ok or the failure’s code |
vantacloud.inference.stop_reason | Why a reply ended: stop, length, tool-calls, content-filter or error |
gen_ai.token.type | input, output, reasoning or cached |
vantacloud.cost.kind | estimate or charge |
vantacloud.edge | An edge’s name: accounts, inference, platform, relay, vantadaemon or sidecar-gateway |
vantacloud.route.class | A kebab-case class of an edge’s routes, one per group of routes the edge’s design page distinguishes, such as protocol or account-api, and unmatched for a request matching none of the edge’s stated calls |
vantacloud.status.class | 1xx, 2xx, 3xx, 4xx or 5xx |
vantacloud.auth.outcome | admitted, refused or none, for a route taking no credential |
vantacloud.throttled | true for a request a throttle refused |
network.io.direction | receive or transmit |
Metrics the platform reports
Section titled “Metrics the platform reports”The sidecar and the platform report these for every workload, with no code of yours.
| Metric | Instrument | Unit | Labels |
|---|---|---|---|
vantacloud.capability.call.ack | Histogram | s | vantacloud.capability, vantacloud.operation, vantacloud.outcome.class |
vantacloud.capability.call.duration | Histogram | s | As above |
vantacloud.db.pool.wait | Histogram | s | vantacloud.db.pool, vantacloud.db.database, vantacloud.operation |
vantacloud.db.connection.held | Histogram | s | As above |
vantacloud.db.lock.waits | Counter | {wait} | vantacloud.db.pool, vantacloud.db.database |
vantacloud.db.statement.duration | Histogram | s | vantacloud.db.pool, vantacloud.db.database, vantacloud.operation, vantacloud.db.schema, db.operation.name, vantacloud.outcome.class |
vantacloud.db.lock.wait | Histogram | s | vantacloud.db.pool, vantacloud.db.database |
vantacloud.db.errors | Counter | {error} | vantacloud.db.pool, vantacloud.db.database, vantacloud.operation, error.kind |
container.cpu.usage | Gauge | {cpu} | none |
container.memory.rss | Gauge | By | none |
vantacloud.container.restarts | Counter | {restart} | none |
vantacloud.container.oom_kills | Counter | {kill} | none |
vantacloud.probe.up | Gauge | 1 | vantacloud.probe.name, vantacloud.probe.target, vantacloud.probe.bound_ms |
vantacloud.probe.duration | Histogram | s | As above |
vantacloud.edge.requests | Counter | {request} | vantacloud.edge, vantacloud.route.class, vantacloud.status.class, vantacloud.auth.outcome, vantacloud.throttled |
vantacloud.edge.sessions | Gauge | {session} | vantacloud.edge, vantacloud.route.class |
vantacloud.edge.bytes | Counter | By | vantacloud.edge, network.io.direction |
Every database series is reported under the service.name of the process
whose connection it measures. The pool wait, the time held, the statements
and the errors carry vantacloud.operation, the operation the connection was
taken for: db.connect for a connection the sidecar brokers, and the
process’s own operation for a process that holds its own connection pool
rather than connecting through the sidecar. The lock waits are sampled from
the engine every 15 seconds for the process’s database role, so they carry
the pool and the database alone.
Statement durations and lock waits are reported for both engines the sidecar
brokers, PostgreSQL and Neo4j, under the same labels, so one query over
vantacloud.db.statement.duration or vantacloud.db.lock.wait covers every
database of a workload.
Lock waits
Section titled “Lock waits”A sample reads the waits of the pool’s own role on its database, under that role’s own connection, so it needs no privilege beyond those the role’s grant gives:
| Engine | Source | A wait |
|---|---|---|
| PostgreSQL | pg_locks joined to pg_stat_activity, filtered to the role, current_user, and to the database | A backend holding a lock request not yet granted, identified by its process and when its wait began |
| Neo4j | SHOW TRANSACTIONS, filtered to the workload’s user, username, to the database, database, and to the waiting transactions, those whose status is Blocked | A transaction seen blocked, identified by its transactionId, from the first sample seeing it blocked to the first no longer seeing it, lasting by the growth of its waitTime |
Neo4j shows every user its own transactions, so the sample takes no SHOW TRANSACTION privilege, and the workload’s user is never given one, since it
would show the user other users’ transactions. A wait counts once on
vantacloud.db.lock.waits in the sample first seeing it, and is recorded on
vantacloud.db.lock.wait with the longest it was seen to last in the sample
first no longer seeing it, so a wait is measured to within one interval and
one shorter than an interval may go unseen.
An edge counts each HTTP request on vantacloud.edge.requests, and a
throttle’s refusals on its vantacloud.throttled label, which an edge
without a throttle omits. A long-lived connection that carries no requests,
a TURN allocation or a machine’s channel, is a session: the edge reports the
sessions open on vantacloud.edge.sessions and their bytes on
vantacloud.edge.bytes.
A call’s ack is the time from the call’s start to the first byte of its
answer, and its duration the time to the answer’s last byte. A probe’s
vantacloud.probe.up is 1 for a probe given the answer its kind expects
within its bound and 0 for every other, one point per probe sent.
Neo4j statements
Section titled “Neo4j statements”On a Neo4j database the sidecar reads the Bolt messages a connection
carries. Each RUN is one statement, timed from the RUN to the end of its
result: the SUCCESS closing the last PULL or DISCARD of its records, or
the FAILURE answering it. A statement pipelined behind another starts
once that one ends, so each is timed alone.
A RUN’s db.operation.name is the first clause of its Cypher, read past
any CYPHER options, EXPLAIN, PROFILE and USE clause, such as MATCH,
OPTIONAL MATCH, CREATE, MERGE, UNWIND, CALL or LOAD CSV, or an
administration command’s verb and object, such as CREATE INDEX or
SHOW TRANSACTIONS.
An explicit transaction’s BEGIN, COMMIT and ROLLBACK messages are each a
statement too, timed from the message to its answer and named for the
message, so a commit’s latency is measured like any other statement’s. Every
Neo4j statement carries vantacloud.db.schema none.
A statement’s vantacloud.outcome.class is ok for a SUCCESS, and for a
FAILURE follows its Neo4j status code’s classification: client-error for
Neo.ClientError, server-error for Neo.TransientError and
Neo.DatabaseError. A statement the client abandons before its result ends,
by a RESET or by closing the connection, is cancelled.
Metrics your code reports
Section titled “Metrics your code reports”Only your code knows these, and the package registers each.
| Metric | Instrument | Unit | Labels |
|---|---|---|---|
vantacloud.operation.ack | Histogram | s | vantacloud.operation, vantacloud.outcome.class |
vantacloud.operation.duration | Histogram | s | As above |
process.memory.usage | Gauge | By | none |
vantacloud.allocator.allocated | Gauge | By | none |
vantacloud.allocator.active | Gauge | By | none |
vantacloud.allocator.resident | Gauge | By | none |
vantacloud.allocator.metadata | Gauge | By | none |
vantacloud.allocator.fragmentation | Gauge | 1 | none |
An operation’s ack is the time from its submission to its acknowledgement: for an HTTP route, the request’s arrival to the first byte of its answer, and for an operation taken from a queue, its enqueueing, the time stored with the job, to the moment the queue records it taken. Its duration runs from the same start to the operation’s end: the answer’s last byte, or the queued work done. Report the ack for every operation, and the duration as well for every operation your callers wait on beyond the ack, such as a streamed answer or a job.
The allocator gauges are reported by a process whose allocator offers them.
fragmentation is 1 - allocated / resident, the share of the memory the
allocator holds that no live allocation uses. process.memory.usage is the
process’s resident set size.
Model requests
Section titled “Model requests”A workload that serves or proxies model requests reports this set.
| Metric | Instrument | Unit | Labels |
|---|---|---|---|
vantacloud.inference.requests | Counter | {request} | gen_ai.request.model, vantacloud.inference.endpoint, vantacloud.inference.outcome |
vantacloud.inference.stop_reasons | Counter | {reply} | gen_ai.request.model, vantacloud.inference.endpoint, vantacloud.inference.stop_reason |
vantacloud.inference.tokens | Counter | {token} | gen_ai.request.model, vantacloud.inference.endpoint, gen_ai.token.type |
vantacloud.inference.time_to_first_token | Histogram | s | gen_ai.request.model, vantacloud.inference.endpoint |
vantacloud.inference.output_rate | Histogram | {token}/s | As above |
vantacloud.inference.duration | Histogram | s | As above, and vantacloud.inference.outcome |
vantacloud.inference.retries | Counter | {attempt} | gen_ai.request.model, vantacloud.inference.endpoint |
vantacloud.inference.endpoint_moves | Counter | {move} | gen_ai.request.model, vantacloud.inference.endpoint, the endpoint moved from |
vantacloud.inference.cost | Counter | {USD} | gen_ai.request.model, vantacloud.inference.endpoint, vantacloud.cost.kind |
The output rate is the reply’s output tokens over the time from its first token to its last. An endpoint move counts a request moved off an endpoint to the next of its route, labelled by the endpoint it moved to.
Histogram boundaries
Section titled “Histogram boundaries”Every duration histogram has the explicit boundaries, in seconds, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.3, 0.5, 0.75, 1, 1.5, 2, 2.5, 5, 10, 30, 60 and 120. Every probe bound and every default latency bound falls on one of them, so a share of answers over a bound is read exactly.
The cardinality rule
Section titled “The cardinality rule”A metric’s labels are the ones the tables above name and no other. A value that identifies a person, an account, a user, a session, a request, a trace or an instance is never a metric label: an id, an email address, a name, an address or a token. Such a value goes on a log record or a span alone, where it is read one record at a time. A label’s values are a small, fixed set known when the code is written, such as the routes of a service, its queues or the models it calls.
The content rule
Section titled “The content rule”No record carries the content a person sent or was answered: no message, prompt, reply, document body, form field or file. A record names content by its id and its size alone.