From d7ced57b1d0434338d842276979fb69da94cb6af Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 25 Sep 2020 06:46:43 -0400 Subject: [PATCH 01/16] Resolve semantic inconsistencies for non traditional messaging Fixes #977 --- .../trace/semantic_conventions/messaging.md | 58 +++++++++++-------- 1 file changed, 35 insertions(+), 23 deletions(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index b133f5929dc..65b49dbf7ed 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -27,16 +27,18 @@ Although messaging systems are not as standardized as, e.g., HTTP, it is assumed that the following definitions are applicable to most of them that have similar concepts at all (names borrowed mostly from JMS): -A *message* usually consists of headers (or properties, or meta information) and an optional body. It is sent by a single message *producer* to: - -* Physically: some message *broker* (which can be e.g., a single server, or a cluster, or a local process reached via IPC). The broker handles the actual routing, delivery, re-delivery, persistence, etc. In some messaging systems the broker may be identical or co-located with (some) message consumers. -* Logically: some particular message *destination*. +A *message* is an envelope around a potentially empty payload. +This envelope may offer the possibility to convey additional metadata, often under the key/value form. +Messages can be delivered to 0, 1, or multiple consumers depending on the dispatching semantic of the protocol. +Traditional messaging brokers, such as JMS, use the concept of topics when a message is dispatched to potentially multiple consumers and queues when a message is dispatched to a single consumer. +In a messaging system such as Apache Kafka, consumer groups are used. Each record, or message, is sent to a single consumer per consumer group. +Whether a specific message is processed as if it was sent to a topic or queue entirely depends on the consumer groups and their composition. ### Destinations -A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. -Two kinds of destinations are distinguished: *topic*s and *queue*s. -A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. +A destination is usually identified by some name unique within the messaging system instance, which might look like a URL or a simple one-word identifier. +Traditional messaging involves two kinds of destinations: *topic*s and *queue*s. +A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all consumers that have *subscribed* to the topic. A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at-least-once mode for messages with [idempotent][] processing). [idempotent]: https://en.wikipedia.org/wiki/Idempotence @@ -47,11 +49,10 @@ The consumption of a message can happen in multiple steps. First, the lower-level receiving of a message at a consumer, and then the logical processing of the message. Often, the waiting for a message is not particularly interesting and hidden away in a framework that only invokes some handler function to process a message once one is received (in the same way that the listening on a TCP port for an incoming HTTP message is not particularly interesting). -However, in a synchronous conversation, the wait time for a message is important. ### Conversations -In some messaging systems, a message can receive a reply message that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. +In some messaging systems, a message can receive a reply message, or possibly multiple, that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. The grouping usually happens through some sort of "In-Reply-To:" meta information or an explicit *conversation ID* (sometimes called *correlation ID*). Sometimes a conversation can span multiple message destinations (e.g. initiated via a topic, continued on a temporary one-to-one queue). @@ -74,6 +75,7 @@ The span name SHOULD be set to the message destination name and the operation be The destination name SHOULD only be used for the span name if it is known to be of low cardinality (cf. [general span name guidelines](../api.md#span)). This can be assumed if it is statically derived from application code or configuration. +Wherever possible, the preference is to use real destination names over logical or aliased names. If the destination name is dynamic, such as a [conversation ID](#conversations) or a value obtained from a `Reply-To` header, it SHOULD NOT be used for the span name. In these cases, an artificial destination name that best expresses the destination, or a generic, static fallback like `"(temporary)"` for [temporary destinations](#temporary-destinations) SHOULD be used instead. @@ -118,6 +120,7 @@ The following operations related to messages are defined for these semantic conv | `messaging.protocol` | string | The name of the transport protocol. | `AMQP`
`MQTT` | No | | `messaging.protocol_version` | string | The version of the transport protocol. | `0.9.1` | No | | `messaging.url` | string | Connection string. | `tibjmsnaming://localhost:7222`
`https://queue.amazonaws.com/80398EXAMPLE/MyQueue` | No | +| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | No | | `messaging.message_id` | string | A value used by the messaging system as an identifier for the message, represented as a string. | `452a7c7c7c7048c2f887f61572b18fc2` | No | | `messaging.conversation_id` | string | The [conversation ID](#conversations) identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | `MyConversationId` | No | | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | @@ -125,7 +128,7 @@ The following operations related to messages are defined for these semantic conv **[1]:** Required only if the message destination is either a `queue` or `topic`. -**Additional attribute requirements:** At least one of the following sets of attributes is required: +**Additional attribute recommendations:** At least one of the following sets of attributes is recommended: * [`net.peer.name`](span-general.md) * [`net.peer.ip`](span-general.md) @@ -140,6 +143,7 @@ The following operations related to messages are defined for these semantic conv Additionally `net.peer.port` from the [network attributes][] is recommended. Furthermore, it is strongly recommended to add the [`net.transport`][] attribute and follow its guidelines, especially for in-process queueing systems (like [Hangfire][], for example). +`messaging.service` refers to the logical name of the external broker or messaging system where a message was sent to, or received from. In an environment such as Kubernetes, it would be the Kubernetes Service Name. These attributes should be set to the broker to which the message is sent/from which it is received. [network attributes]: span-general.md#general-network-connection-attributes @@ -176,6 +180,17 @@ In RabbitMQ, the destination is defined by an _exchange_ and a _routing key_. `messaging.destination` MUST be set to the name of the exchange. This will be an empty string if the default exchange is used. The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_key`, unless it is empty. +#### Apache Kafka + +For Apache Kafka, the following additional attributes are defined: + +| Attribute name | Notes and examples | +| -------------- | ---------------------------------------------------------------------- | +| `messaging.kafka.message_key` | Differs from `messaging.message_id` in that it's not unique, and can be `null`. The type is a String representation of the type of the actual value. | +| `messaging.kafka.consumer_group` | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | +| `messaging.kafka.client_id` | Client Id for the Consumer or Producer that is handling the message. | +| `messaging.kafka.partition` | Partition the message is sent to. | + ## Examples ### Topic with multiple consumers @@ -197,17 +212,16 @@ Process CB: | Span CB1 | | Links | | | | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | -| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | -| `net.peer.port` | `1234` | `1234` | `1234` | | `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"T"` | `"T"` | `"T"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | +| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | | `messaging.operation` | | `"process"` | `"process"` | -| `messaging.message_id` | `"a1"` | `"a1"`| `"a1"` | +| `messaging.kafka.message_key` | `"a1"` | `"a1"` | `"a1"` | ### Batch receiving -Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). +Given is a process P, that sends two messages to a topic Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. @@ -226,17 +240,16 @@ Process C: | Span Recv1 | | Links | | | | Span Prod1 | Span Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | -| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | | `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | +| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | | `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | -| `messaging.message_id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | +| `messaging.kafka.message_key` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | ### Batch processing -Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them separately (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). +Given is a process P, that sends two messages to a topic Q on messaging system MS, and a process C, which receives both of them separately (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). Since each span can only have one parent, C3 should not choose a random parent out of C1 and C2, but rather rely on the implicitly selected parent as defined by the [tracing API spec](../api.md). Similarly, only one value can be set as `message_id`, so C3 cannot report both `a1` and `a2` and therefore attribute is left out. @@ -259,10 +272,9 @@ Process C: | Span Recv1 | Span Recv2 | | Links | | | | | Span Prod1 + Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | -| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | | `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | +| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | | `messaging.operation` | | | `"receive"` | `"receive"` | `"process"` | -| `messaging.message_id` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | +| `messaging.kafka.message_key` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | From c4b7094bf4b540f1edf48e1c9271b785ec0cacd2 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Tue, 29 Sep 2020 16:37:31 -0400 Subject: [PATCH 02/16] Fix table --- specification/trace/semantic_conventions/messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 65b49dbf7ed..08c9af698d9 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -120,7 +120,7 @@ The following operations related to messages are defined for these semantic conv | `messaging.protocol` | string | The name of the transport protocol. | `AMQP`
`MQTT` | No | | `messaging.protocol_version` | string | The version of the transport protocol. | `0.9.1` | No | | `messaging.url` | string | Connection string. | `tibjmsnaming://localhost:7222`
`https://queue.amazonaws.com/80398EXAMPLE/MyQueue` | No | -| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | No | +| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | `myKey` | No | | `messaging.message_id` | string | A value used by the messaging system as an identifier for the message, represented as a string. | `452a7c7c7c7048c2f887f61572b18fc2` | No | | `messaging.conversation_id` | string | The [conversation ID](#conversations) identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | `MyConversationId` | No | | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | From 697ce0a22332ef3b2c4cfe9822c0fe61091a9737 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Wed, 30 Sep 2020 14:42:20 -0400 Subject: [PATCH 03/16] - Add new attribute to messaging yaml - Revert changes to existing examples but change them to use RabbitMQ instead of Kafka - Add specific Kafka example with new attributes --- semantic_conventions/trace/messaging.yaml | 4 ++ .../trace/semantic_conventions/messaging.md | 65 +++++++++++++++---- 2 files changed, 55 insertions(+), 14 deletions(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index f3934b35512..35eac9dae9e 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -47,6 +47,10 @@ groups: type: string brief: 'Connection string.' examples: ['tibjmsnaming://localhost:7222', 'https://queue.amazonaws.com/80398EXAMPLE/MyQueue'] + - id: service + type: string + brief: 'Name of the external broker, or name of the service being interacted with. See note below for a definition.' + examples: ['kafkaService'] - id: message_id type: string brief: 'A value used by the messaging system as an identifier for the message, represented as a string.' diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 08c9af698d9..eba9a634398 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -120,7 +120,7 @@ The following operations related to messages are defined for these semantic conv | `messaging.protocol` | string | The name of the transport protocol. | `AMQP`
`MQTT` | No | | `messaging.protocol_version` | string | The version of the transport protocol. | `0.9.1` | No | | `messaging.url` | string | Connection string. | `tibjmsnaming://localhost:7222`
`https://queue.amazonaws.com/80398EXAMPLE/MyQueue` | No | -| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | `myKey` | No | +| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | `kafkaService` | No | | `messaging.message_id` | string | A value used by the messaging system as an identifier for the message, represented as a string. | `452a7c7c7c7048c2f887f61572b18fc2` | No | | `messaging.conversation_id` | string | The [conversation ID](#conversations) identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | `MyConversationId` | No | | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | @@ -212,16 +212,49 @@ Process CB: | Span CB1 | | Links | | | | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"T"` | `"T"` | `"T"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | -| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | +| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | `"process"` | `"process"` | -| `messaging.kafka.message_key` | `"a1"` | `"a1"` | `"a1"` | + +### Apache Kafka + +Given is a process P, that publishes a message to a topic T1 on Apache Kafka. +One process, CA, receives the message and publishes a new message to a topic T2 that is then received and processed by CB. + +``` +Process P: | Span Prod1 | +-- +Process CA: | Span Rcv1 | + | Span Proc1 | + | Span Prod2 | +-- +Process CB: | Span Rcv2 | +``` + +| Field or Attribute | Span Prod1 | Span Rcv1 | Span Proc1 | Span Prod2 | Span Rcv2 +|-|-|-|-|-|-| +| Span name | `"T1 send"` | `"T1 receive"` | `"T1 process"` | `"T2 send"` | `"T2 receive`" | +| Parent | | Span Prod1 | Span Prod1 | | Span Prod2 | +| Links | | | Span Rcv1 | Span Prod1 | | +| SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | `PRODUCER` | `CONSUMER` | +| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.destination` | `"T1"` | `"T1"` | `"T1"` | `"T2"` | `"T2"` | +| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | +| `messaging.service` | `"myKafka"` | `"myKafka"` | `"myKafka"` | `"myKafka"` | `"myKafka"` | +| `messaging.operation` | | `"receive"` | `"process"` | | `"receive"` | +| `messaging.kafka.message_key` | `"myKey"` | `"myKey"` | `"myKey"` | `"anotherKey"` | `"anotherKey"` | +| `messaging.kafka.consumer_group` | | `"my-group"` | `"my-group"` | | `"another-group"` | +| `messaging.kafka.client_id` | | `"5"` | `"5"` | `"5"` | `"8"` | +| `messaging.kafka.partition` | | `"1"` | `"1"` | | `"3"` | ### Batch receiving -Given is a process P, that sends two messages to a topic Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. @@ -240,16 +273,18 @@ Process C: | Span Recv1 | | Links | | | | Span Prod1 | Span Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | -| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | -| `messaging.kafka.message_key` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | +| `messaging.message_id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | ### Batch processing -Given is a process P, that sends two messages to a topic Q on messaging system MS, and a process C, which receives both of them separately (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them separately (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). Since each span can only have one parent, C3 should not choose a random parent out of C1 and C2, but rather rely on the implicitly selected parent as defined by the [tracing API spec](../api.md). Similarly, only one value can be set as `message_id`, so C3 cannot report both `a1` and `a2` and therefore attribute is left out. @@ -272,9 +307,11 @@ Process C: | Span Recv1 | Span Recv2 | | Links | | | | | Span Prod1 + Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | +| `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | -| `messaging.service` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | | `"receive"` | `"receive"` | `"process"` | -| `messaging.kafka.message_key` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | +| `messaging.message_id` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | From 362e0c1865176d60619d8458fadc5538364d50be Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Thu, 1 Oct 2020 11:24:00 -0400 Subject: [PATCH 04/16] Update ToC, remove `messaging.service`, add information about `peer.service` and `service.name` for Apache Kafka --- semantic_conventions/trace/messaging.yaml | 4 ---- .../trace/semantic_conventions/messaging.md | 16 ++++++++++------ 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index 35eac9dae9e..f3934b35512 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -47,10 +47,6 @@ groups: type: string brief: 'Connection string.' examples: ['tibjmsnaming://localhost:7222', 'https://queue.amazonaws.com/80398EXAMPLE/MyQueue'] - - id: service - type: string - brief: 'Name of the external broker, or name of the service being interacted with. See note below for a definition.' - examples: ['kafkaService'] - id: message_id type: string brief: 'A value used by the messaging system as an identifier for the message, represented as a string.' diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index eba9a634398..dd3b835c61f 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -16,8 +16,10 @@ - [Messaging attributes](#messaging-attributes) * [Attributes specific to certain messaging systems](#attributes-specific-to-certain-messaging-systems) + [RabbitMQ](#rabbitmq) + + [Apache Kafka](#apache-kafka) - [Examples](#examples) * [Topic with multiple consumers](#topic-with-multiple-consumers) + * [Apache Kafka Example](#apache-kafka-example) * [Batch receiving](#batch-receiving) * [Batch processing](#batch-processing) @@ -120,7 +122,6 @@ The following operations related to messages are defined for these semantic conv | `messaging.protocol` | string | The name of the transport protocol. | `AMQP`
`MQTT` | No | | `messaging.protocol_version` | string | The version of the transport protocol. | `0.9.1` | No | | `messaging.url` | string | Connection string. | `tibjmsnaming://localhost:7222`
`https://queue.amazonaws.com/80398EXAMPLE/MyQueue` | No | -| `messaging.service` | string | Name of the external broker, or name of the service being interacted with. See note below for a definition. | `kafkaService` | No | | `messaging.message_id` | string | A value used by the messaging system as an identifier for the message, represented as a string. | `452a7c7c7c7048c2f887f61572b18fc2` | No | | `messaging.conversation_id` | string | The [conversation ID](#conversations) identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | `MyConversationId` | No | | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | @@ -143,7 +144,6 @@ The following operations related to messages are defined for these semantic conv Additionally `net.peer.port` from the [network attributes][] is recommended. Furthermore, it is strongly recommended to add the [`net.transport`][] attribute and follow its guidelines, especially for in-process queueing systems (like [Hangfire][], for example). -`messaging.service` refers to the logical name of the external broker or messaging system where a message was sent to, or received from. In an environment such as Kubernetes, it would be the Kubernetes Service Name. These attributes should be set to the broker to which the message is sent/from which it is received. [network attributes]: span-general.md#general-network-connection-attributes @@ -191,6 +191,9 @@ For Apache Kafka, the following additional attributes are defined: | `messaging.kafka.client_id` | Client Id for the Consumer or Producer that is handling the message. | | `messaging.kafka.partition` | Partition the message is sent to. | +For Apache Kafka producers, `peer.service` should be set to the name of the broker or external service the message will be sent to. +Consumers should set `service.name` as the Resource receiving the message. The value should match `peer.service` set by a Producer. + ## Examples ### Topic with multiple consumers @@ -220,7 +223,7 @@ Process CB: | Span CB1 | | `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | `"process"` | `"process"` | -### Apache Kafka +### Apache Kafka Example Given is a process P, that publishes a message to a topic T1 on Apache Kafka. One process, CA, receives the message and publishes a new message to a topic T2 that is then received and processed by CB. @@ -238,14 +241,15 @@ Process CB: | Span Rcv2 | | Field or Attribute | Span Prod1 | Span Rcv1 | Span Proc1 | Span Prod2 | Span Rcv2 |-|-|-|-|-|-| | Span name | `"T1 send"` | `"T1 receive"` | `"T1 process"` | `"T2 send"` | `"T2 receive`" | -| Parent | | Span Prod1 | Span Prod1 | | Span Prod2 | -| Links | | | Span Rcv1 | Span Prod1 | | +| Parent | | Span Prod1 | Span Rcv1 | | Span Prod2 | +| Links | | | | Span Prod1 | | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | `PRODUCER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | +| `peer.service` | `"myKafka"` | | | `"myKafka"` | | +| `service.name` | | `"myKafka"` | `"myKafka"` | | `"myKafka"` | | `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"T1"` | `"T1"` | `"T1"` | `"T2"` | `"T2"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | -| `messaging.service` | `"myKafka"` | `"myKafka"` | `"myKafka"` | `"myKafka"` | `"myKafka"` | | `messaging.operation` | | `"receive"` | `"process"` | | `"receive"` | | `messaging.kafka.message_key` | `"myKey"` | `"myKey"` | `"myKey"` | `"anotherKey"` | `"anotherKey"` | | `messaging.kafka.consumer_group` | | `"my-group"` | `"my-group"` | | `"another-group"` | From f981ac0a3d9cbdc64bc8fe1df145cc5db0c99946 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Thu, 1 Oct 2020 14:33:10 -0400 Subject: [PATCH 05/16] Fix trailing space --- specification/trace/semantic_conventions/messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index dd3b835c61f..53d52170c40 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -226,7 +226,7 @@ Process CB: | Span CB1 | ### Apache Kafka Example Given is a process P, that publishes a message to a topic T1 on Apache Kafka. -One process, CA, receives the message and publishes a new message to a topic T2 that is then received and processed by CB. +One process, CA, receives the message and publishes a new message to a topic T2 that is then received and processed by CB. ``` Process P: | Span Prod1 | From a7272ce20673f6e3a81ece1f287b20e535e30b93 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 2 Oct 2020 11:30:08 -0400 Subject: [PATCH 06/16] - Update messaging.yaml to align with document - Minor update to wording of Apacha Kafka service and peer names for Producers and Consumers --- semantic_conventions/trace/messaging.yaml | 42 +++++++++++++++++-- .../trace/semantic_conventions/messaging.md | 7 +++- 2 files changed, 44 insertions(+), 5 deletions(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index f3934b35512..eeba00b81b6 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -68,9 +68,13 @@ groups: brief: 'The compressed size of the message payload in bytes.' examples: 2048 constraints: - - any_of: - - 'net.peer.name' - - 'net.peer.ip' + - ref: net.peer.name + required: + conditional: If available. + - ref: net.peer.ip + tag: connection-level + required: + conditional: If available. - include: network - id: messaging.producer @@ -114,3 +118,35 @@ groups: brief: > Semantic convention for servers that consume messages received from messaging systems and always send back replies directed to the producers of these messages. + + - id: messaging.kafka + prefix: messaging.kafka + extends: messaging + brief: > + Attributes for Apache Kafka + attributes: + - id: message_key + type: string + brief: > + Message keys in Kafka are used for grouping alike messages to ensure they're processed on the same partition. + They differ from `messaging.message_id` in that they're not unique. + If the key is `null`, the attribute MUST NOT be set. + note: > + If the key type is not string, it's string representation has to be supplied for the attribute. + examples: 'myKey' + - id: consumer_group + type: string + brief: > + Name of the Kafka Consumer Group that is handling the message. + Only applies to consumers, not producers. + examples: 'my-group' + - id: client_id + type: string + brief: > + Client Id for the Consumer or Producer that is handling the message. + examples: '5' + - id: partition + type: string + brief: > + Partition the message is sent to. + examples: '2' diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 53d52170c40..6debd1f250d 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -184,15 +184,18 @@ The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_ke For Apache Kafka, the following additional attributes are defined: + | Attribute name | Notes and examples | | -------------- | ---------------------------------------------------------------------- | | `messaging.kafka.message_key` | Differs from `messaging.message_id` in that it's not unique, and can be `null`. The type is a String representation of the type of the actual value. | | `messaging.kafka.consumer_group` | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | | `messaging.kafka.client_id` | Client Id for the Consumer or Producer that is handling the message. | | `messaging.kafka.partition` | Partition the message is sent to. | + -For Apache Kafka producers, `peer.service` should be set to the name of the broker or external service the message will be sent to. -Consumers should set `service.name` as the Resource receiving the message. The value should match `peer.service` set by a Producer. +For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or external service the message will be sent to. +Consumers SHOULD identify the [Resource](../../resource/semantic_conventions/README.md#service) that received the message. +The `service.name` of a Consumer's Resource SHOULD match the `peer.service` of the Producer. ## Examples From 930491181edf025b46467c9715b236bce841d3f0 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 2 Oct 2020 11:34:42 -0400 Subject: [PATCH 07/16] Fix "ref" positioning --- semantic_conventions/trace/messaging.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index eeba00b81b6..73f62702563 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -67,7 +67,6 @@ groups: type: number brief: 'The compressed size of the message payload in bytes.' examples: 2048 - constraints: - ref: net.peer.name required: conditional: If available. @@ -75,6 +74,7 @@ groups: tag: connection-level required: conditional: If available. + constraints: - include: network - id: messaging.producer From aa657fe1bca77bb0ccb1da125d208a7541213e21 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Mon, 5 Oct 2020 10:20:42 -0400 Subject: [PATCH 08/16] Updated from running `make table-generation` --- .../trace/semantic_conventions/messaging.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 6debd1f250d..e8d39749926 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -126,14 +126,11 @@ The following operations related to messages are defined for these semantic conv | `messaging.conversation_id` | string | The [conversation ID](#conversations) identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | `MyConversationId` | No | | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | | `messaging.message_payload_compressed_size_bytes` | number | The compressed size of the message payload in bytes. | `2048` | No | +| [`net.peer.ip`](span-general.md) | string | Remote address of the peer (dotted decimal for IPv4 or [RFC5952](https://tools.ietf.org/html/rfc5952) for IPv6) | `127.0.0.1` | Conditional
If available. | +| [`net.peer.name`](span-general.md) | string | Remote hostname or similar, see note below. | `example.com` | Conditional
If available. | **[1]:** Required only if the message destination is either a `queue` or `topic`. -**Additional attribute recommendations:** At least one of the following sets of attributes is recommended: - -* [`net.peer.name`](span-general.md) -* [`net.peer.ip`](span-general.md) - `messaging.destination_kind` MUST be one of the following: | Value | Description | @@ -185,12 +182,14 @@ The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_ke For Apache Kafka, the following additional attributes are defined: -| Attribute name | Notes and examples | -| -------------- | ---------------------------------------------------------------------- | -| `messaging.kafka.message_key` | Differs from `messaging.message_id` in that it's not unique, and can be `null`. The type is a String representation of the type of the actual value. | -| `messaging.kafka.consumer_group` | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | -| `messaging.kafka.client_id` | Client Id for the Consumer or Producer that is handling the message. | -| `messaging.kafka.partition` | Partition the message is sent to. | +| Attribute | Type | Description | Example | Required | +|---|---|---|---|---| +| `messaging.kafka.message_key` | string | Message keys in Kafka are used for grouping alike messages to ensure they're processed on the same partition. They differ from `messaging.message_id` in that they're not unique. If the key is `null`, the attribute MUST NOT be set. [1] | `myKey` | No | +| `messaging.kafka.consumer_group` | string | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | `my-group` | No | +| `messaging.kafka.client_id` | string | Client Id for the Consumer or Producer that is handling the message. | `5` | No | +| `messaging.kafka.partition` | string | Partition the message is sent to. | `2` | No | + +**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or external service the message will be sent to. From 2f442b3ff0e05aff48ce109210e09f26b191aa06 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Wed, 7 Oct 2020 14:49:40 -0400 Subject: [PATCH 09/16] Updates to service.name and peer.service meaning for producers and consumers. Also fixed some references to message.service that lingered --- .../trace/semantic_conventions/messaging.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index e8d39749926..04391496a15 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -192,9 +192,9 @@ For Apache Kafka, the following additional attributes are defined: **[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. -For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or external service the message will be sent to. -Consumers SHOULD identify the [Resource](../../resource/semantic_conventions/README.md#service) that received the message. -The `service.name` of a Consumer's Resource SHOULD match the `peer.service` of the Producer. +For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or service the message will be sent to. +The `service.name` of a Consumer's Resource SHOULD match the `peer.service` of the Producer, when the message is directly passed to another service. +If an intermediary broker is present, `service.name` and `peer.service` will not be the same. ## Examples @@ -222,8 +222,8 @@ Process CB: | Span CB1 | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"T"` | `"T"` | `"T"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | -| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | `"process"` | `"process"` | +| `messaging.message_id` | `"a1"` | `"a1"`| `"a1"` | ### Apache Kafka Example @@ -248,7 +248,7 @@ Process CB: | Span Rcv2 | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | `PRODUCER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | | `peer.service` | `"myKafka"` | | | `"myKafka"` | | -| `service.name` | | `"myKafka"` | `"myKafka"` | | `"myKafka"` | +| `service.name` | | `"myConsumer1"` | `"myConsumer1"` | | `"myConsumer2"` | | `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"T1"` | `"T1"` | `"T1"` | `"T2"` | `"T2"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | `"topic"` | @@ -284,7 +284,6 @@ Process C: | Span Recv1 | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | | `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | -| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | | `messaging.message_id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | @@ -318,6 +317,5 @@ Process C: | Span Recv1 | Span Recv2 | | `messaging.system` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | `"rabbitmq"` | | `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | | `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | -| `messaging.service` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | `"myRabbitMQ"` | | `messaging.operation` | | | `"receive"` | `"receive"` | `"process"` | | `messaging.message_id` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | From 497f8dc2daf672d16a367334fa3fccc915946d64 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 9 Oct 2020 11:54:33 -0400 Subject: [PATCH 10/16] Updates based on feedback --- semantic_conventions/trace/messaging.yaml | 9 ++++--- .../trace/semantic_conventions/messaging.md | 26 +++++++++++++------ 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index 73f62702563..307431d76ff 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -68,6 +68,8 @@ groups: brief: 'The compressed size of the message payload in bytes.' examples: 2048 - ref: net.peer.name + note: > + This should be the IP/hostname of the broker (or other network-level peer) this specific message is sent to/received from. required: conditional: If available. - ref: net.peer.ip @@ -133,6 +135,7 @@ groups: If the key is `null`, the attribute MUST NOT be set. note: > If the key type is not string, it's string representation has to be supplied for the attribute. + If the key can not be represented in string form, don't include its value. examples: 'myKey' - id: consumer_group type: string @@ -144,9 +147,9 @@ groups: type: string brief: > Client Id for the Consumer or Producer that is handling the message. - examples: '5' + examples: 'client-5' - id: partition - type: string + type: number brief: > Partition the message is sent to. - examples: '2' + examples: 2 diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 04391496a15..a2004300361 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -29,20 +29,30 @@ Although messaging systems are not as standardized as, e.g., HTTP, it is assumed that the following definitions are applicable to most of them that have similar concepts at all (names borrowed mostly from JMS): -A *message* is an envelope around a potentially empty payload. -This envelope may offer the possibility to convey additional metadata, often under the key/value form. +A *message* is an envelope with a potentially empty payload. +This envelope may offer the possibility to convey additional metadata, often in key/value form. + +A message is sent by a message *producer* to: + +* Physically: some message *broker* (which can be e.g., a single server, or a cluster, or a local process reached via IPC). The broker handles the actual delivery, re-delivery, persistence, etc. In some messaging systems the broker may be identical or co-located with (some) message consumers. +With Apache Kafka, the physical broker a message is written to depends on the number of partitions, and which broker is the *leader* of the partition the record is written to. +* Logically: some particular message *destination*. + Messages can be delivered to 0, 1, or multiple consumers depending on the dispatching semantic of the protocol. -Traditional messaging brokers, such as JMS, use the concept of topics when a message is dispatched to potentially multiple consumers and queues when a message is dispatched to a single consumer. -In a messaging system such as Apache Kafka, consumer groups are used. Each record, or message, is sent to a single consumer per consumer group. -Whether a specific message is processed as if it was sent to a topic or queue entirely depends on the consumer groups and their composition. ### Destinations A destination is usually identified by some name unique within the messaging system instance, which might look like a URL or a simple one-word identifier. -Traditional messaging involves two kinds of destinations: *topic*s and *queue*s. +Traditional messaging, such as JMS, involves two kinds of destinations: *topic*s and *queue*s. A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all consumers that have *subscribed* to the topic. A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at-least-once mode for messages with [idempotent][] processing). +In a messaging system such as Apache Kafka, all destinations are *topic*s. +Each record, or message, is sent to a single consumer per consumer group. +Consumer groups provide *deliver once* semantics for consumers of a topic within a group. +Whether a specific message is processed as if it was sent to a topic or queue entirely depends on the consumer groups and their composition. +For instance, there can be multiple consumer groups processing records from the same topic. + [idempotent]: https://en.wikipedia.org/wiki/Idempotence ### Message consumption @@ -54,7 +64,7 @@ Often, the waiting for a message is not particularly interesting and hidden away ### Conversations -In some messaging systems, a message can receive a reply message, or possibly multiple, that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. +In some messaging systems, a message can receive one or more reply messages that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. The grouping usually happens through some sort of "In-Reply-To:" meta information or an explicit *conversation ID* (sometimes called *correlation ID*). Sometimes a conversation can span multiple message destinations (e.g. initiated via a topic, continued on a temporary one-to-one queue). @@ -77,7 +87,7 @@ The span name SHOULD be set to the message destination name and the operation be The destination name SHOULD only be used for the span name if it is known to be of low cardinality (cf. [general span name guidelines](../api.md#span)). This can be assumed if it is statically derived from application code or configuration. -Wherever possible, the preference is to use real destination names over logical or aliased names. +Wherever possible, the real destination names after resolving logical or aliased names SHOULD be used. If the destination name is dynamic, such as a [conversation ID](#conversations) or a value obtained from a `Reply-To` header, it SHOULD NOT be used for the span name. In these cases, an artificial destination name that best expresses the destination, or a generic, static fallback like `"(temporary)"` for [temporary destinations](#temporary-destinations) SHOULD be used instead. From d1143e8750b4648018ecf38360c5a060594958a5 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 9 Oct 2020 11:57:28 -0400 Subject: [PATCH 11/16] Fix generated table --- specification/trace/semantic_conventions/messaging.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index a2004300361..1bd4e2aca09 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -137,10 +137,12 @@ The following operations related to messages are defined for these semantic conv | `messaging.message_payload_size_bytes` | number | The (uncompressed) size of the message payload in bytes. Also use this attribute if it is unknown whether the compressed or uncompressed payload size is reported. | `2738` | No | | `messaging.message_payload_compressed_size_bytes` | number | The compressed size of the message payload in bytes. | `2048` | No | | [`net.peer.ip`](span-general.md) | string | Remote address of the peer (dotted decimal for IPv4 or [RFC5952](https://tools.ietf.org/html/rfc5952) for IPv6) | `127.0.0.1` | Conditional
If available. | -| [`net.peer.name`](span-general.md) | string | Remote hostname or similar, see note below. | `example.com` | Conditional
If available. | +| [`net.peer.name`](span-general.md) | string | Remote hostname or similar, see note below. [2] | `example.com` | Conditional
If available. | **[1]:** Required only if the message destination is either a `queue` or `topic`. +**[2]:** This should be the IP/hostname of the broker (or other network-level peer) this specific message is sent to/received from. + `messaging.destination_kind` MUST be one of the following: | Value | Description | @@ -196,10 +198,10 @@ For Apache Kafka, the following additional attributes are defined: |---|---|---|---|---| | `messaging.kafka.message_key` | string | Message keys in Kafka are used for grouping alike messages to ensure they're processed on the same partition. They differ from `messaging.message_id` in that they're not unique. If the key is `null`, the attribute MUST NOT be set. [1] | `myKey` | No | | `messaging.kafka.consumer_group` | string | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | `my-group` | No | -| `messaging.kafka.client_id` | string | Client Id for the Consumer or Producer that is handling the message. | `5` | No | -| `messaging.kafka.partition` | string | Partition the message is sent to. | `2` | No | +| `messaging.kafka.client_id` | string | Client Id for the Consumer or Producer that is handling the message. | `client-5` | No | +| `messaging.kafka.partition` | number | Partition the message is sent to. | `2` | No | -**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. +**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. If the key can not be represented in string form, don't include its value. For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or service the message will be sent to. From ac664f69073a8d4ef8717dc66867c7b2ffb6bc1e Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 9 Oct 2020 12:35:48 -0400 Subject: [PATCH 12/16] Update semantic_conventions/trace/messaging.yaml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Christian Neumüller --- semantic_conventions/trace/messaging.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index 307431d76ff..ce3f830394a 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -135,7 +135,7 @@ groups: If the key is `null`, the attribute MUST NOT be set. note: > If the key type is not string, it's string representation has to be supplied for the attribute. - If the key can not be represented in string form, don't include its value. + If the key has no unambiguous, canonical string form, don't include its value. examples: 'myKey' - id: consumer_group type: string From a3d183bf371746aa9ae5e3229607fe628102a07b Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Fri, 9 Oct 2020 12:37:53 -0400 Subject: [PATCH 13/16] Fix generation --- specification/trace/semantic_conventions/messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index 1bd4e2aca09..a454e6d8c81 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -201,7 +201,7 @@ For Apache Kafka, the following additional attributes are defined: | `messaging.kafka.client_id` | string | Client Id for the Consumer or Producer that is handling the message. | `client-5` | No | | `messaging.kafka.partition` | number | Partition the message is sent to. | `2` | No | -**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. If the key can not be represented in string form, don't include its value. +**[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. If the key has no unambiguous, canonical string form, don't include its value. For Apache Kafka producers, [`peer.service`](./span-general.md#general-remote-service-attributes) SHOULD be set to the name of the broker or service the message will be sent to. From fc461bcde2c6c01e0c5cd6846f7a692ade849678 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Tue, 13 Oct 2020 11:08:20 -0400 Subject: [PATCH 14/16] Adjust description of topic, as there may be no subscribers when a message is added to a topic --- semantic_conventions/trace/messaging.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index ce3f830394a..9e5e07d4fc3 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -26,7 +26,7 @@ groups: brief: "A message sent to a queue" - id: topic value: "topic" - brief: "A message broadcasted to the subscribers of the topic" + brief: "A message sent to a topic" required: conditional: 'Required only if the message destination is either a `queue` or `topic`.' brief: 'The kind of message destination' From 1baeb111835d3368c6d8aa384b230c5a3ab40bf0 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Tue, 13 Oct 2020 11:13:42 -0400 Subject: [PATCH 15/16] Forgot to regenerate tables --- specification/trace/semantic_conventions/messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index a454e6d8c81..e03b4550ea3 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -148,7 +148,7 @@ The following operations related to messages are defined for these semantic conv | Value | Description | |---|---| | `queue` | A message sent to a queue | -| `topic` | A message broadcasted to the subscribers of the topic | +| `topic` | A message sent to a topic | Additionally `net.peer.port` from the [network attributes][] is recommended. From 46b1bdab3f439c5e764673b13e44c3cef2ac56f6 Mon Sep 17 00:00:00 2001 From: Ken Finnigan Date: Thu, 15 Oct 2020 11:25:17 -0400 Subject: [PATCH 16/16] Add "tombstone" to Kafka section as instrumentation currently uses it --- semantic_conventions/trace/messaging.yaml | 5 +++++ specification/trace/semantic_conventions/messaging.md | 1 + 2 files changed, 6 insertions(+) diff --git a/semantic_conventions/trace/messaging.yaml b/semantic_conventions/trace/messaging.yaml index 9e5e07d4fc3..9a5d42402b1 100644 --- a/semantic_conventions/trace/messaging.yaml +++ b/semantic_conventions/trace/messaging.yaml @@ -153,3 +153,8 @@ groups: brief: > Partition the message is sent to. examples: 2 + - id: tombstone + type: boolean + required: + conditional: 'If missing, it is assumed to be false.' + brief: 'A boolean that is true if the message is a tombstone.' diff --git a/specification/trace/semantic_conventions/messaging.md b/specification/trace/semantic_conventions/messaging.md index e03b4550ea3..8bc9b80d644 100644 --- a/specification/trace/semantic_conventions/messaging.md +++ b/specification/trace/semantic_conventions/messaging.md @@ -200,6 +200,7 @@ For Apache Kafka, the following additional attributes are defined: | `messaging.kafka.consumer_group` | string | Name of the Kafka Consumer Group that is handling the message. Only applies to consumers, not producers. | `my-group` | No | | `messaging.kafka.client_id` | string | Client Id for the Consumer or Producer that is handling the message. | `client-5` | No | | `messaging.kafka.partition` | number | Partition the message is sent to. | `2` | No | +| `messaging.kafka.tombstone` | boolean | A boolean that is true if the message is a tombstone. | | Conditional
If missing, it is assumed to be false. | **[1]:** If the key type is not string, it's string representation has to be supplied for the attribute. If the key has no unambiguous, canonical string form, don't include its value.