Repository navigation
NMS-12669: IPC and Kafka docs - #9004
Conversation
The Minion docs described installation per broker but not the Sink and RPC patterns that carry Core, Minion, and Sentinel traffic. The new page follows the core/ipc sink and rpc APIs and the strategy selection in SinkStrategy and RpcStrategy.
The Twin pattern that replicates configuration from Core to Minions was only mentioned by name in the Kafka setup. The section follows the core/ipc/twin publisher and subscriber APIs, the keys registered by Trapd, telemetryd, and the IPFIX provider, and the twin-subscribe shell command.
The page pointed readers at the source distribution for the .proto files without naming them or their message types. It now lists the file paths, the key and message type of each topic, and how to generate consumer classes, following OpennmsKafkaProducer and KafkaPersister; it also corrects the event record key, which is not set.
The page had one sample expression per filter. It now lists tested sample expressions for events, alarms, and metrics, explains how alarm filtering treats cleared and deleted alarms, and notes that nodes follow the event and alarm filters; the expressions match the getters of Event, OnmsAlarm, and KafkaPersister.ResourceFilterContext under the producer's SimpleEvaluationContext.
The tuning page said Sentinel needs the instance ID but only showed Core and Minion, and did not explain why the Sentinel setup also sets group.id. It now names the Sentinel files and explains both properties, following SystemInfoUtils, OnmsKafkaConfigProvider, and the sink Kafka consumer blueprint.
| [[kafka-producer-schema]] | ||
| == Schema | ||
|
|
||
| The Kafka Producer encodes record values with the following Protocol Buffers (proto3) definitions in the {page-component-title} source tree: |
There was a problem hiding this comment.
It feels excessive to have this level of detail in the documentation as opposed to linking to the proto, but there's not really anything wrong with it, I suppose.
There was a problem hiding this comment.
Yeah... we haven't really made a lot of changes to it though so I think it's probably ok to leave it for now? If we do make changes then we can change this section to be more of an example reference and remove some of it.
There was a problem hiding this comment.
Trimmed in 8c30ec4 per Chandra's comments: the source paths, generated class details and the Python example are gone; the topic table stays.
| @@ -0,0 +1,122 @@ | |||
| [[minion-ipc]] | |||
There was a problem hiding this comment.
Overall, really glad to see this get added I think. Feels like it's been missing.
marshallmassengill
left a comment
marshallmassengill
left a comment
There was a problem hiding this comment.
I think this is ok. I would wait for another reviewer before merging though.
| The following Python example parses records from the `alarms` topic with the generated module and the `kafka-python` client: | ||
|
|
||
| [source, python] | ||
| ---- | ||
| from kafka import KafkaConsumer | ||
| import opennms_kafka_producer_pb2 as onms | ||
|
|
||
| consumer = KafkaConsumer("alarms", bootstrap_servers="my-kafka-ip-1:9092") | ||
| for record in consumer: | ||
| if record.value is None: | ||
| print(f"Alarm deleted: {record.key.decode()}") | ||
| continue | ||
| alarm = onms.Alarm() | ||
| alarm.ParseFromString(record.value) | ||
| print(alarm.reduction_key, onms.Severity.Name(alarm.severity)) | ||
| ---- | ||
|
|
||
| Use the `.proto` files from the same version as your {page-component-title} instance. |
There was a problem hiding this comment.
Not sure if we need this python script example. Not usual for OpenNMS docs.
There was a problem hiding this comment.
Removed in 8c30ec4, along with the protoc command that led into it.
|
|
||
| Neither file declares a `package`, so the message types use the root namespace. | ||
| The Java classes that {page-component-title} generates from them are in the `org.opennms.features.kafka.producer.model` package, in the outer classes `OpennmsModelProtos` and `CollectionSetProtos`. | ||
|
|
There was a problem hiding this comment.
All of this can be removed
There was a problem hiding this comment.
Removed in 8c30ec4. The section now names the two .proto files in one sentence and keeps the topic table.
| In the example above, the filter is configured such that only metrics associated with the given location are forwarded. | ||
| Available properties are nodeId, nodeLabel, foreignSource, foreignId, location, resourceTypeName, resourceName, resourceId, ifIndex and instance. | ||
|
|
||
| === Filter examples |
There was a problem hiding this comment.
Haven't verified these examples. It would be good to validate these examples before we publish them on docs.
There was a problem hiding this comment.
Validated: every sample expression was evaluated with the producer's own SpEL context (SimpleEvaluationContext.forReadOnlyDataBinding().withInstanceMethods()) against matching and non-matching sample events, alarms and metric resources, and each returned the documented result; T(...) and constructor calls are rejected as stated. That check also corrected two lines in 8c30ec4: the node topic is fed by topology vertices too, and the combined example no longer sets an empty metricFilter.
…ter notes Drop the source paths, namespace and generated Java class details and the protoc and Python consumer examples from the schema section; it now names the two .proto files and keeps the topic table. The node topic is also fed by topology vertices, not only by events and alarms, and the combined filter example no longer sets an empty metricFilter. Every sample filter expression was evaluated against sample events, alarms and metric resources with the producer's evaluation context.
marshallmassengill
left a comment
marshallmassengill
left a comment
There was a problem hiding this comment.
Updates look good to me.
Documents the Minion IPC and the Kafka producer, split out of #8984; content checked against the code on develop, one commit per ticket.
deployment/…/minion/ipc.adoc(new)core/ipc/sink/api,core/ipc/rpc/apideployment/…/minion/ipc.adoc,minion/message-broker/kafka.adoccore/ipc/twin/api,AbstractTwinPublisher.java.protodefinitionsoperation/…/kafka-producer/kafka-producer.adocfeatures/kafka/producer/src/main/proto/*.proto,OpennmsKafkaProducer.javaoperation/…/kafka-producer/configure-kafka.adocOpennmsKafkaProducer.java(SpEL filters)reference/…/tuning-kafka.adoc,system-properties.adocblueprint-ipc-server.xml,KafkaTopicProvider.javaThe Kafka event-record key is also corrected.
Not addressed
The docs build adds no warnings (develop's two existing ones remain).
External References