> For the complete documentation index, see [llms.txt](https://engineering.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md). # Delivery guarantees Whether a message can get lost or be processed twice. **A delivery guarantee says what happens to a message when something fails along the way.** There are two levels of delivery. * At-most-once: a message is never repeated. It can get lost. This breaks the first rule on the [main page](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity.md). * At-least-once: a message never gets lost. It can arrive twice. **UMH delivers every message at least once and processes it exactly once.** A plugin is idempotent if processing a message twice gives the same result as processing it once: the same output and the same state of an external system (for example, a cache). Every plugin and every data flow is [idempotent](#idempotence). [Designing Data-Intensive Applications](https://dataintensive.net/) (DDIA) calls this exactly-once semantics. It cites Viktor Klang, who calls it "effectively-once" for "message processing with at-least-once + idempotent operations". Take a message that says "part 4711 has finished at station 3". Because the part has a unique part number, it does not matter how often this message arrives: the result stays the same. A message that only says "the current part has finished" is different. If "current" automatically switches to the next part, a second copy of the message marks the next part as finished too. Read: DDIA chapter 12, "Fault Tolerance" and "Idempotence"; [Redpanda Connect, delivery guarantees](https://docs.redpanda.com/connect/get-started/about/). ## Acknowledgement **An input acknowledges a message when it tells its source that the message is handled.** The source then does not send it again. A Kafka input, such as the one that reads from the UNS, acknowledges by committing its offset: the position in the topic before which every message counts as handled. A processor does not acknowledge anything itself. When a processor drops a message, the input acknowledges that message as handled. So a processor drops a message only on purpose, for example in a filter. A library that acknowledges by itself, such as an OPC UA subscription, counts as part of the plugin that uses it. ## Durability before acknowledgement **Every message is on disk before UMH acknowledges it to its sender. No plugin keeps a message only in memory.** A plugin or data flow only acknowledges a message once it was written to a persistent store, for example the UNS or a persistent cache. A persistent store keeps the data after a crash. In the process-locking example, the bridge must not confirm a request before the UNS has it. Otherwise a crash loses a request the machine believes was accepted. Read: DDIA chapter 12, "Acknowledgments and redelivery". ## Store and forward **When the next system is offline, its messages wait in the UNS on disk. This is called store and forward.** The retention of the UNS decides how long a system can be offline. If a system stays offline longer, the UNS deletes the oldest messages that the data flow has not handled yet. Because message brokers such as Kafka work this way, the user expects this loss. The user needs to be aware of the retention, for example through the docs or the UI. ## Consumer offsets Suppose the request for part 102 comes later in the topic than the request for part 101. The MES has approved part 102 but not part 101 yet. Because Kafka commits one offset for everything before it, committing after part 102 would also mark part 101 as handled. After a crash, the input would continue behind part 102. The request for part 101 would then never reach the MES. **A Kafka input therefore commits its offset only up to the first message that is not handled yet.** Read: DDIA chapter 12, "Acknowledgments and redelivery". ## Duplicate request In the example, the machine sends its values again after an error. Both attempts carry the same part number and station. This pair is the idempotency key: a value in the message that stays the same when the message is repeated. The MES uses it to recognise a part it has already approved. **Every plugin and every data flow passes the idempotency key on unchanged up to the MES.** If the bridge gave each attempt a new key, the MES would approve the part twice. Read: DDIA chapter 13, "Uniquely identifying requests" and "The end-to-end argument". ## Idempotence **Every plugin that causes an effect is idempotent.** Idempotent means it has the same effect whether it runs once or twice. At-least-once delivery repeats messages, for example when a data flow calls the MES and crashes before it commits its offset. After the restart it calls the MES again. **The idempotency key depends on the kind of data.** For a request or a record, it is a business key, such as the part number and the station. For time-series data, it is the topic and the timestamp. A timestamp that a processor adds does not work. It changes on every attempt. A plugin that reads from the UNS can also recognise a message it has already read by its Kafka offset. DDIA uses this as its example of idempotence. If a bridge writes the same message twice, for example after a crash, the UNS stores two copies with two different offsets. Both copies have the same idempotency key. A plugin therefore recognises the second copy as a repeat by its idempotency key. Read: DDIA chapter 12, "Idempotence". ## Settings that weaken a guarantee **A plugin keeps its guarantees with every setting it offers.** A setting can change how a plugin keeps them, such as the number of retries. No setting can switch them off. Every such setting would add variants that we and the user have to think through. [Opinionated simplicity](/product/product-standards/opinionated-simplicity.md) rules them out. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.