> For the complete documentation index, see [llms.txt](https://engineering.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity.md).

# Data integrity

How UMH keeps data from getting lost, doubled or wrong on its way through.

Owner: Jeremy

A factory can only build on data the way it builds on [electricity](/company/why-we-exist.md) if the data is reliable. Take a machine that reports every finished part to the MES. Only after the MES approves the part can the machine move on to the next step. This is called process locking. [An example: process locking](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity.md#an-example-process-locking) further below explains it step by step.

* If a report gets lost, the machine never gets an answer. The process stays locked. Production at this station stops until someone steps in.
* If a report reaches the MES twice, the MES counts one part too many.
* If a report mixes the measurement of one part with the number of the next, a bad part moves on.

People who see such errors do not trust the data. They are right to block the rollout of UMH or of any other product.

In our experience, teams at the start of their digitization rarely know the patterns that prevent these errors. The errors then appear in many places at once. The whole application starts to look unreliable. Projects stall. The blame lands on the system integrator or the platform. The successful projects we know were run by teams that understood these patterns and found them hard. **Example:** some of these teams built their own [handshake](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#read-skew-across-several-tags) with the PLC, the controller of the machine. The PLC keeps the values of one part unchanged until the platform confirms that it has read them all. The measurement and the part number then always belong to the same part.

UMH builds these patterns into the platform. We decide, for example, how a bridge reads a PLC, how a data flow handles a repeated message and what happens to a message that fails. Users get most of these patterns without having to know them.

Reading or writing a protocol such as S7 or OPC UA is relatively easy. UMH combines the protocol with delivery guarantees for the whole chain of data flows, from the machine to the MES and back. A delivery guarantee says whether a message can get lost on its way or be processed twice.

UMH can therefore look more complicated than other products at times. Most of them do not handle these patterns at all. The complexity that remains is needed by the people who build the use cases. Without it, they would build an unreliable application. [Simplicity](/engineering/development-process/how-to-build/coding-standards/code-quality/simplicity.md) describes how we remove the rest.

An unreliable application is easy to start. It is extremely hard to roll out.

UMH moves data with [data flows](https://docs.umh.app/usage/data-flows):

* Data flows pass data to each other through the [UNS](https://docs.umh.app/usage/unified-namespace) (Unified Namespace), the message store in the middle of UMH.
* A data flow is built from [benthos-umh plugins](https://docs.umh.app/benthos-umh). A plugin is an input, a processor or an output, such as the OPC UA input or the UNS output.
* A [bridge](https://docs.umh.app/usage/data-flows/bridges) is a data flow that connects a machine or another system, such as the MES, with the UNS.

## Rules

* No data gets lost by accident. A plugin drops data only on purpose and counts every drop. [Error management](/engineering/development-process/how-to-build/coding-standards/architecture/error-management.md) says how a plugin handles an error.
* A message that arrives twice changes nothing the second time.
* A message combines only values that belong together, such as the part number and the measurement of one part. The bridge reads such values from the same scan cycle of the PLC, for example as one struct. [Read skew](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#read-skew-across-several-tags) describes the ways to do this and what each protocol guarantees.
* A value never looks better than it is. A value the source marks as bad, uncertain or stale keeps that mark in the UNS.
* Every data flow works with event time, the time when something happened. Because UMH handles the edge cases around it, such as messages that arrive out of order, twice or late, the user can safely think in event time. Where event time is really hard to keep, UMH can lift these strong constraints and tells the user, for example for a [projection](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#combining-tags-that-change-at-different-rates) that works at processing time.
* These rules hold for every message and for any [state](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md) a plugin keeps.
* Every plugin states its delivery guarantees in its benthos-umh documentation.
* If the bridge reads the PLC over a protocol that can lose a value, the MES can miss a part even when every later plugin keeps its guarantees. The weakest plugin therefore defines the delivery guarantees and the integrity of the whole chain of data flows. The user knows this guarantee, for example from the docs or the UI.
* The sub-pages add the rules for each topic: [delivery guarantees](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md), [state](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md), [reasoning about time](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/reasoning-about-time.md) and [retries and dead-letter queues](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/retries-and-dead-letter-queues.md).

## An example: process locking

**In process locking, a part moves to the next production step only after another tool, for example an** [**MES**](https://en.wikipedia.org/wiki/Manufacturing_execution_system) **(manufacturing execution system), has approved it.** We use it as the example because lost data here stops production. There are also many places where data can get lost. The process combines data from several messages, calls a system we cannot roll back and makes a machine wait for an answer.

1. A carrier with a part arrives at a station. The machine sends two values to UMH in an HTTP request: the part number and a measurement. The part number together with the station names this one request. It stays the same when the machine sends the values again.
2. A [bridge](https://docs.umh.app/usage/data-flows/bridges) receives the request with an [`http_server` input](https://docs.redpanda.com/connect/components/inputs/http_server/). It writes both values into the [UNS](https://docs.umh.app/usage/unified-namespace). It keeps the connection to the machine open. If an error occurs, it answers with an HTTP error status such as 503. The machine then sends the request again.
3. A [stream processor](https://docs.umh.app/usage/data-flows/stream-processor) reads the part number and the measurement. It combines them with other tags, for example the order that currently runs at this station (read from the UNS). Such a tag can come from a different data source and change at a different rate. [State](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#combining-tags-that-change-at-different-rates) explains how to combine them.
4. The [write flow](https://docs.umh.app/usage/data-flows/bridges) of the MES bridge sends the part number, the measurement and the order to the MES. The MES uses the part number and the station to recognise a request it has already seen. A write flow ends in one of three ways:
   * The MES confirms the part, or says that it already approved this part at this station. The write flow writes the approval into the UNS.
   * The MES answers with an error. The write flow writes the error into the [dead-letter queue](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/retries-and-dead-letter-queues.md) in the UNS.
   * The MES does not answer. The write flow tries again a limited number of times. Then it writes a timeout into the dead-letter queue.
5. The machine's bridge watches the UNS for the result of the write flow for this part and station. If the MES approved the part, the bridge returns the approval to the machine as a [synchronous response](https://docs.redpanda.com/connect/guides/sync_responses/). Only then does the part move on.
6. If the result is an error or a timeout, the bridge answers with an error. The machine then decides what happens to the part.
7. Dashboards read the part data and the approvals from the UNS, for example for OEE (overall equipment effectiveness).

In this example, the machine sends its data over HTTP. Many machines use S7 or OPC UA instead. There the bridge reads the part's data from the PLC. It must read values that belong together from the same [scan cycle](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#read-skew-across-several-tags).

In steps 5 and 6 it writes the approval or the error back to the PLC. Steps 3, 4 and 7 stay the same.

**Each step can fail in a way one of the sub-pages describes.** The bridge in step 2 must not confirm a request it could still lose: [delivery guarantees](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md). The stream processor in step 3 must keep its state through a restart. It takes the order from a projection at processing time: [state](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md) and [reasoning about time](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/reasoning-about-time.md). The MES call in step 4 and the wait in steps 5 and 6 need limits and a defined outcome: [retries and dead-letter queues](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/retries-and-dead-letter-queues.md).

**Process locking is a control task. It therefore runs as a** [**control loop**](/engineering/development-process/how-to-build/coding-standards/architecture/control-loops.md)**.** The loop checks in every cycle whether each part has its answer. It keeps the state of each request in the UNS under the part number and the station.

## Our approach

**Every part of UMH that moves data keeps these guarantees, from a single plugin up to several umh-cores.** benthos-umh runs every data flow. A data flow is built from plugins: inputs, processors and outputs. Data flows combine through the UNS.

**We check every plugin, every data flow and their combinations.** Two plugins can each be correct on their own. Their combination can still lose a message, for example when the input acknowledges a message before the output has sent it. [Designing Data-Intensive Applications](https://dataintensive.net/) (DDIA, 2nd edition) calls this the end-to-end argument (chapter 13, "The end-to-end argument").

**We can only keep these guarantees because we limit what users can configure.** [Opinionated simplicity](/product/product-standards/opinionated-simplicity.md) says which settings a user gets. [Fewer options](/engineering/development-process/how-to-build/coding-standards/code-quality/simplicity.md#fewer-options) explains why each option makes correctness harder.

## Documenting the guarantees of each plugin

**Every plugin has a section "Delivery guarantees" in its** [**benthos-umh documentation**](https://docs.umh.app/benthos-umh)**.** Redpanda Connect does the same, for example for its [`dedupe` processor](https://docs.redpanda.com/connect/components/processors/dedupe/#delivery-guarantees). The section answers one question per sub-page:

* **Delivery and acknowledgement:** which guarantee does the plugin give? When does it acknowledge a message? See [delivery guarantees](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md).
* **State:** does the plugin keep state? Where? Does it survive a restart? See [state](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md).
* **Time:** which time does it use? Does it check the assumptions for time-series data? See [reasoning about time](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/reasoning-about-time.md).
* **Errors:** how does it retry? Does it block or use the dead-letter queue after the last retry? See [retries and dead-letter queues](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/retries-and-dead-letter-queues.md).
* **Limits:** what do the guarantees depend on? What is the plugin not suited for?

For example, the JavaScript cache in the `nodered_js` processor would state:

* **Delivery:** at-least-once. A repeated message changes the cache again.
* **State:** in memory, one store per processor, lost on restart. Two messages can overwrite each other's update when the pipeline runs more than one thread.
* **Limits:** suits only values where a lost or repeated update does no harm. Do not count events with it.

We open a ticket for every limitation we document. A pull request that changes how a plugin behaves also changes its "Delivery guarantees" section.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.