> For the complete documentation index, see [llms.txt](https://engineering.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md). # State What a plugin remembers between messages and how it keeps that correct. **State is what a** [**plugin**](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity.md) **remembers from earlier messages, such as the order that currently runs at a station.** It can sit in a cache, in local files, in the [UNS](https://docs.umh.app/usage/unified-namespace) or in a third-party API. State is where a plugin can break the rules most easily. Read: [Flink, stateful stream processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/stateful-stream-processing/). **Changes to state follow the four properties of a database transaction.** DDIA calls them ACID. * Atomicity: an operation either completes or is aborted. If it fails partway, everything it did is discarded. A read of several tags from a PLC is not atomic. [Read skew](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#read-skew-across-several-tags) explains why. * Consistency: the data meets the rules that must always hold for it. The application defines these rules. Our rules for it are not written yet. * Isolation: operations that run at the same time do not see each other's half-finished work. A read of several tags from a PLC does not have it either. * Durability: data that was accepted is not forgotten, even after a crash. In UMH, a plugin's state must survive a crash. A backup of the `/data` directory must contain it. Read: DDIA chapter 8, "The Meaning of ACID". ## Read skew across several tags **Values that must belong together come from the same scan cycle and do not change while UMH reads them.** A PLC runs its program in a loop. One pass through that loop is called a scan cycle and takes a few milliseconds. If UMH reads several tags one after another, a scan cycle can finish in between. The first values then come from an older pass than the last ones. A fast read makes this rare. The chance is still never zero. DDIA calls this read skew. There are several ways to prevent it. Each depends on the protocol and needs a change in the PLC. * Struct: the values sit in one struct in the PLC. UMH reads the struct as one tag. We recommend this way. * Handshake: the PLC writes the values and then sets a "ready" tag. It changes the values again only after UMH has read them and reset the tag. * OPC UA method: the PLC returns all values in one call. If the PLC supports none of these, the remaining risk depends on the protocol. Each protocol plugin states in its docs what it guarantees. * OPC UA gives each value a timestamp from its source. A reader can use it to check which values belong together, if the server sets it correctly. * Modbus values carry only the time at which UMH read them. Values from separate requests can come from different scan cycles. Read: DDIA chapter 8, "Snapshot Isolation and Repeatable Read". ## Combining tags that change at different rates **A projection only holds state. Only an explicit trigger starts an action in another system.** Tags can come from different sources and change at different rates. A data flow that combines them therefore has two separate jobs. A projection keeps the latest value of each tag, or a value computed from them, such as the current average. It updates whenever one tag changes. It can wait a set time, such as 10 seconds, to collect several changes into one update. This is called debouncing. The projection lives in the UNS. Other data flows read it there like a cache. A projection shows the current state of each tag at processing time. Using processing time and not event time means that in some cases it can miss data (e.g., if a message arrives late). ### Triggers A trigger is an event that says something has happened, such as "part has finished" or "order done". It can come from the same machine or from another system. When it arrives, the data flow takes the latest projection. It writes the whole payload to the other system, for example the MES. Whoever sends the trigger decides that the data is complete. UMH does not guess it from timing. Two examples: * Process locking: the arriving part data is the trigger. The order comes from the projection. * A record per order: a projection holds the current average of a measurement. When a separate system reports that the order is done, the data flow writes that average to another system. Read: DDIA chapter 12, "Stream–table join (stream enrichment)" and "Table–table join (materialized view maintenance)"; [Enterprise Integration Patterns, content enricher](https://www.enterpriseintegrationpatterns.com/patterns/messaging/DataEnricher.html). ### A projection is at processing time A projection is always at processing time. The data flow takes the projection as it is when it processes the trigger. The user needs to be aware of this, for example through the docs or the UI. Processing time and the trigger's event time differ in these cases: * The data flow lags behind, for example after a restart, a network outage or a backlog. Triggers from ten minutes ago get the current order. * A data flow replays old data. Every trigger gets the current projection. * The projection changes at almost the same moment as the trigger. The result depends on which message is processed first. * The sources have different delays. A bridge that polls the MES every minute keeps the projection up to a minute behind. In normal operation the difference is the end-to-end latency, from milliseconds to a few seconds. That is fine for values that change much more slowly, such as an order that runs for hours. If this difference matters for a data flow, it keeps a history of every change with its event time instead of a projection. For each trigger, it looks up the value that was valid at the trigger's event time. Before that, it waits until the history has reached the trigger's event time. This adds latency to every trigger. ## Rebuilding state after a failure **State a plugin needs survives a restart. Otherwise the plugin rebuilds it before it handles the next message.** In the example, umh-core restarts. The copy of the latest order in the stream processor's memory is empty. If it does not read the order from the UNS again, the next request gets no order. Read: DDIA chapter 12, "Rebuilding state after a failure". ## Preventing lost updates **State that several messages change at once is changed by one message at a time.** Two messages run through the same cache at the same time. Both read "5" and both write "6". One update is lost. Read: DDIA chapter 8, "Preventing Lost Updates". ## Time dependence of joins **A message records which version of each value it used.** A reader can then find a message that used an old value. In the example, the order at the station changes. The new order has not reached the stream processor yet. The next request gets the old order. This is the lag described in [A projection is at processing time](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#a-projection-is-at-processing-time). Read: DDIA chapter 12, "Time dependence of joins". --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.