> For the complete documentation index, see [llms.txt](https://engineering.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/reasoning-about-time.md). # Reasoning about time Which timestamps a message carries and what UMH assumes about them. Some rules about time hold for all data. For time-series data, UMH makes three more assumptions. Relational data does not get these assumptions. ## Event time versus processing time **Every message has at least two timestamps: when something happened ("event time") and when a plugin is processing it ("processing time").** A plugin can process a message long after it happened. When a data flow replays the messages of last week, their event time stays in last week and their processing time is today. Designing Data-Intensive Applications (DDIA) compares this with the Star Wars films. The episode number is like the event time. The date you watched the film is like the processing time. DDIA warns: > Confusing event time and processing time leads to bad data. Read: DDIA chapter 12, "Reasoning About Time"; [Flink, timely stream processing](https://nightlies.apache.org/flink/flink-docs-stable/docs/concepts/time/). **Every data flow works with event time, the time when something happened.** That is the time the user has in mind when looking at the data. A result based on processing time would change whenever a message arrives late or a data flow replays old messages. Because UMH handles the edge cases around event time, such as messages that arrive out of order, twice or late, the user can safely think in event time. Where event time is really hard to keep, it can be easier to lift these strong constraints. UMH tells the user whenever this happens, for example for a [projection](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/state.md#combining-tags-that-change-at-different-rates) that works at processing time. Event time is the time the source measured the value. For OPC UA, it is the source timestamp. A machine that sends no time of its own gets the time the bridge reads its value. Read: DDIA chapter 12, "Event time versus processing time". ## Ingestion time **A message also keeps the time the bridge wrote it into the UNS.** Kafka stores this time with every message. Flink calls it ingestion time. The difference to the event time shows how long a value took to reach the UNS. ## Timestamps for ordering events **A plugin decides which of two messages is newer by their event time, or for relational data by their** [**version**](#relational-data)**.** It recognises a repeated message by its [idempotency key](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/delivery-guarantees.md#idempotence). ## Invalid timestamps **A plugin refuses a timestamp that cannot be right.** Examples are a timestamp of 0 or a time in seconds where milliseconds are expected. The message goes to the [dead-letter queue](/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/retries-and-dead-letter-queues.md). ## Time-series data [**Time-series data**](https://docs.umh.app/usage/unified-namespace/payload-formats) **is one value with its timestamp, such as a temperature reading. For it, UMH makes three assumptions:** * A message states an absolute value such as 21.5 °C. It never states a difference such as +0.5 °C. Applying the same message twice gives the same result. * Within one topic, event time never goes backwards. Gaps are allowed. * Within one topic, each timestamp has one value. A second message with the same timestamp and the same value is a repeat. A second message with the same timestamp but a different value is an error. **Where the timestamps of time-series data come from depends on the protocol the bridge reads.** Most PLCs send no time of their own. The bridge stamps each value when it reads it. It writes the values of one topic into the UNS in the sequence it read them. With OPC UA, the device sets the source timestamp. A value that did not change keeps its timestamp. A value that the device sends again can carry an older time than the one before it. The plugins after the bridge then skip it as a repeat or as a late message. **Because of the assumptions, a plugin can process every message as soon as it arrives.** A newer message tells the plugin that nothing older is still on its way. Downsampling and counting work only because of these assumptions. The topic and the timestamp together form the idempotency key. If a bridge writes the same message twice, for example after a crash, both copies have the same topic and the same timestamp. A plugin that has already processed the first copy recognises the second one by this idempotency key and skips it. **Every plugin that processes time-series data checks the assumptions.** * It skips a repeat. * It skips a late message and counts it. A late message has an older timestamp than the last one. It stays in the UNS. * It sends a message with a conflicting value to the dead-letter queue. **A clock that NTP sets back makes values look late.** The bridge stamps the values after the correction with an earlier time than the values before it. Plugins filter these values out until the clock has passed the last timestamp again. Keep edge clocks in sync to keep this gap short. ## Relational data [**Relational data**](https://docs.umh.app/usage/data-modeling/payload-shapes) **is a record of several fields that belong together, such as a production order or a part.** None of the three assumptions apply to it. Records can arrive late. A system can also send a correction for an earlier time. * A business key identifies a record, for example the part number and the station. * A correction arrives as an update or a delete for that key. * A plugin keeps the version of each key. It refuses an update with an older version. * The version comes from the source if possible, such as a change counter in the ERP. A clock on a device can be wrong. * A delete keeps its version. Otherwise a late, older update brings the record back. Suppose the MES bridge loses its network for ten minutes. It then sends an older production order change after a newer one. Without the version check, the stream processor replaces the current production order with the older one. Read: DDIA chapter 9, "Timestamps for ordering events". ## Unreliable clocks **Timestamps are in UTC and count milliseconds since 1970.** Local time and daylight saving time never reach a timestamp. **Measure durations with a monotonic clock.** A monotonic clock only moves forward. A timeout measured with it stays correct when the time of day changes. **A deadline that must survive a restart is stored as an absolute time.** A monotonic clock starts again after a restart. **Never build a unique key from a device's clock alone.** Two devices with unsynchronised clocks can stamp different events with the same time. Clocks also drift. DDIA chapter 9 assumes a drift of up to 200 parts per million. That is 17 seconds a day for a clock that syncs once a day. Every edge host therefore keeps its clock in sync, for example with NTP. Read: DDIA chapter 9, "Monotonic Versus Time-of-Day Clocks" and "Clock Synchronization and Accuracy". --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://engineering.umh.app/engineering/development-process/how-to-build/coding-standards/architecture/data-integrity/reasoning-about-time.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.