How data flows

This page follows one value, the temperature of a mixer, from the machine to a report. It explains what happens at each stage and where the value is if something on the way is not reachable.

1. Read

The edge gateway assigned to the device reads it. How depends on the protocol:

• Polled protocols (Modbus, Siemens S7, EtherNet/IP and many others): the gateway asks the device for all its tags at the device's Read every interval.
• Devices that send by themselves publish to the gateway's MQTT broker or post to its HTTP endpoint. Each device has its own token to prove who it is.
• The simulator makes values on the gateway itself.

The gateway gives each value its time (when it was read) and its quality: Good when the read worked, Bad when it failed. A device that keeps failing is not polled at full speed: the gateway tries it again at growing intervals until it answers, so one unreachable machine does not slow the others down.

2. Shape

Before the value goes anywhere, the gateway lets your own logic change it:

• Functions on the gateway see each value first.
• Pipelines that shape data run next, in the order you set: convert a unit, scale a value, rename a tag, add context, drop a value or route it. What comes out of them is what every later stage receives.

Then the gateway compares the values with the previous ones and marks which have changed. Pipelines that react to data (raise an alarm, notify, write to a device) and the rules of the gateway work on this result. Pipelines, rules and functions explains the difference.

3. Publish and route

From the gateway, the value goes three ways at once:

• The Unified Namespace. The gateway publishes the value on its built-in MQTT broker, under the place of the device, for example v1/Meridian/Lyon_Foods_Plant/Processing/Dough/DOCS_TUTORIAL_MIXER_01/telemetry/Temperature. Any MQTT client with access can subscribe. See Unified Namespace.
• Outbound targets. Each target ticked on the device receives the value from the gateway. A target's Transport Strategy decides what it gets: every value (Pass Through), only values that changed (On Change), or only what pipelines send to it (Pipelines Only).
• The hub. The historian, called SYSTEM INTERNAL DB in the device's target list, receives the value. Every new device has it in its list; a device removed from it has no history. The gateway hands the value to the message server, which brings it to the hub.

4. Buffer

A remote gateway does not send values straight to the hub. It first stores them with its own message server, on the gateway's disk, and the hub copies them from there. While the link to the hub is down, or the hub is stopped, values keep collecting on the gateway. When the link is back, the hub receives what it missed, with the times the values were read, so the history has no gap.

• How long the gateway keeps values it could not deliver is set per gateway: Buffer Retention (hours) on the gateway, 48 hours by default, from 1 to 720. Values older than that are dropped, oldest first.
• The gateway needs disk for that many hours at its data rate.
• The SYSTEM gateway runs next to the hub, so it has no buffer setting.

Alarms and notifications raised on a remote gateway are buffered the same way: the gateway keeps each one until the hub confirms it has it, and sends them in the order they were raised.

5. Store and show

At the hub, the value:

• is written to the historian (the ClickHouse database), with the device, the tag, its unit and its time;
• updates the live views: the device page, the UNS Explorer, the Operations Center;
• becomes available to the Trend Workspace, Dashboards and Reports, which read the historian.

By default the historian keeps every value. A retention period can be set in the configuration; see Configuration.

The other direction: configuration

Configuration flows from the hub to the gateways along the same path. When you save a device, the hub sends it to the device's gateway; the gateway applies it and confirms. The verify step of the device setup wizard shows these stages as Saved, Sent to the gateway, Device connected and First values. A gateway that is offline receives the change when it connects again, and a gateway that starts receives its full configuration from the hub.

What's next

• Architecture describes the hub, the gateways and the message server.
• Data is missing helps when a value does not arrive where you expect it.