# How data flows

> The path of one value from the machine through the gateway, the namespace, the outbound targets and the historian, and what happens when a link is down.

Source: https://proxus.io/docs/concepts/data-flow

This page follows one value, the temperature of a mixer, from the machine to a report. It explains what
happens at each stage and where the value is if something on the way is not reachable.

## 1. Read

The edge gateway assigned to the device reads it. How depends on the protocol:

- **Polled protocols** (Modbus, Siemens S7, EtherNet/IP and many others): the gateway asks the device
  for all its tags at the device's **Read every** interval.
- **Devices that send by themselves** publish to the gateway's MQTT broker or post to its HTTP endpoint. Each
  device has its own token to prove who it is.
- **The simulator** makes values on the gateway itself.

The gateway gives each value its **time** (when it was read) and its **quality**: **Good** when the read
worked, **Bad** when it failed. A device that keeps failing is not polled at full speed: the gateway tries it
again at growing intervals until it answers, so one unreachable machine does not slow the others down.

## 2. Shape

Before the value goes anywhere, the gateway lets your own logic change it:

- **Functions** on the gateway see each value first.
- **Pipelines** that shape data run next, in the order you set: convert a unit, scale a value, rename a tag,
  add context, drop a value or route it. What comes out of them is what every later stage receives.

Then the gateway compares the values with the previous ones and marks which have changed. Pipelines that react
to data (raise an alarm, notify, write to a device) and the **rules** of the gateway work on this result.
[Pipelines, rules and functions](https://proxus.io/docs/concepts/pipelines-and-rules) explains the difference.

## 3. Publish and route

From the gateway, the value goes three ways at once:

- **The Unified Namespace.** The gateway publishes the value on its built-in MQTT broker, under the place of the
  device, for example `v1/Meridian/Lyon_Foods_Plant/Processing/Dough/DOCS_TUTORIAL_MIXER_01/telemetry/Temperature`.
  Any MQTT client with access can subscribe. See [Unified Namespace](https://proxus.io/docs/concepts/unified-namespace).
- **Outbound targets.** Each target ticked on the device receives the value from the gateway. A target's
  **Transport Strategy** decides what it gets: every value (**Pass Through**), only values that changed
  (**On Change**), or only what pipelines send to it (**Pipelines Only**).
- **The hub.** The historian, called **SYSTEM INTERNAL DB** in the device's target list, receives the value. Every
  new device has it in its list; a device removed from it has no history. The gateway hands the value to the
  message server, which brings it to the hub.

## 4. Buffer

A remote gateway does not send values straight to the hub. It first stores them with its own message server, on
the gateway's disk, and the hub copies them from there. While the link to the hub is down, or the hub is
stopped, values keep collecting on the gateway. When the link is back, the hub receives what it missed, with the
times the values were read, so the history has no gap.

- How long the gateway keeps values it could not deliver is set per gateway: **Buffer Retention (hours)** on
  the gateway, 48 hours by default, from 1 to 720. Values older than that are dropped, oldest first.
- The gateway needs disk for that many hours at its data rate.
- The **SYSTEM** gateway runs next to the hub, so it has no buffer setting.

Alarms and notifications raised on a remote gateway are buffered the same way: the gateway keeps each one until
the hub confirms it has it, and sends them in the order they were raised.

> **Note:**
The buffer protects the way to the hub. Outbound targets have their own durable queue on the gateway: data for a
target waits there while the target is unreachable, for the same **Buffer Retention (hours)**, and is written in
order when the target is back. The target's status on the [Outbound Targets](https://proxus.io/docs/connect/outbound-targets)
screen shows whether it is delivering and how many messages are waiting.

## 5. Store and show

At the hub, the value:

- is written to the **historian** (the ClickHouse database), with the device, the tag, its unit and its time;
- updates the **live views**: the device page, the UNS Explorer, the Operations Center;
- becomes available to the **Trend Workspace**, **Dashboards** and **Reports**, which read the historian.

By default the historian keeps every value. A retention period can be set in the configuration;
see [Configuration](https://proxus.io/docs/administer/configuration).

## The other direction: configuration

Configuration flows from the hub to the gateways along the same path. When you save a device, the hub sends it
to the device's gateway; the gateway applies it and confirms. The verify step of the device setup wizard shows
these stages as **Saved**, **Sent to** the gateway, **Device connected** and **First values**. A gateway that is
offline receives the change when it connects again, and a gateway that starts receives its full
configuration from the hub.

## What's next

- [Architecture](https://proxus.io/docs/concepts/architecture) describes the hub, the gateways and the message server.
- [Data is missing](https://proxus.io/docs/troubleshoot/data) helps when a value does not arrive where you expect it.
