# How pipelines work

> The two kinds of pipeline, story and canvas, drafts and revisions, Shadow and Active, external actions and the limits a pipeline must stay within.

Source: https://proxus.io/docs/automate/pipeline-concepts

A pipeline is a small program of connected steps that an edge gateway runs on the data it reads. You design it in
the hub; the gateway runs it next to the machines. This page explains the ideas behind the
[pipeline editor](https://proxus.io/docs/automate/pipelines).

## Two kinds of pipeline

Every pipeline has a type. The goal you start from sets it, and the editor says which one it is.

| Type | On screen | What it does |
| --- | --- | --- |
| **Automation** | Reacts to stored data | Starts from an event: new tag values, a value change, no data for a while, a device or gateway status change, a log entry, gateway load figures, or a schedule. It adds alarms, notifications, calculated tags, deliveries or device writes. The stored data itself is not changed. |
| **Transport Pipeline** | Changes data before it is stored | Runs on every message a device sends, before it is stored or sent to outbound targets. It converts, renames, sets units and quality, adds context, drops or sets aside values. What it outputs is what gets stored. |

A Transport Pipeline has exactly one start, **Transport Input**. An Automation pipeline has between one and
eight sources.

### Order of Transport Pipelines

When several Transport Pipelines run on the same gateway, a message passes through them in turn. They are sorted
by **Stage** (**Normalize**, **Enrich**, **Quality**, **Route**, in that order) and then by **Order** within a stage.
**Overwrite** allows a later pipeline to overwrite a field an earlier one wrote.

**On error** decides what happens to a message when a step fails:

- **Pass Original**: the message continues as it was before this pipeline.
- **Drop**: the message is dropped.
- **Quarantine**: the message is set aside for inspection.

## Story and canvas

The same pipeline can be edited in two ways:

- **Build** shows it as a story in five parts: **When** (what starts it), **For** (gateways, sites, devices, tags),
  **Steps**, **Then** (what it does), **Otherwise** (what it does when the condition does not hold). Behind the
  story are the same connected steps the canvas shows.
- **Canvas** shows those connected steps (nodes) and lets you build what the story cannot say, such as several
  branches.

Changing values on the canvas is read back into the story. If you change the shape of the pipeline on the canvas,
the story no longer matches it; the **Build** tab then offers **Open canvas** or **Rebuild from story**. A pipeline
made on the canvas can be replaced with a story with **Replace with a story**. Every node is listed in
[Pipeline nodes](https://proxus.io/docs/reference/pipeline-nodes).

## Draft, revision, deployment

- The **draft** is what you edit. **Save** keeps it, but the gateways keep running what was published before.
- **Validate** checks the draft against each gateway in **Runs on**: the steps must form a valid graph (no loops,
  every step reachable from a source, settings complete), and the devices and tags it needs are looked up on that
  gateway.
- **Publish Shadow** or **Publish Active** makes a numbered **revision**, a copy that never changes, and sends it to
  every gateway in **Runs on**. Publishing an unchanged draft in the same mode makes no new revision.
- Each gateway then has a **deployment** of that revision. The **Deployments** tab shows, per gateway, whether it
  was applied, how many devices it could bind (**Coverage**), whether the gateway still reports (**Runtime**) and
  the last error.
- **Rollback** publishes an earlier revision again. **Disable** removes the pipeline from all gateways. Only a
  disabled pipeline can be archived, and only a pipeline that was never published can be deleted
  (**Delete draft**).

A gateway removed from **Runs on** is told to stop the pipeline at the next publish.

## Shadow and Active

| Mode | What runs | What happens |
| --- | --- | --- |
| **Shadow** | The whole pipeline, on real data, on the gateway | Nothing is changed. Alarms, messages, deliveries and device writes are counted as "would" actions only. |
| **Active** | The whole pipeline | Alarms are opened and closed, messages sent, values stored, and external actions done. |

Publish in Shadow first and look at the **Deployments** and **Trace** tabs for a while; then publish Active. The
pipeline list shows each pipeline's mode in the **Published** column.

## Ways to check a draft before publishing

| Tool | Where | What it uses |
| --- | --- | --- |
| **If it ran now** | Build and Review | The latest value of each device, every few seconds with **Live**. |
| **Rehearse on history** | Review & publish | The stored values of the last **24 hours** or **7 days**, up to 25,000 values per device. |
| **Sandbox** | Sandbox tab | Generated scenarios, or a sample you write or load from the gateway. |
| **Trace** | Trace tab | Live samples from a published pipeline that has a **Keep a sample for Trace** step. |

None of them sends, writes or opens anything.

## External actions

Three kinds of step act outside Proxus: **Call an API** and **Send to an API**, **Write to a device**, and **Send to a
target** (except **Store as tag**, which keeps the result in Proxus). In Active mode they run only on gateways where
the configuration file allows it:

```toml
[Flows]
EffectsEnabled = false
```

With `EffectsEnabled = false` (the default), such steps are refused on that gateway and the pipeline reports
"External actions disabled". Alarms, notifications and **Store as tag** are not affected. In Shadow, external
actions never run. Set the key on each gateway that should do them; see
[Configuration keys](https://proxus.io/docs/reference/configuration).

Device writes stay within the minimum and maximum you set on the step, and can only target a tag that is not
read-only and is a number or a true/false value.

## Limits

A pipeline is checked against these limits when you validate it:

| Limit | Value |
| --- | --- |
| Name | 1 to 120 characters |
| Nodes per pipeline | 1 to 128 |
| Connections per pipeline | 256 |
| Pipeline definition size | 256 KB |
| Sources in an Automation pipeline | 1 to 8 |
| Branches from one node | 16 |
| Tags in one message | 1024 |
| Values in a window (moving average and similar) | 1024 |
| Formula length | 1024 characters |
| Memory for steps that keep state (windows, dead bands, summaries) | 8 MB per pipeline, 64 MB per gateway |
| Pipelines per gateway | 256 |

## What's next

- [Build a pipeline](https://proxus.io/docs/automate/pipelines)
- [Pipeline nodes](https://proxus.io/docs/reference/pipeline-nodes)
- [Pipelines, rules and functions](https://proxus.io/docs/concepts/pipelines-and-rules)