How pipelines work

A pipeline is a small program of connected steps that an edge gateway runs on the data it reads. You design it in the hub; the gateway runs it next to the machines. This page explains the ideas behind the pipeline editor.

Two kinds of pipeline

Every pipeline has a type. The goal you start from sets it, and the editor says which one it is.

A Transport Pipeline has exactly one start, Transport Input. An Automation pipeline has between one and eight sources.

Order of Transport Pipelines

When several Transport Pipelines run on the same gateway, a message passes through them in turn. They are sorted by Stage (Normalize, Enrich, Quality, Route, in that order) and then by Order within a stage. Overwrite allows a later pipeline to overwrite a field an earlier one wrote.

On error decides what happens to a message when a step fails:

• Pass Original: the message continues as it was before this pipeline.
• Drop: the message is dropped.
• Quarantine: the message is set aside for inspection.

Story and canvas

The same pipeline can be edited in two ways:

• Build shows it as a story in five parts: When (what starts it), For (gateways, sites, devices, tags), Steps, Then (what it does), Otherwise (what it does when the condition does not hold). Behind the story are the same connected steps the canvas shows.
• Canvas shows those connected steps (nodes) and lets you build what the story cannot say, such as several branches.

Changing values on the canvas is read back into the story. If you change the shape of the pipeline on the canvas, the story no longer matches it; the Build tab then offers Open canvas or Rebuild from story. A pipeline made on the canvas can be replaced with a story with Replace with a story. Every node is listed in Pipeline nodes.

Draft, revision, deployment

• The draft is what you edit. Save keeps it, but the gateways keep running what was published before.
• Validate checks the draft against each gateway in Runs on: the steps must form a valid graph (no loops, every step reachable from a source, settings complete), and the devices and tags it needs are looked up on that gateway.
• Publish Shadow or Publish Active makes a numbered revision, a copy that never changes, and sends it to every gateway in Runs on. Publishing an unchanged draft in the same mode makes no new revision.
• Each gateway then has a deployment of that revision. The Deployments tab shows, per gateway, whether it was applied, how many devices it could bind (Coverage), whether the gateway still reports (Runtime) and the last error.
• Rollback publishes an earlier revision again. Disable removes the pipeline from all gateways. Only a disabled pipeline can be archived, and only a pipeline that was never published can be deleted (Delete draft).

A gateway removed from Runs on is told to stop the pipeline at the next publish.

Shadow and Active

Publish in Shadow first and look at the Deployments and Trace tabs for a while; then publish Active. The pipeline list shows each pipeline's mode in the Published column.

Ways to check a draft before publishing

None of them sends, writes or opens anything.

External actions

Three kinds of step act outside Proxus: Call an API and Send to an API, Write to a device, and Send to a target (except Store as tag, which keeps the result in Proxus). In Active mode they run only on gateways where the configuration file allows it:

[Flows]
EffectsEnabled = false

With EffectsEnabled = false (the default), such steps are refused on that gateway and the pipeline reports "External actions disabled". Alarms, notifications and Store as tag are not affected. In Shadow, external actions never run. Set the key on each gateway that should do them; see Configuration keys.

Device writes stay within the minimum and maximum you set on the step, and can only target a tag that is not read-only and is a number or a true/false value.

Limits

A pipeline is checked against these limits when you validate it:

What's next

• Build a pipeline
• Pipeline nodes
• Pipelines, rules and functions