# Troubleshoot automation

> What pipeline, rule and function errors mean, why automation does not deploy or does not act, and what to check for each.

Source: https://proxus.io/docs/troubleshoot/automation

Pipelines, rules and functions run on edge gateways. Most problems are one of three kinds: the automation is not on
the gateway, it is on the gateway but sees no data, or it runs but its action fails. This page covers each.

## Health Checks findings

| Finding | Meaning | What to do |
| --- | --- | --- |
| `Pipeline is published but runs nowhere` | It is not assigned to a gateway, or no gateway was sent it. | Choose gateways under **Runs on** and publish again. |
| `Pipeline reads no device` | Its input matches no device on its gateways. | Check the source's **Which data** conditions and **Required tags**. |
| `Rule reads nothing we can resolve` | No tag or device matches the rule's condition. | Check that the tags and devices in the condition exist and are spelled as configured. |
| `The edge rejected this object` | The gateway refused the configuration. | Open the object, read the error, fix it and deploy again. |
| `The edge runs a different version` | What is configured is not what the gateway runs. | Deploy again, or discard the change. |
| `Everything runs on one gateway` | Many rules, functions and targets depend on one gateway. | A warning about risk, not an error. |

## Pipelines

### Deployment

The **Deployments** tab shows one row per gateway. **Deployment** is one of:

| State | Meaning |
| --- | --- |
| `Desired` | Published; the hub is about to send it. |
| `Pending` | Sent; waiting for the gateway, or for devices it needs (`Waiting for devices`). |
| `Active` | The gateway runs exactly this revision. |
| `Failed` | The gateway refused it. **Last error** says why. |
| `Drifted` | The gateway runs something other than this revision (`Running plan differs`). Publish again. |
| `Disabled` | Stopped on this gateway. |

The pipeline's health: `Healthy`, `Syncing`, `Offline` (the gateway stopped reporting), `Attention required` or
`No runtime evidence`. `Heartbeat stale` and `Never reported` mean the gateway has not sent the pipeline's status
recently or ever: check that the gateway is online.

### Errors

Errors appear as short texts on the canvas, in **Deployments** and in **Trace**:

| Text | Meaning | What to do |
| --- | --- | --- |
| `Pipeline could not be compiled`, `Validation failed` | The definition has an error. **Validate** lists it per node. | Fix the node named. |
| `No gateway selected` | **Runs on** is empty. | Choose a gateway. |
| `Steps form a loop`, `Too many steps`, `Too many connections`, `Definition too large` | A limit of [pipelines](https://proxus.io/docs/reference/pipeline-nodes#limits-of-a-pipeline) is exceeded. | Split the pipeline. |
| `Invalid formula`, `Formula too complex` | The **Calculate** formula is wrong or too long. | Check names and brackets. |
| `Formula failed` | At run time: division by zero, or a missing or invalid value. | Guard the input with a **Switch**, or connect **error** to an **Error Sink**. |
| `Bad input quality` | A node that calculates got a value whose quality is not Good. | Check the device's tag errors. |
| `Waiting for both inputs` | **Latest Join** has seen only one of its two values. | Normal after start; check that both devices send. |
| `Late data ignored` | A reading arrived older than the time window. | Check the device clock or the window. |
| `External actions disabled` | **Device Write**, **Call Api** or **Target Output** in Active mode, but the gateway has `[Flows] EffectsEnabled = false`. | Set it to `true` on that gateway and restart it. |
| `Action rate limit reached` | The same action came faster than **Minimum interval (ms)**. | Raise the interval, or add a **Debounce**. |
| `Action queue full` | More actions than the gateway can run. | Fewer actions, or raise `[Flows] EffectQueueCapacity`. |
| `Read-only address`, `Write address not found`, `Device write failed`, `Unsupported write data type` | **Device Write** failed. | The tag must exist, be writable and be numeric; the device must allow writes. |
| `API call failed`, `API timed out`, `API unreachable`, `API returned HTTP <code>` | **Call Api** failed. | Test the REST API target with **Check**. |
| `Sending to target failed`, `Target not running`, `Nothing to send` | **Target Output** failed. | **Check** on the target. |
| `Notification channel not found` | The **Notification** node names a channel that does not exist. | Pick an existing channel. |
| `Checkpoint storage unavailable`, `Checkpoint too large`, `Checkpoint storage full` | The gateway cannot save the pipeline's state. | Free disk space on the gateway; use smaller windows. |

### It runs but does nothing

- **It is published in Shadow mode.** Shadow records what would happen in **Trace** but does not act. Use
  **Publish Active**.
- **Its effects are off on the gateway** (`External actions disabled`, see above).
- **A Filter or Switch sends everything to false.** Look at **Trace**, or run a sample in **Sandbox**.
- **A Schedule Input did not fire while the gateway was off.** Missed runs are not made up.

## Rules

Rules run on the gateway of the device they watch. Their errors go to the gateway's log, not to a screen. Open
[System logs](https://proxus.io/docs/administer/system-logs) and look for:

| Log message | Meaning |
| --- | --- |
| `Error while evaluating rule '<name>': ...` or `Error evaluating rule '<name>': ...` | The condition failed at run time, for example on a value of the wrong type. |
| `Failed to compile criteria for rule '<name>' ...` | The condition cannot be built. Check the tag names and operators. |
| `<action> limit reached for rule '<name>'. Next ... allowed at ...` | The rule fired more often than its action limit allows; the extra actions are skipped. |

If a rule never fires, check that its device is connected, that the tag in the condition exists with that exact
name, and that the gateway is online. See [Rules](https://proxus.io/docs/automate/rules).

Notifications from rules that do not arrive: test the channel with **Test** (see
[Notification channels](https://proxus.io/docs/reference/notification-channels#test-a-channel)). A send that fails is tried again
for about an hour before it is given up.

## Functions

### Build errors

**Build** compiles the function and reports errors. Messages:

| Message | Meaning |
| --- | --- |
| `Compilation successful!` | Ready to deploy. |
| `Compile Error` with lines such as `CS0103 (line 12, column 9): The name 'x' does not exist in the current context` | A C# error, with its position. |
| `Blacklisted namespace used: System.IO. Reason: For security reasons, direct filesystem access is not permitted.` | The code uses a namespace functions may not use. See [Functions SDK](https://proxus.io/docs/develop/functions-sdk#what-the-compiler-offers). |
| `Function Code and Name cannot be empty.` | |
| `A function named <name> already exists. Please choose a different name.` | |
| `Unable to find the actor type in the compiled assembly.` | No class derives from `FunctionBase`. |
| `Actor ctor failed: ...` | The constructor threw an exception. Pass the three arguments to `FunctionBase` unchanged and do no work in the constructor. |

### Deploy errors

| Message | Meaning |
| --- | --- |
| `Function '<name>' deployed to <n> gateway(s): <names>` | Deployed. |
| `Deployment timed out (no gateway confirmation)`, `Gateway did not respond to deploy request` | The gateway is offline or busy. |
| `Failed to deploy to <n> gateway(s): <reasons>` | The gateway refused it. The reason follows. |
| `The function is currently being processed. Please wait.` | Another build or deploy of this function is running. |

### It deployed but does nothing, or stopped

- **It subscribes to nothing.** Add the subscription before `base.OnStarted()`; see
  [Functions SDK](https://proxus.io/docs/develop/functions-sdk#subscriptions).
- **It was undeployed after too many exceptions.** After `ExceptionThresholdPerFunction` exceptions (10 by default),
  the gateway undeploys the function and logs that the exception count reached the maximum. Fix the cause and
  deploy again.
- **Its log is silent.** `LogLevelPolicy` limits log lines per level; after the limit a warning says so and further
  lines are dropped until the window ends.
- **`Save skipped: missing or invalid DeviceId in MetaData.`** The function saved a reading without a device ID.

Read the function's lines under **Logs** on the function, or in [System logs](https://proxus.io/docs/administer/system-logs).

## Related

- [Pipeline nodes](https://proxus.io/docs/reference/pipeline-nodes)
- [Functions SDK](https://proxus.io/docs/develop/functions-sdk)
- [Troubleshoot gateways](https://proxus.io/docs/troubleshoot/gateways)
