# Kubernetes

> Deploy Proxus on Kubernetes with the shipped manifests: local, high-availability and data variants, what high availability covers, and how to apply them.

Source: https://proxus.io/docs/operate/kubernetes

At the end of this guide, Proxus runs on your Kubernetes cluster, either as a single copy for tests or as a
high-availability (HA) installation. Docker Compose remains the standard way to run Proxus; use Kubernetes when
your site runs its workloads there.

## Before you start

- A Kubernetes cluster and `kubectl` with access to it.
- The `Deployment/kubernetes` folder of the Proxus 2.0.0 release.
- For HA: at least three nodes, a storage class that offers `ReadWriteMany` volumes, the NGINX ingress controller,
  and the two database operators named below.

## The variants

| Folder | What it deploys | Namespace |
| --- | --- | --- |
| `local/` | One of each service, like the Docker Compose stack. Not HA. For tests and development. | `proxus` |
| `ha/` | The HA application: two hub replicas, a three-node message broker cluster, one pod per gateway, Secrets, disruption budgets and an ingress. | `proxus-ha` |
| `ha-data/` | HA databases for `ha/`: a three-instance PostgreSQL cluster (CloudNativePG) and a two-replica ClickHouse cluster with a three-node Keeper (Altinity operator). | `proxus-data`, `clickhouse` |
| `nats-trust/` | Shared by the others: creates the message broker's credentials and TLS certificate once and publishes them as a Secret. | (included) |

> **Warning:**
Always apply a folder with `kubectl apply -k <folder>`. The manifests take their namespace from the folder's
`kustomization.yaml`; `kubectl apply -f` on a single file puts the resources in the wrong namespace.

## What high availability means for Proxus

| Part | In `ha/` | What happens when a pod or node fails |
| --- | --- | --- |
| Hub (`proxus-ui`) | 2 replicas, active-active, behind an ingress with sticky sessions | The other replica keeps serving. Users of the failed replica are moved to the other one and may have to sign in again. |
| Message broker (`hub-server`) | 3-node cluster, every stream kept on all three (`NATS__Replicas=3`). The servers connect to each other by pod name over TLS (port `6222`). | No message is lost when one broker pod fails. |
| Gateway (`proxus-server-gateway-1`) | Exactly 1 pod, `Recreate` | Kubernetes restarts the pod. Reading pauses until it is back. |
| PostgreSQL | 3 instances, managed by CloudNativePG | The operator promotes a replica. |
| ClickHouse | 2 replicas with a 3-node Keeper | The other replica answers. |

A gateway is never run as two copies: two pods with the same gateway ID would read the same devices twice. Do not
set `replicas` above 1 on a gateway deployment. High availability for a gateway means a fast restart, not a second
running copy.

## Try it with `local/`

```bash
kubectl apply -k Deployment/kubernetes/local
kubectl -n proxus get pods,svc,pvc
kubectl -n proxus port-forward svc/proxus-ui 8080:8080
```

Then open `http://localhost:8080`. On the first start, `hub-server` and `proxus-server` stay in `ContainerCreating`
until the hub has created the broker credentials Secret `proxus-nats-trust`. This is expected. The hub serves
HTTPS on port `8443` as well.

The local variant has the database passwords in the manifests and keeps one copy of everything. Do not use it in
production. To remove it, including its volumes:

```bash
kubectl delete -k Deployment/kubernetes/local
```

## Deploy the HA variant

1. Install the two database operators, CloudNativePG and the Altinity ClickHouse operator, following their own
   instructions. The README in `Deployment/kubernetes` names the versions that were tested.
2. Set your database passwords. In `ha/secrets.yaml`, change `user`, `password` and `database` of the Secret
   `postgres-credentials`; they are all `proxus` in the shipped file. Use the same values for the PostgreSQL cluster
   in `ha-data/`, and change the ClickHouse credentials there too.
3. Check the storage. In `ha/storage.yaml`, the hub's shared volumes (`proxus-ui-config-rwx`,
   `proxus-ui-modules-rwx`, `nats-hub-config`) must be `ReadWriteMany` on a storage class that supports it before
   the hub replicas run on different nodes.
4. Set the host name. In `ha/ingress-nginx-sticky.yaml`, replace `proxus.local` with your DNS name and add a `tls`
   section with your certificate Secret. Keep the cookie affinity annotations: they keep each browser on one hub
   replica.
5. If you use managed or existing databases instead of `ha-data/`, point the two ExternalName services in
   `ha/external-data-services.yaml` at them.
Then apply the data layer, then the application:

```bash
kubectl apply -k Deployment/kubernetes/ha-data
kubectl apply -k Deployment/kubernetes/ha
```

Check that everything runs:

```bash
kubectl -n proxus-ha get pods,pdb,svc
```

Two `proxus-ui` pods, three `hub-server` pods and one `proxus-server-gateway-1` pod should be `Running`.

The broker credentials and the broker's TLS certificate (Secret `proxus-nats-trust`) are created by the hub on its
first start. Every connection to the broker uses TLS and checks the server against the platform CA. The hub renews
the certificate before it expires. Do not write or edit that Secret by hand.

The Certificate Store key is kept in `keys/STORE/certificate-store.key` on the hub's shared config disk
(`proxus-ui-config-rwx`). Back it up with that disk, or give it to the hub from your own Secret with the environment
variable `PROXUS_CERTIFICATE_STORE_KEY` (a base64 value of 32 bytes). See
[Certificates](https://proxus.io/docs/administer/certificates#protect-the-stores-private-keys).

> **Note:**
The manifests refer to images tagged `latest-arm64`. On nodes of another processor architecture, set the image tags
to the matching Proxus images, for example with an `images:` entry in your own kustomization that uses the
shipped folder as its base.

## Connect edge gateways from outside the cluster

Remote edge gateways connect to the message broker on port `7422`. In the shipped manifests, the `hub-server`
service is only reachable inside the cluster. Expose port `7422` to the networks of your gateways, for example with
a `LoadBalancer` service, and use that address when you
[install a gateway](https://proxus.io/docs/connect/edge-gateways/install-a-gateway). The broker's certificate covers every address an
install command is created for. If gateways reach the broker under another name, add it to `[NATS]` `TlsHostNames` or
`[Service]` `PublicHostNames`.

## Limits of the current manifests

- ClickHouse credentials in `ha-data/` are written in the manifest, not in a Secret.
- There is no NetworkPolicy. Add one if your cluster requires default-deny.
- The hub does not get the Docker socket on Kubernetes, so the **Restart** button of the Config Editor has no effect.
  Restart with `kubectl -n proxus-ha rollout restart deployment/proxus-ui`.
- After an upgrade to 2.0.0, restart the `hub-server` pods once after the hub has started
  (`kubectl -n proxus-ha rollout restart statefulset/hub-server`), then run every remote gateway's install command
  again. See [Upgrade](https://proxus.io/docs/operate/upgrade#upgrading-to-200).
- Back up the config volumes of the hub and of each gateway. See
  [Backup and restore](https://proxus.io/docs/operate/backup-and-restore).

## What's next

- [Upgrade](https://proxus.io/docs/operate/upgrade)
- [Monitoring](https://proxus.io/docs/operate/monitoring)
- [Security](https://proxus.io/docs/operate/security)