Kubernetes

At the end of this guide, Proxus runs on your Kubernetes cluster, either as a single copy for tests or as a high-availability (HA) installation. Docker Compose remains the standard way to run Proxus; use Kubernetes when your site runs its workloads there.

Before you start

• A Kubernetes cluster and kubectl with access to it.
• The Deployment/kubernetes folder of the Proxus 2.0.0 release.
• For HA: at least three nodes, a storage class that offers ReadWriteMany volumes, the NGINX ingress controller, and the two database operators named below.

The variants

What high availability means for Proxus

A gateway is never run as two copies: two pods with the same gateway ID would read the same devices twice. Do not set replicas above 1 on a gateway deployment. High availability for a gateway means a fast restart, not a second running copy.

Try it with local/

kubectl apply -k Deployment/kubernetes/local
kubectl -n proxus get pods,svc,pvc
kubectl -n proxus port-forward svc/proxus-ui 8080:8080

Then open http://localhost:8080. On the first start, hub-server and proxus-server stay in ContainerCreating until the hub has created the broker credentials Secret proxus-nats-trust. This is expected. The hub serves HTTPS on port 8443 as well.

The local variant has the database passwords in the manifests and keeps one copy of everything. Do not use it in production. To remove it, including its volumes:

kubectl delete -k Deployment/kubernetes/local

Deploy the HA variant

1. Install the two database operators, CloudNativePG and the Altinity ClickHouse operator, following their own instructions. The README in Deployment/kubernetes names the versions that were tested.
2. Set your database passwords. In ha/secrets.yaml, change user, password and database of the Secret postgres-credentials; they are all proxus in the shipped file. Use the same values for the PostgreSQL cluster in ha-data/, and change the ClickHouse credentials there too.
3. Check the storage. In ha/storage.yaml, the hub's shared volumes (proxus-ui-config-rwx, proxus-ui-modules-rwx, nats-hub-config) must be ReadWriteMany on a storage class that supports it before the hub replicas run on different nodes.
4. Set the host name. In ha/ingress-nginx-sticky.yaml, replace proxus.local with your DNS name and add a tls section with your certificate Secret. Keep the cookie affinity annotations: they keep each browser on one hub replica.
5. If you use managed or existing databases instead of ha-data/, point the two ExternalName services in ha/external-data-services.yaml at them. Then apply the data layer, then the application:

kubectl apply -k Deployment/kubernetes/ha-data
kubectl apply -k Deployment/kubernetes/ha

Check that everything runs:

kubectl -n proxus-ha get pods,pdb,svc

Two proxus-ui pods, three hub-server pods and one proxus-server-gateway-1 pod should be Running.

The broker credentials and the broker's TLS certificate (Secret proxus-nats-trust) are created by the hub on its first start. Every connection to the broker uses TLS and checks the server against the platform CA. The hub renews the certificate before it expires. Do not write or edit that Secret by hand.

The Certificate Store key is kept in keys/STORE/certificate-store.key on the hub's shared config disk (proxus-ui-config-rwx). Back it up with that disk, or give it to the hub from your own Secret with the environment variable PROXUS_CERTIFICATE_STORE_KEY (a base64 value of 32 bytes). See Certificates.

Connect edge gateways from outside the cluster

Remote edge gateways connect to the message broker on port 7422. In the shipped manifests, the hub-server service is only reachable inside the cluster. Expose port 7422 to the networks of your gateways, for example with a LoadBalancer service, and use that address when you install a gateway. The broker's certificate covers every address an install command is created for. If gateways reach the broker under another name, add it to [NATS] TlsHostNames or [Service] PublicHostNames.

Limits of the current manifests

• ClickHouse credentials in ha-data/ are written in the manifest, not in a Secret.
• There is no NetworkPolicy. Add one if your cluster requires default-deny.
• The hub does not get the Docker socket on Kubernetes, so the Restart button of the Config Editor has no effect. Restart with kubectl -n proxus-ha rollout restart deployment/proxus-ui.
• After an upgrade to 2.0.0, restart the hub-server pods once after the hub has started (kubectl -n proxus-ha rollout restart statefulset/hub-server), then run every remote gateway's install command again. See Upgrade.
• Back up the config volumes of the hub and of each gateway. See Backup and restore.

What's next

• Upgrade
• Monitoring
• Security