# Getting Started

Get started with ServersCTL in minutes. Install the free Linux agent using the one-shot command from the control panel and begin monitoring, managing, and protecting your servers from a single unified platform.

# Introduction

#### A message from the developers.

ServersCTL started life as a solution to a problem many of us face: HAProxy is often the single point of failure in an otherwise resilient infrastructure.

The original goal was simple. Build a way to monitor HAProxy instances and automatically move traffic if a load balancer becomes unavailable without VRRP or a floating/failover IP address. Once the first working version was in place, it became clear that the underlying concept was far more powerful than we had anticipated.

By combining a lightweight agent with infrastructure-focused automation, we found ourselves solving many of the day-to-day challenges that infrastructure managers encounter. Monitoring, backups, disaster recovery, DNS failover, cPanel replication, service management, object deployments, DIFF, Versioning and operational visibility could all be brought together into a single platform.

[ServersCTL](https://serversctl.com) and [BalCTL](https://balctl.com) are not intended to replace the tools you already use. Instead, they aim to sit alongside them, providing practical automation and visibility where it matters most.

Today, we are releasing the Linux agent free of charge for anyone to use. Our hope is that it saves you time, reduces operational headaches, helps keep your services online, and perhaps one day saves your bacon when you need a backup or a failover plan.

We release this with no limits on the number of pools you can create or the number of servers you can manage. A Pro subscription gives you **access to all features for HAProxy and Generic Linux Servers running RHEL, AlmaLinux, Rocky Linux, Ubuntu, and Debian. This includes integrations for cPanel, MariaDB, and OpenLiteSpeed.**

If you find this product useful, support us by upgrading your account, contributing ideas or even reporting a bug. Everything helps.

We hope you find it useful.

#### What the agent is

The ServerCTL/BalCTL agent is a small <span class="font-semibold">Python 3.9+</span> process (stdlib only, no pip packages) that runs on each <span class="font-semibold">pool member Linux VM</span>. It:

1. <span class="font-semibold">Heartbeats</span> to the control plane over outbound HTTPS (proves liveness, reports host/HAProxy state).
2. <span class="font-semibold">Pulls jobs</span> from the control plane after each successful heartbeat (install, reload, backup, restore, firewall, etc.).
3. <span class="font-semibold">Optionally self-updates</span> from a published zip bundle.

It is deployed as a <span class="font-semibold">systemd</span> service: `<span class="md-inline-path-filename">balctl-heartbeat.service</span>`, with secrets in `<span class="md-inline-path-prefix">/etc/balctl/</span><span class="md-inline-path-filename">agent.env</span>`.

<span class="font-semibold">Current release:</span> check `AGENT_VERSION` in `<span class="md-inline-path-prefix">agents/</span><span class="md-inline-path-filename">balctl_heartbeat.py</span>` or `python3 balctl_heartbeat.py --version`.

# Requirements

### Server Stack

#### Operating system

- <span class="font-semibold">Supported:</span> Debian/Ubuntu and <span class="font-semibold">RHEL family</span> (AlmaLinux, Rocky, CentOS).
- <span class="font-semibold">Init:</span> <span class="font-semibold">systemd</span> (required — agent is designed as a systemd unit).
- <span class="font-semibold">Architecture:</span> Linux x86\_64 (typical VPS; agent uses standard distro package managers).

#### Runtime dependencies  


<div class="ui-scroll-area" id="bkmrk-component-required-n"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Component</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Required</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Notes</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Python 3.9+</span></div></td><td><div class="md-table-cell-content">Yes</div></td><td><div class="md-table-cell-content">Stdlib only; on EL8 minimal images may need `python39`</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">systemd</span></div></td><td><div class="md-table-cell-content">Yes</div></td><td><div class="md-table-cell-content">Service: `<span class="md-inline-path-filename">balctl-heartbeat.service</span>`</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">wget</span> or <span class="font-semibold">curl</span></div></td><td><div class="md-table-cell-content">Install-time</div></td><td><div class="md-table-cell-content">Download `<span class="md-inline-path-filename">agent.zip</span>`</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">unzip</span></div></td><td><div class="md-table-cell-content">Install-time</div></td><td><div class="md-table-cell-content">Extract bundle</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">sudo / root</span></div></td><td><div class="md-table-cell-content">For full feature set</div></td><td><div class="md-table-cell-content">Heartbeat itself can run unprivileged; most panel jobs need root</div></td></tr></tbody></table>

</div></div></div>#### Optional packages (installed by agent jobs when needed)

<div class="ui-scroll-area" id="bkmrk-package-when-haproxy"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Package</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">When</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">haproxy</span></div></td><td><div class="md-table-cell-content">Install HAProxy job or `BALCTL_PROVISION_HAPROXY=1`</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">socat</span></div></td><td><div class="md-table-cell-content">Admin socket drain/ready, runtime HAProxy commands</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">firewalld</span></div></td><td><div class="md-table-cell-content">RHEL-family firewall jobs (auto-installed on first “Refresh rules” if missing, agent v78+)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">ufw</span></div></td><td><div class="md-table-cell-content">Debian firewall backup jobs</div></td></tr></tbody></table>

</div></div></div>### Network requirements

#### Outbound HTTPS (required)

The VM must reach:

<div class="ui-scroll-area" id="bkmrk-destination-purpose-"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Destination</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Purpose</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">`https://serversctl.com`</span> (or your `BALCTL_API_BASE`)</div></td><td><div class="md-table-cell-content">Heartbeat, job claim/complete, backup upload/download</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">`https://download.serversctl.com/agent.zip`</span></div></td><td><div class="md-table-cell-content">Self-update (default)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">`https://api.ipify.org`</span> (optional)</div></td><td><div class="md-table-cell-content">Public IPv4 discovery when `BALCTL_PROBE_PUBLIC_IP=1`</div></td></tr></tbody></table>

</div></div></div></div></div></div>All agent API traffic must use <span class="font-semibold">HTTPS</span> — the agent refuses plaintext `BALCTL_API_BASE` / `BALCTL_UPDATE_URL` (v28+).

#### Inbound (not required)

Panel-driven operations use the <span class="font-semibold">outbound job queue</span>. No inbound SSH or agent port is required if the agent runs as <span class="font-semibold">root</span> for privileged jobs.

#### IP allowlisting (enrollment)

When you create a pool member, you must supply <span class="font-semibold">at least one allowed source IPv4</span>. This is the VM’s <span class="font-semibold">outbound/egress IP</span> as seen when it calls the control plane — <span class="font-semibold">not necessarily</span> its SSH IP or the IP traffic should hit.

The control plane validates <span class="font-semibold">`CF-Connecting-IP`</span> against the enrolled allowlist on every agent request. Mismatch → <span class="font-semibold">403</span>.

#### Enrollment requirements

Before the agent can heartbeat, create the member in the dashboard (<span class="font-semibold">Add pool member</span>):

<div class="ui-scroll-area" id="bkmrk-field-requirement-ho"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Field</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Requirement</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Hostname</span></div></td><td><div class="md-table-cell-content">Must match the JSON `hostname` the agent sends (case-insensitive). Override with `BALCTL_HOSTNAME` if OS hostname differs.</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Allowed source IPs</span></div></td><td><div class="md-table-cell-content">One or more IPv4 addresses (comma-separated). Must include egress IP to control plane.</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Enrollment secret</span></div></td><td><div class="md-table-cell-content"><span class="font-semibold">48 hex characters</span>, no hyphens. Shown <span class="font-semibold">once</span> in the modal. <span class="font-semibold">Not</span> the member UUID on the card.</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Member template</span></div></td><td><div class="md-table-cell-content">e.g. <span class="font-semibold">HAProxy balancer</span> — determines which panel commands are available.</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content"><span class="font-semibold">Linux family</span></div></td><td><div class="md-table-cell-content">Debian/Ubuntu vs RHEL — affects generated install one-liner.</div></td></tr></tbody></table>

</div></div></div></div></div></div>#### Authentication model

<div class="ui-scroll-area" id="bkmrk-header%3A%C2%A0authorizatio"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Header: `Authorization: Bearer <48-char-enrollment-secret>`
- Secret stored server-side as SHA-256 hash only.
- <span class="font-semibold">401</span> = wrong/unknown secret.
- <span class="font-semibold">403</span> = IP not allowlisted, or hostname mismatch, or missing `CF-Connecting-IP`.

  
</div></div></div>

# Register & Install the Agent

## Installation

When adding a pool, you will be asked whether the pool will be for HAProxy nodes.

**Only select the HAProxy Pool Template if the backend server(s) are currently running, or will be running, HAProxy.**

**For all other server types**—including cPanel, OpenLiteSpeed, MySql/MariaDB, and general-purpose Linux hosts—select **Generic Linux**.

Choosing the correct template ensures that the UI enables the appropriate features and management tools for that server type. If you accidentally select the wrong template, remove any members from the pool and recreate the pool on the correct template.

### Recommended: one-shot from the dashboard

<p class="callout info">You can access the UI using [https://serversctl.com/app](https://serversctl.com/app) or [https://balctl.com/app](https://balctl.com/app). Only the public websites are different.</p>

1. Register an account at [https://serversctl.com/app/](https://serversctl.com/app/)
2. Add a new pool. [https://serversctl.com/app/sites](https://serversctl.com/app/sites)
    - Servers can be pooled together. For example, cPanel Servers, LiteSpeed, MariaDB/MySQL all use the **Generic Linux template,** or HAProxy servers use the dedicated **HAProxy template**.
3. Select **Add Member.** The modal generates a paste-ready command that: 
    - Ensures `unzip` + `python3` (via apt or dnf/yum).
    - Downloads `<a href="https://download.serversctl.com/agent.zip">https://download.serversctl.com/agent.zip</a>`.
    - Runs `balctl-agent.sh --enrol --key … --hostname … --api-base …`.
    - Writes `<span class="md-inline-path-prefix">/etc/balctl/</span><span class="md-inline-path-filename">agent.env</span>`, runs `--update`, enables systemd.
4. The ServersCTL UI will now start to report agent information. 
    - See troubleshooting if you have problems.

<p class="callout success">To add further members to a pool. Keep using the Add Member button.</p>

### Files after install

<div class="ui-scroll-area" id="bkmrk-path-role-%2Fusr%2Flocal"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Path</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Role</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content">`<span class="md-inline-path-prefix">/usr/local/bin/</span><span class="md-inline-path-filename">balctl_heartbeat.py</span>`</div></td><td><div class="md-table-cell-content">Agent binary</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`<span class="md-inline-path-prefix">/etc/systemd/system/</span><span class="md-inline-path-filename">balctl-heartbeat.service</span>`</div></td><td><div class="md-table-cell-content">systemd unit</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`<span class="md-inline-path-prefix">/etc/balctl/</span><span class="md-inline-path-filename">agent.env</span>`</div></td><td><div class="md-table-cell-content">Secrets + config (`chmod 600`)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`<span class="md-inline-path-prefix">/var/lib/balctl/</span>`</div></td><td><div class="md-table-cell-content">State stamps (e.g. `<span class="md-inline-path-filename">.haproxy-provisioned</span>`)</div></td></tr></tbody></table>

</div></div></div>### Bundle contents (`<span class="md-inline-path-filename">agent.zip</span>`)

Flat zip: `<span class="md-inline-path-filename">balctl_heartbeat.py</span>`, `<span class="md-inline-path-filename">balctl-agent.sh</span>`, `<span class="md-inline-path-filename">balctl-heartbeat.env.example</span>`, `<span class="md-inline-path-filename">balctl-heartbeat.service</span>`, `<span class="md-inline-path-filename">README.md</span>`, `LICENSE`, `<span class="md-inline-path-filename">INSTALL_VM.txt</span>`.

---

## Configuration (`<span class="md-inline-path-prefix">/etc/balctl/</span><span class="md-inline-path-filename">agent.env</span>`)

If your configured hostname is different from the hostname sent to ServersCTL, use `BALCTL_HOSTNAME`

<div class="ui-scroll-area" id="bkmrk-variable-required-de"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Variable</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Required</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Default</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm">Purpose</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_ENROLLMENT_SECRET`</div></td><td><div class="md-table-cell-content"><span class="font-semibold">Yes</span> (heartbeat)</div></td><td><div class="md-table-cell-content">—</div></td><td><div class="md-table-cell-content">48 hex chars from modal</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_API_BASE`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">`https://serversctl.com`</div></td><td><div class="md-table-cell-content">Control plane origin</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_INTERVAL_SEC`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">`1`</div></td><td><div class="md-table-cell-content">Bootstrap interval only; control plane returns authoritative `heartbeatIntervalSec` (1–60s)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_HOSTNAME`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">OS hostname/FQDN</div></td><td><div class="md-table-cell-content">Override reported hostname</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_DECLARE_IP`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">—</div></td><td><div class="md-table-cell-content">Fixed IPv4 in heartbeat JSON (for DNS failover)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_PROBE_PUBLIC_IP`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">off</div></td><td><div class="md-table-cell-content">Discover public IPv4 via HTTPS each heartbeat</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_PUBLIC_IP_URL`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">`https://api.ipify.org`</div></td><td><div class="md-table-cell-content">Probe URL</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_UPDATE_URL`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">`https://download.serversctl.com/agent.zip`</div></td><td><div class="md-table-cell-content">Self-update zip</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_PROVISION_HAPROXY`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">off</div></td><td><div class="md-table-cell-content">One-shot HAProxy install on first start (root)</div></td></tr><tr class="border-border border-b"><td><div class="md-table-cell-content">`BALCTL_ALLOW_AGENT_DOWNGRADE`</div></td><td><div class="md-table-cell-content">No</div></td><td><div class="md-table-cell-content">off</div></td><td><div class="md-table-cell-content">Allow installing older agent from zip (not recommended)</div></td></tr></tbody></table>

</div></div></div>Systemd loads this via `EnvironmentFile=/etc/balctl/agent.env`. Manual `sudo python3 …` runs merge missing vars from the same file.

# Part I - HAProxy Server Pools

# Overview & concepts

### What is an HAProxy pool?

An <span class="font-semibold" data-streamdown="strong">HAProxy pool</span> is a ServerCTL deployment preset for the <span class="font-semibold" data-streamdown="strong">edge tier</span>: public DNS, one or more enrolled Linux VMs running HAProxy, and optional automatic promotion when the active host fails.

ServerCTL is the <span class="font-semibold" data-streamdown="strong">control plane</span>. It does not terminate customer traffic itself. It:

- Enrols VMs via the <span class="font-semibold" data-streamdown="strong">ServersCTL agent</span>
- Publishes a managed <span class="font-semibold" data-streamdown="strong">A record</span> through Cloudflare or cPanel/WHM
- Tracks <span class="font-semibold" data-streamdown="strong">heartbeats</span> (~1s check-ins) and <span class="font-semibold" data-streamdown="strong">systemd HAProxy</span> health
- Queues <span class="font-semibold" data-streamdown="strong">remote jobs</span> (install, reload, backup, drain) that run on the next heartbeat

<span class="font-semibold" data-streamdown="strong">Status:</span> HAProxy pools are well tested and in public beta.

### Core terminology

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-term-meaning-pool-on"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Term</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Meaning</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Pool</span></div></td><td><div class="md-table-cell-content">One site/deployment in the dashboard</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Member</span></div></td><td><div class="md-table-cell-content">One enrolled VM (node) with hostname, allowed egress IP, and enrollment secret</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Active member</span></div></td><td><div class="md-table-cell-content">The host whose IPv4 the managed DNS A record points at</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Standby</span></div></td><td><div class="md-table-cell-content">Enrolled member not currently receiving DNS traffic</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Failover hostname</span></div></td><td><div class="md-table-cell-content">Public FQDN clients use (e.g. `<span class="md-inline-path-filename">lb.example.com</span>`)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Member template</span></div></td><td><div class="md-table-cell-content">Role at enroll time — for HAProxy pools use <span class="font-semibold" data-streamdown="strong">HAProxy balancer</span></div></td></tr></tbody></table>

</div></div></div>### Architecture (high level)

<div class="composer-message-codeblock" id="bkmrk-clients-%E2%86%92-dns-%28cloud"><div class="ui-code-block"><div class="ui-code-block-content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-default-code ui-code-block-default-code"><div class="ui-default-code__content"><div class="ui-default-code__line"><div class="ui-default-code__line-content">Clients → DNS (Cloudflare / cPanel) → A record → Active HAProxy VM</div></div><div class="ui-default-code__line"><div class="ui-default-code__line-content">↑</div></div><div class="ui-default-code__line"><div class="ui-default-code__line-content">ServerCTL Worker updates DNS</div></div><div class="ui-default-code__line"><div class="ui-default-code__line-content">↑</div></div><div class="ui-default-code__line"><div class="ui-default-code__line-content">Standby HAProxy VMs ← agent heartbeats + jobs</div></div></div></div></div></div></div></div></div></div><span class="font-semibold" data-streamdown="strong">Health for failover:</span> A member is unhealthy when:

1. No heartbeat within the <span class="font-semibold" data-streamdown="strong">failover delay</span> window (10–120 seconds), or
2. HAProxy is monitored, and <span class="font-semibold" data-streamdown="strong">systemd reports HAProxy inactive</span>

<p class="callout warning"><span class="font-semibold" data-streamdown="strong">Important:</span> Clients must use the <span class="font-semibold" data-streamdown="strong">failover hostname</span>, not a member’s raw IP. ServerCTL moves the A record; your apps keep the same DNS name.</p>

### What HAProxy pools include vs other presets

HAProxy pools uniquely enable:

- Remote HAProxy jobs (install, reload, backup)
- HAProxy systemd probe on member cards
- <span class="font-semibold" data-streamdown="strong">Disaster Recovery</span> tab (cross-member restore, 2+ members)
- Traffic-flow diagram on Overview
- HAProxy <span class="font-semibold" data-streamdown="strong">Status</span> tab

Generic Linux pools hide HAProxy-specific jobs unless the agent detects HAProxy on the host.

# Create your first pool & Enroll your first member

## Create your first pool

### Step 1 — Add pool

1. Go to <span class="font-semibold" data-streamdown="strong">Pools</span> → <span class="font-semibold" data-streamdown="strong">Add pool</span>
2. Choose the <span class="font-semibold" data-streamdown="strong">HAProxy template</span>
3. Name the pool (e.g. `production-edge`)
4. After create, you land in the pool with a setup banner

### Step 2 — Connect DNS (Settings)

ServerCTL needs API access to authoritative DNS to create/update the failover <span class="font-semibold" data-streamdown="strong">A record</span>.

<span class="font-semibold" data-streamdown="strong">Cloudflare</span>

- API token: <span class="font-semibold" data-streamdown="strong">Zone · DNS · Edit</span> (+ zone read)
- Cloudflare <span class="font-semibold" data-streamdown="strong">Account ID</span>
- Select the zone that will host your public hostname

<span class="font-semibold" data-streamdown="strong">cPanel / WHM</span>

- WHM hostname and port (usually 2087 or 443)
- WHM username + API token
- Zone domain (apex), e.g. `<span class="md-inline-path-filename">example.com</span>`

You can save reusable Cloudflare credentials under <span class="font-semibold" data-streamdown="strong">Settings → API providers</span> and link them to pools without re-entering tokens.

### Step 3 — Enrol the first member

On <span class="font-semibold" data-streamdown="strong">Overview</span> → <span class="font-semibold" data-streamdown="strong">Add member</span>:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-field-notes-member-t"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Field</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Notes</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Member template</span></div></td><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">HAProxy balancer</span></div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Hostname</span></div></td><td><div class="md-table-cell-content">Must match JSON `hostname` from the agent; set `BALCTL_HOSTNAME` on the VM if OS hostname differs</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Allowed source IPs</span></div></td><td><div class="md-table-cell-content">VM <span class="font-semibold" data-streamdown="strong">outbound</span> IPv4 to `<span class="md-inline-path-filename">serversctl.com</span>` (egress), not necessarily SSH address</div></td></tr></tbody></table>

</div></div></div>After creating, copy the <span class="font-semibold" data-streamdown="strong">one-shot install command</span> **immediately and run it in the HAProxy Server** — the enrollment secret is shown <span class="font-semibold" data-streamdown="strong">once</span>.

The command:

- Downloads the agent bundle
- Runs `balctl-agent.sh --enrol --key … --hostname …`
- Writes `<span class="md-inline-path-prefix">/etc/balctl/</span><span class="md-inline-path-filename">agent.env</span>`
- Installs and starts `<span class="md-inline-path-filename">balctl-heartbeat.service</span>`

Within a few seconds, the member tab should show a green heartbeat.

### Step 4 — Set the public failover hostname

<span class="font-semibold" data-streamdown="strong">Settings</span> or <span class="font-semibold" data-streamdown="strong">Managed DNS</span> tab:

- Set DNS label (e.g. `lb` → `<span class="md-inline-path-filename">lb.example.com</span>`)
- Choose orange-cloud (proxied) vs DNS-only as needed
- On Overview, <span class="font-semibold" data-streamdown="strong">Make active</span> on the member that should receive traffic

### Step 6 — Add a standby (High Availability)

Repeat enrollment on a second VM. Enable <span class="font-semibold" data-streamdown="strong">Automatic failover</span> in Settings when ready for unattended promotion.

# Pool workspace (Overview, Settings, DNS, DR, Monitoring)

## Pool workspace

The pool page has a tab bar with three groups:

1. <span class="font-semibold" data-streamdown="strong">Overview</span> (pool home)
2. <span class="font-semibold" data-streamdown="strong">Member tabs</span> (one per enrolled host)
3. <span class="font-semibold" data-streamdown="strong">Pool tools</span> (DR, Monitoring, Settings, Managed DNS)

### Overview tab

For HAProxy pools, Overview answers:

- Is traffic on the active node?
- Are standbys ready?
- Will DNS move if HAProxy or the agent fails?

<span class="font-semibold" data-streamdown="strong">[![HAProxy-H2.png](https://docs.serversctl.com/uploads/images/gallery/2026-06/scaled-1680-/haproxy-h2.png)](https://docs.serversctl.com/uploads/images/gallery/2026-06/haproxy-h2.png)</span>

<span class="font-semibold" data-streamdown="strong">Hero panel:</span> Traffic-flow diagram — Cloudflare/DNS → active HAProxy → standbys.

<span class="font-semibold" data-streamdown="strong">Actions:</span>

- <span class="font-semibold" data-streamdown="strong">Add member</span>
- <span class="font-semibold" data-streamdown="strong">Cut DNS to standby</span> — manual DNS cutover to next ready standby (requires connected DNS)
- <span class="font-semibold" data-streamdown="strong">Settings</span> shortcut

<span class="font-semibold" data-streamdown="strong">KPI tiles:</span> healthy members, failover-ready count, backups, cron jobs, last failover time.

Member cards show <span class="font-semibold" data-streamdown="strong">Active</span> vs <span class="font-semibold" data-streamdown="strong">Standby</span>, heartbeat state, and <span class="font-semibold" data-streamdown="strong">Make active</span> on standbys.

### Settings tab

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-section-purpose-pool"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Section</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Purpose</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Pool name</span></div></td><td><div class="md-table-cell-content">Rename the pool</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">API providers</span></div></td><td><div class="md-table-cell-content">Cloudflare credentials, WHM links</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Balancer failover</span></div></td><td><div class="md-table-cell-content">Auto-failover toggle, recovery time (10–120s)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Remove pool</span></div></td><td><div class="md-table-cell-content">Destructive — deletes pool and related data</div></td></tr></tbody></table>

</div></div></div>Failover hostname, proxied vs DNS-only, and Dynamic DNS sync live on the <span class="font-semibold" data-streamdown="strong">Managed DNS</span> tab (not only Settings).

### Managed DNS tab

- Failover DNS label and FQDN preview
- Orange cloud vs DNS-only
- <span class="font-semibold" data-streamdown="strong">Dynamic DNS sync</span> — optional; updates A record when active member’s public IPv4 changes on heartbeat
- DNS connectivity test
- Current A record target IP

### Disaster Recovery tab

Visible when the pool has <span class="font-semibold" data-streamdown="strong">2+ members</span> (HAProxy preset only).

<span class="font-semibold" data-streamdown="strong">Cross-member restore:</span> Pick a target member, choose a snapshot from another host’s backups, restore scoped HAProxy files onto the target.

<p class="callout info"><span class="font-semibold" data-streamdown="strong">Requires Pro</span> or active trial for cross-member restore.</p>

### Monitoring tab

Pool-wide alert settings and failover notification preferences (email when auto-failover promotes a standby).

Per-member monitoring is under each member’s <span class="font-semibold" data-streamdown="strong">Monitoring</span> tab.

### Protection tab

Only appears when <span class="font-semibold" data-streamdown="strong">2+ cPanel members</span> exist — not core HAProxy-only pools. Document separately if you mix cPanel hosts into an HAProxy pool.

# Pool members & enrollment

### Member tab layout

Click a member in the tab bar to open its <span class="font-semibold" data-streamdown="strong">workspace</span>. Sub-tabs:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-tab-purpose-control-"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Tab</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Purpose</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Control panel</span></div></td><td><div class="md-table-cell-content">Host ops: reboot, updates, hostname, TLS domain (non-HAProxy PEM)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Security</span></div></td><td><div class="md-table-cell-content">UFW firewall, SSH enable/disable, firewall backup</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Status</span></div></td><td><div class="md-table-cell-content">Live HAProxy traffic stats from heartbeat (`show stat`)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Cron &amp; Jobs</span></div></td><td><div class="md-table-cell-content">Scheduled tasks + job timeline</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Restore Backups</span></div></td><td><div class="md-table-cell-content">List snapshots, scoped backup/restore</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Recipes</span></div></td><td><div class="md-table-cell-content">One-click enable flows (admin socket, SSH, Let’s Encrypt, agent update)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Monitoring</span></div></td><td><div class="md-table-cell-content">Member-level alert thresholds</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Settings</span></div></td><td><div class="md-table-cell-content">Display name, hostname, allowed IPs, geo, remove member</div></td></tr></tbody></table>

</div></div></div>HAProxy-specific <span class="font-semibold" data-streamdown="strong">Management</span> actions (install, reload, drain, TLS failover) are surfaced on <span class="font-semibold" data-streamdown="strong">Control panel</span> and via <span class="font-semibold" data-streamdown="strong">Recipes</span> — the dedicated HAProxy tab exists in code but is hidden until product-ready.

### Enrollment security model

Each heartbeat must satisfy:

1. <span class="font-semibold" data-streamdown="strong">Bearer token</span> — 48-character enrollment secret (hashed in D1)
2. <span class="font-semibold" data-streamdown="strong">`CF-Connecting-IP`</span> — must match allowed source IP(s)
3. <span class="font-semibold" data-streamdown="strong">JSON `hostname`</span> — must match enrolled hostname

Mismatch → <span class="font-semibold" data-streamdown="strong">403</span> (IP) or credential errors.

### Agent environment

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-variable-purpose-bal"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Variable</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Purpose</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">`BALCTL_API_BASE`</div></td><td><div class="md-table-cell-content">Worker URL (e.g. `https://serversctl.com`)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">`BALCTL_ENROLLMENT_SECRET`</div></td><td><div class="md-table-cell-content">From Add member modal</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">`BALCTL_HOSTNAME`</div></td><td><div class="md-table-cell-content">Override OS hostname</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">`BALCTL_DECLARE_IP`</div></td><td><div class="md-table-cell-content">Declare public IPv4 in heartbeat</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">`BALCTL_PROBE_PUBLIC_IP=1`</div></td><td><div class="md-table-cell-content">Probe public IP if not declared</div></td></tr></tbody></table>

</div></div></div>Agent runs as <span class="font-semibold" data-streamdown="strong">root</span> for HAProxy install, backup/restore, admin socket, and cert writes.

# DNS failover & traffic cutover

### Manual cutover

<span class="font-semibold" data-streamdown="strong">Make active</span> on a standby member → ServerCTL sets it as primary and updates the managed A record to its public IPv4.

<span class="font-semibold" data-streamdown="strong">Cut DNS to standby</span> on Overview → promotes next <span class="font-semibold" data-streamdown="strong">failover-ready</span> standby (same DNS update, overview-oriented workflow).

[![HAProxy-HM4.png](https://docs.serversctl.com/uploads/images/gallery/2026-06/scaled-1680-/haproxy-hm4.png)](https://docs.serversctl.com/uploads/images/gallery/2026-06/haproxy-hm4.png)

### Automatic failover

Enable in <span class="font-semibold" data-streamdown="strong">Settings → Balancer failover</span>.

When enabled, ServerCTL periodically evaluates the active member. Promotion triggers when:

- Heartbeat age exceeds <span class="font-semibold" data-streamdown="strong">failover delay</span>, or
- HAProxy is monitored and <span class="font-semibold" data-streamdown="strong">inactive</span>

A healthy standby is promoted; DNS is updated; optional email alert fires.

### Failover delay

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-setting-range-recove"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Setting</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Range</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Recovery time</div></td><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">10–120 seconds</span></div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Community (free)</div></td><td><div class="md-table-cell-content">Fixed at <span class="font-semibold" data-streamdown="strong">120s</span></div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Pro / trial</div></td><td><div class="md-table-cell-content">Faster presets (e.g. 10s, 30s)</div></td></tr></tbody></table>

</div></div></div>Agents' heartbeat independently (~1s); failover delay is <span class="font-semibold" data-streamdown="strong">not</span> the heartbeat interval.

### Failover-ready criteria

A standby is <span class="font-semibold" data-streamdown="strong">ready</span> when:

- Recent heartbeat within the failover window, <span class="font-semibold" data-streamdown="strong">and</span>
- HAProxy is not down (when monitored)

### Dynamic DNS Sync

Optional for HAProxy pools when the <span class="font-semibold" data-streamdown="strong">active</span> member’s WAN IPv4 changes (DHCP/ISP churn). Each heartbeat can push the new public IP to Cloudflare without manual DNS edits.

### Proxied vs DNS-only

- <span class="font-semibold" data-streamdown="strong">Orange cloud (proxied):</span> Traffic through Cloudflare; good for HTTP/S when origin IP hiding matters.
- <span class="font-semibold" data-streamdown="strong">DNS-only (grey cloud):</span> Clients connect directly to member IPv4 — required for raw TCP services (e.g. non-HTTP on custom ports).

# HAProxy operations

### Install &amp; lifecycle

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-action-command-id-no"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Action</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Command ID</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Notes</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Install HAProxy</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.provision</span>`</div></td><td><div class="md-table-cell-content">Fresh VM</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Re-install</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.provision</span>` + `force: true`</div></td><td><div class="md-table-cell-content">Overwrite install path</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Reload</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.reload</span>`</div></td><td><div class="md-table-cell-content">After config edits</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Provision standby from backup</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">standby.provision_from_backup</span>`</div></td><td><div class="md-table-cell-content">Clone config from backup onto standby</div></td></tr></tbody></table>

</div></div></div>Jobs are enqueued to the API; the agent claims and runs them on the <span class="font-semibold" data-streamdown="strong">next heartbeat</span>.

### Admin stats socket (drain / ready)

Runtime backend control requires a <span class="font-semibold" data-streamdown="strong">Unix admin socket</span> in `<span class="md-inline-path-filename">haproxy.cfg</span>`:

```
stats socket /run/haproxy/admin.sock mode 600 level admin expose-fd listeners
stats timeout 2m
```

Enable via <span class="font-semibold" data-streamdown="strong">Recipe: Enable HAProxy admin stats socket</span> or <span class="font-semibold" data-streamdown="strong">Enable admin stats socket</span> action.

**Requires <span class="font-semibold" data-streamdown="strong">socot</span> + agent as <span class="font-semibold" data-streamdown="strong">root</span>. This is <span class="font-semibold" data-streamdown="strong">not</span> a public HTTP stats page.**

### Backend server states

From Management/topology table:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-action-command-id-ha"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Action</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Command ID</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">HAProxy runtime</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Drain</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.server_drain</span>`</div></td><td><div class="md-table-cell-content">`set server … state drain`</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Ready</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.server_ready</span>`</div></td><td><div class="md-table-cell-content">`state ready`</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Maintenance</div></td><td><div class="md-table-cell-content">`<span class="md-inline-path-filename">haproxy.server_maint</span>`</div></td><td><div class="md-table-cell-content">`state maint`</div></td></tr></tbody></table>

</div></div></div>### TLS (Let’s Encrypt on HAProxy)

<span class="font-semibold" data-streamdown="strong">Recipe: Let’s Encrypt (failover / HAProxy)</span>

- Uses <span class="font-semibold" data-streamdown="strong">DNS-01</span> via Cloudflare for the <span class="font-semibold" data-streamdown="strong">pool failover FQDN</span>
- Agent writes combined PEM: `<span class="md-inline-path-prefix">/etc/haproxy/certs/</span><span class="md-inline-path-filename"><hostname>.pem</span>`
- <span class="font-semibold" data-streamdown="strong">One-time operator step:</span> add `ssl crt /etc/haproxy/certs/<hostname>.pem` in config, validate, reload
- Renew from Management or cron preset `<span class="md-inline-path-filename">tls.acme_renew_force</span>`

The pool must have Cloudflare linked and a failover label set before the recipe applies.

### Status tab

Shows live traffic from agent heartbeat enrichment — <span class="font-semibold" data-streamdown="strong">not</span> a duplicate of the Overview topology diagram. Use for session rates, backend health columns, etc.

# Backups & disaster recovery

### What gets backed up

HAProxy backup job captures:

- `<span class="md-inline-path-prefix">/etc/haproxy/</span><span class="md-inline-path-filename">haproxy.cfg</span>` and `conf.d/*.cfg`
- `<span class="md-inline-path-prefix">/etc/haproxy/certs/</span><span class="md-inline-path-filename">*</span>`
- Let’s Encrypt material under `<span class="md-inline-path-prefix">/etc/letsencrypt/</span>`
- Paths referenced by `ssl crt` in config under `<span class="md-inline-path-prefix">/etc/</span>`
- Optional <span class="font-semibold" data-streamdown="strong">UFW</span> rules (`<span class="md-inline-path-filename">backup.ufw</span>`) — separate job

Storage: <span class="font-semibold" data-streamdown="strong">Per member S3</span>.

Path pattern:

<div class="composer-message-codeblock" id="bkmrk-%2F%7Buserid%7D%2Fsites%2F%7Bsit"><div class="ui-code-block"><div class="ui-code-block-content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-default-code ui-code-block-default-code"><div class="ui-default-code__content"><div class="ui-default-code__line"><div class="ui-default-code__line-content">/{userId}/sites/{siteId}/snapshots/{snapshotId}/</div></div></div></div></div></div></div></div></div></div>### Restore flows

<span class="font-semibold" data-streamdown="strong">Same member:</span> Restore Backups tab → pick snapshot → scoped restore → agent validates with `haproxy -c` → reload.

<span class="font-semibold" data-streamdown="strong">Cross-member (DR tab):</span> Restore another member’s snapshot to a target VM — typically after an outage or a bad config push.

<span class="font-semibold" data-streamdown="strong">Fresh VM rebuild:</span>

1. Enrol new/replacement member
2. Optional: Install HAProxy
3. Restore snapshot
4. Make active when ready

### Standby provisioning

<span class="font-semibold" data-streamdown="strong">Provision standby from backup</span> clones, HAProxy config from a backup onto a standby host — faster than manual copy for DR drills.

# Recipes & scheduled jobs

### Recipes (member → Recipes tab)

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-recipe-when-enable-h"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Recipe</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">When</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Enable HAProxy admin stats socket</div></td><td><div class="md-table-cell-content">HAProxy detected</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Enable SSH access</div></td><td><div class="md-table-cell-content">Always available</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Let’s Encrypt (failover / HAProxy)</div></td><td><div class="md-table-cell-content">HAProxy + Cloudflare + failover FQDN</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Update agent</div></td><td><div class="md-table-cell-content">When agent version outdated</div></td></tr></tbody></table>

</div></div></div>Recipes show steps, completion state, and optional <span class="font-semibold" data-streamdown="strong">disable</span> actions (e.g. remove admin socket lines).

### Cron &amp; Jobs tab

Control-plane cron (UTC) enqueues jobs on the next agent heartbeat.

Common presets:

- HAProxy backup
- `<span class="md-inline-path-filename">haproxy.reload</span>`
- TLS force renew
- `<span class="md-inline-path-filename">failover.evaluate</span>` (pool-level failover check)

Separate from per-member <span class="font-semibold" data-streamdown="strong">backup schedule</span> on Restore Backups — both can exist.

### Job timeline

All agent jobs appear in <span class="font-semibold" data-streamdown="strong">Cron &amp; Jobs</span> with status: pending → running → completed/failed. Remote actions from Overview cards also enqueue here.

# Agent reference

### Heartbeat payload (HAProxy-relevant)

The agent sends JSON including:

- `ip` — declared/probed IPv4
- `hostname`
- `haproxy` block — monitored, active, topology, listeners, optional `show stat` summary

ServerCTL validates IP against enrollment and stores the latest row per node.

### CLI essentials

```
sudo balctl_heartbeat.py --version
sudo balctl_heartbeat.py --provision-haproxy # local install
sudo balctl_heartbeat.py --update # from configured agent.zip URL
sudo journalctl -u balctl-heartbeat.service -f
```

<div class="composer-message-codeblock" id="bkmrk-"><div class="ui-code-block"><div class="ui-code-block-content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-default-code ui-code-block-default-code"><div class="ui-default-code__content"><div class="ui-default-code__line"><div class="ui-default-code__line-content">  
</div></div></div></div></div></div></div></div></div></div>### Job loop

1. `POST /api/agents/heartbeat`
2. Server returns pending jobs
3. Agent executes, posts `POST /api/agents/jobs/complete`

Full agent docs: `<span class="md-inline-path-prefix">agents/</span><span class="md-inline-path-filename">README.md</span>` in the repo.

# Troubleshooting & FAQ

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-symptom-likely-cause"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Symptom</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Likely cause</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Fix</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">403 on heartbeat</span></div></td><td><div class="md-table-cell-content">Wrong allowed IP or hostname</div></td><td><div class="md-table-cell-content">Update allowed IPs; set `BALCTL_HOSTNAME`</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">401 unknown credential</span></div></td><td><div class="md-table-cell-content">Used member UUID instead of enrollment secret</div></td><td><div class="md-table-cell-content">Re-enroll; use 48-char secret from modal</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">No HAProxy on card</span></div></td><td><div class="md-table-cell-content">No config / unit not detected</div></td><td><div class="md-table-cell-content">Install or ensure `<span class="md-inline-path-prefix">/etc/haproxy/</span><span class="md-inline-path-filename">haproxy.cfg</span>` exists</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Drain buttons missing</span></div></td><td><div class="md-table-cell-content">No admin socket</div></td><td><div class="md-table-cell-content">Run admin socket recipe</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Backup shows · D1 not · R2</span></div></td><td><div class="md-table-cell-content">R2 not bound when backup ran</div></td><td><div class="md-table-cell-content">Fix Worker binding; run new backup</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Auto-failover didn’t run</span></div></td><td><div class="md-table-cell-content">Only one member, auto off, or no healthy standby</div></td><td><div class="md-table-cell-content">Add standby; enable auto; check readiness</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">DNS didn’t update</span></div></td><td><div class="md-table-cell-content">DNS not connected, private IP in heartbeat, label unset</div></td><td><div class="md-table-cell-content">Connect provider; use public IPv4; set label</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Let’s Encrypt recipe greyed out</span></div></td><td><div class="md-table-cell-content">No Cloudflare or no failover FQDN</div></td><td><div class="md-table-cell-content">Complete DNS setup first</div></td></tr></tbody></table>

</div></div></div><span class="font-semibold" data-streamdown="strong">Support bundle:</span> If contacting support, include the pool name, member hostname, `journalctl` excerpt, screenshot of member health badge.

---

## Appendix — Plan gating (for operators)

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-feature-community-pr"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Feature</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Community</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell">Pro / trial</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Failover delay</div></td><td><div class="md-table-cell-content">120s only</div></td><td><div class="md-table-cell-content">10s–120s</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Cross-member DR restore</div></td><td><div class="md-table-cell-content">Locked</div></td><td><div class="md-table-cell-content">Available</div></td></tr><tr class="border-border border-b" data-streamdown="table-row"><td><div class="md-table-cell-content">Premium DNS/provider modals</div></td><td><div class="md-table-cell-content">Gated</div></td><td><div class="md-table-cell-content">Available</div></td></tr></tbody></table>

</div></div></div>

# Part II - Linux Server Pools

# Member Tab Bar

<table id="bkmrk-tab-always-in-nav%3F-a" style="width: 95.119%;"><thead class="bg-muted/80"><tr class="border-border border-b"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" style="width: 23.05%;">Tab</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" style="width: 19.0553%;">Always in nav?</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" style="width: 57.8947%;">Active when</th></tr></thead><tbody class="divide-y divide-border bg-muted/40"><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Control panel</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Security</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">OpenLiteSpeed</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes (Generic)</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Content when OpenLiteSpeed detected; else frosted <span class="font-semibold">not detected</span> overlay</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">MariaDB</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes (Generic)</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Content when MariaDB/MySQL detected or cPanel-managed MySQL</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">cPanel</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes (Generic)</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Content when cPanel detected else frosted overlay</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Status</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always — host/agent health summary</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Cron &amp; Jobs</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Restore Backups</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Recipes</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Monitoring</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr><tr class="border-border border-b"><td style="width: 23.05%;"><div class="md-table-cell-content"><span class="font-semibold">Settings</span></div></td><td style="width: 19.0553%;"><div class="md-table-cell-content">Yes</div></td><td style="width: 57.8947%;"><div class="md-table-cell-content">Always</div></td></tr></tbody></table>

# Overview Tab

### Member Overview

<span class="font-semibold" data-streamdown="strong">Purpose:</span> <span class="font-semibold" data-streamdown="strong">Host-level</span> operations on this Server — OS family, uptime, services, quick actions, TLS.

[![Member-Workspace.png](https://docs.serversctl.com/uploads/images/gallery/2026-07/scaled-1680-/member-workspace.png)](https://docs.serversctl.com/uploads/images/gallery/2026-07/member-workspace.png)

#### What you see

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-section-content-heal"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><table style="width: 79.4048%; height: 238.375px;"><thead class="bg-muted/80" data-streamdown="table-header"><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell" style="width: 15.8804%; height: 29.7969px;">Section</th><th class="whitespace-nowrap px-4 py-2 text-left font-semibold text-sm" data-streamdown="table-header-cell" style="width: 83.9673%; height: 29.7969px;">Content</th></tr></thead><tbody class="divide-y divide-border bg-muted/40" data-streamdown="table-body"><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Health strip</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Agent version, heartbeat age, firewall summary</div></td></tr><tr style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Console</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Open a secure SSH session to the member</div></td></tr><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">KPI row</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">CPU, memory, disk, load</div></td></tr><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Quick actions</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Reboot, shutdown, install updates, backup (stack-aware), Let’s Encrypt (when applicable)</div></td></tr><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Services</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Running units relevant to detected stacks</div></td></tr><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">SSL / TLS</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Certificate expiry, sync domain from DNS</div></td></tr><tr class="border-border border-b" data-streamdown="table-row" style="height: 29.7969px;"><td style="width: 15.8804%; height: 29.7969px;"><div class="md-table-cell-content"><span class="font-semibold" data-streamdown="strong">Recent activity</span></div></td><td style="width: 83.9673%; height: 29.7969px;"><div class="md-table-cell-content">Latest completed jobs</div></td></tr></tbody></table>

</div></div></div>

# Security Tab

## Member Security

**<span class="font-semibold" data-streamdown="strong">Tab:</span> <span class="font-semibold" data-streamdown="strong">Security</span>.**

Use the **Security** tab to check and manage host-level firewall and SSH access for one server in a hosting pool. It is for server access control, not website, database, or cPanel account security.

[![Member-Security.png](https://docs.serversctl.com/uploads/images/gallery/2026-07/scaled-1680-/member-security.png)](https://docs.serversctl.com/uploads/images/gallery/2026-07/member-security.png)

### What you can do

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-area-what-it-is-for-"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Area</th><th>What it is for</th></tr></thead><tbody><tr><td>**Firewall status**</td><td>See whether a supported firewall backend is detected, which backend is in use, whether it is enabled, and when the rule list was last refreshed.</td></tr><tr><td>**Security actions**</td><td>Refresh firewall rules, enable or disable the firewall, manage SSH access, back up firewall rules, and add firewall rules.</td></tr><tr><td>**Firewall rules**</td><td>Review the latest stored rule snapshot and delete rules when needed.</td></tr></tbody></table>

</div></div></div></div></div>Supported firewall views are based on what the member reports, commonly UFW or firewalld.

### Free and Pro

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-feature-community-pr"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View firewall status from the latest member report</td><td>Included</td><td>Included</td></tr><tr><td>View SSH status from the latest member report</td><td>Included</td><td>Included</td></tr><tr><td>View stored firewall rule snapshots</td><td>Included</td><td>Included</td></tr><tr><td>Refresh firewall rules</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Enable or disable firewall</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Add firewall rules</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Delete firewall rules</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Enable or disable SSH service</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Back up firewall rules</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div></div></div></div>Community gives visibility. Pro gives controlled action: rule changes, firewall state changes, SSH access control, and firewall backups before risky changes.

#### Firewall status

Open **Pool**, choose the member, then select **Security**.

The firewall status area shows:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-whether-a-supported-"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Whether a supported firewall was detected.
- The firewall backend, such as UFW or firewalld.
- Whether the firewall is enabled or disabled.
- When the rule snapshot was last refreshed.

</div></div></div>If the firewall is not detected, the member may not have a supported firewall installed or the agent may not have enough information yet. Refresh rules after fixing the host.

#### Refreshing rules

Use **Refresh rules** to pull a fresh firewall rule list from the host.

Refresh is useful after:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-you-changed-firewall"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- You changed firewall rules outside ServersCTL.
- A backup or restore changed host security files.
- The rules table is empty or stale.
- You want to confirm the current live state before editing.

</div></div></div>The rules table updates after the job completes. Check **Cron &amp; Jobs** if the refresh does not appear to finish.

### Managing the firewall

Use **Firewall** to enable or disable the detected firewall backend.

Before enabling the firewall:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-make-sure-ssh-is-all"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Make sure SSH is allowed from your access IP or management network.
- Make sure required web, database, control panel, and monitoring ports are allowed.
- Refresh the rule list so you can see what will apply.

</div></div></div>Before disabling the firewall:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-understand-that-inbo"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Understand that inbound filtering is reduced or stopped until it is enabled again.
- Use this only for controlled troubleshooting or planned maintenance.

</div></div></div>Existing rules are preserved. Refresh rules after the job completes so the table reflects the new state.

#### Adding rules

Use **Add rule** to allow, deny, or reject traffic by protocol, port, and source.

The form asks for:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-field-meaning-action"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody><tr><td>**Action**</td><td>Allow, deny, or reject matching traffic.</td></tr><tr><td>**Protocol**</td><td>TCP or UDP.</td></tr><tr><td>**Port**</td><td>A single port or a valid range.</td></tr><tr><td>**Source**</td><td>`any` or an IPv4 CIDR range.</td></tr><tr><td>**Comment**</td><td>Optional note to make the rule easier to identify later.</td></tr></tbody></table>

</div></div></div></div></div>Use narrow sources where possible. For example, allow SSH from your office or VPN range instead of allowing it from anywhere.

#### Deleting rules

Use **Delete** from the firewall rules table when a rule is no longer needed.

Deleting a rule can immediately change live traffic. Check the action, port, protocol, source, and comment before confirming.

#### SSH access

Use **SSH access** to enable or disable the SSH service on the host.

Disabling SSH may close current sessions and block new SSH logins. The balctl agent should continue running, so you can turn SSH back on from the Security tab when the member is still checking in.

Use this carefully. If the agent stops checking in and SSH is disabled, you may need console access from your server provider.

#### Firewall backup

Use **Firewall backup** before large firewall changes, migrations, or security cleanups.

Firewall backups are stored with the member's backups so they can be found from the backup and restore areas. Keep a backup before removing broad rules or changing access to production services.

#### What this tab does not manage

The Security tab is host-level only.

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-section-content-fire"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Not managed here</th><th>Where to manage it</th></tr></thead><tbody><tr><td>cPanel users, packages, and account access</td><td>cPanel tab or WHM</td></tr><tr><td>AutoSSL and website certificates</td><td>cPanel or OpenLiteSpeed tools, depending on the server</td></tr><tr><td>Database users and grants</td><td>MariaDB / MySQL tab or database tools</td></tr><tr><td>DNS provider security</td><td>Managed DNS or provider settings</td></tr></tbody></table>

</div></div></div></div></div>

# OpenLiteSpeed Tab

## Member OpenLiteSpeed

**<span class="font-semibold" data-streamdown="strong">Tab:</span> <span class="font-semibold" data-streamdown="strong">OpenLiteSpeed</span>.  
Sub Tabs: Overview, Recovery Wizard**

Use the **OpenLiteSpeed** tab to manage standalone OpenLiteSpeed servers in a hosting pool. It gives you one place to check the web server, create recovery points, restore sites, and move sites to another compatible OpenLiteSpeed server.

This tab is for servers running OpenLiteSpeed directly. cPanel servers use the cPanel tools instead.

### What is included

<div id="bkmrk-area-what-you-use-it"><div><table><thead><tr><th>Area</th><th>What you use it for</th></tr></thead><tbody><tr><td>**Overview**</td><td>Check server health, open the WebAdmin panel, reload or restart OpenLiteSpeed, test configuration, update PHP packages, and create a server config backup.</td></tr><tr><td>**Backups**</td><td>Create site, web-file, database, and full recovery points for OpenLiteSpeed applications.</td></tr><tr><td>**Transfer Wizard**</td><td>Move an OpenLiteSpeed site to another compatible OpenLiteSpeed member.</td></tr></tbody></table>

</div></div>#### Community and Pro

<div id="bkmrk-feature-community-pr-1"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View detected OpenLiteSpeed status</td><td>Included</td><td>Included</td></tr><tr><td>Open WebAdmin link and view basic server details</td><td>Included</td><td>Included</td></tr><tr><td>Refresh detected applications</td><td>Included</td><td>Included</td></tr><tr><td>View linked databases when discovered</td><td>Included</td><td>Included</td></tr><tr><td>Create OpenLiteSpeed backups</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Restore OpenLiteSpeed backups</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Transfer sites between OpenLiteSpeed servers</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Automatic DNS cutover after restore or transfer</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Pro turns this tab from a read-only view into a recovery and migration workspace. It is designed for operators who need to move sites quickly, keep usable recovery points, and reduce manual rebuild work.

### Overview

Open **Pool**, choose the OpenLiteSpeed member, then select **OpenLiteSpeed**.

The **Overview** area shows:

- WebAdmin access for the selected member.
- Current service state and key server details.
- Actions for reload, restart, configuration test, backup, upgrade, and PHP package updates.
- Recent log output when available.

Use **Config test** before restarting a busy server. If the test fails, fix the configuration first so the web server does not reload into a broken state.

#### Backups

Open **Backups** to work with detected OpenLiteSpeed applications.

Each application card shows the site name, document root, PHP and TLS information when available, linked databases when discovered, and the latest recovery point.

To create a recovery point:

1. Select the application.
2. Choose **Backup now**.
3. Pick the backup type.
4. Confirm the storage estimate and start the backup.

Common backup types:

<div id="bkmrk-backup-type-use-it-w"><div><table><thead><tr><th>Backup type</th><th>Use it when</th></tr></thead><tbody><tr><td>**Full site backup**</td><td>You want the safest recovery point for a real application.</td></tr><tr><td>**Files only**</td><td>You only changed web files, themes, plugins, uploads, or static content.</td></tr><tr><td>**Databases only**</td><td>You only need a database snapshot for a linked schema.</td></tr></tbody></table>

</div></div>Recovery points can be restored, downloaded, inspected, or deleted from the same workspace.

#### OpenLiteSpeed Transfer Wizard

Use the OLS **Transfer Wizard** when you want to move an OpenLiteSpeed site to another compatible OpenLiteSpeed member.

#### Transfer a Site (OpenLiteSpeed to OpenLiteSpeed)

To transfer a site:

1. Open **Transfer Wizard**.
2. Select the source application.
3. Choose a compatible destination member.
4. Confirm whether DNS will be updated manually or automatically.
5. Start the transfer and watch the progress panel.

Destination servers must be compatible. cPanel servers are not valid OpenLiteSpeed transfer targets, even if they also run LiteSpeed or expose similar services.

#### DNS cutover

After a restore or transfer, DNS still decides where visitors go.

<div id="bkmrk-option-availability-"><div><table style="width: 100%;"><thead><tr><th style="width: 23.2381%;">Option</th><th style="width: 18.1206%;">Availability</th><th style="width: 58.6412%;">What happens</th></tr></thead><tbody><tr><td style="width: 23.2381%;">**Manual DNS update**</td><td style="width: 18.1206%;">Community and Pro</td><td style="width: 58.6412%;">The site is moved or restored, then you update the A record yourself.</td></tr><tr><td style="width: 23.2381%;">**Automatic DNS update**</td><td style="width: 18.1206%;">Pro</td><td style="width: 58.6412%;">The platform updates the saved DNS provider record after the operation completes.</td></tr></tbody></table>

</div></div>Automatic DNS is useful when you want a cleaner migration flow and already have a DNS key saved for the domain or account.

##### Good operating habits

- Run a full backup before major changes.
- Use configuration test before restarting OpenLiteSpeed.
- Keep enough storage for at least one known-good recovery point per important site.
- Confirm DNS after any restore or transfer.
- Use Pro for production migration and recovery workflows where speed matters.

##### Direct Transfer security model

Direct Transfer is not an open file receiver and it is not general-purpose SSH access. The API acts as the control plane for a single copy operation:

1. **The authenticated operator starts a transfer.** The dashboard calls the Worker with the selected source member, destination member, cPanel username, route mode, and DNS handling choice. The Worker checks plan access, pool ownership, WHM/cPanel compatibility, account state, agent versions, destination disk, and route availability before creating the run.
2. **API creates a one-use session.** The Worker mints a random session ID, records the exact source node, destination node, account username/domain, expected byte count, destination address, source IP filter, status, and expiry. Session APIs reject unknown, expired, terminal, wrong-role, or wrong-status sessions.
3. **Only enrolled agents can advance it.** Source and destination agents call back using their normal enrolled heartbeat credentials. The Worker verifies that the calling node is the recorded source or destination for that session before accepting prepare, public-key registration, progress, transfer-complete, or restore-complete updates.
4. **The source generates an ephemeral SSH key.** The source agent creates a session-scoped keypair for this transfer. The private key stays on the source server. The public key is sent to the Worker, which queues an authorised job for the destination agent.
5. **Destination installs a restricted receiver key.** The destination agent writes the public key to root `authorized_keys` with a forced command for chunk receive only, plus `from=<source-ip>`, `no-port-forwarding`, `no-X11-forwarding`, `no-agent-forwarding`, `no-pty`, and `no-user-rc`. A matching `session.json` must exist before any chunk is accepted.
6. **Chunks are individually checked.** The source agent streams fixed-size chunks over SSH. The forced receiver accepts only `sessionId`, chunk index, and chunk SHA-256, reads the chunk from stdin, verifies the chunk hash, and writes it under that session's temporary receive directory.
7. **Archive integrity is checked before restore.** After transfer, the destination agent assembles the archive and verifies the final SHA-256 recorded on the session/run before calling `restorepkg`.
8. **Session cleanup is automatic.** On success, failure, expiry, or revoke, the destination removes the matching `authorized_keys` line and deletes the temporary session directory. The Worker will also queue `cpanel_account_sync_revoke` if the run fails before normal cleanup.

The session lifetime is time-limited. It has a 30-minute minimum and scales with package size using a conservative transfer-rate estimate, so a large package has enough time to complete without leaving a permanent receiver open.

# MariaDB/MySQL Tab

### Standalone database server

#### Overview

Use the **MariaDB / MySQL** tab to check database service health, refresh the live database inventory, create database backups, and restore database versions when needed.

This tab appears for Generic Linux members. It is useful for dedicated database servers, OpenLiteSpeed servers with local databases, and cPanel servers where MySQL is managed by WHM.

[![MariaDB-MySQL-Overview.png](https://docs.serversctl.com/uploads/images/gallery/2026-06/scaled-1680-/mariadb-mysql-overview.png)](https://docs.serversctl.com/uploads/images/gallery/2026-06/mariadb-mysql-overview.png)

#### What is included

<div id="bkmrk-subtab-what-you-use-"><div><table><thead><tr><th>Subtab</th><th>What you use it for</th></tr></thead><tbody><tr><td>**Overview**</td><td>Check database service health, listener status, version, replication role, recent backup age, logs, and service actions.</td></tr><tr><td>**Databases**</td><td>View live schemas, refresh discovery, see backup versions, back up selected schemas, restore versions, and delete old database backups.</td></tr></tbody></table>

</div></div>#### Free and Pro

<div id="bkmrk-feature-community-pr-2"><div><table style="width: 86.0714%;"><thead><tr><th style="width: 63.3385%;">Feature</th><th style="width: 25.741%;">Community</th><th style="width: 10.9204%;">Pro</th></tr></thead><tbody><tr><td style="width: 63.3385%;">View service health and heartbeat-reported database status</td><td style="width: 25.741%;">Included</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">View version, listener, uptime, and basic metrics when available</td><td style="width: 25.741%;">Included</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">View live database inventory</td><td style="width: 25.741%;">Included</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Refresh live database discovery</td><td style="width: 25.741%;">Included</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">View existing backup versions</td><td style="width: 25.741%;">Included when available</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Back up all databases</td><td style="width: 25.741%;">Upgrade required</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Back up selected schemas</td><td style="width: 25.741%;">Upgrade required</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Restore database backup versions</td><td style="width: 25.741%;">Upgrade required</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Delete managed database backup versions</td><td style="width: 25.741%;">Upgrade required</td><td style="width: 10.9204%;">Included</td></tr><tr><td style="width: 63.3385%;">Choose a Storage Pool for database backups</td><td style="width: 25.741%;">Upgrade required</td><td style="width: 10.9204%;">Included</td></tr></tbody></table>

</div></div>Pro is for real database operations: scheduled recovery coverage, one-click backups before risky changes, schema-level restore, and storage-aware backup management.

##### Overview subtab

Open **Pool**, choose the member, select **MariaDB / MySQL**, then open **Overview**.

The Overview subtab shows:

- Database service status.
- Server version.
- Replication or Galera role when reported.
- Last all-databases backup age.
- Listener status, usually port 3306.
- Connections, running threads, queries per second, uptime, and buffer pool signals when available.
- Error and slow-query log tails from the latest member report.

Available actions include:

<div id="bkmrk-action-use-it-when-r"><div><table><thead><tr><th>Action</th><th>Use it when</th></tr></thead><tbody><tr><td>**Restart**</td><td>The database service needs a controlled restart.</td></tr><tr><td>**Config test**</td><td>You want to check database configuration before restarting.</td></tr><tr><td>**Flush privileges**</td><td>Database grants or users were changed and privileges need reloading.</td></tr><tr><td>**Backup all**</td><td>You want one recovery point containing all databases on the member.</td></tr><tr><td>**Harden**</td><td>You want the platform to apply supported database hardening guidance or actions.</td></tr></tbody></table>

</div></div>On cPanel servers, the tab may show a cPanel host notice. Use the cPanel tab for WHM account work, but use **Databases** here to refresh and back up discovered application schemas.

##### Databases subtab

Open **Databases** to work with the live schema inventory.

The Databases subtab shows:

- Live schemas currently found on the server.
- Size and table count when available.
- Database engine details where reported.
- Backup version count for schemas with stored recovery points.
- The latest backup age for each backed-up schema.
- A discovery cache age so you know how fresh the list is.

Use **Refresh live** when you have added databases, restored accounts, moved sites, or changed application database settings. On cPanel and web hosts, discovery can also find application-linked schemas from known config files.

##### Backing up databases

To back up all databases:

1. Open **Overview**.
2. Choose **Backup all**.
3. Pick the Storage Pool if more than one is available.
4. Review the estimate and start the backup.

To back up selected schemas:

1. Open **Databases**.
2. Open the menu on the schema card.
3. Choose **Backup**.
4. Confirm the schemas and Storage Pool.
5. Start the backup.

The backup modal warns when the selected Storage Pool does not have enough remaining space.

##### Restoring versions

To restore a database version on Pro:

1. Open **Databases**.
2. Open the menu on a schema with backup versions.
3. Choose **Versions**.
4. Select the version you want.
5. Choose **Restore** and confirm.

Restoring a database can overwrite live data. Check the version date, source member, and schema name before confirming.

##### Deleting versions

Use **Delete** from the versions modal when you no longer need a stored database backup. Deletion is permanent, so keep at least one known-good recovery point for important schemas.

### cPanel-managed MySQL

On cPanel servers, MySQL is shared across accounts. That means:

- WHM account tasks stay in the cPanel tab.
- Database inventory and schema backups stay in MariaDB / MySQL.
- Use **Refresh live** after adding or restoring cPanel accounts.
- Application-linked schemas may appear after discovery even when root-level metrics are limited.

### Galera (`wsrep`) (In Alpha)

When Galera is enabled, the agent will report the wsrep state. <span class="font-semibold" data-streamdown="strong">ServersCTL does not</span> run quorum, SST, or writer election. <span class="font-semibold" data-streamdown="strong">DNS is active ≠ Galera primary.</span>

# cPanel Tab

Use the **cPanel** member tab to manage a WHM server in a ServersCTL hosting pool. It brings server health, WHM accounts, account transfers, recovery actions, and cPanel service controls into one workspace for the selected server.

The cPanel tab is for WHM/cPanel servers. OpenLiteSpeed-only servers use the OpenLiteSpeed tab instead.

## Free and Pro

<div id="bkmrk-feature-community-pr-3"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View cPanel overview and service state</td><td>Included</td><td>Included</td></tr><tr><td>View Operations health, metrics, logs, and WHM API status</td><td>Included</td><td>Included</td></tr><tr><td>View all WHM accounts and account details</td><td>Included</td><td>Included</td></tr><tr><td>Manual Agent Transfer between cPanel servers</td><td>Included</td><td>Included</td></tr><tr><td>Create, suspend, unsuspend, modify, or terminate accounts</td><td>Upgrade required</td><td>Included</td></tr><tr><td>One-time cPanel login links</td><td>Upgrade required</td><td>Included</td></tr><tr><td>cPanel account backups and AutoSSL actions</td><td>Upgrade required</td><td>Included</td></tr><tr><td>cPanel service restarts, config checks, hardening, and config backup</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Automatic DNS update during Agent Transfer</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Migrate &amp; Recovery live WHM transfers</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Pool Protection and scheduled replication</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Community is enough for visibility and manual account copy workflows. Pro is for production hosting operations: account management, automated DNS cutover, recovery, live transfers, backups, and service control.

## Requirements

For the full cPanel workspace:

- The server must be enrolled as a pool member.
- The balctl agent must be checking in.
- cPanel/WHM must be detected.
- WHM must be linked for account and migration tools.
- A second compatible cPanel member is needed for transfer workflows.
- Saved DNS keys are needed for automatic DNS updates.

If the tab is visible but actions are unavailable, check the member **Status** tab, WHM link state, and plan level.

## Subtabs

<div id="bkmrk-subtab-what-you-use--1"><div><table><thead><tr><th>Subtab</th><th>What you use it for</th></tr></thead><tbody><tr><td>**Overview**</td><td>See cPanel protection state, service badges, quick actions, WHM links, and account recovery topology.</td></tr><tr><td>**Operations**</td><td>Check cPanel service health, disk, network, WHM API status, metrics, logs, and server-level actions.</td></tr><tr><td>**Accounts**</td><td>List WHM accounts, inspect account details, create accounts, suspend, unsuspend, back up, log in, run AutoSSL, and terminate accounts.</td></tr><tr><td>**Agent Transfer**</td><td>Copy one account directly to another compatible cPanel server using the installed agents.</td></tr><tr><td>**Migrate &amp; Recovery**</td><td>Start and monitor WHM live transfer sessions between cPanel members.</td></tr></tbody></table>

</div></div>Some subtabs only appear after the server is detected as cPanel and WHM is linked for the pool.

### Overview subtab

Use **Overview** as the first stop for a cPanel member.

The Overview subtab shows:

- The cPanel protection and recovery topology.
- Whether accounts are protected or have standby coverage.
- Service stack badges.
- Quick links to WHM and Webmail when available.
- Quick actions such as account creation or backup where available.
- WHM audit information when the server is linked.

Use this subtab to understand whether the selected server is acting as the live source, standby, or an unprotected cPanel host.

### Operations subtab

Use **Operations** for server-level cPanel health and maintenance.

The Operations subtab shows:

- cPanel health and license state.
- Web, mail, cPanel, database, and related service state.
- Listener and network information.
- Disk usage and server metrics.
- WHM API status and DNS drift information.
- cPanel and mail log tails.

Common Pro actions:

<div id="bkmrk-action-use-it-when-r-1"><div><table style="width: 72.8571%;"><thead><tr><th style="width: 24.3446%;">Action</th><th style="width: 75.6554%;">Use it when</th></tr></thead><tbody><tr><td style="width: 24.3446%;">**Restart web**</td><td style="width: 75.6554%;">Apache/httpd is down or web traffic needs a controlled restart.</td></tr><tr><td style="width: 24.3446%;">**Restart mail**</td><td style="width: 75.6554%;">Exim or Dovecot needs a controlled restart.</td></tr><tr><td style="width: 24.3446%;">**Restart cPanel**</td><td style="width: 75.6554%;">WHM/cPanel services are unhealthy.</td></tr><tr><td style="width: 24.3446%;">**Config check**</td><td style="width: 75.6554%;">You want to check cPanel configuration health before changes.</td></tr><tr><td style="width: 24.3446%;">**Backup config**</td><td style="width: 75.6554%;">You want a cPanel metadata/config recovery point.</td></tr><tr><td style="width: 24.3446%;">**Harden ports**</td><td style="width: 75.6554%;">You want supported cPanel firewall/service hardening.</td></tr><tr><td style="width: 24.3446%;">**Refresh snapshot**</td><td style="width: 75.6554%;">You want the latest WHM service snapshot and DNS drift state.</td></tr></tbody></table>

</div></div>Check **Cron &amp; Jobs** after starting any action.

### Accounts subtab

Use **Accounts** to work with WHM accounts on the selected server.

The Accounts subtab shows:

- All WHM accounts returned for the member.
- Domain, username, IP, plan, email, disk usage, and status where available.
- Filters for all, active, and suspended accounts.
- Account details in a drawer or modal.
- Protection and DNS hints when configured.

Community users can view the full list and inspect details. Pro users can also run account actions.

Pro account actions:

<div id="bkmrk-action-what-it-does-"><div><table><thead><tr><th>Action</th><th>What it does</th></tr></thead><tbody><tr><td>**Create account**</td><td>Creates a WHM account with username, domain, password, plan, and contact email.</td></tr><tr><td>**Suspend / Unsuspend**</td><td>Changes whether an account can be used.</td></tr><tr><td>**Login**</td><td>Opens a one-time cPanel login session.</td></tr><tr><td>**Backup**</td><td>Creates a cPanel account backup.</td></tr><tr><td>**Change package**</td><td>Moves the account to another WHM package.</td></tr><tr><td>**Reset password**</td><td>Generates and shows a new password once.</td></tr><tr><td>**Run AutoSSL check**</td><td>Starts an AutoSSL check for accounts.</td></tr><tr><td>**Terminate**</td><td>Permanently removes the account after confirmation.</td></tr></tbody></table>

</div></div>Termination is destructive. The confirmation asks you to type the domain so accidental deletion is harder.

### Agent Transfer subtab

**Agent Transfer** is the member **cPanel** inner tab for one-shot **cPanel → cPanel** account copies between enrolled pool members. Agents package the source account, move the archive over a direct route or through user storage, restore on the destination with `restorepkg`, and optionally swing the DNS after the copy succeeds.

#### Direct Transfer security model

Direct Transfer is not an open file receiver and it is not general-purpose SSH access. The API acts as the control plane for a single copy operation:

1. **The authenticated operator starts a transfer.** The dashboard calls the Worker with the selected source member, destination member, cPanel username, route mode, and DNS handling choice. The Worker checks plan access, pool ownership, WHM/cPanel compatibility, account state, agent versions, destination disk, and route availability before creating the run.
2. **API creates a one-use session.** The Worker mints a random session ID, records the exact source node, destination node, account username/domain, expected byte count, destination address, source IP filter, status, and expiry. Session APIs reject unknown, expired, terminal, wrong-role, or wrong-status sessions.
3. **Only enrolled agents can advance it.** Source and destination agents call back using their normal enrolled heartbeat credentials. The Worker verifies that the calling node is the recorded source or destination for that session before accepting prepare, public-key registration, progress, transfer-complete, or restore-complete updates.
4. **The source generates an ephemeral SSH key.** The source agent creates a session-scoped keypair for this transfer. The private key stays on the source server. The public key is sent to the Worker, which queues an authorised job for the destination agent.
5. **Destination installs a restricted receiver key.** The destination agent writes the public key to root `authorized_keys` with a forced command for chunk receive only, plus `from=<source-ip>`, `no-port-forwarding`, `no-X11-forwarding`, `no-agent-forwarding`, `no-pty`, and `no-user-rc`. A matching `session.json` must exist before any chunk is accepted.
6. **Chunks are individually checked.** The source agent streams fixed-size chunks over SSH. The forced receiver accepts only `sessionId`, chunk index, and chunk SHA-256, reads the chunk from stdin, verifies the chunk hash, and writes it under that session's temporary receive directory.
7. **Archive integrity is checked before restore.** After transfer, the destination agent assembles the archive and verifies the final SHA-256 recorded on the session/run before calling `restorepkg`.
8. **Session cleanup is automatic.** On success, failure, expiry, or revoke, the destination removes the matching `authorized_keys` line and deletes the temporary session directory. The Worker will also queue `cpanel_account_sync_revoke` if the run fails before normal cleanup.

The session lifetime is time-limited. It has a 30-minute minimum and scales with package size using a conservative transfer-rate estimate, so a large package has enough time to complete without leaving a permanent receiver open.

This is the direct account transfer workspace. It is separate from scheduled Protection and separate from the WHM live transfer wizard.

To transfer an account:

1. Open **Agent Transfer** on the source cPanel member.
2. Choose or drag an active account.
3. Choose a compatible destination cPanel member.
4. Review the route preflight.
5. Choose DNS handling.
6. Start the transfer.
7. Watch the transfer spotlight until it completes.

DNS handling:

<div id="bkmrk-option-availability--1"><div><table><thead><tr><th>Option</th><th>Availability</th><th>What happens</th></tr></thead><tbody><tr><td>**Manual DNS update**</td><td>Community and Pro</td><td>The account is copied. You update DNS yourself when ready.</td></tr><tr><td>**Update A record automatically**</td><td>Pro</td><td>ServersCTL uses a saved Cloudflare or cPanel DNS key after restore succeeds.</td></tr></tbody></table>

</div></div>The direct route is preferred. If direct transfer cannot work and user storage is available, the transfer can use storage as the fallback route. If storage is required but unavailable, add storage or use another transfer method.

After a successful direct transfer, the temporary transfer access is cleaned up. The account archive may remain on the destination server for the user to remove when they are finished with it.

### Migrate &amp; Recovery subtab

Use **Migrate &amp; Recovery** for live WHM transfer sessions between cPanel members.

This is a Pro workflow for native WHM transfers. It shows outgoing and incoming sessions for the selected server, including route, direction, status, DNS cutover state, and last update time.

To start a live WHM transfer:

1. Open **Migrate &amp; Recovery**.
2. Choose **Start transfer**.
3. Pick the account.
4. Pick the target cPanel server.
5. Review preflight checks.
6. Decide whether DNS cutover should run.
7. Confirm and monitor the session.

Use the refresh button to update transfer sessions. If a completed transfer had DNS cutover enabled but DNS failed, use the retry option after fixing the DNS key or zone issue.

#### Agent Transfer vs Migrate &amp; Recovery vs Protection

<div id="bkmrk-tool-best-for-agent-"><div><table style="width: 97.1429%;"><thead><tr><th style="width: 27.841%;">Tool</th><th style="width: 72.159%;">Best for</th></tr></thead><tbody><tr><td style="width: 27.841%;">**Agent Transfer**</td><td style="width: 72.159%;">One-off account copy using ServersCTL agents. Community can use manual DNS; Pro can automate DNS.</td></tr><tr><td style="width: 27.841%;">**Migrate &amp; Recovery**</td><td style="width: 72.159%;">Pro live WHM transfer sessions between cPanel members.</td></tr><tr><td style="width: 27.841%;">**Pool Protection**</td><td style="width: 72.159%;">Pro scheduled standby replication and failover readiness across protected accounts.</td></tr></tbody></table>

</div></div>Use Agent Transfer when you want a direct copy now. Use Protection when you want ongoing standby coverage. Use Migrate &amp; Recovery when you specifically want the WHM transfer workflow.

#### WHM Binding

Full WHM API when member matches pool `host`. Run <span class="font-semibold" data-streamdown="strong">WHM link check</span> recipe after DNS connect.

# Status Tab

Use the **Status** tab to see whether a server is online, reporting correctly, and ready for work. It is the first place to check when a member looks stale, jobs are not moving, or a server has not checked in recently.

### What you can see

The Status tab shows a live operational summary for the selected member:

<div id="bkmrk-area-what-it-tells-y"><div><table><thead><tr><th>Area</th><th>What it tells you</th></tr></thead><tbody><tr><td>**Heartbeat**</td><td>Whether the member is checking in and how recent the last signal was.</td></tr><tr><td>**Agent version**</td><td>The installed agent version and whether an update is available.</td></tr><tr><td>**Host details**</td><td>Hostname, operating system, uptime, and basic identity details.</td></tr><tr><td>**Resources**</td><td>CPU, memory, disk, and other capacity signals when reported by the member.</td></tr><tr><td>**Security basics**</td><td>Firewall and SSH status when available.</td></tr><tr><td>**Updates**</td><td>Whether the server appears to need package updates.</td></tr></tbody></table>

</div></div>Some specialist members may show a tailored status view for their role, such as HAProxy.

### Free and Pro

<div id="bkmrk-feature-community-pr-4"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View heartbeat and online state</td><td>Included</td><td>Included</td></tr><tr><td>View agent version and host details</td><td>Included</td><td>Included</td></tr><tr><td>View reported resource usage</td><td>Included</td><td>Included</td></tr><tr><td>View firewall, SSH, and update signals when reported</td><td>Included</td><td>Included</td></tr><tr><td>Use advanced tabs that act on status signals</td><td>Limited</td><td>Included</td></tr><tr><td>Faster operational recovery using backup, transfer, and automation tools</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>The Status tab is included because every user needs to know whether their member is healthy. Pro adds the tools around that signal: backup, restore, transfer, scheduling, and automation.

### How to use it

1. Open the pool.
2. Select the member.
3. Open **Status**.
4. Check the heartbeat first.
5. Review resource, update, firewall, and agent signals.

If the server is online but a feature is not behaving as expected, check **Cron &amp; Jobs** next. That tab shows whether recent actions are queued, running, completed, or failed.

##### Common checks

<div id="bkmrk-what-you-see-what-it"><div><table><thead><tr><th>What you see</th><th>What it usually means</th></tr></thead><tbody><tr><td>**Recent heartbeat**</td><td>The agent is checking in and the member can be managed.</td></tr><tr><td>**Stale heartbeat**</td><td>The server, network, or agent may need attention.</td></tr><tr><td>**Agent update available**</td><td>Update the agent before testing new features.</td></tr><tr><td>**High disk usage**</td><td>Backups, restores, and transfers may fail unless space is freed.</td></tr><tr><td>**Firewall warning**</td><td>Remote access or service traffic may be blocked.</td></tr></tbody></table>

</div></div>

# Cron & Jobs Tab

## Member Cron &amp; Jobs

Use **Cron &amp; Jobs** to see scheduled work and recent server tasks for a member. It helps you confirm whether backups, discovery runs, refreshes, and other actions are queued, running, completed, or failed.

### What it is for

<div id="bkmrk-area-what-you-use-it-1"><div><table><thead><tr><th>Area</th><th>What you use it for</th></tr></thead><tbody><tr><td>**Schedules**</td><td>View and manage recurring jobs for the selected member.</td></tr><tr><td>**Recent jobs**</td><td>Check the latest actions sent to the member.</td></tr><tr><td>**Run now**</td><td>Trigger a supported scheduled task immediately when available.</td></tr><tr><td>**History**</td><td>See whether work completed or needs attention.</td></tr></tbody></table>

</div></div>Times are shown in the product interface for easy reading. When you are coordinating with logs or support, include the member name and the job time shown in the UI.

### Free and Pro

<div id="bkmrk-feature-community-pr-5"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View recent jobs and outcomes</td><td>Included</td><td>Included</td></tr><tr><td>See existing schedules</td><td>Included</td><td>Included</td></tr><tr><td>Create recurring schedules</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Edit, pause, resume, or delete schedules</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Run scheduled jobs on demand</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Use backup and discovery automation</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Pro is where Cron &amp; Jobs becomes automation instead of just visibility. It is intended for teams that want regular backups, routine discovery refreshes, and repeatable maintenance without manual checklists.

### Using recent jobs

Open **Cron &amp; Jobs** and look at the recent job list.

Each job usually shows:

- What action was requested.
- The current state.
- When it started or finished.
- The member that handled it.
- Any visible failure reason.

If a job fails, use the failure message to decide the next step. Common causes are stale agent heartbeat, missing storage, low disk space, incompatible service type, or a feature that requires Pro.

### Using schedules

Schedules are useful for repeated operational work.

To create a schedule on Pro:

1. Open **Cron &amp; Jobs**.
2. Choose **Add schedule**.
3. Pick the job type.
4. Set the timing.
5. Confirm the target and save.

Use schedules for routine backups and refresh tasks. Avoid scheduling risky changes unless the result is easy to verify and recover from.

##### Good operating habits

- Check recent jobs after starting a backup, restore, transfer, or discovery run.
- Keep schedules simple and clearly named.
- Pause schedules during major migrations if they could interfere.
- Use Pro schedules for regular backup and discovery coverage.

# Restore Backups Tab

Use **Restore Backups** to find recovery points for a member and put them back when something needs to be recovered. It is the recovery workspace for server backups, service backups, application backups, and other stored recovery points that belong to the member.

### What you can do

<div id="bkmrk-area-what-it-is-for--1"><div><table><thead><tr><th>Area</th><th>What it is for</th></tr></thead><tbody><tr><td>**Recovery timeline**</td><td>Browse available backups by date and type.</td></tr><tr><td>**Backup details**</td><td>Inspect size, source, version, and what the recovery point contains.</td></tr><tr><td>**Restore**</td><td>Put a compatible backup back onto a server.</td></tr><tr><td>**Download**</td><td>Download a recovery point when available.</td></tr><tr><td>**Delete**</td><td>Remove old backups you no longer want to keep.</td></tr><tr><td>**Run backup**</td><td>Create a fresh backup when the member and plan support it.</td></tr></tbody></table>

</div></div>### Free and Pro

<div id="bkmrk-feature-community-pr-6"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View available recovery points</td><td>Included when storage is connected</td><td>Included</td></tr><tr><td>Inspect backup details</td><td>Included</td><td>Included</td></tr><tr><td>Download supported backups</td><td>Included when allowed by storage policy</td><td>Included</td></tr><tr><td>Create new cloud backups</td><td>Upgrade or storage required</td><td>Included with connected storage</td></tr><tr><td>Restore backups from the UI</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Delete managed recovery points</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Cross-member restore to compatible servers</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Pro gives you the recovery controls that matter during an incident: create fresh backups, restore faster, and recover to compatible members without turning the process into manual file handling.

### Finding a backup

1. Open the pool.
2. Select the member.
3. Open **Restore Backups**.
4. Use the timeline or filters to find the backup type you need.
5. Open the backup details before restoring.

Check the source member, backup type, date, and size before you restore. For full server or site backups, confirm the destination is compatible.

### Restoring a backup

To restore on Pro:

1. Select the recovery point.
2. Choose **Restore**.
3. Review the compatibility and warning messages.
4. Confirm the target.
5. Start the restore.
6. Watch the job progress in **Cron &amp; Jobs** if needed.

Restores are intentionally gated because they can overwrite live files, service configuration, databases, or accounts. If a restore target is incompatible, choose another server or use the correct product tab for that service.

### Creating a fresh backup

Use **Run backup** when you want a new recovery point before maintenance, upgrades, migrations, or risky changes.

If the button is unavailable, the member may need connected storage, a supported backup type, a healthy agent heartbeat, or a Pro plan.

### Compatibility rules

Backups should be restored to compatible servers.

<div id="bkmrk-backup-type-restore-"><div><table><thead><tr><th>Backup type</th><th>Restore expectation</th></tr></thead><tbody><tr><td>OpenLiteSpeed site backups</td><td>Restore to compatible OpenLiteSpeed members.</td></tr><tr><td>cPanel account backups</td><td>Restore to compatible cPanel members.</td></tr><tr><td>Database backups</td><td>Restore where the database engine and target are suitable.</td></tr><tr><td>Service configuration backups</td><td>Restore only where the operating system and service match.</td></tr></tbody></table>

</div></div>Do not use a backup to push Ubuntu service files onto a RHEL server, or cPanel account backups onto a non-cPanel host.

#### Tips

- Create a fresh backup before major work.
- Check Status before restoring.
- Check Cron &amp; Jobs after starting the restore.
- Keep storage connected and healthy so recovery points can be created automatically.
- Use Pro for production recovery, cross-member restores, and faster rollback.

# Recipes Tab

Use the **Recipes** tab to run guided setup, hardening, verification, TLS, and maintenance actions on one server. Recipes are designed for repeatable server tasks where the platform can do the routine work and then show whether the result is active, inactive, running, or needs review.

Recipes apply to the selected member only. Pool-wide features such as Protection, Managed DNS, and Storage Pools are managed from their own pool tabs.

### What recipes are

Recipes are guided actions for common server tasks:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-enable-or-disable-ss"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Enable or disable SSH access.
- Harden cPanel, OpenLiteSpeed, or database firewall rules.
- Verify cPanel services.
- Check WHM linking.
- Back up cPanel configuration.
- Run AutoSSL after failover.
- Issue Let's Encrypt certificates where supported.
- Update the balctl agent.
- Show Galera state when detected.

</div></div></div></div></div></div>The Recipes tab shows only recipes that make sense for the selected server. A cPanel server will show cPanel recipes. A database server will show database recipes. An OpenLiteSpeed server will show OpenLiteSpeed recipes.

### Free and Pro

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-feature-community-pr-7"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View available recipe cards</td><td>Included</td><td>Included</td></tr><tr><td>View Active, Inactive, Running, and Needs review state</td><td>Included</td><td>Included</td></tr><tr><td>View read-only detected recipes such as Galera state</td><td>Included</td><td>Included</td></tr><tr><td>Run agent update when available</td><td>Included where allowed</td><td>Included</td></tr><tr><td>Run service verification recipes</td><td>Upgrade may be required</td><td>Included</td></tr><tr><td>Run hardening recipes</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Run TLS and certificate recipes</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Run cPanel, OpenLiteSpeed, database, SSH, and backup actions</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Disable supported recipe changes</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div></div></div></div></div></div></div>Community is useful for visibility. Pro turns Recipes into a guided operations toolkit for production hosts.

### Recipe card states

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-state-meaning-inacti"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>State</th><th>Meaning</th></tr></thead><tbody><tr><td>**Inactive**</td><td>The recipe is available but not currently active on this member.</td></tr><tr><td>**Running**</td><td>A recipe job has been queued or is still being applied.</td></tr><tr><td>**Active**</td><td>The member reports the expected result.</td></tr><tr><td>**Needs review**</td><td>The recipe was seen before or may need attention, but the latest member report does not show it as fully active.</td></tr></tbody></table>

</div></div></div></div></div></div></div></div>If a card says the agent must be updated first, run **Update balctl agent**, wait for the member to check in again, then return to the recipe.

### Running a recipe

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-open-the-pool.-selec-2"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">1. Open the pool.
2. Select the member.
3. Open **Recipes**.
4. Choose the recipe card.
5. Select **Enable**, **Run again**, **Verify link**, or **Update agent**, depending on the card.
6. Watch the card state.
7. Check **Cron &amp; Jobs** if you want more job detail.

</div></div></div></div></div></div>Some recipes can be disabled again from the card menu. Disable actions can affect access or service behavior, so read the confirmation before continuing.

## Common recipes

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-recipe-use-it-when-e"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div><div><table><thead><tr><th>Recipe</th><th>Use it when</th></tr></thead><tbody><tr><td>**Enable SSH access**</td><td>SSH is disabled and you want the host SSH service available again.</td></tr><tr><td>**Harden cPanel ports**</td><td>You want cPanel/WHM service ports allowed and reviewed.</td></tr><tr><td>**Verify cPanel services**</td><td>You want a read-only cPanel health refresh without restarting services.</td></tr><tr><td>**WHM link check**</td><td>You want to confirm the server matches the linked WHM host and DNS setup.</td></tr><tr><td>**Backup cPanel configuration**</td><td>You want a WHM/cPanel configuration recovery point before changes.</td></tr><tr><td>**AutoSSL after failover**</td><td>DNS has moved to this cPanel host and you want AutoSSL checked.</td></tr><tr><td>**Harden database**</td><td>You want MySQL/MariaDB access rules reviewed.</td></tr><tr><td>**Harden OpenLiteSpeed**</td><td>You want HTTP, HTTPS, and WebAdmin access rules reviewed.</td></tr><tr><td>**Let's Encrypt (this server)**</td><td>You want host TLS for a domain under a linked DNS provider.</td></tr><tr><td>**Let's Encrypt (failover / HAProxy)**</td><td>You want TLS for a failover hostname on a HAProxy member.</td></tr><tr><td>**Update balctl agent**</td><td>A newer agent is available or a feature requires a newer agent.</td></tr><tr><td>**Galera cluster**</td><td>Galera has been detected and you want to see cluster-related status.</td></tr></tbody></table>

</div></div></div></div></div></div></div></div>Install recipes for cPanel, OpenLiteSpeed, and MariaDB/MySQL may be launched from the relevant product area instead of appearing as general recipe cards. After installation is detected, the matching member tab becomes available or active.

### TLS recipes

TLS recipes need a compatible DNS setup because certificate validation may require DNS changes.

Before running TLS recipes:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-make-sure-the-domain"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Make sure the domain is controlled by a saved DNS provider.
- Confirm the member is healthy and checking in.
- Update the agent if the recipe asks for a newer version.
- Know where the certificate will be used after it is issued.

</div></div></div></div></div></div>Use the Control panel or service-specific tab after the certificate is issued if the web server still needs configuration.

### Hardening recipes

Hardening recipes usually adjust or verify firewall/service access for the selected stack.

Before running a hardening recipe:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-check-the-security-t"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Check the **Security** tab so you understand current firewall state.
- Make sure SSH remains allowed from a trusted source.
- Confirm the service ports you expect customers to use.
- Create a backup where available before major changes.

</div></div></div></div></div></div>Hardening helps with standard access rules, but it does not replace a full security review.

### WHM and cPanel recipes

cPanel recipes appear when cPanel is detected.

Use them to:

<div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover" id="bkmrk-check-the-whm-link.-"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content"><div class="ui-scroll-area" data-direction="horizontal" data-scroll-padding="4" data-visibility="hover"><div class="ui-scroll-area__viewport"><div class="ui-scroll-area__content">- Check the WHM link.
- Verify cPanel services.
- Harden cPanel ports.
- Back up cPanel configuration.
- Trigger AutoSSL after failover.

</div></div></div></div></div></div>Account management still belongs in the **cPanel** tab. Scheduled standby replication still belongs in the pool **Protection** tab.

# Monitoring Tab

Use the member **Monitoring** tab to choose alert thresholds for one server. This is different from pool **Monitoring**, which handles fleet-level and protection/failover alerts.

Member Monitoring answers: when should this specific server email me because heartbeat, CPU, memory, disk, cPanel service, or cPanel account quota needs attention?

## What you configure

<div id="bkmrk-area-what-it-is-for-"><div><table><thead><tr><th>Area</th><th>What it is for</th></tr></thead><tbody><tr><td>**Heartbeat**</td><td>Email when the agent stops checking in.</td></tr><tr><td>**CPU**</td><td>Email on sustained high CPU usage.</td></tr><tr><td>**Memory**</td><td>Email on sustained memory pressure.</td></tr><tr><td>**Disk space**</td><td>Email when the root filesystem reaches a selected usage level.</td></tr><tr><td>**cPanel service alerts**</td><td>Email when selected core cPanel services are down.</td></tr><tr><td>**Hosting account disk quota**</td><td>Email when WHM accounts on the member reach a disk threshold.</td></tr><tr><td>**Recovery notifications**</td><td>Email when the member recovers after an alert state.</td></tr></tbody></table>

</div></div>Alert recipients are managed in member **Settings**. Monitoring controls what triggers an alert; Settings controls who receives it.

## Free and Pro

<div id="bkmrk-feature-community-pr"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View live monitoring state</td><td>Included</td><td>Included</td></tr><tr><td>Heartbeat alert at community timing</td><td>Included</td><td>Included</td></tr><tr><td>Faster heartbeat alert presets</td><td>Upgrade required</td><td>Included</td></tr><tr><td>CPU, memory, and disk threshold alerts</td><td>Upgrade required</td><td>Included</td></tr><tr><td>cPanel service-down alerts</td><td>Upgrade required</td><td>Included</td></tr><tr><td>cPanel hosting account quota alerts</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Recovery notification emails</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Community gives basic heartbeat awareness. Pro gives proper server operations alerts, with faster timing and resource/service thresholds.

## Alerts

### Infrastructure alerts

The first section covers the server itself.

<div id="bkmrk-alert-useful-when-he"><div><table><thead><tr><th>Alert</th><th>Useful when</th></tr></thead><tbody><tr><td>**Heartbeat**</td><td>You want to know the agent stopped checking in.</td></tr><tr><td>**CPU**</td><td>You want alerts for sustained compute pressure.</td></tr><tr><td>**Memory**</td><td>You want alerts before memory pressure affects services.</td></tr><tr><td>**Disk space**</td><td>You want warnings before the root filesystem fills.</td></tr></tbody></table>

</div></div>CPU and memory alerts need telemetry from the agent and are based on sustained usage, not one short spike.

### cPanel alerts

When cPanel is detected, Monitoring also shows cPanel-specific alert presets.

Use cPanel service alerts for services such as web, mail, database, and other core WHM services reported by the member. Use hosting account disk quota alerts to watch WHM account disk usage across accounts on that server.

These alerts are Pro features because they are production hosting controls, not just basic host reachability.

### Saving monitoring settings

1. Open the pool.
2. Select the member.
3. Open **Monitoring**.
4. Choose the alert presets you want.
5. Click **Save monitoring settings**.

If a preset is disabled, it usually requires Pro or an active trial, or the matching service has not been detected on that member.

# Settings Tab

Use member **Settings** to manage identity, allowed source IPs, alert recipients, agent update behavior, WHM access, and member removal for one enrolled server.

This tab is for the selected member only. Pool name, pool deletion, Managed DNS, Storage Pools, and Protection settings live in their own pool-level areas.

## What you can manage

<div id="bkmrk-section-what-it-is-f"><div><table><thead><tr><th>Section</th><th>What it is for</th></tr></thead><tbody><tr><td>**Member details**</td><td>Friendly name, agent hostname, and allowed source IPs.</td></tr><tr><td>**Alert recipients**</td><td>Account email toggle and additional team email addresses for monitoring alerts.</td></tr><tr><td>**Agent updates**</td><td>Installed agent version, update channel, and auto-update setting.</td></tr><tr><td>**WHM API key**</td><td>cPanel account discovery and recovery access for this member.</td></tr><tr><td>**Danger zone**</td><td>Remove the member from the pool.</td></tr></tbody></table>

</div></div>## Free and Pro

<div id="bkmrk-feature-community-pr"><div><table><thead><tr><th>Feature</th><th>Community</th><th>Pro</th></tr></thead><tbody><tr><td>View member details</td><td>Included</td><td>Included</td></tr><tr><td>View alert recipient settings</td><td>Included</td><td>Included</td></tr><tr><td>Send alerts to account email where available</td><td>Included</td><td>Included</td></tr><tr><td>Edit member name, hostname, and allowed IPs</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Add extra team alert recipients</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Change agent update channel and auto-update</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Add, edit, rotate, or remove WHM API key</td><td>Upgrade required</td><td>Included</td></tr><tr><td>Remove member from pool</td><td>Upgrade required</td><td>Included</td></tr></tbody></table>

</div></div>Community keeps settings visible. Pro gives the operational controls needed for teams and production hosting.

## Member details

Member details control how the server is identified and allowed to connect.

<div id="bkmrk-field-meaning-friend"><div><table><thead><tr><th>Field</th><th>Meaning</th></tr></thead><tbody><tr><td>**Friendly name**</td><td>Display name shown in the pool UI.</td></tr><tr><td>**Agent hostname**</td><td>Hostname the member should report as.</td></tr><tr><td>**Allowed source IPs**</td><td>Public egress IP addresses allowed to contact ServersCTL for this member.</td></tr></tbody></table>

</div></div>Be careful when editing allowed IPs. If the member's real egress IP is removed, the agent may stop checking in until the allowed list is corrected.

## Alert recipients

Settings controls who receives monitoring emails.

You can:

- Send alerts to the account email.
- Add extra team email addresses on Pro.
- Remove extra recipients.
- Save recipient changes.

The alert rules themselves are configured in the member **Monitoring** tab.

## Agent updates

The Agent updates card shows the installed agent build and lets Pro users choose:

<div id="bkmrk-setting-meaning-prod"><div><table><thead><tr><th>Setting</th><th>Meaning</th></tr></thead><tbody><tr><td>**Production**</td><td>Normal release channel for most users.</td></tr><tr><td>**Stable**</td><td>Conservative channel when available.</td></tr><tr><td>**Development**</td><td>Early channel for testing newer builds.</td></tr><tr><td>**Auto-update**</td><td>Lets the member update itself according to the selected channel.</td></tr></tbody></table>

</div></div>Use Production for normal production servers unless support asks you to use another channel.

## WHM API key

When cPanel is detected, Settings shows a **WHM API key** card.

The member WHM key is used for cPanel account discovery and local WHM operations for that server, such as account management and recovery checks. It is not the same as a DNS provider key.

Available Pro actions:

<div id="bkmrk-action-what-it-does-"><div><table><thead><tr><th>Action</th><th>What it does</th></tr></thead><tbody><tr><td>**Add WHM API key**</td><td>Links this cPanel member to WHM access.</td></tr><tr><td>**Rotate key**</td><td>Replaces the saved key.</td></tr><tr><td>**Edit**</td><td>Updates WHM connection details.</td></tr><tr><td>**Remove**</td><td>Removes the saved WHM access for this member.</td></tr></tbody></table>

</div></div>DNS cutover keys belong in Managed DNS, not in member Settings.

## Remove member

The Danger zone lets Pro users remove the member from the pool.

Removing a member detaches it from the pool so it can be enrolled again with a fresh key. Do this only when you are sure the server should no longer be managed from this pool.

Before removing a member:

- Check whether it is used by Protection, Agent Transfer, or recovery workflows.
- Confirm you no longer need its backups or history from the pool view.
- Make sure you know how to reinstall or re-enroll the agent if needed.

# Part III - Storage Pools

Store files, back up to S3, check diffs, and schedule deployments across all members.

# Introduction

## Storage Pools

Storage Pools are the central distribution platform for ServersCTL. Publish backups, configuration files and deployment packages from browsers, agents or external software, then compare, version, schedule and deploy them safely across your infrastructure.

**Storage Pools** hold things you care about in production:

- OpenLiteSpeed site and vhost backups.
- cPanel account backups.
- MariaDB / MySQL dumps.
- HAProxy, nginx, PHP, SSL, cron, firewall, and other configuration files.
- Browser uploads and deployment packages.

Each pool is a **named, regional destination** with its own quota, S3 credentials, and object browser. Storage is **account-scoped**: one login can have **multiple pools** (for example *London Recovery* and *Toronto DR*). Pools are **not** tied to a single server pool—you can send or restore from a Storage Pool to **any enrolled member** in your account when compatibility rules allow.

The product thinks in **objects** (for example *example.com vhost*, *all-databases dump*, *nginx site config*), not raw folder paths. Repeated backups of the same site become **new versions** under the same object.

<div class="qMYqUG_convSearchResultHighlightRoot" id="bkmrk-storage-quotas-stora"><div class="" data-is-intersecting="true" data-turn-id-container="d86299f2-5e8f-4713-8be7-cc36ac39db34"><section class="text-token-text-primary w-full focus:outline-none has-data-writing-block:pointer-events-none [&:has([data-writing-block])>*]:pointer-events-auto R6Vx5W_threadScrollVars scroll-mb-[calc(var(--scroll-root-safe-area-inset-bottom,0px)+var(--thread-response-height))] scroll-mt-[calc(var(--header-height)+min(200px,max(70px,20svh)))]" data-testid="conversation-turn-622" data-turn="assistant" data-turn-id="d86299f2-5e8f-4713-8be7-cc36ac39db34" data-turn-id-container="d86299f2-5e8f-4713-8be7-cc36ac39db34" dir="auto">### Where to find Storage Pools

<table><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Location

</th><th colspan="1" rowspan="1">What you get

</th></tr><tr><td colspan="1" rowspan="1">**Left sidebar → Storage Pools**

</td><td colspan="1" rowspan="1">Account overview and list of all pools

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools`**

</td><td colspan="1" rowspan="1">Storage Pools overview (usage, recent objects, live activity)

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools/new`**

</td><td colspan="1" rowspan="1">Create a new pool (region + capacity + name)

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools/{pool}`**

</td><td colspan="1" rowspan="1">**Objects** — browse, upload, discover, deploy, restore

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools/{pool}/activity`**

</td><td colspan="1" rowspan="1">Live incoming/outgoing jobs

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools/{pool}/credentials`**

</td><td colspan="1" rowspan="1">S3 endpoint and keys for WHM/cPanel/scripts

</td></tr><tr><td colspan="1" rowspan="1">**`/app/storage-pools/{pool}/settings`**

</td><td colspan="1" rowspan="1">Slug, region, quota breakdown

</td></tr></tbody></table>

The sidebar footer also shows **aggregate storage used / quota** across all pools.

</section></div></div>

# Plans, costs, and quota

Storage capacity is billed at the **account** level. Each Storage Pool has its own **assigned quota** slice of that account capacity.

### Included with Pro

<table id="bkmrk-plan-base-storage-co"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Plan

</th><th colspan="1" rowspan="1">Base storage

</th></tr><tr><td colspan="1" rowspan="1">**Community**

</td><td colspan="1" rowspan="1">**0 bytes** — you must purchase add-on capacity before backups can upload

</td></tr><tr><td colspan="1" rowspan="1">**Pro** (active subscription)

</td><td colspan="1" rowspan="1">**5 GiB included**

</td></tr><tr><td colspan="1" rowspan="1">**14-day trial**

</td><td colspan="1" rowspan="1">Same as Pro while trial is active

</td></tr></tbody></table>

When you create a Storage Pool, you can apply the **Included 5 GiB (Pro)** capacity option **once** per account (until that entitlement is fully claimed). The create wizard shows how much included storage is still available.

### Storage Pool add-ons (monthly)

If you need more space, purchase a **Storage** add-on. Prices are **per month**; checkout uses the **region** of the pool or vault you are protecting.

<table id="bkmrk-capacity-monthly-pri"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Capacity

</th><th colspan="1" rowspan="1">Monthly price

</th></tr><tr><td colspan="1" rowspan="1">10 GiB

</td><td colspan="1" rowspan="1">$4

</td></tr><tr><td colspan="1" rowspan="1">50 GiB

</td><td colspan="1" rowspan="1">$12

</td></tr><tr><td colspan="1" rowspan="1">100 GiB

</td><td colspan="1" rowspan="1">$20

</td></tr><tr><td colspan="1" rowspan="1">250 GiB

</td><td colspan="1" rowspan="1">$40

</td></tr><tr><td colspan="1" rowspan="1">500 GiB

</td><td colspan="1" rowspan="1">$65

</td></tr></tbody></table>

**How add-ons combine with Pro:** add-on bytes **add to** your account quota. A Pro account with a 10 GiB add-on has **15 GiB total** (5 GiB included + 10 GiB add-on), not 10 GiB replacing the included amount.

**Where to buy add-ons:** the **Storage Pool** tier picker appears when:

- A backup or upload is blocked for insufficient quota
- You use the replication wizard **Cloud Backup Storage** step
- Legacy pool **Storage** flows that still surface the tier cards

Purchases go through **Stripe Checkout** (`Account` billing). Upgrading an existing storage subscription changes the current Stripe item with proration instead of creating a duplicate subscription.

### What counts toward quota

- Final stored objects and backup archives
- Temporary upload staging while a large backup is still assembling (hidden from the object browser but counted until complete)
- Active upload **reservations** while an agent backup is in progress

When the quota is full, new uploads fail with a clear **insufficient storage** message until you delete objects or buy more capacity.

### Feature access by plan

<table id="bkmrk-action-community-pro"><colgroup><col></col><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Action

</th><th colspan="1" rowspan="1">Community

</th><th colspan="1" rowspan="1">Pro / active trial

</th></tr><tr><td colspan="1" rowspan="1">Browse &amp; download objects

</td><td colspan="1" rowspan="1">Yes

</td><td colspan="1" rowspan="1">Yes

</td></tr><tr><td colspan="1" rowspan="1">Run backups into storage

</td><td colspan="1" rowspan="1">Only with purchased quota

</td><td colspan="1" rowspan="1">Yes (with quota)

</td></tr><tr><td colspan="1" rowspan="1">Browser upload

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes (max **5 GiB per file**)

</td></tr><tr><td colspan="1" rowspan="1">Delete objects

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes

</td></tr><tr><td colspan="1" rowspan="1">Restore to server

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes (compatible targets)

</td></tr><tr><td colspan="1" rowspan="1">Deploy config to server

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes

</td></tr><tr><td colspan="1" rowspan="1">Configuration Discovery

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes

</td></tr><tr><td colspan="1" rowspan="1">Deployment Wizard

</td><td colspan="1" rowspan="1">Locked

</td><td colspan="1" rowspan="1">Yes

</td></tr></tbody></table>

### Regions and endpoints

When you **create** a Storage Pool, you choose a **region** (physical vault cluster). Pick the region closest to your servers and restore targets for lower latency.

### Billing regions (Stripe)

Add-on checkout is priced per vault region:

<table id="bkmrk-region-key-label-sho"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Region key

</th><th colspan="1" rowspan="1">Label shown in billing

</th></tr><tr><td colspan="1" rowspan="1">`eu-west`

</td><td colspan="1" rowspan="1">**EU West**

</td></tr><tr><td colspan="1" rowspan="1">`ca-east`

</td><td colspan="1" rowspan="1">**CA East**

</td></tr><tr><td colspan="1" rowspan="1">`sgp-central`

</td><td colspan="1" rowspan="1">**SG Central**

</td></tr></tbody></table>

Additional regions may appear in the create wizard, marked **Coming soon**.

### Your pool’s hostname

Each pool gets a dedicated endpoint on **serversctl.com** storage infrastructure:

```text
{pool-slug}.{region}.storage.serversctl.com
```

Example: slug `london-recovery` in EU West → `london-recovery.eu-west.storage.serversctl.com`

Use this hostname for:

- **WHM / cPanel → Additional Destinations → S3 Compatible**
- S3 backup tools and scripts
- Agent off-site backup authentication

**Credentials** for the pool are under **Storage Pools → {pool} → Credentials**.

# Create your first Storage Pool

1. Open **Storage Pools** in the left sidebar.
2. Click **Create Storage Pool** (or go to `/app/storage-pools/new`).
3. **Step 1 — Region:** select an available region (healthy clusters show **S3 endpoint**).
4. **Step 2 — Capacity:**
    
    
    - **Included 5 GiB (Pro)** — free, uses your Pro entitlement (shown only while remaining).
    - **Paid tiers** — 10 GiB through 500 GiB monthly options (when checkout is enabled for your account).
5. **Step 3 — Name:** give the pool a clear name (for example *Frankfurt Recovery*). The wizard suggests a name from the region.
6. Click **Create Storage Pool**.

You land on the pool **Objects** page. Activation provisions vault space, DNS, TLS, and S3 credentials for that pool.

**Tip:** use separate pools for separate purposes (production backups vs config baselines vs a second geography). Always secure your account with Two-Factor authentication.

# Navigation overview

## Account overview (`/app/storage-pools`)

- Total usage across pools
- Per-pool cards (region, used %, recent activity)
- Links into each pool’s Objects and Activity

### Per-pool tabs

<table id="bkmrk-tab-purpose-objects-"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Tab

</th><th colspan="1" rowspan="1">Purpose

</th></tr><tr><td colspan="1" rowspan="1">**Objects**

</td><td colspan="1" rowspan="1">Main work surface — browse, upload, discover, deploy, restore, delete

</td></tr><tr><td colspan="1" rowspan="1">**Activity**

</td><td colspan="1" rowspan="1">**Live** jobs only (running backups, uploads, Send To, deploy, restore)

</td></tr><tr><td colspan="1" rowspan="1">**Credentials**

</td><td colspan="1" rowspan="1">S3 URL, bucket/slug, access key, secret — for WHM and automation

</td></tr><tr><td colspan="1" rowspan="1">**Settings**

</td><td colspan="1" rowspan="1">Read-only pool metadata (slug, region, quota, default flag)

</td></tr></tbody></table>

## Objects and versions

### Object browser layout

Objects are grouped **by source server** first (member folders), not dumped in a flat list.

1. **Top level:** folders per source member (hostname / site name).
2. **Inside a member:** service folders such as **Backups**, **OpenLiteSpeed**, **MariaDB**, **NGINX**, **WordPress**, **SSL**, **Cron**, **Firewall**, **PHP**, **cPanel**, **Services**.
3. **Inside a service folder:** individual objects.

**Backups** from a server stay under that server’s folder, then **Backups** — so an old site backup does not create a second root folder for the domain name.

**Search** flattens the tree and shows matching objects across the pool.

### Object row actions

<table id="bkmrk-button-when-it-appea"><colgroup><col></col><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Button

</th><th colspan="1" rowspan="1">When it appears

</th><th colspan="1" rowspan="1">What it does

</th></tr><tr><td colspan="1" rowspan="1">**Open**

</td><td colspan="1" rowspan="1">Always

</td><td colspan="1" rowspan="1">Version history, metadata, compare, download

</td></tr><tr><td colspan="1" rowspan="1">**Deploy**

</td><td colspan="1" rowspan="1">Object is deployable (for example OLS vhost config, discovered `.conf`)

</td><td colspan="1" rowspan="1">Push config to compatible members

</td></tr><tr><td colspan="1" rowspan="1">**Restore**

</td><td colspan="1" rowspan="1">Object is a restorable backup snapshot

</td><td colspan="1" rowspan="1">Guarded restore on compatible members

</td></tr></tbody></table>

Row menu adds **Send latest to inbox**, **Deploy latest**, **Restore latest**, **Download**, **Delete** (Pro).

### Versions

Each object has one or more **versions** (v1, v2, …). Version states include **active**, **archived**, and **deprecated**. Deploy and restore the default to the **latest active** version unless you pick another in the UI.

**Open** an object to:

- See all versions with size, date, and source channel (agent backup, WHM S3, browser upload, discovery, and so on)
- **View file** — inline text preview for configs (binary/archives prompt download instead)
- **Compare** — diff against the previous version (text) or metadata/size for archives
- **Download** a specific version
- **Send**, **Deploy**, or **Restore** a specific version

# Getting data into a Storage Pool

## Managing Data

### Member backups

From any enrolled server’s member tabs:

<table id="bkmrk-source-typical-path-"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Source

</th><th colspan="1" rowspan="1">Typical path

</th></tr><tr><td colspan="1" rowspan="1">**OpenLiteSpeed → Backups**

</td><td colspan="1" rowspan="1">Full / site / web backup jobs

</td></tr><tr><td colspan="1" rowspan="1">**MariaDB → Backup**

</td><td colspan="1" rowspan="1">Database dump

</td></tr><tr><td colspan="1" rowspan="1">**cPanel → account backup**

</td><td colspan="1" rowspan="1">WHM `pkgacct` style packages

</td></tr><tr><td colspan="1" rowspan="1">**Backups tab**

</td><td colspan="1" rowspan="1">Snapshot backups

</td></tr></tbody></table>

When the job supports it, you **choose the destination Storage Pool** before the job runs. Progress appears under **Activity** (incoming lane).

Agent backups flow: **check quota → collect → dump/package → upload → confirm stored**. Install a current agent (v204+ recommended) for step-by-step byte progress.

### WHM / cPanel S3 destination

1. Open **Storage Pools → {pool} → Credentials**.
2. Copy **S3 endpoint**, **bucket** (your pool slug), **access key**, and **secret**.
3. In WHM: **Backup → Additional Destinations → S3 Compatible**.
4. Set endpoint to your pool hostname, bucket to slug, path as documented on the Credentials page.
5. Run WHM backups — they land as objects in that pool.

Quota is enforced on the vault; over-quota uploads receive **507 Insufficient Storage**.

### Browser upload (Pro)

1. Open pool **Objects**.
2. Click **Upload**.
3. Select file(s) — large files upload in **16 MiB chunks** (max **5 GiB per file**).
4. After upload, **publish**: attach as a new version of an existing object, keep as a separate object, or cancel.

### Configuration Discovery (Pro)

Baseline server configs without running a full backup:

1. On **Objects**, click **Run Discovery**.
2. Pick **one member** to scan (account-wide member list).
3. Choose categories (HAProxy, nginx, OpenLiteSpeed, PHP, SSL, MariaDB, cPanel, cron, firewall, apps, and more) or select all.
4. Options: include SSL private keys (off by default), include system package units under `/lib/systemd/system` (off by default), **force fresh baseline** (ignore agent cache).
5. Confirm — a read-only scan runs on that member only. Discovered text files become objects; **unchanged files do not create new versions**.

Live progress shows on the Objects page until complete. Requires agent **v208+** for discovery (older agents may still be selectable; the job reports the real result).

**Privacy:** application configs (WordPress, Laravel, Docker Compose, and similar) store **redacted** copies — passwords and secrets are stripped. Redacted objects show **Secrets redacted** and cannot be deployed back to production.

### Track Changes (daily discovery)

On a **member folder** row menu → **Track Changes**:

- Enables a **daily** scheduled discovery for that member and pool.
- Folder shows **Tracked daily.**
- Only **changed** files create new versions (agent v219+ cache).

### Agent CLI upload (large files)

On the server (agent **v195+**):

```bash
sudo /usr/local/bin/balctl_heartbeat.py --send-to-storage /path/to/file-or-folder
```

The agent packages the path, checks quota, uploads as an **Agent upload** object. Useful when browser limits are awkward.

### Protection replication

**Protection** jobs (cPanel / OpenLiteSpeed standby) can replicate into vault storage tied to your account quota. If space is insufficient, the replication wizard prompts for **Cloud Backup Storage** add-ons.

# Send To, Deploy, and Restore

### Send To inbox

Delivers a copy to a member’s **storage inbox** (`/var/lib/balctl/storage-inbox/`) **without** changing production files. Use for inspection or manual steps.

1. Open object or version → **Send To** (or row menu).
2. Select one or more target members.
3. Confirm — progress appears in **Activity** (outgoing).

### Deploy

Writes configuration to **production paths** on selected members. The agent:

- Backs up the current file under `/tmp/balctl-storage-deploy/`
- Writes allowlisted paths only
- Validates and reloads services where supported
- **Rolls back** on failure

**OpenLiteSpeed vhost** deploy uses a guarded vhost path. **Discovered configs** deploy to the same member or another member with matching OS/service (agent v215+).

1. Click **Deploy** on a deployable object, or **Deployment Wizard** for multiple objects.
2. Pick **version** if prompted.
3. Select **compatible targets** (incompatible members are shown but blocked).
4. Review paths, optional **reload OpenLiteSpeed**, optional advanced path override.
5. **Deploy now** or **schedule** (wizard).

A live **deploy spotlight** shows progress per target member.

### Restore

For **backup objects** linked to a snapshot (OLS site/web/full, cPanel account, database dumps):

1. Click **Restore** on object or version.
2. Select **compatible** members (OLS backups → OLS members, cPanel → cPanel, and so on).
3. Confirm — guarded restore runs on each target; **Activity** tracks progress.

Restore **replaces** production data for that backup scope — use with care.

### Deployment Wizard

**Objects → Deployment Wizard** (Pro, when deployable/restorable objects exist):

1. **Source member** — same grouping as the object browser.
2. **Objects** — pick deploy and/or restore objects (one action type per run).
3. **Targets** — compatible members only.
4. **Paths** (deploy) or **Confirm** (restore).
5. **Schedule** (deploy only) — now or future time.

# Activity, S3, Settings & Deleting Objects

## Activity tab

Shows **live work only** — not historical logs.

<table id="bkmrk-lane-examples-incomi"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Lane

</th><th colspan="1" rowspan="1">Examples

</th></tr><tr><td colspan="1" rowspan="1">**Incoming**

</td><td colspan="1" rowspan="1">Running backups, browser/S3 uploads, discovery archiving

</td></tr><tr><td colspan="1" rowspan="1">**Outgoing**

</td><td colspan="1" rowspan="1">Send To, deploy jobs, restore jobs

</td></tr></tbody></table>

Refreshes every few seconds. Completed or failed jobs disappear from Activity; the finished object remains in **Objects**.

## Credentials and S3 setup

**Storage Pools → {pool} → Credentials**

<table id="bkmrk-field-use-endpoint-s"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Field

</th><th colspan="1" rowspan="1">Use

</th></tr><tr><td colspan="1" rowspan="1">**Endpoint**

</td><td colspan="1" rowspan="1">S3 URL for WHM, rclone, restic, etc.

</td></tr><tr><td colspan="1" rowspan="1">**Bucket**

</td><td colspan="1" rowspan="1">Pool slug

</td></tr><tr><td colspan="1" rowspan="1">**Access key / Secret**

</td><td colspan="1" rowspan="1">Authentication

</td></tr></tbody></table>

Regenerate keys from the panel if compromised. Keys are scoped to your account and pool.

**WHM tips:**

- Use the **regional** endpoint hostname shown for your pool.
- Do not point WHM at a shared global server name — each pool has its own hostname and vault assignment.
- Ensure server egress IP is allowed (same network rules as agent heartbeat where applicable).

## Settings

**Settings** shows:

- **Slug** — used in URLs and S3 bucket name
- **Region** — vault cluster
- **Quota / Included** — capacity assigned to this pool
- **Default pool** — whether this pool is the account default for new uploads

Rename and delete pool flows (if exposed in UI) respect active usage — check quota and objects before removing a pool.

## Deleting objects

**Pro:** select objects with checkboxes → **Delete selected**, or delete from row menu / object details.

**Deletion removes the logical object and version history from your storage. There is no way to recover a file after deletion.**

# Troubleshooting & Glossary

### “Insufficient storage” / backup won’t start

- Check sidebar **storage used / quota**.
- Delete old objects or purchase a **Cloud Backup Storage** add-on.
- Pro: confirm you have not exhausted the **included 5 GiB** without add-ons.
- Large jobs reserve space before upload — wait for stuck jobs to expire (up to 24h) or cancel the job.

### Upload button disabled

- **Community** plan — upgrade to Pro or buy storage quota.
- Trial ended — subscribe to Pro.

### Deploy / Restore target greyed out

- Target member lacks the right stack (for example cPanel backup → non-cPanel server).
- Object is redacted or not deployable/restorable.
- Refresh members by reopening the modal (panel refreshes account member list).

### Discovery stuck or slow

- Member must be online with agent check-in.
- Watch the progress panel — phases are agent-reported (v219+).
- **Force fresh baseline** re-uploads everything and takes longer.

### WHM S3 backup fails

- Verify endpoint, bucket, keys from **Credentials**.
- Confirm quota headroom.
- Check vault region matches what you purchased in Stripe.

### Objects missing after backup “succeeded”

- Large archives may still be **assembling** — watch **Activity** until **Confirm stored**.
- Look inside the correct **member → Backups** folder.
- Use **Search** by filename or domain.

## Glossary

<table id="bkmrk-term-meaning-object-"><colgroup><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Term

</th><th colspan="1" rowspan="1">Meaning

</th></tr><tr><td colspan="1" rowspan="1">**Object**

</td><td colspan="1" rowspan="1">A named thing in a pool (site backup, config file, dump)

</td></tr><tr><td colspan="1" rowspan="1">**Version**

</td><td colspan="1" rowspan="1">A point-in-time revision of an object

</td></tr><tr><td colspan="1" rowspan="1">**Artefact**

</td><td colspan="1" rowspan="1">Internal platform type (OLS vhost, cPanel account, etc.) — you see the friendly object name

</td></tr><tr><td colspan="1" rowspan="1">**Publish**

</td><td colspan="1" rowspan="1">Finalize an upload into the object catalog

</td></tr><tr><td colspan="1" rowspan="1">**Send To**

</td><td colspan="1" rowspan="1">Copy to member inbox without production changes

</td></tr><tr><td colspan="1" rowspan="1">**Deploy**

</td><td colspan="1" rowspan="1">Write config to production with rollback

</td></tr><tr><td colspan="1" rowspan="1">**Restore**

</td><td colspan="1" rowspan="1">Replay a backup snapshot on a compatible server

</td></tr><tr><td colspan="1" rowspan="1">**Discovery**

</td><td colspan="1" rowspan="1">Read-only inventory of config files into the pool

</td></tr></tbody></table>

# Troubleshooting

# Error Messages

###   


<div class="balctl-member-action-result__actions" id="bkmrk-"></div><div class="balctl-member-action-result__actions" id="bkmrk--1"></div>

# Partner WHMCS Program

Additional ways to earn income for Partners

# Partner Program Overview & Prerequisites

## 1. Partner overview and prerequisites

The ServersCTL Partner Program provides an immediate additional income stream for anyone who operates more than one cPanel server. By installing the ServersCTL agent on each cPanel server and configuring the servers inside a **Generic Linux Pool** within the ServersCTL app, WHMCS can act as an orchestrator to manage replicated cPanel accounts.

Partner mode connects three parts:

1. A **source cPanel pool** containing the servers that currently host customer accounts.
2. One or more approved **recovery destinations** controlled by the hosting provider.
3. A paired **WHMCS installation** that handles ordering, billing, customer access and support.

The provider remains the merchant and support owner. ServersCTL supplies the replication, verification, recovery and routing control plane behind the provider's service.

Before activation, confirm:

- You are signed in as the ServersCTL account owner.
- You have a Generic Linux pool containing at least one cPanel server.
- The ServersCTL agent is enrolled and reporting a recent heartbeat from each server.
- You have an eligible second cPanel member or a linked Cloud DR recovery target.
- Every source and recovery server used by a WHMCS product is also configured as an enabled cPanel server in WHMCS.
- WHMCS uses the same normalised server hostname reported by the ServersCTL agent.

## 2. Activate your Partner account

Open **Account → Partners**.

### Existing ServersCTL account

1. Under **Import your cPanel pool**, choose the pool that hosts your WHMCS customer accounts.
2. Optionally enter your HTTPS support URL. This is shown when a customer needs assistance.
3. Select **Activate Free Partner**.
4. Review the imported pool, recovery destinations and current capacity.

Pools that are not cPanel pools, or that have no recently reporting agent, are not available for import.

### New provider starting from WHMCS

A provider without a ServersCTL login can begin from the WHMCS module connection screen:

1. Enter the company name and owner email.
2. Accept the Partner terms and submit the signed registration request.
3. ServersCTL creates a provisional Partner workspace and connects the installation.
4. Open the owner claim email within 24 hours.
5. Choose a password of at least 12 characters to verify the owner and activate the account.

Infrastructure setup and hidden product drafts are available while the account is provisional. Customer provisioning and public status publication remain locked until the owner completes the claim.

## 3. Configure pools and recovery destinations

The **Imported pool** card controls the infrastructure available to Partner integrations.

### Source pool

Choose the cPanel pool containing the live customer accounts sold from WHMCS. Account inventory is read from the enrolled cPanel servers and matched to WHMCS services by cPanel username and server identity.

Changing the source pool changes the inventory offered to the paired integration. Review existing protections before switching a live installation.

### Standby targets

Select the cPanel servers that WHMCS customers may use as recovery destinations.

- **Pool member** means another eligible cPanel member from the provider's infrastructure.
- **Cloud DR** means a linked ServersCTL recovery target.
- Leaving every target unchecked allows any eligible destination returned by the Partner API.

Free Partner uses direct server-to-server transfer and one recovery target by default. ServersCTL-hosted storage fallback requires an eligible paid Partner capability or an active Cloud DR subscription for the exact source pool and domain.

### Support URL

Enter an HTTPS URL owned by the hosting provider, such as its WHMCS support department. Unsafe URL schemes are rejected. Customer-facing Partner pages do not link customers to ServersCTL support or pricing.

## 4. Connect and manage WHMCS installations

Download the current module from **Account → Partners → WHMCS module**, then follow the [WHMCS module guide](https://docs.serversctl.com/WHMCS_PARTNER_MODULE.md).

### Approve a connection

1. Activate **ServersCTL Replication** in WHMCS.
2. Open the module and copy the displayed one-time pairing code.
3. In ServersCTL, open **Account → Partners → Connect WHMCS securely**.
4. Paste the code and select **Approve this WHMCS**.
5. Confirm that the success message shows the expected WHMCS domain and public egress IP.

The pairing code expires after 15 minutes. Approval binds the installation to the public source IP observed by ServersCTL.

### Changed WHMCS public IP

If the WHMCS server's public egress IP changes, installation authentication is rejected. Open the module to generate a fresh pairing code, then approve it from **Account → Partners**. The new approval replaces the old IP binding.

### Disconnect an installation

Only the ServersCTL account owner can disconnect a paired installation.

1. Find the installation under **WHMCS module**.
2. Select **Disconnect**.
3. Confirm the warning.

Disconnecting revokes that installation's sessions, nonces and pairing. Existing imported infrastructure, Partner customers and protection records are retained. The module must present a new one-time code before it can reconnect.

## 5. Plans, capacity and API access

The **Capacity** card shows the authoritative limits currently returned by ServersCTL:

- Protected end customers used and allowed.
- Paired WHMCS installations used and allowed.
- Custom API keys used and allowed.
- Permitted replication schedules.
- Recovery targets per customer.
- Direct-only or storage-fallback transfer capability.

The current Free Partner defaults are:

- No expiry.
- Five protected customers.
- One WHMCS installation.
- Weekly replication.
- One recovery target.
- No customer manual sync.
- Direct server-to-server transfer with no general API storage fallback.

Paid Partner capabilities and prices may change. The values shown in **Capacity** and enforced by the Partner API are authoritative.

### Upgrade

Select **Upgrade Partner** from the Capacity card to open the Partner checkout. WHMCS product pricing is still controlled separately by the hosting provider.

### Advanced API access

Permanent `sk_partner_…` keys are for custom server-to-server integrations only. The official WHMCS module uses secure pairing and does not need one.

To create a custom-integration key:

1. Select **Create key** under **Advanced API access**.
2. Copy the plaintext key immediately; it is shown once.
3. Store it in a secret manager.
4. Revoke it from the same card when it is no longer needed.

The API root shown on the page ends in `/api/partner/v1`. See the [Partner API technical reference](https://docs.serversctl.com/WHMCS_PARTNER_API.md) for routes and security requirements.

## 6. Security and data handling

- WHMCS never stores a ServersCTL password.
- A new module creates its own 2048-bit RSA installation identity.
- Pairing requests are signed and use timestamps and single-use nonces.
- Installation sessions use short-lived, scoped tokens and are locked to the approved public source IP.
- The module cannot use installation sessions to create or manage permanent Partner API keys.
- SSH passwords used by guided server enrolment remain only in the current WHMCS request and are not sent to ServersCTL.
- Cloudflare API tokens are encrypted while an order is pending, transferred through the authenticated Partner API, then removed from WHMCS after activation.
- Downstream customer pages use the provider's brand. ServersCTL plan, price and support prompts remain visible only to the provider.

## 7. Troubleshooting and FAQ

### No eligible cPanel pools

Confirm that the pool uses the Generic Linux profile, contains a cPanel server and has a recently reporting agent. Refresh **Account → Partners** after the next heartbeat.

### No standby servers available

Add another cPanel member or link an eligible Cloud DR target. Confirm the destination agent is live and cPanel is detected.

### Pairing code expired

Reload the WHMCS addon page to request a new code, then approve it within 15 minutes.

### WHMCS reconnects with a new code

The most common cause is a changed public egress IP. Verify the IP shown in ServersCTL belongs to the WHMCS host before approving it.

### An API key is requested during normal setup

Do not create or paste a permanent Partner key. Current module installations use secure pairing. The legacy-key field exists only for automatic migration of an older installation and is cleared after migration.

### Does disconnecting WHMCS delete customer replicas?

No. Disconnect revokes only the selected installation's access. Infrastructure and Partner customer/protection records remain in ServersCTL.

### Do downstream customers need ServersCTL accounts?

No. They order, pay, view status and use any permitted continuity actions inside the provider's WHMCS client area.

# ServersCTL WHMCS cPanel Replication Module

## 1. Requirements and preparation

The module is designed for self-hosted WHMCS 8.13 LTS and 9.x. WHMCS Cloud does not support the required custom add-on and server modules.

Before installation, confirm:

- PHP has the cURL and OpenSSL extensions enabled.
- WHMCS can make outbound HTTPS requests to the configured ServersCTL Partner API.
- You can write to the WHMCS `modules/addons/` and `modules/servers/` directories.
- Each production and recovery node is an enabled cPanel server in WHMCS.
- The WHMCS server hostname exactly matches the normalised hostname reported by the corresponding ServersCTL agent.
- Source and recovery servers have enrolled, recently reporting ServersCTL agents.
- The source pool's Managed DNS configuration controls the protected domains.
- The WHMCS host has a stable public egress IP, or you can approve a new pairing after it changes.

<p class="callout info">Take a normal WHMCS database and file backup before installing or upgrading any third-party module.</p>

### 1.a. DNS Requirements for Replication

Replication requires reliable access to update the domain’s DNS records. This is especially important during a high-availability event, when ServersCTL activates the replicated cPanel account after the primary server becomes unavailable.

Do not use an API key from the primary cPanel server for DNS management. If that server goes offline, ServersCTL may be unable to update the DNS records and direct traffic to the recovery server.

There are two methods for managing DNS currently. You must use one of these methods to achieve High Availability.

##### cPanel DNS Cluster

[DNS Clusters | cPanel &amp; WHM Documentation](https://docs.cpanel.net/whm/clusters/dns-cluster/)

A cPanel DNS Cluster provides centralised DNS management for your cPanel servers. DNS changes made on the cluster are propagated to its connected servers. When configured appropriately, changes made within WHM or cPanel are also synchronised with the DNS Cluster hosts.

To use a cPanel DNS Cluster with ServersCTL, add an API DNS key from one of your cluster hosts. Customers point their domains to the cluster’s nameservers, and ServersCTL uses the API key to manage the required DNS records.

##### Cloudflare DNS

Customers using Cloudflare can provide a scoped API token that grants ServersCTL permission to read and update DNS records for the relevant zone. The customer enters the API Key during the order process.

## 2. Download and install

1. Sign in to ServersCTL.
2. Open **Account → Partners**.
3. Under **WHMCS module**, select **Download WHMCS module**.
    
    
    - You can also download from the CDN. [ServersCTL WHMCS cPanel Replication Module](https://download.serversctl.com/whmcs/serversctl_replication.zip).
4. Extract `serversctl_replication.zip` at the WHMCS root.

The ZIP is rooted at `modules/` and installs both components:

<table id="bkmrk-component-installed-"><colgroup><col></col><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Component

</th><th colspan="1" rowspan="1">Installed path

</th><th colspan="1" rowspan="1">Purpose

</th></tr><tr><td colspan="1" rowspan="1">Addon module

</td><td colspan="1" rowspan="1">`modules/addons/serversctl_replication/`

</td><td colspan="1" rowspan="1">Pairing, admin workspace, inventory, product catalogue, widget and customer continuity pages

</td></tr><tr><td colspan="1" rowspan="1">Server module

</td><td colspan="1" rowspan="1">`modules/servers/serversctl_protection/`

</td><td colspan="1" rowspan="1">WHMCS provisioning and lifecycle actions for protection services

</td></tr></tbody></table>

Do not move the bundled server module into the addon directory.

### Activate the addon

1. In WHMCS, open **System Settings → Addon Modules**.
2. Find **ServersCTL Replication**.
3. Select **Activate**.
4. Grant access to the required WHMCS administrator roles.
5. Open the module from the WHMCS addon menu.

Activation creates the module's local mapping tables and the editable **ServersCTL Replication - DNS Cutover** email template. Deactivating the addon later does not stop existing protection at ServersCTL, but WHMCS management is unavailable until it is reactivated.

## 3. Pair WHMCS with ServersCTL

The current module uses cryptographic installation pairing. Do not create a permanent Partner API key for normal setup.

1. Open **ServersCTL Replication** in WHMCS.
2. Copy the one-time pairing code and note the displayed public source IP.
3. In a separate browser tab, sign in to ServersCTL as the account owner.
4. Open **Account → Partners**.
5. Paste the code under **Connect WHMCS securely**.
6. Select **Approve this WHMCS**.
7. Return to WHMCS and reload the addon.

The code expires after 15 minutes. Approval binds the module identity to the WHMCS domain and observed public egress IP. ServersCTL passwords and permanent API keys never enter WHMCS.

### Create a new Partner account from WHMCS

If the provider has no ServersCTL account, use the registration form shown with the pairing code. Enter the company name and owner email, accept the terms, then open the emailed 24-hour claim link. The owner must set a password of at least 12 characters.

Customer provisioning remains locked until the claim is completed.

## 4. Configure the addon

Open **System Settings → Addon Modules → ServersCTL Replication → Configure**.

<table id="bkmrk-setting-recommended-"><colgroup><col></col><col></col><col></col></colgroup><tbody><tr><th colspan="1" rowspan="1">Setting

</th><th colspan="1" rowspan="1">Recommended value

</th><th colspan="1" rowspan="1">Effect

</th></tr><tr><td colspan="1" rowspan="1">**API Base URL**

</td><td colspan="1" rowspan="1">Production Partner API URL ending in `/api/partner/v1`

</td><td colspan="1" rowspan="1">Selects the ServersCTL control plane

</td></tr><tr><td colspan="1" rowspan="1">**Legacy API Key (migration only)**

</td><td colspan="1" rowspan="1">Empty

</td><td colspan="1" rowspan="1">Used once only when upgrading an old key-based installation

</td></tr><tr><td colspan="1" rowspan="1">**Customer-facing brand name**

</td><td colspan="1" rowspan="1">Your hosting brand

</td><td colspan="1" rowspan="1">Overrides the WHMCS company name in customer continuity pages

</td></tr><tr><td colspan="1" rowspan="1">**Customer-facing logo URL**

</td><td colspan="1" rowspan="1">HTTPS SVG or PNG

</td><td colspan="1" rowspan="1">Overrides the WHMCS theme logo; a transparent image at least 320 px wide is recommended

</td></tr><tr><td colspan="1" rowspan="1">**Dark continuity experience**

</td><td colspan="1" rowspan="1">On

</td><td colspan="1" rowspan="1">Uses the dark continuity workspace; disable to retain the light WHMCS presentation

</td></tr><tr><td colspan="1" rowspan="1">**WHMCS status-area visibility**

</td><td colspan="1" rowspan="1">Off initially

</td><td colspan="1" rowspan="1">`authenticated` or `public` exposes only aggregate, redacted status data to compatible templates

</td></tr><tr><td colspan="1" rowspan="1">**Customer target choice**

</td><td colspan="1" rowspan="1">Automatic

</td><td colspan="1" rowspan="1">`choose-one` or `choose-multiple` lets customers select from approved destinations

</td></tr><tr><td colspan="1" rowspan="1">**Allow Cloudflare DNS**

</td><td colspan="1" rowspan="1">As required

</td><td colspan="1" rowspan="1">Lets customers choose Cloudflare and enter a scoped token during purchase

</td></tr><tr><td colspan="1" rowspan="1">**Allow customer test cutover**

</td><td colspan="1" rowspan="1">According to policy

</td><td colspan="1" rowspan="1">Shows guarded recovery and failback controls to eligible customers

</td></tr><tr><td colspan="1" rowspan="1">**Customer recovery action**

</td><td colspan="1" rowspan="1">Replicate &amp; Cut Over

</td><td colspan="1" rowspan="1">Takes and restores a fresh copy before DNS movement; DNS only uses the latest verified copy

</td></tr><tr><td colspan="1" rowspan="1">**DNS cutover email notifications**

</td><td colspan="1" rowspan="1">On

</td><td colspan="1" rowspan="1">Sends the editable WHMCS template after manual or automatic routing changes

</td></tr><tr><td colspan="1" rowspan="1">**Offer weekly/daily/8-hour/4-hour/hourly replication**

</td><td colspan="1" rowspan="1">Match your catalogue

</td><td colspan="1" rowspan="1">A frequency appears only when enabled, priced in WHMCS and allowed by the Partner plan

</td></tr></tbody></table>

The Partner API is the feature authority. Editing the open-source PHP module cannot grant schedules, targets, manual sync or storage capability that the Partner account does not have.

## 5. Connect servers and destinations

Open **ServersCTL Replication → Servers &amp; destinations**.

The setup sequence is:

1. **Connect servers** — install the ServersCTL agent on each cPanel server involved.
2. **Choose account servers** — select the source pool containing live WHMCS accounts.
3. **Allow destinations** — choose the recovery cPanel servers customers may use.
4. **Protect accounts** — return to **Account protection** and enable services.

### Existing ServersCTL fleet

Select the existing source cPanel pool and approved recovery destinations. Add the provider's HTTPS support URL, then save.

### New fleet

The guided setup can create a cPanel pool and enrol a server:

1. Enter the server connection details.
2. Verify the displayed SSH host fingerprint against a trusted source.
3. Explicitly confirm the fingerprint.
4. Allow WHMCS to check root privilege, supported Linux/cPanel software and available disk space.
5. Wait for the agent installation and fresh heartbeat.

The SSH password is held only for the current PHP request and is not saved or sent to ServersCTL. WHMCS constructs the fixed install command locally and verifies the signed release manifest and file hashes before executing the downloaded agent as root.

## 6. Create and price the protection catalogue

From **Servers &amp; destinations**, select **Create protection catalogue**.

The action is idempotent and can safely repair an existing catalogue. It creates:

- Weekly replication addon.
- Daily replication addon.
- 8-hour replication addon.
- 4-hour replication addon.
- Hourly replication addon.
- Additional Replication Target addon.

Weekly pricing is created at zero by default. Premium frequency addons remain disabled until the provider sets recurring prices in WHMCS.

### Configure sales

1. Open the generated WHMCS product addons.
2. Set recurring prices for every billing cycle and currency you intend to sell.
3. Enable only the addon frequencies you want customers to order.
4. Confirm those same frequencies are enabled in the ServersCTL Replication addon settings.
5. Verify the Partner Capacity card allows them.

Customers see only the intersection of host-enabled, WHMCS-priced and Partner-authorized frequencies. The first selected destination is included in the frequency addon; each additional selected destination adds one **Additional Replication Target** line.

Do not manually create or request internal ServersCTL product IDs.

## 7. Protect existing accounts

Open **ServersCTL Replication → Account protection**.

The inventory cross-references active WHMCS shared-hosting and reseller services with live cPanel `listaccts` data. It identifies protected, unprotected, missing and unmatched accounts.

To protect a matched service:

1. Find the customer and cPanel account.
2. Select an approved recovery destination.
3. Choose the replication policy and permitted customer actions.
4. Start protection.
5. Confirm that the account reports an active protection state and a recovery point after the first completed copy.

### Admin-managed accounts

Select **Manually protect an account** to show discovered cPanel accounts that do not match a WHMCS service. These are labelled **Admin-managed**.

Protecting an Admin-managed account does not create or modify a WHMCS client, hosting service, order or invoice. It is excluded from customer ordering.

### Other administration tabs

- **Infrastructure map** embeds the real, read-only and redacted cPanel pool overview, including linked Cloud DR recovery agents referenced by the pool's protection jobs. Use **Open fullscreen** for the wallboard view.
- **Servers &amp; destinations** controls source, recovery and support routing and creates the protection catalogue.
- **DNS credentials** manages per-customer Cloudflare credentials and bindings. Customers are labelled by domain, cPanel account, WHMCS customer and service number.

## 8. Customer ordering and continuity console

The customer opens the provider-branded **Hosting Protection** or **Account Continuity** page in WHMCS.

The purchase wizard:

1. Lists eligible active shared-hosting or reseller accounts owned by that customer.
2. Lets the customer choose an available replication frequency.
3. Uses automatic target selection by default, or offers approved targets according to module policy.
4. Lets the customer choose cPanel DNS or Cloudflare when enabled.
5. Creates an addon-only WHMCS order attached to the parent hosting service.
6. Provisions protection after invoice payment or staff activation. ServersCTL activation completes before WHMCS marks the order active; a rejected remote activation leaves the add-on pending for a safe retry. Retrying that account reuses the same add-on and cannot create duplicate active protection billing.

For Cloudflare, the customer must supply the 32-character Account ID and an API token with Zone DNS read/edit access to the exact zone. Successful credential verification immediately creates the domain-specific Managed DNS binding, even when a later protection step needs to be retried. The secret is encrypted only while the order is pending and is removed from WHMCS after successful activation.

Weekly zero-value orders are closed as paid immediately and do not remain in the customer's unpaid-invoice count.

The continuity console can show:

- Current live and standby server roles.
- Last protected copy and replication schedule.
- Recovery readiness and recent activity.
- Manual replication when allowed.
- DNS and credential state.
- Guarded recovery and failback when allowed.
- A customer self-service **Remove protection** action when safe.

Customer pages use the provider's brand. They do not display ServersCTL pricing, upgrade prompts or support links.

## 9. Recovery, DNS and lifecycle behavior

### Recovery requirements

Recovery preflight checks:

- An active protected customer and valid entitlement.
- A completed protected copy.
- A reachable receiving agent and standby endpoint.
- Website/TLS readiness.
- Authoritative DNS control for the protected domain.
- An exact enabled WHMCS cPanel server matching the destination hostname.

For host-managed DNS, customer readiness messages use the WHMCS company/Customer Brand Name only. Credential labels, DNS hostnames and key timestamps remain operator-only.

The confirmation is one-time and expires. Never use a stale preflight for a later cutover.

### DNS only

Uses the latest verified protected copy, then moves all enabled cPanel account A records together. This includes the apex and service names such as `mail`, `webmail`, `cpanel` and `ftp`.

### Replicate &amp; Cut Over

Starts a durable workflow that:

1. Takes a fresh full account copy.
2. Restores it to the recovery server.
3. Waits for agent, DNS, HTTP and TLS readiness.
4. Re-runs safety checks.
5. Moves authoritative DNS.
6. Updates the WHMCS hosting service's assigned server.

The workflow continues if the browser closes. A workflow is claimed by only one evaluator before routing changes begin, so overlapping scheduler runs cannot repeat confirmation or overwrite a successful cutover with a stale-route error. After cutover, native WHMCS cPanel login and quick actions use the active recovery host. Failback restores DNS and the WHMCS server assignment to production. If the synchronous WHMCS request is delayed after routing changes, the modal polls authoritative state and completes as soon as the requested route is visible.

### Email notifications

When enabled, manual and automatic cutovers send **ServersCTL Replication - DNS Cutover** through WHMCS. Edit this template under the normal WHMCS email-template settings. Duplicate notifications for the same active server are suppressed.

### Suspension, cancellation and termination

- Addon suspension pauses the customer's protection access.
- Unsuspension resumes it.
- Addon cancellation or termination requests guarded cleanup.
- Successful parent cPanel service termination cascades cleanup to mapped protection addons.
- Future-dated cancellation does not remove protection before WHMCS terminates the service.
- Cleanup waits if DNS is still routed to recovery and is retried by daily WHMCS maintenance after the safety condition is resolved.
- Removing protection leaves the parent hosting service intact.
- Open continuity consoles poll live state and automatically reconcile to the unprotected view when protection is removed in another tab or by staff; customers do not need to refresh manually.

The WHMCS cron is also used to reconcile the hosting service's assigned server with the authoritative DNS-active node after automatic cutover or an interrupted request.

## 10. Upgrade, disconnect and deactivate

### Upgrade

1. Back up WHMCS.
2. Download the latest ZIP from **Account → Partners**.
3. Extract it over the existing module directories.
4. Open the addon and verify health, pairing and the installed version.
5. Run **Create protection catalogue** to repair catalogue definitions without duplicating them.
6. Test one non-production or designated recovery-test service.

An installation upgraded from an old key-based release can use its legacy Partner key once to bind the exact installation identity. The module clears the key after successful migration. Old API values containing only the ServersCTL site origin are resolved to `/api/partner/v1`.

### Disconnect

Disconnect from **ServersCTL → Account → Partners**, not by deleting database rows. Disconnect revokes the selected installation's access while retaining imported infrastructure and protection records.

### Deactivate

WHMCS addon deactivation leaves active protection running at ServersCTL. Reactivate and pair the addon to manage it again.

## 11. Troubleshooting

### Pairing code is missing or expired

Reload the addon page and request a fresh code. Approve it within 15 minutes while signed in as the ServersCTL account owner.

### Module returns an IP or authentication error

Compare the WHMCS server's current public egress IP with the IP shown under **Account → Partners**. A changed IP requires fresh pairing.

### “PHP OpenSSL is required”

Enable the PHP OpenSSL extension for the PHP runtime serving WHMCS, then restart the relevant PHP/web service and reactivate the addon.

### Connection fails

Confirm the API URL is HTTPS and ends in `/api/partner/v1`, PHP cURL is enabled, DNS resolves and the WHMCS server can make outbound HTTPS requests. Do not work around the error by adding a permanent Partner key.

### No accounts are listed

Confirm:

- The source pool is selected.
- Its cPanel agents are live.
- The WHMCS products are shared-hosting or reseller services in Active or Suspended state.
- The WHMCS cPanel username matches a discovered live account.
- The cPanel server hostname matches the ServersCTL agent hostname.

Use **Manually protect an account** only for accounts intentionally not sold through WHMCS.

### A frequency does not appear to customers

It must be enabled in addon settings, have a valid recurring WHMCS price and be allowed by the Partner plan. Check all three.

### Recovery preflight cannot map the destination

Add the recovery node as an enabled cPanel server in WHMCS. Its hostname must exactly match the hostname reported by the ServersCTL agent.

### Cloudflare recovery is disconnected

Open the customer's DNS page, add a scoped Cloudflare credential and bind it to the exact protected domain. Credentials belonging to another customer or the provider's general DNS vault are not used as fallback.

### Cancellation remains pending

Check whether authoritative DNS still points to recovery. Return service to production first. Daily WHMCS maintenance retries cleanup after the safety condition is satisfied.

### Safe release test

Use only a designated recovery-test account. Verify a completed fresh copy, recovery preflight, authoritative DNS movement, HTTP response, WHMCS server reassignment and successful failback. Never run destructive recovery testing against an unrelated customer.

# Uninstall The Agent

## Fully remove the ServersCTL agent

Use this runbook when a server must stop heartbeating to ServersCTL/BalCTL and all local agent files should be removed before re-enrolling, moving between staging and production, or retiring the host.

The uninstall removes:

- `balctl-heartbeat.service`
- `/etc/balctl/agent.env` and local discovered credential cache
- `/usr/local/bin/balctl_heartbeat.py`
- local agent state under `/var/lib/balctl`
- local agent logs and temporary installer files when present

It does not remove customer workloads or platform software such as cPanel, OpenLiteSpeed, HAProxy, MariaDB/MySQL, Nginx, firewalld, CSF, websites, databases, DNS zones, or backup archives outside `/var/lib/balctl`.

Before removing the agent, delete or disable the matching member in the panel if the server is not being re-enrolled. That revokes the old enrollment path and stops the UI from expecting more heartbeats from this host.

### Debian and Ubuntu

Run as `root`, or paste as a user with `sudo` access:

```
sudo systemctl stop balctl-heartbeat.service 2>/dev/null || true
sudo systemctl disable balctl-heartbeat.service 2>/dev/null || true
sudo systemctl stop balctl-agent-post-update.service 2>/dev/null || true
sudo systemctl reset-failed balctl-heartbeat.service balctl-agent-post-update.service 2>/dev/null || true

sudo rm -f /etc/systemd/system/balctl-heartbeat.service
sudo rm -f /etc/systemd/system/balctl-agent-post-update.service
sudo systemctl daemon-reload

sudo rm -f /usr/local/bin/balctl_heartbeat.py
sudo rm -f /usr/local/bin/balctl-agent.sh
sudo rm -rf /etc/balctl
sudo rm -rf /var/lib/balctl
sudo rm -rf /var/log/balctl
sudo rm -rf /tmp/balctl-* /tmp/balctl_agent_* /tmp/balctl-storage-deploy
```

If the original installer was extracted in the current directory, remove the extracted bundle too:

```
rm -f agent.zip balctl-agent.sh balctl_heartbeat.py balctl-heartbeat.service balctl-agent-version.txt
rm -rf systemd
```

Verify removal:

```
systemctl status balctl-heartbeat.service --no-pager
systemctl list-unit-files | grep -i balctl || true
pgrep -af balctl_heartbeat || true
test ! -e /etc/balctl/agent.env && echo "agent.env removed"
test ! -d /var/lib/balctl && echo "agent state removed"
```

Expected result: `balctl-heartbeat.service` is not found or inactive, no `balctl_heartbeat.py` process is listed, `/etc/balctl/agent.env` is gone, and `/var/lib/balctl` is gone.

Do not remove `python3`, `unzip`, `wget`, or `curl` unless you are certain they were installed only for the agent. They are common system tools and may be required by the OS or other services.

### RHEL, AlmaLinux, Rocky Linux, and CentOS

Run as `root`, or paste as a user with `sudo` access:

```
sudo systemctl stop balctl-heartbeat.service 2>/dev/null || true
sudo systemctl disable balctl-heartbeat.service 2>/dev/null || true
sudo systemctl stop balctl-agent-post-update.service 2>/dev/null || true
sudo systemctl reset-failed balctl-heartbeat.service balctl-agent-post-update.service 2>/dev/null || true

sudo rm -f /etc/systemd/system/balctl-heartbeat.service
sudo rm -f /etc/systemd/system/balctl-agent-post-update.service
sudo systemctl daemon-reload

sudo rm -f /usr/local/bin/balctl_heartbeat.py
sudo rm -f /usr/local/bin/balctl-agent.sh
sudo rm -rf /etc/balctl
sudo rm -rf /var/lib/balctl
sudo rm -rf /var/log/balctl
sudo rm -rf /tmp/balctl-* /tmp/balctl_agent_* /tmp/balctl-storage-deploy
```

If the original installer was extracted in the current directory, remove the extracted bundle too:

```
rm -f agent.zip balctl-agent.sh balctl_heartbeat.py balctl-heartbeat.service balctl-agent-version.txt
rm -rf systemd
```

Verify removal:

```
systemctl status balctl-heartbeat.service --no-pager
systemctl list-unit-files | grep -i balctl || true
pgrep -af balctl_heartbeat || true
test ! -e /etc/balctl/agent.env && echo "agent.env removed"
test ! -d /var/lib/balctl && echo "agent state removed"
```

Expected result: `balctl-heartbeat.service` is not found or inactive, no `balctl_heartbeat.py` process is listed, `/etc/balctl/agent.env` is gone, and `/var/lib/balctl` is gone.

Do not remove `python3`, `python39`, `unzip`, `wget`, or `curl` Unless you are certain they were installed only for the agent. RHEL-family systems often rely on Python packages for operating system tooling.

### Re-enrol after removal

After a full removal, use the current install command from the panel. Do not reuse an old command or old enrollment secret.

Production enrollments should point at:

- API base: `https://serversctl.com`
- Agent zip: `https://serversctl.com/agent/production/agent.zip`

Staging enrollments should point at:

- API base: `https://staging.serversctl.com`
- Agent zip: `https://staging.serversctl.com/api/agent-assets/development/agent.zip`

The hostname in the command must match the panel member hostname, and the server's egress IP must be in that member's allowed source IP list.