Salt on macOS — How kri Deploys and Operates the Fleet Control Plane

PKG install · launchd plists · PAM auth · rest_cherrypy TLS · multi-master failover

Install & Service

Installing Salt-Master + Running It as a Service

kri orchestrates the entire Salt infrastructure via Ansible: when you provision a new Mac Mini as a salt-master, kri’s API dispatches an Ansible playbook from inside a Docker container that installs, configures, and starts salt-master and salt-api as native launchd daemons on the bare metal. Minions are other Mac Minis on the same LAN; they connect to the master over ZeroMQ (ports 4505/4506) using direct LAN IPs — no DNS or VPN required. kri’s API then controls the fleet exclusively through salt-api over HTTPS, keeping the Docker boundary clean.

End-to-End Architecture — kri → Ansible → Salt

Why Onedir PKG?

Salt’s “onedir” bundle is a self-contained /opt/salt/ tree — Python interpreter, Salt, all dependencies — installed as a macOS PKG. No Homebrew, no system Python, no pip.

Benefits for an air-gapped fleet: the .pkg is bundled inside the kri docker image under playbooks/files/, copied to the target via Ansible’s copy: module, and installed with installer -pkg. Zero outbound internet from the Mac Mini.

salt_version: "3007.14"
# Bundled at build time:
playbooks/files/salt-3007.14-py3-arm64.pkg

Role Task Order

• 1

  install_macos.yml

  Copies & installs the PKG, sets PATH

• 2

  configure.yml

  Writes /etc/salt/master.d/kri.conf + salt-api.conf

• 3

  pki.yml

  Creates PKI directories for master key

• 4

  pillar.yml / states.yml

  Creates /etc/salt/pillar + /etc/salt/states

• 5

  api_user.yml

  Creates krisalt system user (PAM)

• 6

  api_tls.yml

  Self-signed cert/key for salt-api HTTPS

• 7

  service_macos.yml

  Writes plists, loads + starts both daemons

macOS File Layout After Install

/opt/salt/ # onedir bundle — binary + Python salt-master salt-api salt-key salt-call salt-minion Python lib/ … /etc/salt/ # configuration (owned root:wheel) master.d/ kri.conf # interface, pillar_roots, external_auth, netapi_enable_clients salt-api.conf # rest_cherrypy port + TLS paths pki/ master/ # master keypair (auto-generated on first start) api/ salt-api.crt # self-signed TLS cert salt-api.key # private key (never committed) pillar/ # per-minion pillar data (ingest_url, node_token) states/ # salt states (grain_report, heartbeat …) /Library/LaunchDaemons/ # launchd plist files com.saltstack.salt.master.plist com.saltstack.salt.api.plist /var/log/salt/ # log files master master-error api api-error

The launchd Plist — Why Salt Starts Automatically

macOS uses launchd as its init system. A plist in /Library/LaunchDaemons/ (root-owned) defines a system-level daemon that launchd manages. Two keys make salt survive reboots:

<key>RunAtLoad</key>   <!-- start the daemon when launchd loads the plist (boot or launchctl load) -->
<true/>

<key>KeepAlive</key>   <!-- restart the process automatically if it exits -->
<true/>

<key>ThrottleInterval</key>
<integer>5</integer>  <!-- wait 5 s before restart-on-crash to avoid tight loops -->

RunAtLoad means launchd starts salt-master immediately when the plist is loaded — at boot, or via launchctl load -w. The -w flag removes any Disabled key so it persists across reboots.

KeepAlive means launchd will restart the process if it crashes or exits — exactly like Restart=always in systemd.

salt-master.plist (condensed)

<key>Label</key>
<string>com.saltstack.salt.master</string>

<key>ProgramArguments</key>
<array>
  <string>/opt/salt/salt-master</string>
  <string>--log-level=info</string>
</array>

<key>UserName</key>
<string>root</string>  # manages PKI dirs
<key>RunAtLoad</key> <true/>
<key>KeepAlive</key>  <true/>

salt-api.plist (condensed)

<key>Label</key>
<string>com.saltstack.salt.api</string>

<key>ProgramArguments</key>
<array>
  <string>/opt/salt/salt-api</string>
  <string>--log-level=info</string>
</array>

<key>UserName</key>
<string>root</string>
<key>RunAtLoad</key> <true/>
<key>KeepAlive</key>  <true/>

Ansible Loads the Daemon with launchctl

Writing the plist file is not enough — launchd must be told to pick it up. The role does:

# Unload first (idempotent — ignore errors)
/bin/launchctl unload /Library/LaunchDaemons/com.saltstack.salt.master.plist 2>/dev/null || true
# -w clears the Disabled key so it persists across reboots
/bin/launchctl load -w /Library/LaunchDaemons/com.saltstack.salt.master.plist
# Explicit start (redundant with RunAtLoad, but safe)
/bin/launchctl start com.saltstack.salt.master

The same sequence applies to com.saltstack.salt.api. The role waits for port 4505 (master) and port 4507 (api) to open before continuing, so Ansible fails fast if the daemon doesn’t come up.

Important: On macOS 13+ (Ventura+), prefer launchctl kickstart -k system/com.saltstack.salt.master for restarting rather than stop+start. It is atomic and avoids the window where the service is stopped but not yet restarted.

Boot → Salt Ready — Service Start Flow

Managing the Service Day-to-Day

Action	Command
Check status	launchctl list | grep salt
Restart master	sudo launchctl kickstart -k system/com.saltstack.salt.master
Restart api	sudo launchctl kickstart -k system/com.saltstack.salt.api
Stop master	sudo launchctl bootout system/com.saltstack.salt.master
Load plist again	sudo launchctl bootstrap system /Library/LaunchDaemons/com.saltstack.salt.master.plist
Check master log	tail -f /var/log/salt/master
List minion keys	sudo /opt/salt/salt-key -L

Minion Bootstrap

How the Minion Finds Its Master

There’s no DNS magic, no service discovery, no consul. The minion finds its master because Ansible writes the master’s IP address directly into /etc/salt/minion during bootstrap. Here’s the complete picture.

/etc/salt/minion — The Config That Wires Everything

The bootstrap playbook (bootstrap_mac_mini.yml) writes this config on every node:

master:
  - 192.0.2.64    # mm (primary salt-master, macOS)
  - 192.0.2.10    # cylon (secondary, Linux)

master_type: failover    # try masters in order, switch on failure
master_alive_interval: 60  # probe master every 60 s
master_tries: -1           # keep trying forever (don't give up)
random_master: True        # randomise order on start (load spread)

id: Abhisheks-Mac-mini     # minion ID — set at bootstrap, stable
log_level: info
log_file: /var/log/salt/minion
encryption_algorithm: OAEP-SHA1  # required by salt 3006+

No /etc/hosts entry needed. The master address is a LAN IP (e.g. 192.0.2.64). The minion resolves it via normal unicast routing — no DNS, no host file. This is intentional: adding a hostname-based entry would require DNS infrastructure that may not exist. IP addresses are stable for fixed Mac Minis.

Minion → Master Authentication Flow (First Contact)

Key Pre-Seeding

On first contact, the minion sends its public key to the master and waits for acceptance (TOFU — Trust On First Use). This can take minutes if an admin has to manually accept.

kri optionally pre-seeds the master’s public key onto the minion at bootstrap time:

# Written to:
/etc/salt/pki/minion/minion_master.pub

# Sourced from:
deploy/salt-pki/master.pub  # stable across provisions

With the master’s key already trusted, the minion skips re-authentication on reconnect — it just verifies the master’s signature and connects immediately.

Multi-Master Failover

When master_type: failover is set with a list of masters, the minion:

• 1

  Connect to first reachable master

  From the random-shuffled list (random_master: True)

• 2

  Probe master every 60 s

  master_alive_interval checks the connection is still healthy

• 3

  Failover on detect

  If probe fails, minion tries the next master in the list

• 4

  Never gives up

  master_tries: -1 means infinite retry — good for reboots

Note: master_shuffle: True was renamed to random_master: True in Salt 3006. Always use random_master with 3007.

How Ansible Builds the Master List

The bootstrap playbook receives salt_masters as an extra-var (list of IPs). It uses a Jinja2 loop to produce the YAML list format:

# In bootstrap_mac_mini.yml
master:
{% for m in salt_masters %}
  - {{ m }}
{% endfor %}

kri passes salt_masters from the SaltMaster DB table — it queries all enabled masters and builds the list at dispatch time. This means the minion config always reflects the current set of kri-managed masters, not a hardcoded value.

Why OAEP-SHA1?

Salt 3006+ changed the default encryption algorithm for the minion→master authentication handshake. The minion must declare the algorithm explicitly or authentication silently fails.

Salt Version	Default Algorithm	Config Required
< 3006	OAEP (implied)	No
≥ 3006	None enforced — must declare	encryption_algorithm: OAEP-SHA1

Without this key, Salt 3007 minions connecting to a Salt 3007 master produce confusing auth errors. Always set it explicitly.

Salt-API & Auth

salt-api: The HTTP Control Plane

salt-api is a CherryPy-based HTTP server that wraps the salt bus. kri’s API/worker containers use it to run commands on minions, manage keys, and query fleet status — all over HTTPS from a Docker container to the native salt-master on the Mac Mini.

Why salt-api Exists

Salt’s native transport (ZeroMQ) is not accessible from Python code running in a separate process or container without installing the full salt Python package. salt-api solves this: it exposes an HTTP/JSON interface that any language or container can call.

kri uses the rest_cherrypy backend (salt’s built-in WSGI server). It handles auth, dispatches to the salt bus, and returns JSON.

Salt 3006+ Breaking Change: All netapi clients (local, local_async, runner, wheel) are disabled by default. Every client kri uses must be explicitly enabled in master config. Without this, salt-api returns 400 Client disabled even with valid credentials.

salt-api.conf

rest_cherrypy:
  port: 4507
  disable_ssl: false
  ssl_crt: /etc/salt/pki/api/salt-api.crt
  ssl_key: /etc/salt/pki/api/salt-api.key
  webhook_disable_auth: false

TLS is terminated at salt-api itself. The cert is self-signed (generated by api_tls.yml). kri connects with verify=False since the cert is internal — the encryption is still active, only verification is skipped.

netapi_enable_clients

# In /etc/salt/master.d/kri.conf
netapi_enable_clients:
  - local        # target.function calls
  - local_async  # non-blocking exec
  - runner       # fleet-wide status
  - wheel        # key management

Without all four entries, kri’s probe and command endpoints return 400 or 500. This is the most common misconfiguration on an existing salt-master being brought into kri.

Authentication — PAM User + external_auth ACL

salt-api supports multiple auth backends (eauth). kri uses PAM — the simplest approach: a real OS-level user whose credentials are verified by Linux/macOS’s Pluggable Authentication Modules.

The role creates a dedicated system account:

# api_user.yml (macOS)
sysadminctl -addUser krisalt \
  -fullName "kri Salt API" \
  -password "{{ kri_salt_api_password }}" \
  -roleAccount     # no home dir, no login shell

Then kri.conf maps this user to an ACL via external_auth:

external_auth ACL — Least-Privilege by Design

external_auth:
  pam:
    krisalt:
      # Local client — execute these salt functions on minions
      - 'test.ping'
      - 'grains.items'
      - 'state.apply'
      - 'cmd.run'
      - 'pkg.install'
      - 'service.restart'
            … (full function list in kri.conf) …

      # Wheel client — key operations only
      - '@wheel':
        - 'key.list_all'
        - 'key.accept'
        - 'key.reject'
        - 'key.delete'

      # Runner client — fleet status only
      - '@runner':
        - 'manage.up'
        - 'manage.versions'
        - 'manage.status'

The ACL enforces that krisalt can only call functions in this list — nothing else, regardless of what kri’s API sends. This is the second security layer after the PAM password check.

kri → salt-api → salt-master — Request Flow

Client Types — What kri Actually Calls

Client	What It Does	Example Call	ACL Prefix
local	Execute a function on targeted minion(s) (synchronous)	test.ping, state.apply	(function name directly)
local_async	Same as local but returns a job ID immediately	Long-running state runs	(function name directly)
runner	Fleet-wide status commands that run on the master itself	manage.up, manage.versions	@runner
wheel	Manage master-side state: keys, configs	key.list_all, key.accept	@wheel

kri’s Probe — How It Verifies a Master Is Healthy

kri runs a structured probe (SaltMasterProbe) whenever “Test connection” is clicked or the background poll fires. It checks in order:

• 1

  DNS

  socket.getaddrinfo(address) — the master’s IP/hostname resolves

• 2

  TCP 4505 / 4506

  ZeroMQ publish and return ports are open and accepting connections

• 3

  salt_api_auth

  POST to /run with client=runner, fun=manage.up — confirms auth works and the runner client is enabled

• 4

  key_store

  POST with client=wheel, fun=key.list_all — confirms wheel access and PKI is intact

• 5

  version / minions_up

  manage.versions and manage.up — reports fleet version consistency and live minion count

All 7 checks must pass for the master to show as healthy in kri. A single fail makes it unreachable; a warning makes it degraded.

Common Failure Modes and Fixes

Symptom	Root Cause	Fix
salt_api_auth: 400 Client disabled	netapi_enable_clients missing from master config	Add all 4 clients to /etc/salt/master.d/kri.conf and restart
salt_api_auth: 500 on runner.X	Function not in @runner ACL, or calling a local function via runner client	Verify function name is in @runner ACL (e.g. manage.up not test.ping)
key_store: 500	Calling key.list_all via client=runner (should be wheel)	Use client=wheel for all key.* functions
salt_api_auth: 401	Wrong password, or krisalt user not created	Re-run provision to reset the PAM user password
tcp_4505/4506 fail	salt-master not running, or firewall blocking	sudo launchctl kickstart -k system/com.saltstack.salt.master

Worked Example — Two-Step salt-api Flow

Every salt-api session is two calls: obtain a token, then use it. kri’s SaltAPIClient does exactly this on every request.

Step 1 — Login (exchange PAM credentials for a session token):

# POST /login — PAM credentials for krisalt
curl -sk -X POST https://192.0.2.64:4507/login \
  -H 'Content-Type: application/json' \
  -d '{"username":"krisalt","password":"<secret>","eauth":"pam"}'

# Response
{
  "return": [{
    "token":  "e3b0c44298fc1c149afb...",  # use this in step 2
    "expire": 1750524123.4,
    "user":   "krisalt",
    "eauth":  "pam",
    "perms":  ["test.ping", "@wheel", "@runner", ...]
  }]
}

Step 2 — Run a salt-api call (using the token from step 1):

# POST /run — fleet-wide minion status via runner
curl -sk -X POST https://192.0.2.64:4507/run \
  -H 'Content-Type: application/json' \
  -H 'X-Auth-Token: e3b0c44298fc1c149afb...' \
  -d '{"client":"runner","fun":"manage.up"}'

# Response — list of minions currently up
{
  "return": [[
    "Abhisheks-Mac-mini",
    "DKs-Mac-mini",
    "Cyruss-Mac-mini"
  ]]
}

Token lifetime: The default CherryPy token expires after 12 hours. kri re-authenticates automatically on each probe cycle — it never caches tokens across requests to avoid stale-auth failures.

Probe Checks → Exact salt-api Calls

Every check in SaltMasterProbe maps to a concrete socket operation or salt-api POST:

Check Name	Type	Exact Call	What It Verifies
dns	Socket	socket.getaddrinfo(address, None)	Master IP/hostname is resolvable from the kri container
tcp_4505	Socket	socket.connect((address, 4505))	ZeroMQ publish port open (minions subscribe here)
tcp_4506	Socket	socket.connect((address, 4506))	ZeroMQ return port open (minions return results here)
salt_api_auth	HTTP POST	client=runner, fun=manage.up	PAM auth works; runner client enabled; salt bus reachable
key_store	HTTP POST	client=wheel, fun=key.list_all	Wheel ACL works; PKI directory readable by salt-master
version	HTTP POST	client=runner, fun=manage.versions	Consistent Salt version across master and connected minions
minions_up	HTTP POST	client=runner, fun=manage.up	At least one minion is currently connected and responsive

Security Model — Defense in Depth

Four independent layers protect the salt-api surface:

1. PAM password check — krisalt’s OS password must match. No token is issued without a valid PAM session.
2. external_auth ACL — Even with a valid token, every function call is checked against the per-user allowlist in kri.conf. Calling anything not listed returns 403.
3. TLS at rest_cherrypy — All traffic is encrypted via the self-signed cert on port 4507. Credentials are never sent in plaintext.
4. LAN-only, not Tailscale — salt-api binds to the LAN interface (192.168.x.x). Traffic never traverses DERP relays. The attack surface is physically bounded to the local network.

TL;DR — How It All Fits Together

• 1

  kri provisions via Ansible

  Docker → SSH → installs salt 3007.14 PKG, writes configs, creates launchd plists

• 2

  salt-master and salt-api start as root launchd daemons

  KeepAlive + RunAtLoad ensure they survive reboots and crashes

• 3

  Minions connect via LAN IP

  Ansible writes the master IP list directly into /etc/salt/minion — no DNS needed

• 4

  kri controls everything via salt-api HTTPS

  POST /login → token; POST /run with token → salt executes; PAM + ACL enforce least privilege

• 5

  SaltMasterProbe validates health end-to-end

  7 checks: DNS → TCP → auth → wheel → runner — all must pass for status = healthy

Liveness vs Freshness

Liveness ≠ Freshness — Why a Running Minion Still Goes Stale

The single most important operational insight: launchd guarantees process liveness, but kri’s “stale” badge measures data freshness. A salt-minion can be perfectly alive — process running, KeepAlive respawning on crash, RunAtLoad started at boot — while the node simultaneously shows a stale badge in kri, because the two signals are completely independent of each other.

The Two Signals

Signal	What It Means	Who Guarantees It
Process Liveness	salt-minion / salt-master / salt-api process is running	launchd — RunAtLoad starts on boot; KeepAlive respawns on crash with a 5 s throttle
Data Freshness	kri refreshed last_seen_at for this node recently	the presence-refresh chain (pull path via Celery beat + push path via grain_report)

Key insight: kri’s node status is computed purely as now − last_seen_at. Nothing about the badge inspects whether any process is alive. The thresholds (from node_status.classify_status): ≤ 15 min = online, 15 min–4 h = stale, > 4 h = offline.

Two Paths That Refresh last_seen_at

A node stays online only while at least one of the two paths keeps firing. Miss both for 15 minutes and the badge flips to stale.

Pull Path

• 1

  Celery beat fires every 90 s

  sync_minion_presence task — salt_presence_tasks.py:126

• 2

  Calls runner manage.up

  Sent to the default-flagged master only via salt-api HTTPS

• 3

  Sets last_seen_at = now

  For every minion returned in the up-list

Push Path

• 1

  Minion runs grain_report state

  On its own scheduled interval — pillar provides ingest_url + node_token

• 2

  POSTs grains to kri ingest API

  HTTP POST to /ingest with the node’s grain data

• 3

  Sets last_seen_at = now

  ingest.py:242 — ingest handler writes the timestamp directly

mark_stale_nodes runs every 5 min and applies the thresholds: online → stale (gap > 15 min), stale → offline (gap > 4 h). A node goes stale only when both refresh paths stop updating it for 15 minutes.

Presence Refresh Paths — Pull + Push → last_seen_at → Status Badge

Why the Pull Path Breaks

Cause	What Happens	Scope
Wrong master polled — multi-master failover	kri polls only the master flagged is_default. With master_type: failover + random_master: True, a minion may attach to a non-default master (e.g. cylon) while kri polls mm1. manage.up on mm1 will not list it → stale, even though the minion is healthily connected.	Per-node
Default master’s salt-api broken	Any error from salt-api (Salt 3006+ netapi_enable_clients 400, runner/wheel ACL 500, expired PAM password, TLS failure) returns nothing → no node is refreshed. The entire fleet goes stale in 15 min and offline in 4 h.	Fleet-wide
No master flagged default	The query WHERE is_default IS TRUE returns nothing → the task skips entirely → everything stale.	Fleet-wide
manage.up is a live ping with a short timeout	The runner pings every minion and waits a few seconds. A slow, loaded, or briefly unresponsive minion misses that window and is absent from the up-list, causing it to flap in and out of the stale state.	Per-node

Why the Push Path Breaks

• 1

  macOS sleep / App Nap / Power Nap

  A Mac Mini that sleeps suspends all scheduled Salt states. KeepAlive keeps the process registered with launchd, but a sleeping Mac runs no schedule — no heartbeat is sent. Operational fix: sudo pmset -a sleep 0 powernap 0 disablesleep 1 on always-on fleet nodes.

• 2

  Missing pillar

  The grain_report state needs ingest_url + node_token from pillar. If pillar was not rendered or the minion attached to a master that lacks it, the heartbeat silently no-ops — no POST is ever sent.

• 3

  Key pending or rejected

  After a master rebuild or re-key, the minion process runs but cannot exchange jobs. The minion appears alive to launchd but is invisible to both refresh paths.

• 4

  Clock skew

  Freshness is computed as now − last_seen_at. A skewed clock on the minion host or the kri server distorts this calculation — a node can appear stale even though last_seen_at was just written.

On “they don’t start automatically”: The master and salt-api plists ARE configured to auto-start (RunAtLoad) and respawn (KeepAlive) — verified in the kri repo templates. If they genuinely do not come up at boot, the usual macOS causes are: a crash-loop (check sudo launchctl print system/com.saltstack.salt.master and /var/log/salt/master-error), wrong plist location or permissions (must be /Library/LaunchDaemons/, owned root:wheel, mode 0644), or the Mac sleeping. The throttle is 5 s, so a crash-looping daemon retries every 5 s.

Honesty caveats — what is NOT verified here:

The minion launchd plist is installed by the Salt PKG installer, NOT the kri repo — verify its RunAtLoad/KeepAlive keys on the host with sudo launchctl print system/com.saltstack.salt.minion.

Which master each minion is currently attached to is runtime state — check on the master with sudo /opt/salt/salt-run manage.up and on the minion with salt-call --local grains.get master.

Operator’s Triage Checklist — Node Shows Stale

• 1

  Is the minion process alive?

  sudo launchctl print system/com.saltstack.salt.minion on the node — check state is running.

• 2

  Which master is it connected to?

  salt-call --local config.get master on the node — is that address the kri default master?

• 3

  Does the DEFAULT master see it?

  sudo /opt/salt/salt-run manage.up on the default master — is the node’s minion ID in the list?

• 4

  Is salt-api answering for kri?

  kri UI → Masters → “Test connection” on the default master — all 7 probe checks must be green.

• 5

  Is the Mac asleep?

  pmset -g on the node — check sleep and powernap settings. Always-on fleet nodes must have sleep disabled.

• 6

  Is the key accepted?

  sudo /opt/salt/salt-key -L on the default master — the minion ID must appear under Accepted Keys, not Unaccepted or Rejected.

TL;DR: “Stale” is a freshness alarm, not a crash alarm. launchd keeps the processes up; staleness means the refresh chain — minion → ingest, or kri → default-master manage.up — was broken for 15 minutes. In a multi-master fleet the #1 cause is a minion attached to a non-default master that kri never polls. Fleet-wide staleness almost always means the default master’s salt-api is down.