Salt — Complete Visual Guide

Deepak Gupta · 30 May 2026 ·40 min read

Every concept explained with animated diagrams · Salt 3007.x (Chlorine) · Built for the kri Mac Mini fleet platform

Architecture · Data · kri

Salt Architecture — All Components

SaltStack is a remote execution and configuration management system. One salt-master controls many salt-minions over encrypted ZeroMQ channels. Everything is event-driven and asynchronous.

Complete salt architecture — animated

Core Components

🖥️

salt-master

Control plane. One per fleet. Manages keys, serves state files and pillar, listens on :4505/:4506. In kri: runs natively on mm1 (launchd service).

💻

salt-minion

Agent on each managed node. Subscribes to master events, executes commands locally, returns results. Runs via launchd on macOS.

🌐

salt-api

HTTP REST layer over the master. kri API/worker use it to dispatch ad-hoc commands (test.ping, state.apply). Runs alongside salt-master on mm1.

⚡

salt-call

Run salt functions locally on the minion without going to the master. Used in heartbeat scripts and grain collection (salt-call grains.items).

🔑

salt-key

Manage minion RSA keys: list pending/accepted/rejected, accept, reject, delete. kri exposes this via Settings → Minion Keys.

🏃

salt-run

Execute runner modules on the master itself (not on minions). Used for jobs, orchestration, managing jobs cache.

Communication Model

Minion connects

Minion subscribes to master’s ZeroMQ PUBLISH socket (:4505). Stays connected. This is a persistent long-lived TCP connection.

Master broadcasts job

When you run salt mm2 test.ping, master publishes a job to :4505. All minions receive it; only targeted ones execute.

Minion executes

Targeted minion runs the function locally (e.g. calls Python code). Completely async — minion doesn’t block master.

Minion returns

Result is pushed back to master on :4506 (PUSH/PULL socket), tagged with the job ID. Master stores in job cache and returns to caller.

Events fire

Every job, return, state run, and minion start/stop fires an event on the master’s event bus. Reactor subscriptions trigger on matching patterns.

Key insight: Salt never polls minions. All communication is push-based from the master, and minions maintain persistent TCP connections to :4505. This scales to thousands of minions on a single master.

kri Integration

kri uses Salt as its fleet control plane. Every Mac Mini runs a salt-minion; mm1 runs the salt-master as a native launchd service. Here is the complete integration map.

kri + Salt complete integration (v0.1.194)

kri Salt Configuration Reference

Setting	Value	Where
salt_master_address	198.51.100.75	group_vars/all.yml
salt_version	3007.14	group_vars/all.yml
SALT_API_URL	http://198.51.100.75:8080	.env.docker
SALT_API_USER	krisalt	.env.docker
Heartbeat interval	5 minutes	/etc/salt/minion.d/kri-heartbeat.conf
Pillar dir	/srv/salt/pillar/	mm1 (salt-master)
States dir	/srv/salt/states/	mm1 (synced from git)
PKI dir	/etc/salt/pki/master/	mm1 (launchd)

kri Salt Flows

Node goes online

Minion starts → connects to master → reactor fires → state.apply base.grain_report → minion POSTs grains to kri API → node status: online

Heartbeat (every 5 min)

kri_heartbeat schedule fires → salt-call --local grains.items → HTTP POST to ingest API → refreshes last_seen_at

Quick Actions (UI)

User clicks Test Ping/Reboot → kri API calls Celery → run_salt_cmd → POST to salt-api HTTP → ZeroMQ → minion

Bootstrap

Ansible playbook (bootstrap_mac_mini.yml) installs salt-minion pkg, writes minion config, applies base.heartbeat state, POSTs initial grains

Mark stale / offline

Celery beat runs mark_stale_nodes every 5 min. Nodes stale >15 min, offline >4 hours. Node remains stale/offline until next grain report.

Install salt-master on mm1: ansible-playbook playbooks/setup_salt_master.yml -i playbooks/inventory/hosts.ini -e "kri_salt_api_password=yourpassword"
Then update .env.docker with SALT_API_URL, SALT_API_USER, SALT_API_PASSWORD.

📊 Data Layer — Grains, Pillar & Mine

Data — Grains, Pillar & Mine

Three distinct data stores in Salt, each with a different owner, audience, and update mechanism.

Grains vs Pillar vs Mine — who owns it, who sees it

🌱 Grains

Static/semi-static facts about the minion collected locally at startup. Used for targeting (-G 'os:MacOS') and state conditionals. In kri, grains are POSTed to the ingest API as the node’s heartbeat.

# Collect all grains
salt mm2 grains.items

# Get one grain
salt mm2 grains.get cpuarch

# Set a custom grain
salt mm2 grains.set role worker

# Force re-collection
salt mm2 saltutil.refresh_grains

# Target by grain
salt -G 'cpuarch:arm64' state.apply

In kri: kri_heartbeat.py runs salt-call --local grains.items then POSTs to /api/v1/ingest/grains every 5 minutes.

🔒 Pillar

Secure configuration stored on the master. Each minion only sees its own pillar slice. Never transmitted to other minions. Perfect for tokens, passwords, URLs. Rendered from Jinja2 SLS files.

# /srv/salt/pillar/mm2.sls
fleet_platform:
  ingest_url: http://198.51.100.27/api/v1/ingest
  node_token: pLmxKN...

# /srv/salt/pillar/top.sls
base:
  'mm2':
    - mm2
  'mm3':
    - mm3

# Refresh after editing
salt mm2 saltutil.refresh_pillar

# Read from state
{% set token = pillar['fleet_platform']['node_token'] %}

⛏️ Mine

Minions push selected data to the master, which shares it with all other minions. Useful for service discovery — a load balancer minion reads the IP addresses of all web server minions.

# /etc/salt/minion.d/mine.conf
mine_functions:
  network.ip_addrs: []
  disk.usage: []
  status.loadavg: []

mine_interval: 60  # minutes

# Manually update
salt mm2 mine.update

# Read another minion's mine data
salt mm2 mine.get 'mm3' network.ip_addrs

# Read from a state template
{% set ips = salt['mine.get']('web*','network.ip_addrs') %}

kri does not currently use Mine — nodes report grains directly via HTTP. Mine is useful if you need minions to discover each other.

States · Events · Reactor · Thorium

States & Execution Modules

States describe desired configuration (declarative, idempotent). Execution modules run ad-hoc commands (imperative, immediate).

State rendering and application pipeline

State File Anatomy (.sls)

# base.grain_report
# state ID — must be unique
report_grains_to_fleet_platform:
  # state module.function
  cmd.run:
    - name: |
        python3 - <<'EOF'
        import urllib.request, json
        ...
        EOF
    # Requisites
    - require:
      - file: kri_heartbeat_script
    - onchanges:
      - file: kri_config

# Jinja2 in states
{% set url = pillar.get('fleet_platform',{})
              .get('ingest_url','') %}
{% if url %}
report:
  cmd.run:
    - name: "curl -X POST {{ url }}"
{% endif %}

Requisites — Execution Order

Requisite	Meaning
require	This state only runs if the listed states succeeded
require_in	The listed state must run after this one
watch	Like require, but also runs if the listed state made a change
watch_in	Notify another state to run if this one changed
onchanges	Only run if one of the listed states reported a change
unless	Skip if this shell command returns 0 (exit success)
onlyif	Only run if this shell command returns 0
prereq	Run before listed state only if listed state would change
use	Copy arguments from another state ID
onfail	Run only if the listed state FAILED

top.sls — Highstate Routing

The top file maps minions to state trees. Running state.highstate applies everything in top.sls that matches the minion.

# /srv/salt/states/top.sls
base:
  '*':           # all minions
    - base.grain_report
    - base.heartbeat

  'G@os:MacOS':  # grain match
    - macos.defaults

  'mm1':         # exact match
    - role.salt_master

  'web*':        # glob
    - nginx
    - ssl_cert

production:    # saltenv
  'prod*':
    - production.config

State vs Execution Modules

State module	Execution module
file.managed	file.read
pkg.installed	pkg.install
service.running	service.start
cmd.run	cmd.run
schedule.present	schedule.list

State modules are called in SLS files and enforce desired state. Execution modules are called ad-hoc via salt minion module.function. Some share the same name but have different semantics.

Rule: States = idempotent desired state. Execution = one-time imperative action.

Common State Modules

file.managed

Ensure file content, ownership, permissions

pkg.installed

Ensure package is installed at version

service.running

Ensure daemon is running and enabled

cmd.run

Run shell command, optionally conditional

user.present

Ensure OS user exists with correct settings

git.latest

Keep a git repo at a specific branch/tag

pip.installed

Ensure Python package installed in venv

schedule.present

Create a scheduled job on the minion

Orchestrate, Runners & Formulas

Orchestrate coordinates complex multi-stage workflows across multiple minions. Runners execute on the master itself. Formulas are reusable, community-maintained state collections.

🎼 Orchestrate — Multi-Stage Workflows

Orchestrate runs on the master (via salt-run) and coordinates state runs across multiple minions in a specific order. Unlike regular states (per-minion), orchestrate spans the whole fleet.

# /srv/salt/states/orch/deploy.sls
# Step 1: Deploy to staging first
deploy_staging:
  salt.state:
    - tgt: staging*
    - sls: app.deploy

# Step 2: Run integration tests
run_tests:
  salt.function:
    - tgt: test-runner
    - fun: cmd.run
    - arg:
      - pytest /tests/
    - require:
      - deploy_staging  # only after staging

# Step 3: Deploy to production
deploy_prod:
  salt.state:
    - tgt: prod*
    - sls: app.deploy
    - require:
      - run_tests  # only if tests pass

# Run it
salt-run state.orchestrate orch.deploy

🏃 Runners — Master-Side Modules

Runners execute on the master process itself, not on minions. Used for fleet-wide operations, job management, and orchestration.

# Common runners
# Manage online/offline minions
salt-run manage.present     # online
salt-run manage.absent      # offline
salt-run manage.status      # all with status

# Job history
salt-run jobs.list_jobs     # recent jobs
salt-run jobs.lookup_jid 2026053014000001

# Mine data
salt-run mine.get '*' network.ip_addrs

# Cache
salt-run cache.grains mm2   # cached grains

# Orchestrate
salt-run state.orchestrate orch.deploy
salt-run state.orchestrate orch.upgrade   pillar='{"version": "2.5.0"}'

📦 Formulas — Reusable State Collections

Formulas are pre-built, community-maintained Salt state collections. Like Ansible roles or Helm charts — install nginx, postgres, docker with a single include.

# Install a formula (clone to states dir)
cd /srv/salt/states
git clone https://github.com/saltstack-formulas/nginx-formula nginx

# Use in your state or top.sls
# /srv/salt/states/top.sls
base:
  'web*':
    - nginx

# Configure via pillar
# /srv/salt/pillar/nginx.sls
nginx:
  server:
    config:
      worker_processes: auto
  vhosts:
    mysite:
      enabled: True
      name: example.com

# Popular formulas
# nginx-formula, postgres-formula
# docker-formula, users-formula
# git-formula, openssh-formula

🌍 SaltEnv — Multiple Environments

SaltEnv lets you serve different state trees for different environments (base, dev, staging, prod) from the same master. Minions request a specific saltenv.

# /etc/salt/master.d/environments.conf
file_roots:
  base:
    - /srv/salt/states/base
  dev:
    - /srv/salt/states/dev
    - /srv/salt/states/base   # fallback
  prod:
    - /srv/salt/states/prod
    - /srv/salt/states/base

pillar_roots:
  base:
    - /srv/salt/pillar/base
  prod:
    - /srv/salt/pillar/prod
    - /srv/salt/pillar/base

# Apply a specific environment
salt mm2 state.apply saltenv=prod

# Pin minion to an env
# /etc/salt/minion
saltenv: prod

⚡ Events, Reactor, Schedule & Automation

Events, Reactor, Schedule & Beacons

Salt’s event bus is the nervous system. Every action produces an event. Reactor listens and triggers responses. Schedule and Beacons add time-based and condition-based automation.

Event bus flow — everything produces events, reactor subscribes

⚡ Reactor

The reactor listens to event patterns and triggers actions. kri uses it to fire grain_report the instant a minion connects — eliminating the heartbeat delay (from 5 min to <2 seconds).

# /etc/salt/master.d/kri.conf
reactor:
  - 'salt/minion/*/start':
    - salt://reactor/grain_report_on_start.sls
  - 'salt/beacon/*/disk_usage':
    - salt://reactor/alert_disk.sls

# /srv/salt/states/reactor/grain_report_on_start.sls
fire_grain_report:
  local.state.apply:
    - tgt: "{{ data['id'] }}"
    - arg:
      - base.grain_report

The reactor runs on the master and is event-driven. It can trigger local commands, state applies, runner modules, or wheel functions.

📅 Schedule

Run jobs on a timer — on the minion (without master involvement) or on the master. kri uses a minion-side schedule stored in /etc/salt/minion.d/ so it persists across restarts and cache clears.

# Minion-side — survives cache clears
# /etc/salt/minion.d/kri-heartbeat.conf
schedule:
  kri_heartbeat:
    function: cmd.run
    job_kwargs:
      cmd: /opt/salt/bin/python3.10 /usr/local/bin/kri_heartbeat.py
    minutes: 5
    enabled: True
    run_on_start: True  # fires on minion start

# Master-side (via salt-run)
# runs on master, NOT on minions
salt-run schedule.add nightly_backup \
  function=state.apply \
  job_kwargs="{'arg': ['backup']}" \
  hours=24

schedule.present (state module) writes to volatile /var/cache/salt/minion/schedule.p. kri writes to /etc/salt/minion.d/ for persistence.

🔦 Beacons

Beacons monitor the local system and fire events when conditions are met — without a master initiating anything. Useful for disk full alerts, process monitoring, log file watching.

# /etc/salt/minion.d/beacons.conf
beacons:
  disk_usage:         # monitor disk
    - /: 80%          # fire if / > 80%
    - /data: 90%
    - interval: 60

  ps:                 # process monitor
    - processes:
        salt-master: running
    - interval: 30

  log:                # log file watcher
    - file: /var/log/salt/minion
    - tags:
        salt.ERROR: {}
    - interval: 10

Beacon events: salt/beacon/{minion}/{type}/{data}

🔄 Returners

By default, job results go back to the master. Returners let you also send results to external systems — databases, log aggregators, monitoring platforms — without changing states.

# Send all results to Elasticsearch
salt mm2 state.apply base.grain_report \
  --return=elasticsearch

# Configure returner in minion
# /etc/salt/minion.d/returner.conf
elasticsearch.host: "logs.example.internal:9200"
elasticsearch.index: "salt"

# Built-in returners include:
# sqlite3, mysql, postgres, redis
# elasticsearch, splunk, carbon (Graphite)
# sentry, slack, hipchat

kri acts as a custom returner — the ingest API receives grain data pushed via HTTP POST from the minion’s heartbeat script.

Advanced — Syndic, Proxy Minions & More

Advanced Salt patterns for hierarchical deployments, managing non-standard devices, and scaling to thousands of nodes.

🔗 Syndic — Hierarchical Masters

The Syndic allows chaining salt-masters. A top-level master controls sub-masters (syndics), which each control their own minions. Used for geo-distributed fleets or organizational separation.

# Architecture:
# master-of-masters → syndic1 → mm1,mm2
#                  → syndic2 → mm3,mm4

# On syndic master:
# /etc/salt/master
syndic_master: top-master.example.com

# /etc/salt/minion
master: top-master.example.com

# Command flows down the chain:
# salt '*' test.ping (on top master)
# → syndic1 fans out to mm1,mm2
# → syndic2 fans out to mm3,mm4
# Results bubble back up

🔌 Proxy Minions — Non-Linux Devices

Proxy minions manage devices that can’t run a salt-minion: network switches, IoT devices, APIs, legacy systems. The proxy runs on a Linux host and translates salt commands.

# Proxy minion talks to a REST API device
# /etc/salt/proxy
master: salt-master.example.com

# /srv/salt/pillar/switch01.sls
proxy:
  proxytype: napalm        # network device
  driver: ios
  host: 192.0.2.1
  username: admin
  password: secret

# Start the proxy
salt-proxy --proxyid=switch01 -d

# Use normally
salt switch01 net.interfaces
salt switch01 net.arp

☁️ salt-cloud — Provision Cloud VMs

Salt-cloud provisions VMs on AWS, GCP, Azure, DigitalOcean etc., installs the salt-minion, and adds them to the fleet automatically. Infrastructure as code.

# /etc/salt/cloud.providers.d/aws.conf
my-aws:
  driver: ec2
  id: AKIAIOSFODNN7EXAMPLE
  key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  keyname: my-keypair
  securitygroup: default
  location: us-east-1

# /etc/salt/cloud.profiles.d/web.conf
web-server:
  provider: my-aws
  image: ami-0c55b159cbfafe1f0
  size: t3.micro
  minion:
    grains:
      role: web

# Spin up and auto-register
salt-cloud -p web-server prod-web-01

# Destroy
salt-cloud -d prod-web-01

⚡ Thorium — Reactive System

Thorium is Salt’s advanced reactive system — a layer above the Reactor. Where Reactor matches single events, Thorium aggregates multiple events over time and fires when conditions are met (e.g., “3 disk alerts in 5 minutes”).

# /srv/salt/thorium/detect_disk_crisis.sls
# Accumulate beacon events from minions
check_disk:
  thresh.count:
    - count: 3          # 3 events needed
    - seconds: 300      # within 5 minutes
    - event_data:
        tag: salt/beacon/*/disk_usage

# When threshold met: fire action
alert_disk_crisis:
  local.state.apply:
    - tgt: "{{ data['id'] }}"
    - arg:
      - disk.emergency_cleanup
    - require:
      - check_disk

# Enable Thorium in master config
# /etc/salt/master.d/thorium.conf
thorium_roots:
  base:
    - /srv/salt/thorium

kri note: kri uses the simpler Reactor for immediate on-connect events. Thorium would be used for complex fleet-wide condition detection (e.g., cascade failures).

📊 Salt Versions — Release Codenames

⚗️

3005

Phosphorus

🟡

3006

Sulfur (LTS)

🟢

3007

Chlorine ← kri uses

🔵

3008

Argon (master)

Versions named after chemical elements. LTS releases (Sulfur/3006) get extended support. kri minions run 3007.14 (Chlorine); salt-master on mm1 runs 3008.0 (Argon) — backward compatible.

⚗️ Custom Module Development

Every Salt module is a Python file. The Salt loader injects special dunder (double-underscore) variables that give your module access to grains, pillar, other modules, and configuration.

Dunder Variables

Variable	What it gives you
salt	Call other execution modules: salt’cmd.run’
grains	Read minion grains: grains[‘os’]
pillar	Read pillar data: pillar.get(‘key’, ‘default’)
opts	Master/minion config options
utils	Utility functions shared across modules
states	Call state functions from an execution module
context	Per-minion persistent cache (survives between calls)
virtualname	Override the module name exposed to users

Writing a Custom Execution Module

# /srv/salt/states/_modules/kri_info.py
# salt '*' kri_info.node_summary

def node_summary():
    """Return a kri-specific node summary."""
    os = __grains__['os']
    mem = __grains__['mem_total']
    uptime = __salt__['cmd.run']('uptime -p')
    token = __pillar__.get('fleet_platform', {}).get('node_token', 'none')
    return {'os': os, 'mem_gb': mem//1024, 'uptime': uptime, 'token_set': token != 'none'}

# Custom state module
# /srv/salt/states/_states/kri_node.py
def registered(name, token):
    """Ensure node is registered with kri."""
    result = {'name': name, 'changes': {}, 'result': True, 'comment': ''}
    current = __salt__['kri_info.node_summary']()
    if current['token_set']:
        result['comment'] = 'Already registered'
        return result
    # ... register logic
    result['changes']['registered'] = True
    return result

# Deploy custom modules
salt mm2 saltutil.sync_modules

✅ Testing with salt-check

salt-check lets you write and run unit tests for your salt states directly on the minion, without mocking.

# /srv/salt/states/base/test_grain_report.sls
# Tests for the grain_report state
check_heartbeat_script_exists:
  salt_check.run:
    - assertions:
      - assertion: assertTrue
        expr_form: True
        result: "{{ salt['file.file_exists']('/usr/local/bin/kri_heartbeat.py') }}"

check_heartbeat_is_executable:
  salt_check.run:
    - assertions:
      - assertion: assertEqual
        result: "{{ salt['cmd.run']('test -x /usr/local/bin/kri_heartbeat.py && echo ok') }}"
        expected: "ok"

# Run the tests
salt mm2 salt_check.run_state_tests base.grain_report
salt mm2 salt_check.run_highstate_tests

📋 Job Cache & Event Tags

Every salt command creates a job with a unique JID. Results are cached on the master. Event bus tags follow a structured pattern you can use in Reactor subscriptions.

# Job management
salt-run jobs.list_jobs              # recent jobs
salt-run jobs.lookup_jid 2026053014  # get result
salt-run jobs.active                  # running jobs
salt-run jobs.exit_success <jid>     # did it pass?

# /etc/salt/master.d/jobs.conf
keep_jobs: 72  # hours to keep job cache
job_cache: True

# Event tag patterns (use in Reactor)
salt/job/{jid}/new           # job published
salt/job/{jid}/ret/{minion}  # result returned
salt/minion/{id}/start       # minion connected
salt/minion/{id}/refresh_pillar
salt/state/ret               # state run complete
salt/beacon/{id}/{type}      # beacon fired
salt/presence/present        # presence heartbeat
salt/presence/change         # minion gone/appeared

🔐 Advanced Data — Vault & GPG Encrypted Pillar

HashiCorp Vault Integration

Use Vault as a pillar backend so secrets never touch the salt-master filesystem. Minions request their secrets from Vault via the master as a proxy.

# /etc/salt/master.d/vault.conf
ext_pillar:
  - vault:
      path: secret/data/nodes/{minion_id}

vault:
  url: https://vault.example.internal:8200
  auth:
    method: token
    token: s.xxxxxxxxxxx

# Minion accesses pillar normally
{% set secret = pillar['my_secret'] %}

# Vault path is per-minion by ID
# vault write secret/data/nodes/mm2 token=xxx

GPG-Encrypted Pillar

Encrypt secrets directly in pillar files using GPG. Salt decrypts on the master before sending to the minion. Secrets are safe to commit to git.

# Encrypt a value
echo -n "my-secret-password" | gpg   --armor --batch --encrypt   -r salt-master@example.com

# /srv/salt/pillar/mm2.sls
fleet_platform:
  node_token: |
    -----BEGIN PGP MESSAGE-----
    hQEMA7...encrypted...data
    -----END PGP MESSAGE-----

# /etc/salt/master.d/gpg.conf
decrypt_pillar: True
decrypt_pillar_default: gpg
gpg_keydir: /etc/salt/gpgkeys

Security · Targeting · CLI · Reference

Security, Targeting & Keys

Salt uses RSA public-key cryptography for all master-minion authentication. Every minion has a unique key pair. The master is the trust anchor.

Full key lifecycle — from first boot to trusted minion

PKI Directory Structure

# On salt-master (mm1)
/etc/salt/pki/master/
  master.pem      # RSA private key (NEVER share)
  master.pub      # RSA public key (sent to minions)
  minions/        # accepted minion pub keys
    mm2           # one file per accepted minion
    mm3
  minions_pre/    # pending (not yet accepted)
    mm4
  minions_rejected/ # explicitly rejected
  minions_denied/   # denied (accepted then revoked)

# On salt-minion (each Mac Mini)
/etc/salt/pki/minion/
  minion.pem        # minion private key
  minion.pub        # minion public key (sent to master)
  minion_master.pub # master's public key (for auth)

Key States & Commands

Command	Action
salt-key -L	List all keys by state (accepted/pending/rejected)
salt-key -a mm3	Accept a specific pending key
salt-key -A -y	Accept ALL pending keys (auto-trust, risky)
salt-key -r mm4	Reject a key (moved to minions_rejected)
salt-key -d mm4	Delete a key entirely (decommission)
salt-key -f mm2	Show fingerprint of a specific key
salt-key -F	Show all fingerprints
salt-key —gen-keys=test	Generate a key pair (for testing)

In kri: use Settings → Minion Keys to accept/reject. Never use -A -y in production — always inspect fingerprints first.

External Authentication (salt-api)

When using salt-api, clients authenticate via eauth (external authentication) before they can call salt functions. kri uses PAM auth with a dedicated system user.

# /etc/salt/master.d/kri.conf
external_auth:
  pam:
    krisalt:       # OS user on mm1
      - '.*'      # allow all functions
      - '@runner' # allow runner modules
      - '@wheel'  # allow key management

# HTTP call to salt-api
POST http://198.51.100.75:8080/run
{
  "client": "local",
  "tgt": "mm2",
  "fun": "test.ping",
  "username": "krisalt",
  "password": "...",
  "eauth": "pam"
}

Encryption Layers

Transport encryption

All ZeroMQ traffic is encrypted with a rotating AES session key. The AES key is exchanged using RSA during the authentication handshake.

Pillar encryption

Pillar data is encrypted per-minion using the minion’s public key. Other minions cannot decrypt pillar data sent to a different minion.

Tailscale (kri)

In kri, all traffic also travels through Tailscale’s WireGuard tunnel, adding a second encryption layer and network isolation.

salt-api TLS

salt-api should use TLS in production. In kri we rely on Tailscale’s encryption, so TLS is disabled locally (disable_ssl: true).

Targeting

Every salt command needs a target. Salt supports 8 different targeting methods. Mix them with compound expressions for surgical precision.

Targeting methods — how each selects minions

Targeting Syntax Reference

Flag	Type	Example
(none)	Glob (default)	mm* web[12]
-G	Grain	os:MacOS cpuarch:arm64
-P	Grain PCRE	os:Mac.*
-E	PCRE minion ID	mm[0-9]+
-L	Comma list	mm1,mm2,mm3
-N	Nodegroup	production
-I	Pillar	role:primary
-J	Pillar PCRE	role:web.*
-S	Subnet/IP	192.0.2.0/24
-C	Compound	G@os:MacOS and not mm1

Nodegroups

Named groups of minions defined in the master config. Can include any targeting expression. Reusable across commands and states.

# /etc/salt/master.d/kri.conf
nodegroups:
  fleet: 'mm1 or mm2 or mm3'
  arm_minis: 'G@cpuarch:arm64'
  prod: 'I@env:production'
  all_except_master: '* and not mm1'

# Use them
salt -N fleet test.ping
salt -N arm_minis state.apply base.grain_report
salt -N prod state.highstate

# In top.sls
base:
  'N@arm_minis':
    - macos.arm_optimizations

Batch Execution — Limiting Blast Radius

For large fleets, apply changes in batches to avoid taking down everything at once.

# Apply to 10% at a time
salt --batch=10% \
  '*' state.apply

# Apply to 3 at a time
salt --batch=3 \
  'mm*' pkg.upgrade

# Async — don't wait for results
salt --async mm2 \
  state.apply base.grain_report

# Get result of async job
salt-run jobs.lookup_jid \
  20260530140000123456

# Timeout — don't wait forever
salt -t 30 mm2 state.apply

# Show progress
salt --progress \
  '*' test.ping

📋 CLI, Config Files, Ports & Logs

CLI Reference — All Salt Commands

Complete reference for salt, salt-call, salt-key, salt-run, and salt-api.

`salt` — Run commands on minions

Command	Description
salt ’*’ test.ping	Ping all minions
salt mm2 grains.items	Get all grains from mm2
salt mm2 state.apply base.grain_report	Apply a specific state
salt mm2 state.highstate	Apply all states in top.sls
salt mm2 state.apply test=True	Dry run — show what would change
salt mm2 cmd.run ‘uptime’	Run a shell command
salt mm2 saltutil.refresh_grains	Force grains re-collection
salt mm2 saltutil.refresh_pillar	Reload pillar from master
salt mm2 saltutil.sync_all	Sync all custom modules/states
salt -G ‘os:MacOS’ test.ping	Target by grain
salt -L ‘mm1,mm2,mm3’ test.ping	Target by list
salt —batch=2 ’*’ state.apply	Apply 2 at a time
salt —async mm2 state.apply	Fire and forget
salt -t 10 mm2 cmd.run ‘uptime’	10 second timeout
salt mm2 system.reboot	Reboot the minion OS
salt mm2 schedule.list	List scheduled jobs on minion

`salt-call` — Run locally on minion

Run salt functions locally without needing the master. Used in heartbeat scripts and bootstrapping.

# Run on the minion itself (no master)
salt-call --local grains.items

# Apply a state locally
salt-call --local state.apply base.grain_report

# Get pillar (needs master)
salt-call pillar.items

# JSON output for scripting
salt-call --local --out=json grains.items

# kri heartbeat uses this:
raw = subprocess.check_output(
  ["/opt/salt/salt-call", "--local",
   "grains.items", "--out=json"]
)

`salt-run` — Runner modules (master-side)

Execute runner modules on the master itself. Not dispatched to minions.

# List all recent jobs
salt-run jobs.list_jobs

# Look up a specific job result
salt-run jobs.lookup_jid 2026053014000012

# Check which minions are active
salt-run manage.present
salt-run manage.absent

# Cache status
salt-run manage.status

# Orchestrate (multi-state workflow)
salt-run state.orchestrate orch.deploy

# Mine stats
salt-run mine.get '*' network.ip_addrs

`salt-key` — Key management

# List all keys by state
salt-key -L

# Accept specific        # Accept all pending
salt-key -a mm3 -y            salt-key -A -y

# Reject                  # Delete (decommission)
salt-key -r mm4 -y            salt-key -d mm4 -y

# Show fingerprints       # Show specific
salt-key -F                   salt-key -f mm2

# Generate standalone key pair (testing)
salt-key --gen-keys=test --gen-keys-dir=/tmp/

🔌 Ports Reference

Port	Protocol	Direction	Purpose	Who listens
4505	TCP (ZeroMQ PUB)	master → minions	Command publish channel. Minions subscribe and stay connected.	salt-master on mm1
4506	TCP (ZeroMQ PULL)	minions → master	Return channel. Minions push job results back here.	salt-master on mm1
8080	HTTP (REST)	kri → master	salt-api endpoint. kri worker POSTs commands here.	salt-api on mm1
80 / 443	HTTP / HTTPS	browser → kri	kri web frontend + API. nginx reverse proxy.	Docker host
8000	HTTP	nginx → api	FastAPI internal port (not exposed externally).	Docker container
5432	PostgreSQL	internal	TimescaleDB. Only accessible inside Docker network.	Docker container
6379	Redis	internal	Celery broker + rate limiting. Internal only.	Docker container

Firewall: Only 4505, 4506, and 8080 need to be reachable on mm1 from other Mac Minis. In kri these are protected by Tailscale — no public exposure.

📄 Configuration Files

File / Directory	Purpose
/etc/salt/master	Main salt-master config
/etc/salt/master.d/*.conf	Drop-in master config files (kri.conf, salt-api.conf)
/etc/salt/minion	Minion config (master address, id, log level)
/etc/salt/minion.d/*.conf	Drop-in minion config — kri writes kri-heartbeat.conf here
/etc/salt/pki/master/	Master RSA keys + all minion public keys
/etc/salt/pki/minion/	Minion private key + cached master.pub
/srv/salt/states/	State files (.sls) served via salt://
/srv/salt/pillar/	Pillar files — per-node secrets and config
/Library/LaunchDaemons/com.saltstack.salt.master.plist	macOS launchd plist for salt-master
/Library/LaunchDaemons/com.saltstack.salt.api.plist	macOS launchd plist for salt-api
/Library/LaunchDaemons/com.saltstack.salt.minion.plist	macOS launchd plist for salt-minion
/usr/local/bin/kri_heartbeat.py	kri heartbeat script (managed by base.heartbeat state)

📋 Log Files & Debugging

Log File	What it contains
/var/log/salt/master	salt-master activity, auth events, job dispatch
/var/log/salt/master-error	salt-master errors and exceptions
/var/log/salt/api	salt-api HTTP requests and responses
/var/log/salt/minion	Minion connection, job execution, state results
/var/log/salt/minion-error	Minion errors (look here first when things break)

# Tail master log
tail -f /var/log/salt/master

# Tail minion log on mm2 (via SSH)
ssh 198.51.100.62 "sudo tail -f /var/log/salt/minion"

# Increase verbosity (troubleshooting)
# In /etc/salt/minion:
log_level: debug

# Watch events in real time (on master)
salt-run state.event pretty=True

# Check job history
salt-run jobs.list_jobs

🔗 salt-ssh — Agentless Mode

salt-ssh runs salt commands over SSH without installing a minion. No ZeroMQ, no PKI keys, no persistent connection. Uses a thin client library injected over SSH. Perfect for bootstrapping new nodes or managing legacy systems.

salt-ssh vs salt-minion

Feature	salt-ssh	salt-minion
Agent needed	No — SSH only	Yes
Transport	SSH (port 22)	ZeroMQ (4505/4506)
Auth	SSH keys/password	RSA key pairs
Speed	Slower (per-connection)	Fast (persistent)
Scale	Hundreds	Tens of thousands
State support	Full	Full
Grains	Limited (no cache)	Full + cached
Best for	Bootstrap, one-offs	Production fleet

Roster — ssh Inventory

The roster file defines SSH targets. Like an Ansible inventory.

# /etc/salt/roster
mm1:
  host: 198.51.100.75
  user: dk
  sudo: True
  priv: /root/.ssh/id_ed25519

mm2:
  host: 198.51.100.62
  user: dk
  sudo: True

# Use it
salt-ssh mm1 test.ping
salt-ssh '*' state.apply base.grain_report
salt-ssh mm1 grains.items

# kri uses Ansible (not salt-ssh) for
# bootstrap because Ansible has better
# macOS support and the bootstrap playbook
# installs the full salt-minion afterwards

🏗️ High Availability — Multi-Master

Multi-Master Setup

Minions can connect to multiple masters simultaneously. If one master fails, minions failover to the next. Both masters must have identical PKI keys (same master.pem/pub).

# /etc/salt/minion (on each minion)
master:
  - mm1.tailscale           # primary
  - mm1-backup.tailscale    # failover

master_alive_interval: 30   # check every 30s
master_tries: -1            # retry forever

# Both masters MUST share PKI
# Copy master.pem + master.pub to backup
# Both masters share accepted minion keys
# Use shared file system or DB for job cache

# kri note: single master on mm1 is sufficient
# for a home lab. Multi-master for production
# fleets with SLA requirements.

Peer Runner — Minion-to-Minion

Allow minions to request data from other minions or trigger runners, via the master as a proxy. Useful for service discovery and coordination.

# Allow mm2 to call runners on master
# /etc/salt/master.d/peer.conf
peer:
  mm2:                  # this minion ID
    - network.ip_addrs  # can call these
    - grains.items
    - mine.get

peer_run:
  mm2:
    - manage.present    # can call these runners

# From mm2 (via salt-call)
salt-call publish.publish 'mm3' network.ip_addrs
salt-call publish.runner manage.present