VCF Salt — build it the right way

Get the bundle

Download VMware_Salt_Raas-<VERSION>-XXXXXXXX.el8_Installer.tar.gz from the Broadcom Support Portal. Verify with sha256sum. Stage on every node.

tar -xzvf VMware_Salt_Raas-*.tar.gz
cd Config-*/sse-installer
sudo rpmkeys --import keys/*.asc

Install Salt — masters

Run on both master boxes.

cd ~/Config-*/rpm
sudo rpm -ivh salt-*.rpm
sudo rpm -ivh salt-master-*.rpm
sudo rpm -ivh salt-minion-*.rpm
sudo rpm -ivh salt-cloud-*.rpm
sudo rpm -ivh salt-api-*.rpm

Install Salt — data node

Run on the RaaS + DB + Valkey box. Just the minion — no master here.

cd ~/Config-*/rpm
sudo rpm -ivh salt-*.rpm
sudo rpm -ivh salt-minion-*.rpm

Mirror master keys

This is the magic. Both masters use the same master.pem and master.pub. That's how a minion can talk to either master without re-keying — and how blue/green works later.

# on master 1 — generate the keys (first time only)
# the files don't exist until you start the service once
sudo systemctl start salt-master
sudo systemctl stop salt-master

# on master 2 — paste in master 1's content
sudo chmod 600 /etc/salt/pki/master/master.pem
sudo vim   /etc/salt/pki/master/master.pem
sudo chmod 400 /etc/salt/pki/master/master.pem

sudo vim   /etc/salt/pki/master/master.pub

Treat master.pem like a root password. Use a vault or out-of-band transfer — never commit it.

Set auto-accept

The other half of the magic. Edit /etc/salt/master on both masters.

user: root
auto_accept: True

If your security policy bans blanket auto-accept, use auto_accept_grain patterns — but the rule must match on both masters.

Configure minions and start

Three things on each box: tell it about the masters, give it a name, start the right services.

a) Point at both masters — same file on every box

# /etc/salt/minion.d/master.conf
master:
  - prod-master1.example.com
  - prod-master2.example.com

b) Set the minion ID — different on each box

Names are your choice. Pick something that describes the role, not the hostname. The IDs appear later in the pillar (step 8). A good convention: <env>-<role><N>.

# on master 1
echo "prod-master1" | sudo tee /etc/salt/minion_id

# on master 2
echo "prod-master2" | sudo tee /etc/salt/minion_id

# on the data node (RaaS + PG + Valkey)
echo "prod-raas" | sudo tee /etc/salt/minion_id

c) Start services — masters run three, data node runs one

# on both masters
sudo systemctl enable --now \
  salt-master salt-minion salt-api

# on the data node
sudo systemctl enable --now salt-minion

Verify keys and ping

From master 1. Accept all pending keys and confirm every minion answers.

sudo salt-key -L     # list pending
sudo salt-key -A     # accept all
sudo salt '*' test.ping

All three minions (both masters + the data node) should return True. If not, restart salt-minion on the silent box and retry.

Set up the pillar — the recipe Salt will read

This is the moment Salt starts configuring itself. Pillar is the data Salt reads to know what to build. Once you write these files, master 1 has the recipe — the highstate in step 10 reads them and uses them to install PostgreSQL, Valkey, and RaaS on the data node, and the master plugin on both masters. Steps 1–7 were hand-install. From here on, you write config and Salt does the rest.

8.1 — Stage the installer files

On master 1 only:

cd ~/Config-*/sse-installer
sudo mkdir -p /srv/salt/pillar
sudo cp -r salt/sse        /srv/salt/
sudo cp -r salt/top.sls    /srv/salt/
sudo cp -r pillar/sse      /srv/salt/pillar/
sudo cp -r pillar/top.sls  /srv/salt/pillar/

8.2 — Edit /srv/salt/pillar/top.sls

List the minion IDs of every box in this deployment. Salt uses this to know which boxes get the SSE pillar data.

{# Pillar Top File #}

{# Define VCF Salt servers #}
{% load_yaml as sse_servers %}
  - prod-raas
  - prod-master1
  - prod-master2
{% endload %}

base:
  {# Assign pillar data to each server #}
  {% for server in sse_servers %}
  '{{ server }}':
    - sse
  {% endfor %}

8.3 — Edit /srv/salt/pillar/sse/sse_settings.yaml

The file has five sections. Most fields are fine at default. The fields you actually need to set are highlighted in cyan below.

# Section 1: Servers — which minion does what (use minion IDs from step 06)
servers:
  pg_server:    prod-raas
  redis_server: prod-raas       # variable retained — actual DB is Valkey
  eapi_servers:
    - prod-raas
  salt_masters:
    - prod-master1
    - prod-master2           # comment out for single-master

# Section 2: PostgreSQL — endpoint is IP or DNS of the data node
pg:
  pg_endpoint: raas.example.com
  pg_port: 5432
  pg_username: salteapi
  pg_password: abc123                 # default — change after install
  pg_hba_by_ip: True
  pg_hba_by_fqdn: True
  pg_cert_cn: localhost
  pg_cert_name: localhost

# Section 3: Valkey — same data node as PostgreSQL
redis:
  redis_endpoint: raas.example.com
  redis_port: 6379
  redis_username: saltredis
  redis_password: def456              # default — change after install

# Section 4: eAPI / RaaS service
eapi:
  eapi_username: root                 # default — change after install
  eapi_password: salt                 # default — change after install
  eapi_endpoint: raas.example.com
  eapi_ssl_enabled: True
  eapi_ssl_validation: False
  eapi_standalone: False              # multi-node deploy = False
  eapi_failover_master: False         # False = active/active (recommended)
  eapi_key: <openssl rand -hex 32>
  eapi_server_cert_cn: localhost
  eapi_server_cert_name: localhost

# Section 5: Identifiers — must be unique per install
ids:
  customer_id: <uuidgen>
  cluster_id: saltmaster_cluster_1

openssl rand -hex 32

cat /proc/sys/kernel/random/uuid

The variable is still redis_server — the database is Valkey, but don't rename it. Both eapi_key and customer_id must be unique per install.

Install the pip wheels Air-gap only

Skip this step if your nodes have internet. The SSEAPE master plugin still needs to be installed (see the master plugin block below) — but the dependency wheels can come from PyPI directly.

Run on every node. The bundle ships these wheels in the pip/ folder. The SSEAPE master plugin is in sse-installer/.

cd ~/Config-*/pip
sudo salt-pip install PyJWT-*.whl
sudo salt-pip install pika-*.whl
sudo salt-pip install pyspnego-*.whl
sudo salt-pip install smbprotocol-*.whl
sudo salt-pip install pypsexec-*.whl

# master plugin — both masters only
cd ~/Config-*/sse-installer/salt/sse/eapi_plugin/files
sudo salt-pip install SSEAPE-*-py3-none-any.whl

With internet: sudo salt-pip install PyJWT pika pyspnego smbprotocol pypsexec pulls the same wheels from PyPI. The SSEAPE plugin still has to come from the bundle (it's not on PyPI).

Run the highstate

Data layer first, then masters. The order matters and there's a one-line toggle in the middle. From master 1.

10a · Refresh and apply RaaS:

sudo salt '*' saltutil.refresh_grains
sudo salt '*' saltutil.refresh_pillar
sudo salt prod-raas state.highstate

10b · On the RaaS node, enable autoaccept for the master plugin:

# SSH into prod-raas, then:
sudo vim /etc/raas/raas
# add this line under the existing config, save:
master_autoaccept: true

sudo systemctl restart raas

Without this, the SSEAPE master plugin can't register with RaaS during the next highstate and the masters fail with a key-rejection error. We enable it here, run the masters, then strip it back out in step 11.

10c · Apply the masters:

sudo salt prod-master1 state.highstate
sudo salt prod-master2 state.highstate

Watch tail -f /var/log/messages on the RaaS node. If a master highstate fails, bounce its salt-minion and retry. Tip: snapshot the VMs before this step.

Lock it down

Post-install cleanup. Strip the install-time autoaccept, point each master's minion at itself, drop the cross-keys. Order matters — do RaaS first, then each master in turn.

11a · Remove autoaccept on RaaS:

# SSH into prod-raas, then:
sudo sed -i '/master_autoaccept: true/d' /etc/raas/raas
sudo systemctl restart raas

This was set in 10b so the master plugin could register. With install done, it has to come back out — leaving it on means any new master plugin registers without review.

11b · Lock down master 1:

# on prod-master1
echo "master: localhost" | sudo tee \
  /etc/salt/minion.d/master.conf
sudo systemctl restart salt-master
sudo systemctl restart salt-minion
sudo salt '*' test.ping              # minions still reachable?
sudo salt-key -d prod-master2        # drop the cross-key
sudo salt '*' test.ping              # nothing else fell off?

11c · Lock down master 2:

# on prod-master2
echo "master: localhost" | sudo tee \
  /etc/salt/minion.d/master.conf
sudo systemctl restart salt-master
sudo systemctl restart salt-minion
sudo salt '*' test.ping
sudo salt-key -d prod-master1
sudo salt '*' test.ping

The two test.ping calls per master are the sanity checks. If the count drops between them, you deleted a key you shouldn't have — restore from snapshot and try again.