Salt Cheat Sheet

SERVICES


Start / Stop / Restart service on Minion

salt 'target' service.start "service name"  (start/stop/restart)


Restart Minion on Win target

salt 'target' cmd.run 'start powershell "Restart-Service -Name salt-minion"'


Restart Minion on Linux target

salt 'target' cmd.run 'service salt-minion restart"'


Execute a script remotely

salt target cmd.exec_code python 'import sys; print sys.version'

2.7.8 GCC 4.9.1


salt target cmd.exec_code sh 'echo $PATH'

/usr/local/bin:/usr/local/sbin


Check service on minion

salt target service.status httpd


Check if service is available

salt target service.available httpd


get all services

salt target service.get_all


reload a service config (avoids restart)

salt target service.reload httpd


start | stop | restart a service

salt target service.start httpd


TARGETING

by OS grain

salt -G os:Windows cmd.run "net stop Firewall"  


by other grains

salt –G 'server_type:app and env:prod' state.highstate


target EC2 instances only

salt -G uuid:ec2\* test.ping


compound match

salt –C 'server_type:web and clo*' state.sls nginx


list based match
salt –L 'hostname1,hostname2,hostname3' state.sls ntp


Nodegroup match

salt –N ny_db_servers cmd.run 'ps –ef | grep mysql'


regex OR

salt -E "(nyweb|db5)" test.ping


by pillar value

salt -I 'role:webserver' test.ping


KEY MANAGEMENT


Add Minions to Master

salt-key -L (show pending to be accepted)

salt-key -A (accept all pending)

salt-key -a target (accept by hostname)


Remove inactive minions from Salt

salt-run manage.down removekeys=True


Remove minions by name

salt-key -D targetName

SERVER DIAGNOSTICS


Test Connection

salt 'target' test.ping


Diagnostics

salt target status.all_status // gets all info

status.cpu_info

status.cpustats

status.uptime

status.diskusage  // or disk.usage

status.loadavg

status.meminfo

status.netdev  // network device

status.netstats //network stats

status.procs

status.version //system version

status.vmstats //virtual mem stats

status.w  //who is logged in


Show Minions by State (Up/Down)

salt-run manage.up

salt-run manage.down

salt-run manage.status  (show all by status)


show available file roots

on master

salt-run config.get file_roots

on minion

salt-call config.get file_roots

Compliance and Audit

to get a compliance result, run a State check with test=True

salt \* state.highstate test=True

This will return any differences from existing configuration to whats in the Top file


Show Salt Master version

salt --versions-report


Show Salt Minion version

salt-call --versions-report


Start Minion in Debug mode

salt-master --log-level=debug


Restart everything on Master:

pkill salt-minion  //Kill minion

pkill salt-syndic  // Kill Syndic

salt-run cache.clear_all   //Clear all cache

salt '*' saltutil.sync_grains   //Sync grains

salt-master -d  //Start master daemon

salt-minion -d  //Start minion daemon

salt-syndic -d  //Start syndic daemon


Agent Env Info


show all information about a minion (lots of data)

salt minion status.all_status


show memory

salt minion status.meminfo


show disk usage

salt minion status.diskusage


show who is logged in

salt minion status.w

GRAINS

Show Grain data

salt '*' grains.ls   

salt '*' grains.items


get specific Grain 

salt cent7 grains.get selinux

cent7:

    ----------

    enabled:

        True

    enforced:

        Enforcing


get multiple grains

salt cent7 grains.item selinux serialnumber zmqversion


set a Grain data on a node

salt cent7 grains.set 'apps:Myapp:port' 2500


salt cent7 grains.item apps

cent7:

    ----------

    apps:

        ----------

        Myapp:

            ----------

            port:

                2500



All grain data is stored on the minion in /etc/salt/grains file

if adding more data manually, refresh Grains on the Master to pick up changes

salt target saltutil.refresh_modules


Use grain in a state file

apache:

  pkg.installed:

    {% if grains['os'] == 'RedHat' %}

    - name: httpd


show JSON output

salt target grains.item ipv4 --out=json

{

    "target": {

        "ipv4": [

            "10.0.2.15", 

            "127.0.0.1", 

            "192.168.56.102"

        ]

    }

}


Use grain as a variable

{% set nodename = grains['nodename'] %}

base:

  '*':

    - common

    - packages

    - users

    - servers.{{ nodename }}



MINE


show mine data

salt \* mine.get \* x509.get_pem_entries


PACKAGES AND INSTALLATION

Verbose output (timeout 300 sec)

salt 'target' state.hightstate -t 300 -v


Show package version

salt 'target' pkg.version apache


install package on minions

salt 'target' pkg.install apache


Uninstall pkg

salt 'target' pkg.remove 'npp'

salt 'target' pkg.purge 'npp'


Show Installed Packages or Software

salt 'target' pkg.list_pkgs


show all packages that need updates

salt target pkg.list_upgrades


upgrade all packages

salt target pkg.upgrade


Windows (Chocolatey)

install chocolatey

salt wintarget chocolatey.bootstrap  


install pkg using choco

salt mrxwin7 chocolatey.install 7zip 


JOBS AND PROCESS CONTROL


Show all Salt jobs run history

salt-run jobs.list_jobs


Show active Salt jobs

salt-run jobs.active // returns a Job ID


Show currently running processes on a minion

salt '*' saltutil.running


Kill active job

salt 'target' saltutil.kill_job $JOB_ID

salt '*' saltutil.term_job <job id>


Clear Job cache

salt '*' saltutil.clear_cache


REACTOR


examples of reactor matching


/etc/salt/master.d/reactor.conf

reactor:

  - 'sayhello':

    - /srv/reactor/test.sls


/srv/reactor/test.sls

{% if data['id'].startswith('web') %}

sayhello:

  local.state.apply:

    - tgt: {{ data['id'] }}

    - arg:

      - say-hello

  

  local.cmd.run:

    - tgt: minion1

    - arg: 

      - "echo 'hello' > /tmp hello"


{% endif %}


you can kick of this Reaction via an Event

minion> salt-call event.send "sayhello" "{ name: Joe, age: 23 }"

DEBUG

Run highstate in debug

salt-call -l debug state.highstate


Run specific state in debug

salt-call -l debug state.sls elasticsearch


show highstate process (debug YAML syntax errors)

salt-call state.show_highstate


call a highstate, show only changes, timeout=10min
salt-call state.highstate test=true --state-output=changes -t 600


show specific State details

salt 'target' state.show_sls apache 


show only Changed and Failed during run

modify /etc/salt/master  and /etc/salt/minion, restart Master after change

state_verbose: True

state_output: mixed


start minion in debug, see connection errors

salt-minion -l debug


https://docs.saltstack.com/en/latest/topics/troubleshooting/minion.html


if Master not seeing Minion key requests, add IPTables rules to Master,

root@master# iptables -I INPUT -s 172.31.23.0/24 -p tcp -m multiport --dports 4505,4506 -j ACCEPT

root@master# iptables -I INPUT -s 172.31.25.0/24 -p tcp -m multiport --dports 4505,4506 -j ACCEPT


# reject everything else,

root@master# iptables -A INPUT -p tcp -m multiport --dports 4505,4506 -j REJECT


Log Jinja variables to Minion

{% do salt.log.error('testing jinja logging') -%}

show Options passed to a State (ie, test=true)

{% do salt.log.error(opts['test']) -%}

Output variables from State file,

show_var:

- test.show_notification:

    - text:  This is my var {{ var }}

Exit w failure message

fail_run:

  test.fail_without_changes:

    - name: your message here


SDB - Simple Database

store credentials in SDB, pass them to pillar in encrypted form, each minion can use credentials from pillars to run formulas

Scheduler

cat minion.d/_schedule.conf 

schedule:  __mine_interval: {enabled: true, function: mine.update, jid_include: true, maxrunning: 2,    minutes: 60, return_job: false, run_on_start: true}  job1: {enabled: true, function: test.ping, jid_include: true, maxrunning: 1, name: job1,    run: true, seconds: 30}  job2:    args: [date >> /tmp/date]    enabled: true    function: cmd.run    jid_include: true    maxrunning: 1    name: job2    seconds: 10

minion needs to be restarted to pick up schedule

Masterless Agent

install salt on host

cd /opt

python3 -m venv salt

cd /opt/salt

./bin/pip install salt

ln -s /opt/salt/bin/salt-call /usr/bin/salt-call

create minion config file

vim /etc/salt/minion

add contents


master_type: disablepub_ret: falsemine_enabled: falsefile_client: local
file_roots:
  base:    -  /srv/saltstack/salt/state
pillar_roots:
  base:    -  /srv/saltstack/salt/pillar

copy Saltstack repo to the host, place in /srv/saltstack

make sure salt-minion service is DISABLED and STOPPED

test salt-call state apply

salt-call test.pingsalt-call state.apply formula.myapp 


Slots - get result of salt cmd execution into a state file


content-from-slots:  file.managed:    - name: /tmp/things.txt    - contents: __slot__:salt:test.echo("hello world")

DOCS & FILES

Modules List

https://docs.saltstack.com/en/latest/salt-modindex.html

State List
https://docs.saltstack.com/en/latest/ref/states/all/index.html#all-salt-states

Salt source files
/usr/lib/python<version>/site-packages/salt

BOOTSTRAP

Bootstrap Install

https://repo.saltstack.com/#bootstrap


wget -O bootstrap_salt.sh https://bootstrap.saltstack.com 


install Master

sh bootstrap_salt.sh -M  


install specific Salt version

sh bootstrap_salt.sh git v2015.8.8

FILE OPERATIONS

Check if file contains a string (true/false)

salt '*' file.contains /etc/ssh/sshd_config 'Port'


Check if a file on the Salt minions contains a certain regex (search file on minions):

salt "*" file.contains_regex /etc/resolv.conf "timeout.4"


check if file is a file or directory

salt target file.stats /etc/hosts


Find a file

salt '*' file.find /etc name=host\*.\*

result

- /etc/host.conf

- /etc/hosts.allow

- /etc/hosts.deny'


copy small file ( > 100kb)  from Master to minion

salt-cp 'target' /opt/file (source)  /opt (destination)


copy dir from Master /srv/salt area to minion

salt 'target' cp.get_dir salt://myDir /target/dir


copy large file from Master /srv/salt/distribution folder to minion

salt 'target' cp.get_file salt://distribution/myFile.tar   /tmp/myFile.tar


copy file from one minion to another using MinionFS (only works after Salt 2016.3.2)

https://docs.saltstack.com/en/latest/topics/tutorials/minionfs.html


add MinionFS to master conf file

vi /etc/salt/master


add this lines: 

minionfs_mountpoint: salt://minionfs

file_recv: True


restart Master


get the file from the minion and store it in MinionFS

salt 'target' cp.push /path/to/file/or/dir/on/minion


files are stored on Master in here:

/var/cache/salt/master/minions/<minion>



copy file from Minion to Master

on Master, set "file_recv: True"  in /etc/salt/master, restart Master


to copy file, 


salt \* cp.push /path/to/file/on/minion


all files are stored on Master /var/cache/salt/master/minions/<minion>/files


add host entry to a minion

salt target hosts.add_host 192.168.55.100 hostname


Replace contents of file with new value

salt '*' file.sed /etc/ssh/sshd_config 'Port 22' 'Port 2201'


Create folder

salt '*' file.makedirs /tmp/testFolder/ (for windows use native Win syntax, ie C:/temp/dir)


Delete folder

salt '*' file.remove /tmp/testFolder


Create new file

salt '*' file.touch /tmp/testFolder/emptyFile


manage file content in State file

/etc/fstab:

  file.line: 

    - content: "proc    /proc"

    - mode: insert

    - after


replace contents of file

update_modprobe:

  file.replace:

    - name: /etc/modprobe.d/salt_cis.conf

    - pattern: "^install {{ fs }} /bin/true"

    - repl: install {{ fs }} /bin/true

    - append_if_not_found: True


create symlink

symlink: file.symlink: - name: /path/to/A - target: /symlink/path/A

create directory

/home/qb/q3:

  file.directory:

- user: qb

- group: qb

- dir_mode: 755

- file_mode: 755

- require:

     - user: qb


replace file contents using Augeas

sshd_config:

  augeas.change:

    - context: /files/etc/ssh/sshd_config

    - changes:

      - set Port 8888

      - set PasswordAuthentication yes

      - set PubkeyAuthentication yes


copy directory from Master to minion

app_ta_nix_dir:

file.recurse:

     - name: /opt/splunkforwarder/etc/apps/Splunk_TA_nix

     - source: salt://{{ slspath }}/files/apps/Splunk_TA_nix

     - makedirs: True

     - user: splunk

     - group: splunk

     - file_mode: 0755


File Managed

try several files if 1st one doesnt exist

monit_config:

    file.managed:

        - name: /etc/monit/monit.conf

        - source: 

            - salt://{{ slspath }}/files/configs/host/{{ grains.id }}.j2

            - salt://{{ slspath }}/files/configs/profile/{{ salt['pillar.get']('profile') }}.j2

            - salt://{{ slspath }}/files/configs/default.j2

        - template: jinja

        - makedirs: True

        - mode: 0600

        - user: monit

        - group: monit


USER AND GROUP MGMT


Set user's password to 123123

salt '*' shadow.set_password user02

'$6$EYk3o52W$DaSUIfHpYMBkSShFYXdODyrHbmQlCNKFghNl9FZzZshUn240GCOn5szQ3piyBMtt/x4m.'


Generate a password

salt 'target' shadow.gen_password myP@ssword

$6$nTul6WP1$EJ6THWEYKgOuGjqSEhnv8ZcYET6z/sDsSB.YBoyImRWEoDjguvcUahnY3UuNtNpECVhwxsjWI6ucvCc1


Additional ways to generate you own password:

Add User

salt target user.add Joe

Remove User

salt '*' user.delete Joe remove=True force=True

Show all users on a target

salt target user.list_users


Info on all users on a target

salt target user.getent


Info on specific user

salt target user.info Joe


Add User to Group

salt target user.chgroups Joe Administrator, LocalAdmin True

// or

salt target group.adduser admins Joe

Remove User from Group

salt target group.deluser admins Joe

Show users Groups

salt target user.list_groups Joe

Change Users Shell

salt '*' user.chshell user02 /bin/bash

get info on all groups

salt target group.getent

get info o a particular group

salt target group.info splunk

Delete group

salt target group.delete splunk

STATES

Highstate 

salt '*' state.highstate

Deploy specific state

salt '*' state.apply webserver

Run multiple state executions on same minion at once (by design Salt limits 1 state per minion at once)

salt target state.sls yourState concurrent=True


Requisites 

https://docs.saltstack.com/en/latest/ref/states/requisites.html

unless

vim:

  pkg.installed:

- unless:

   - rpm -q vim-enhanced

   - ls /usr/bin/vim


onlyif

set_RTC:

  cmd.run:

- name: "/usr/bin/timedatectl set-local-rtc 0"

- onlyif: "/usr/bin/timedatectl  | grep "RTC in local TZ" | grep yes"



require

bar:

  pkg.installed:

- require:

   - sls: foo


onchanges

extract_package:

  archive.extracted:

- name: /usr/local/share/myapp

- source: /usr/local/share/myapp.tar.xz

- archive_format: tar

- onchanges:

   - file: Deploy server package


watch

ntpd:

  service.running:

- watch:

   - file: /etc/ntp.conf


prereq

prereq allows for actions to be taken based on the expected results of a state that has not yet been executed. The state containing the prereq requisite is defined as the pre-requiring state. The state specified in the prereq statement is defined as the pre-required state.


graceful-down:

  cmd.run:

- name: service apache graceful

- prereq:

   - file: site-code


use

The use requisite is used to inherit the arguments passed in another id declaration. This is useful when many files need to have the same defaults.

/etc/foo.conf:

  file.managed:

- source: salt://foo.conf

- template: jinja

- mkdirs: True

- user: apache

- group: apache

- mode: 755


/etc/bar.conf:

  file.managed:

- source: salt://bar.conf

- use:

   - file: /etc/foo.conf


require_in

vim:

  pkg.installed:

- require_in:

   - file: /etc/vimrc


import YAML data into a state (same with import_json, import_text)

{% import_yaml 'formula/facl/files/configs/test.yaml' as myconfig %}


show_config:

  - test.show_notification:

    - text: {{ myconfig }}

SSH Management (Agentless)

edit the Roster file, include node's IP address and user to use (do this on Master)

vi /etc/salt/roster

# Sample salt-ssh config file

mrxcloud1:

   host: 104.131.102.230      # The IP addr or DNS hostname

   user: fred # Remote executions will be executed as user fred

   passwd: foobarbaz  # The password to use for login, if omitted, keys are used

   sudo: True      # Whether to sudo to root, not enabled by default


Run command on node (salt-ssh -i)

salt-ssh -i mrxcloud1 cmd.run "uname -a"


To run SSH as another user (non-root)

Salt Roster File - Ansible-syntax (configure Roster to have variables and Groups)


PILLAR

Sample Pillar structure to create system users pillar structure

Salt top file calls the Users state

/srv/salt/top.sls


base:

  '*':

    - common

    - users

Pillar top file tells what pillars to load for what nodes

/srv/pillar/top.sls


base:

  '*':

    - users


Users Pillar contains actual data for users

/srv/pillar/users.sls

users:

  spiderman:

    uid: 1280

    fullname: 'spider man'

    shell: /bin/bash

    ssh-keys:

      - ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAt1IFQP9xxx

  black.hood:

    uid: 1281

    fullname: black hood

    shell: '/bin/bash'

    ssh-keys:

      - ssh-rsa AADRN34zf12fdfd343434wAAAQEAwAAAQEA

  supergirl:

    uid: 1282

    fullname: super girl!

    shell: '/bin/bash'

    ssh-keys:

      - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNWRiUmFXjxrp4V

      - ssh-rsa ABCDEF134343434343dfdfdf343111dgfdfdfdf


Salt Users state parses array and creates users,

/srv/salt/users.sls

{% for user, args in pillar.get('users',{}).iteritems() %}

{{ user }}:

  group.present:

    - gid: {{ args['uid'] }}


  user.present:

    - fullname: {{ args['fullname'] }}

    - uid: {{ args['uid'] }}

    - gid: {{ args['uid'] }}

    - shell: {{ args['shell'] }}

    - home: /home/{{ user }}

{% endfor %}


Refresh pillars on all nodes

salt \* saltutil.refresh_pillar

Look at pillar data

salt \* pillar.items

get a Pillar value in a state file or Jinja file  (and pass a default value if no pillar is found)

{{ salt['pillar.get']('role:name', 'default') }}

get Pillar value by passing a variable

{% for rt in salt['pillar.get']('network:routes:{0}:networks'.format(interface)) -%}

get nested pillar value (use semi colon to get to specific key)

salt nycweb01 pillar.get users:joe

get pillar data into another pillar file

pillar1.sls


{% set mypillar = "../pillar/" + grains.get('host') + ".sls" %}
{% import_yaml mypillar as yaml %}
{{ yaml.data }}  # abc

/pillar/host1.sls

data: abc




PORTS & NETWORK

Master - Agent ports: 4505 (master to agent), 4506 (agent to master)


Get IP of a Minion

salt target network.ip_addrs

Ping from Minion

salt target network.ping someHostname

get all active TCP connections on a minion

salt target network.active_tcp

get ARP table 

salt target network.arp

test port connectivity for certain port

salt target network.connect www.google.com 80

get hardware address for a MAC

salt target network.hw_addr eth0

get intet address for interface

salt target network.interface eth0

get all interfaces

salt target network.interfaces

JINJA

 For Loop

{% for usr in 'moe','larry','curly' %}

{{ usr }}:

  group:

    - present

  user:

    - present

    - gid_from_name: True

    - require:

      - group: {{ usr }}

{% endfor %}


If Conditional


{% if var == 2 %}

    Var is 2

{% elif var == 5 %}

var is 5

{% else %}

    var is not 2

{% endif %}


While loop

{% range number from 3 to 6 %}

      {{ number }}

      (...)

  {% endrange %}


Comparisons

{% if 'Watermelon' ends with 'n' %}

 It ends with N

{% endif %}


{% if varA == varB %}

{% if varA != varB %}



get shell command value from inside Jinja

{% set procs = salt['cmd.run']('ps aux') %}


disable tab space in a for loop

{% for server in servers %}

{{ server }}

{% endfor %}


results in: 

   server1

   server2


to disable this, add 


#jinja2: lstrip_blocks: True


to top of the template


Run a state only if a file doesnt exist,

{% if not salt['file.directory_exists']('/opt/q') %}


deploy_kdb:

file.managed:

     - name: /opt/q.tar.gz

     - source: salt://repo/q.tar.gz


extract_kdb:

archive.extracted:

     - name: /opt/

     - source: /opt/q.tar.gz

     - user: kdb

     - group: kdb


{% endif %}


Test for File


{% if not salt['file.file_exists']('/opt/file') %}


render parameter

{{ var_name }}


set a parameter

{% set fruit = 'apple' %}


Iterate dictionary (For Loop)

{% for name, app in applications.iteritems() %}

   {{ name }}

   {{ app['version'] }}

{% endfor %}


sort a list

{% for vm in vcenter['vm_list']|sort %}

    {{ vm }}

{% endfor %}


or by attribute or reverse

{% for vm in vcenter['vm_list']|sort(attribute='osname', reverse = True) %}


get total # of elements in list

{{ myList|length }}


If statement with AND & OR

{% if var is None and var2 == 'blah' % or val3 == 'shmaa' %}


convert variable uppercase / lowercase

{{ somevar | upper }}


set a default value if value doesnt exist

{{ somevar or 'default message here' }}


match by regex

{% if grains.id | regex_match('nyc(.*)', ignorecase=True) %}


remove element from List
{% set myList = ["a", "b", "c"] %}
# remove B
{% set idx = myList.index("b") %}
{% set myList = myList.pop(idx) %}


Jinja Tricks

 

difference

{{ [1, 2, 3] | difference([2, 3, 4]) | join(', ') }}

>> 1


avg, min, max, is_list, 

{{ [1, 2, 3] | avg }}


generate random UID

{{ 'random' | uuid }}


date format

{{ 1457456400 | date_format }}

{{ 1457456400 | date_format('%d.%m.%Y %H:%M') }}

2017-03-08

08.03.2017 17:00

string to number

{{ '5' | to_num }}


run Salt execution module

{{ salt.cmd.run('whoami') }}

{{ salt.group.add('newgroup1') }}

regex match

{{ 'abcd' | regex_match('BC(.*)', ignorecase=True) }}


regex search

{{ 'muppet baby' | regex_search('pet(.*)', ignorecase=True) }}

>> baby


compare_lists, compare_dicts

{{ [1,2,3] | compare_lists([1,2,4]) }}

>> {'new': 4, 'old': 3}


list files in a directory

{{ '/etc/salt/' | list_files | join('\n') }}


escape Jinja syntax

{% raw %}

some text that contains jinja {% characters that need to be escaped

{% endraw %}


iterate a dictionary

parent_dict = [{'A':'val1','B':'val2'}]

{% for item in parent_dict %}

  {% for key,val in item.items() %}

     {{ key }} {{ val }}


ETC

generate random password hash

python -c "import crypt; print(crypt.crypt('password', crypt.mksalt(crypt.METHOD_SHA512)))"