Configuring basic configuration management with a client-server architecture

First up is a bit of an explanation about what Salt is, and why it’s so awesome. Salt was originally designed for high-speed communication (aka remote execution) with large numbers of systems. It then took the next logical step and became a configuration management utility founded on the same principle of fast communication.

So at it’s core Salt is a remote execution engine, that establishes a high-speed, secure and bi-directional communication grid for groups of systems. Then on top of that, Salt provides an extremely fast, flexible and easy-to-use configuration management system.

What I love most about Salt is how easy it is to use as a configuration management tool. The configuration files are written in yaml format and Salt provides many options for templating with jinja (amongst other engines). It’s relatively simple to look at your files and know exactly what’s going on.

We’re going to cover the configuration management portion today in a client-server or master-minions configuration.

Set up the salt-master

Firstly you will need to install salt. You can install via pip with pip install salt, via rpm with yum install salt-master (you will need the EPEL), or via a bootstrap script on GitHub (example one liner: curl -L https://bootstrap.saltstack.com | sudo sh).

Once you’ve installed Salt, you will need to start the service with service salt-master start and enable it at boot with chkconfig salt-master on.

For a CentOS 6 box setting up the salt-master with rpms looks like:

rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
yum install salt-master -y
chkconfig salt-master on

This assumes that you’re not using iptables, but if you are:

\-A INPUT -m state --state new -m tcp -p tcp --dport 4505 -j ACCEPT
\-A INPUT -m state --state new -m tcp -p tcp --dport 4506 -j ACCEPT

Set up the salt-minion

Firstly a bit of prep work is needed so that your minion can find a salt-master. By default the salt-minion looks for ‘salt’ on the network.

You can customize the minion’s configuration file manually or you can set the minion’s hostfile to point at the salt-master. Generally I like to set the hostfile and have the minion bootstrap everything (including the minion configuration). For example:

echo '192.168.250.100 salt' >> /etc/hosts

Next you will need to install salt. You can install via pip with pip install salt, via rpm with yum install salt-minion (you will need the EPEL), or via a bootstrap script on GitHub (example one liner: curl -L https://bootstrap.saltstack.com | sudo sh).

Once you’ve installed Salt, you will need to start the service with service salt-minion start and enable it at boot with chkconfig salt-minion on.

Back on the salt-master, run salt-key -A -y to accept all connected minio keys. Alternatively, if your environment makes sense to do so, set a cronjob to accept keys:

crontab -l | fgrep -i -v 'salt-key -A -y' | echo '\* \* \* \* \* salt-key -A -y' | crontab -

Applying state (configuration management) via the salt-master

In order to apply a system state, you need state files (which are stored in /srv/salt/ by default). There is a large community around SaltStack and salt formulas, and you can view the github page for community driven formulas and states. I also have this cool loadbalancer state (or formula rather) that I’ll walk you through a little here.

So for starters we’re going to build a basic state that installs keepalived. Why? Because I like keepalived. First let’s look at how we need to structure the state:

/srv/salt/loadbalancer/init.sls

state\_name:
  function:
    - arg
    - arg: value
  function:
    - arg
    - arg: value
  function:
    - arg
    - arg: value
  ...

So, for keepalived we’re going to name it appropriately and call the salt.states.pkg to install it:

keepalive:
  pkg:
    - name: keepalived
    - installed

The pkg function takes a name for the package (which can be different from the state name or take a mapped variable… but more on that another time!) and the required state (which in this case is installed).

So now we have a state that installs keepalived. But the service isn’t started?! That’s because we need to add to our state a salt.states.service function to specify the state as enabled at boot and started!

keepalive:
  pkg:
    - name: keepalived
    - installed
  service:
    - name: keepalived
    - enable: True
    - running

Easy enough, very similar language to the pkg function.

But what if the service tries to start before the package is installed? That’re where you need to build in dependancies.

keepalive:
  pkg:
    - name: keepalived
    - installed
  service:
    - name: keepalived
    - enable: True
    - running
    - require:
      - pkg: keepalived

So now we have a good state and it’s well structured. Keepalived will install and start without issue. But it’s just running the default configuration, which isn’t useful. That’s where the salt.states.file comes in.

keepalive:
  pkg:
    - name: keepalived
    - installed
  service:
    - name: keepalived
    - enable: True
    - running
    - require:
      - pkg: keepalived
  file:
    - name: /etc/keepalived/keepalived.conf
    - managed
    - source: salt://loadbalancer/files/keepalived.conf.jinja
    - template: jinja

Pretty self explanatory. But when changes to the configuration file happen, the service doesn’t get restarted. So let’s add a watch argument to the service.

keepalive:
  pkg:
    - name: keepalived
    - installed
  service:
    - name: keepalived
    - enable: True
    - running
    - require:
      - pkg: keepalived
    - watch:
      - file: /etc/keepalived/keepalived.conf
  file:
    - name: /etc/keepalived/keepalived.conf
    - managed
    - source: salt://loadbalancer/files/keepalived.conf.jinja
    - template: jinja

So that was a very high-level overview of how states work. And it covers the basic use case most people are initially interested in solving. There is a lot of documentation for how to build really nice formulas and states on docs.saltstack.com. I recommend you check it out!

To apply the state we run:

salt '\*' <function> \[arguments\]
salt '\*' state.sls \[arguments\]
salt '\*' state.sls loadbalancer

That command will make all your minions into loadbalancers (btw!).

To apply all states based on the /srv/salt/top.sls file:

salt '\*' state.highstate

Example environment with Vagrant

You can see an example environment using SaltStack and Vagrant on my GitHub account here. I also have a Vagrantfile included with the loadbalancer example above.