Operationalizing Puppet Stand-Alone Architecture

Creating a Puppet module that provides a number of features on top of a default Puppet Agent installation.

Problem

The scenario is that we have are looking to run Puppet with the stand-alone architecture (deprecated but still supported in Puppet 8.X.X). We have installed Puppet Agent as described in Install Puppet and wondering how to operationalize it.

Solution

We will build a Puppet module that provides a number of features on top of a default Puppet Agent installation.

• A mechanism to easily perform minor / patch upgrades for Puppet Agent
• Disabling the puppet service; required with the stand-alone architecture
• Enabling the puppet command to be run using the sudo command
• Enabling Puppet Reports
• Enable the use of third-party modules
• Automatic Puppet apply every 30 minutes

Prerequisites

This article assumes that you have a basic understanding of Puppet; the series of articles starting with Puppet Concepts by Example: Part 1 covers what is required.

If you wish to follow along, you will need an Internet connected machine running a Debian (or derivative) operating system with Puppet Agent installed as described in Install Puppet, e.g. this article was written using Debian GNU/Linux 12 (bookworm) using Puppet Agent version 8.10.0.

note: While this article was written specifically with Debian (or derivative) in mind, the underlying ideas still apply to most any Linux operating system.

You will need to know the specific version of the puppet-agent package installed, e.g., here we can see that this article was written using 8.10.0–1bookworm.

# apt-cache policy puppet-agent
puppet-agent:
  Installed: 8.10.0-1bookworm
  Candidate: 8.10.0-1bookworm
...

Using the Puppet Module

While we are going to be building out the Puppet module in the remaining sections, we first see how this Puppet module (can be downloaded here) is used.

The Puppet module, puppet, needs to be copied into the /etc/puppetlabs/code/environments/production/modules folder.

The puppet module has to be included with the required version parameter set to the current version of Puppet Agent; later the parameter can be used to perform minor / patch upgrades for Puppet Agent.

One way to accomplish this is to create a site.pp file in the /etc/puppetlabs/code/environments/production/manifests folder.

lookup('classes', Array[String], 'unique').include

and in the /etc/puppetlabs/code/environments/production/data folder we create a common.yaml file.

---
classes:
  - puppet
puppet::version: 8.10.0-1bookworm

With this in place, we can run Puppet apply:

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-test.c.alf-testing.internal in environment production in 0.48 seconds
Notice: /Stage[main]/Puppet/Service[puppet]/enable: enable changed 'true' to 'false'
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet]/ensure: created
Notice: /Stage[main]/Puppet/File[/var/lib/puppet]/ensure: created
Notice: /Stage[main]/Puppet/File[/var/lib/puppet/reports]/ensure: created
Notice: /Stage[main]/Puppet/File[/etc/puppetlabs/puppet/puppet.conf]/content: content changed '{sha256}486f54346d69ef4aa47df5c7cb1a117b4cf9dac4aad77788eb79dd4d7540cfaf' to '{sha256}1b007fb63c10aca9c5a083f4307551142bd256e3ca121577f52d50ee24076d79'
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet-reports-cleanup]/ensure: defined content as '{sha256}3bb43216c5a7f1c477eee2319904c87d3386ffac6a2255050212c0e0c651f9f1'
Notice: /Stage[main]/Puppet/File[/etc/cron.d/puppet-reports-cleanup]/ensure: defined content as '{sha256}bebd35f7ffcaf71b3414036c12549b79709a50c4d5cfb81539478c5942b9cbfa'
Notice: /Stage[main]/Puppet/Exec[install_librarian_puppet]/returns: executed successfully
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/librarian-puppet]/ensure: created
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet-apply]/ensure: defined content as '{sha256}d48ebeb14fbaa609a01e46ddb628a291b6c09656a011f69876ab5a585b1a49c2'
Notice: /Stage[main]/Puppet/File[/etc/cron.d/puppet-apply]/ensure: defined content as '{sha256}0ac9797d9ab821e477f3b0800a1b3c276fa349c402c1e7127df6f29d7bfd80ce'
Notice: Applied catalog in 6.54 seconds

After the puppet module is first applied, we can now use third-party modules by setting the modules Hiera key with key-value pairs consisting of a module name (key) and module version (value). We also need to include the third-party module.

For example, building off the previous implementation, we would update common.yaml to install and use the puppetlabs-apache version 12.2.0 module.

---
classes:
  - apache
  - puppet
modules:
  puppetlabs-apache: 12.2.0
puppet::version: 8.10.0-1bookworm

note: A bit of confusion is that the module name to install is in form [OWNER]-[MODULE NAME], e.g., puppetlabs-apache to install it and apache to use it.

We now need to use the puppet-apply command (installed by the puppet module) to apply the Puppet configuration.

# puppet-apply
Notice: Compiled catalog for puppet-test.c.alf-testing.internal in environment production in 0.85 seconds
Notice: /Stage[main]/Apache/Package[httpd]/ensure: created
...
Notice: Applied catalog in 14.58 seconds

Create Puppet Module

Having seen how to use the puppet module, let us walk through building it in this and the remaining sections.

We start by creating the puppet module by creating the following folders and init.pp file under the /etc/puppetlabs/code/environments/production/modules folder.

init.pp

class puppet {
  notify { "hello world": }
}

note: Here we assume that the puppet module has been included (see the Using the Puppet Module section above for a way to accomplish this).

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.03 seconds
Notice: hello world
Notice: /Stage[main]/Puppet/Notify[hello world]/message: defined 'message' as 'hello world'
Notice: Applied catalog in 0.01 seconds

note: At this point, the puppet executable is not in the secure path of the sudo command; we will fix that later.

puppet-agent Version

The first feature of the puppet module is to use it to manage the minor / patch version of the puppet-agent package.

note: Upgrading a major version would require enabling a different repository which is outside the scope of this article.

We update init.pp file to:

class puppet (String $version) {
  package { 'puppet-agent':
    ensure => $version,
  }
}

note: Here we assume that the puppet module has to be included with the required version parameter set to the current version of Puppet Agent (see the Using the Puppet Module section above for a way to accomplish this).

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.37 seconds
Notice: Applied catalog in 0.28 seconds

While here we can see this is results in a no-op operation, we can later update module’s version parameter to upgrade the puppet-agent package to a new minor / patch version.

Disable puppet Service

Because we are running Puppet in the stand-alone architecture we need to disable (prevent from running at boot) the puppet service; here we can see that it defaults to being stopped but enabled.

# service puppet status
○ puppet.service - Puppet agent
     Loaded: loaded (/lib/systemd/system/puppet.service; enabled; preset: enabl>
     Active: inactive (dead)
       Docs: man:puppet-agent(8)

We update init.pp to:

class puppet (String $version) {
  package { 'puppet-agent':
    ensure => $version,
  }

  service { 'puppet':
    enable  => false,
    ensure  => stopped,
    require => Package['puppet-agent'],
  }
}

note: For good measure, we ensure that the service is also stopped.

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.46 seconds
Notice: /Stage[main]/Puppet/Service[puppet]/enable: enable changed 'true' to 'false'
Notice: Applied catalog in 0.72 seconds

We can indeed confirm that it worked as expected.

# service puppet status
○ puppet.service - Puppet agent
     Loaded: loaded (/lib/systemd/system/puppet.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:puppet-agent(8)

sudo Secure Path

We can see where the puppet executable is installed.

# which puppet
/opt/puppetlabs/bin/puppet

The problem is that the folder /opt/puppetlabs/bin is not in the sudo secure path which can be corrected by creating a symbolic link from /usr/local/bin/puppet to the executable by adding to init.pp.

class puppet (String $version) {
  ...

  file { '/usr/local/sbin/puppet':
    ensure => 'link',
    require => Package['puppet-agent'],
    target => '/opt/puppetlabs/bin/puppet',
  }
}

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.47 seconds
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet]/ensure: created
Notice: Applied catalog in 0.35 seconds

Indeed we can now run the puppet command using sudo.

$ sudo puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.45 seconds
Notice: Applied catalog in 0.30 seconds

Puppet Reports

Next we enable Puppet reports and ensure that we only keep the most recent ones.

In the puppet module folder create a files folder with three files:

puppet.conf

reports=store
reportdir=/var/lib/puppet/reports

note: This approach replaces puppet.conf; a better approach (outside the scope of this article) would have been to ensure that puppet.conf had these two configuration lines.

puppet-reports-cleanup

#!/usr/bin/env bash
set -euo pipefail

find /var/lib/puppet/reports -mmin +59 -type f -exec rm -f {} \;

puppet-reports-cleanup.cron

0 * * * * root puppet-reports-cleanup

and then adding to init.pp.

class puppet (String $version) {
  ...
 
  file { [ '/var/lib/puppet', '/var/lib/puppet/reports', ]:
    ensure => directory,
  }

  file { '/etc/puppetlabs/puppet/puppet.conf':
    require => Package['puppet-agent'],
    source => 'puppet:///modules/puppet/puppet.conf'
  }

  file { '/usr/local/sbin/puppet-reports-cleanup':
    mode => '0755',
    source => 'puppet:///modules/puppet/puppet-reports-cleanup',
  }

  file { '/etc/cron.d/puppet-reports-cleanup':
    source => 'puppet:///modules/puppet/puppet-reports-cleanup.cron',
  }
}

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.48 seconds
Notice: /Stage[main]/Puppet/File[/var/lib/puppet]/ensure: created
Notice: /Stage[main]/Puppet/File[/var/lib/puppet/reports]/ensure: created
Notice: /Stage[main]/Puppet/File[/etc/puppetlabs/puppet/puppet.conf]/content: content changed '{sha256}e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' to '{sha256}1b007fb63c10aca9c5a083f4307551142bd256e3ca121577f52d50ee24076d79'
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet-reports-cleanup]/ensure: defined content as '{sha256}3bb43216c5a7f1c477eee2319904c87d3386ffac6a2255050212c0e0c651f9f1'
Notice: /Stage[main]/Puppet/File[/etc/cron.d/puppet-reports-cleanup]/ensure: defined content as '{sha256}bebd35f7ffcaf71b3414036c12549b79709a50c4d5cfb81539478c5942b9cbfa'
Notice: Applied catalog in 0.36 seconds

With this in place, we can run the apply a couple of times to generate the reports in /var/lib/puppet/reports.

And if we wait a couple of hours, we will see that all the reports are deleted.

Install librarian-puppet

We prepare to support third-party modules available at Puppet Forge using librarian-puppet by adding to init.pp.

class puppet (String $version) {
  ...
   exec { 'install_librarian_puppet':
    command => '/opt/puppetlabs/puppet/bin/gem install librarian-puppet -v 5.0.0 && /opt/puppetlabs/puppet/bin/gem uninstall minitar -I && /opt/puppetlabs/puppet/bin/gem install minitar -v 0.12',
    creates => '/opt/puppetlabs/puppet/bin/librarian-puppet',
    require => Package['puppet-agent'],
  }

  file { '/usr/local/sbin/librarian-puppet':
    ensure => 'link',
    require => Exec['install_librarian_puppet'],
    target => '/opt/puppetlabs/puppet/bin/librarian-puppet',
  }
}

note: As of the writing of this article, the 5.0.0 version of librarian-puppet is incompatible with the version of minitar installed; so we a had to downgrade minitar.

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.50 seconds
Notice: /Stage[main]/Puppet/Exec[install_librarian_puppet]/returns: executed successfully
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/librarian-puppet]/ensure: created
Notice: Applied catalog in 3.25 seconds

Indeed we can see that the librarian-puppet executable is available.

# librarian-puppet -h
Commands:
  librarian-puppet clean           # Cleans out the cache and install paths.
  librarian-puppet config          # Show or edit the config.
  librarian-puppet help [COMMAND]  # Describe available commands or one specific command
  librarian-puppet init            # Initializes the current directory.
  librarian-puppet install         # Resolves and installs all of the dependencies you specify.
  librarian-puppet outdated        # Lists outdated dependencies.
  librarian-puppet package         # Cache the puppet modules in vendor/puppet/cache.
  librarian-puppet show            # Shows dependencies
  librarian-puppet update          # Updates and installs the dependencies you specify.
  librarian-puppet version         # Displays the version.

Use librarian-puppet

In the puppet module files folder, we create a puppet-apply file which essentially:

1. Uses Puppet lookup to generate a list of third-party modules to install from Hiera
2. Creates a Puppetfile file with the list of third-party modules
3. Uses the Puppetfile file and librarian-puppet to install the third-party modules into the global modules folder, /etc/puppetlabs/code/modules
4. Calls Puppet apply; where third-party modules can now be used

#!/usr/bin/env bash
set -euo pipefail

readonly ENVIRONMENT='/etc/puppetlabs/code/environments/production'
readonly LIBRARIAN='/etc/puppetlabs/code'
readonly INPUT=$(puppet lookup --merge=deep modules "${ENVIRONMENT}")
cat << EOF > "${LIBRARIAN}/Puppetfile"
#!/usr/bin/env ruby
#^syntax detection
forge "https://forgeapi.puppetlabs.com"
EOF
COUNT=0
while IFS= read -r LINE
do
  if [[ "${COUNT}" -eq 0 ]]
  then
    COUNT=$((COUNT+1))
    continue
  fi
  IFS=': '; MODULE=($LINE); unset IFS;
  echo "mod '${MODULE[0]}', '${MODULE[1]}'" >> "${LIBRARIAN}/Puppetfile"
  ((COUNT++)) 
done <<< "${INPUT}"
cd "${LIBRARIAN}"
librarian-puppet install
puppet apply /etc/puppetlabs/code/environments/production

We add the following to init.pp.

class puppet (String $version) {
  ...

  file { '/usr/local/sbin/puppet-apply':
    mode => '0755',
    source => 'puppet:///modules/puppet/puppet-apply',
  }
}

We apply.

# puppet apply /etc/puppetlabs/code/environments/production
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 0.50 seconds
Notice: /Stage[main]/Puppet/File[/usr/local/sbin/puppet-apply]/ensure: defined content as '{sha256}c5aa369ef16df3ebdbf891759fd1bf0a07f92041f145b952850fd406958efc49'
Notice: Applied catalog in 0.35 seconds

We can now use third-party modules by setting the modules Hiera key with key-value pairs consisting of a module name (key) and module version (value). We also need to include the third-party module. See the Using the Puppet Module section above for an example of doing this.

One important observation here is that we have to run puppet apply /etc/puppetlabs/code/environments/production once on a machine with no third-party modules and then switch to using the puppet-apply where we can now use third-party modules.

puppet-apply Cron

The wrap up the puppet Module, we create puppet-apply.cron in the puppet module’s files folder.

0/30 * * * * root puppet-apply

and add to init.pp to run the puppet-apply commands once every 30 minutes.

class puppet (String $version) {
  ...

  file { '/etc/cron.d/puppet-apply':
    source => 'puppet:///modules/puppet/puppet-apply.cron',
  }
}

We apply (again using puppet-apply).

# puppet-apply
Notice: Compiled catalog for puppet-learning.c.alf-testing.internal in environment production in 1.02 seconds
Notice: /Stage[main]/Puppet/File[/etc/cron.d/puppet-apply]/ensure: defined content as '{sha256}0ac9797d9ab821e477f3b0800a1b3c276fa349c402c1e7127df6f29d7bfd80ce'
Notice: Applied catalog in 0.86 seconds