How to run the test suite Edit

Published on Jun 30, 2025.

The testing and development tools have a bunch of dependencies, all managed by bundler. By default the tests use the latest version of Puppet. If you want a specific version of Puppet, you must set an environment variable such as:

export PUPPET_VERSION="~> 8.8.1"

Running the tests in a local ruby environment

Installing Dependencies

Dependencies for running tests are installed as gems via bundler and will run in ruby 2.7 to 3.4 (as of time of writing). It should be trivial to install via your package manager or gem.

Debian/Ubuntu

# apt install ruby-bundler

RedHat (and similar)

As of el9, dnf replaces yum. Use the appropriate package manager to get it installed.

# yum install rubygem-bundler
# dnf install rubygem-bundler

ArchLinux

# pacman -Syu extra/ruby-bundler

By Gem (Last Option)

# gem install bundler

Configuring Bundler

You can install all needed gems for spec tests into the modules directory by running:

bundle config set --local path '.vendor/bundle'
bundle config set --local without 'development system_tests release'
bundle install

If you also want to run acceptance tests, don’t exclude the system_tests group.

If you don’t know if you need to install or update gems, you can just add bundle update && bundle clean as an additional command.

Vox Pupuli helpers

Check out the following page if you want to add a test suite to your module or want to learn more about the Vox Pupuli test helpers:

Linting

Vox Pupuli uses puppet-lint for better code quality. To run it:

bundle exec rake lint

To automatically fix puppet-lint issues you can use:

bundle exec rake lint_fix

Aso run some RuboCop tests against it:

bundle exec rake rubocop

RuboCop also supports save autofixing:

bundle exec rake rubocop:autocorrect

And another autofix option that can fix more, but might break your code:

bundle exec rake rubocop:autocorrect_all

REFERENCE.md update

If REFERENCE.md is now out of date you can fix it with:

bundle exec rake strings:generate:reference

Unit tests

The unit test suite covers most of the code, as mentioned above please add tests if you’re adding new functionality. If you’ve not used rspec-puppet before then feel free to ask about how best to test your new feature.

To run the linter, the syntax checker and the unit tests:

bundle exec rake test

Detailed sub tasks

bundle exec rake spec

If you have multiple cpu cores available we suggest using the following command:

bundle exec rake parallel_spec

Single test file could be run by:

bundle exec rspec spec/classes/myclass_spec.rb
bundle exec rake spec SPEC=spec/classes/myclass_spec.rb

To limit test execution to a certain os or os release you can set the environment variable SPEC_FACTS_OS.

export SPEC_FACTS_OS=centos
export SPEC_FACTS_OS=centos-7

Running Acceptance Tests

The unit tests just check the code runs, not that it does exactly what we want on a real machine. For that we’re using Beaker.

This fires up a new virtual machine or container and runs a series of simple tests against it after applying the module.

Installing Dependencies for Acceptance Tests

Follow the steps in Installing Dependencies making sure to install the system_tests group of dependencies.

bundle config set --local without 'development release'

Beaker Hypervisors

Beaker uses a local hypervisor or container stack to build required OS/Distributions to test. By default this is Docker, but it supports different hypervisors via additional gems.

Note: it also works with Podman but you need to point it to the appropriate socket, for example with systemctl start --user podman.socket && export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock

Follow the instructions from Docker or your preferred/known working method to install.

For libvirt / vagrant, use the distribution’s packages to configure.

Remember to add your user to the group that allows access to docker/libvirt.

Environment Variables and hostnames

For Vox Pupuli’s acceptance testing suite, Beaker is managed by a set of environment variables.

  • BEAKER_HYPERVISOR Sets the Hypervisor, vagrant or vagrant_libvirt for VM based testing It is a good idea to export the BEAKER_HYPERVISOR variable in your shell configuration.
  • BEAKER_DESTROY Should the test environment be removed at the end of a test
    • BEAKER_DESTROY=onpass Only removes the environment if everything passes, allowing review of the system in the state it was in when the test suite failed.
    • BEAKER_DESTROY=no Always leave an artifact, which likely will break future runs without cleanup.
    • BEAKER_DESTROY=yes (DEFAULT) always clean up.
  • BEAKER_PROVISION Should we ensure a clean system is built to run the suite.
    • If BEAKER_DESTROY is set to no, BEAKER_PROVISION=yes will fail the run (because of the existing box/container)
    • If BEAKER_DESTROY is set to no, BEAKER_PROVISION=no will run the test suite against the system anyways, which may cause environmental issues if the test suite doesn’t perfectly put things to a default state.
  • BEAKER_SETFILE What should we be testing. Beaker will call beaker-hostgenerator to create a defaultconfiguration based on this, using the known configurations for Vox Pupuli.
  • BEAKER_PUPPET_COLLECTION What implementation and version of Puppet are we testing against

These are all defined in voxpupuli-acceptance to review how they are used and more specialized features.

Getting setfiles for a module

Installed as part of the voxpupuli test suite is metadata2gha, which as its name suggests, is a helper to generate a json blob containing the variables for all supported oses.

# From the base directory of the module
$ bundle exec metadata2gha | grep puppet_beaker_test_matrix | cut -d= -f2- | jq
[
# ... truncated for length
  {
    "name": "Puppet 8 - Debian 12",
    "env": {
      "BEAKER_PUPPET_COLLECTION": "puppet8",
      "BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet8}"
    }
  },
  {
    "name": "Puppet 7 - Debian 12",
    "env": {
      "BEAKER_PUPPET_COLLECTION": "puppet7",
      "BEAKER_SETFILE": "debian12-64{hostname=debian12-64-puppet7}"
    }
  },
# ... truncated for length
  {
    "name": "OpenVox 8 - Rocky 9",
    "env": {
      "BEAKER_PUPPET_COLLECTION": "openvox8",
      "BEAKER_SETFILE": "rocky9-64{hostname=rocky9-64-openvox8}"
    }
  },
  {
    "name": "OpenVox 7 - Rocky 9",
    "env": {
      "BEAKER_PUPPET_COLLECTION": "openvox7",
      "BEAKER_SETFILE": "rocky9-64{hostname=rocky9-64-openvox7}"
    }
  },
# ... truncated for length
]

This returns names that show Puppet implementation (OpenVox or Puppet) / language version, and operating system / version. It also provides the environment variables needed to be applied to run Beaker for this combination.

Running Beaker

Running Beaker is handled via a rake task bundle exec rake beaker, and either exporting variables or setting them for a single run are both supported:

  1. export variables, then run bundle exec rake beaker

    export BEAKER_SETFILE="ubuntu2404-64"
bundle exec rake beaker

  2. run bundle exec rake beaker with inline shell variables

    BEAKER_SETFILE="ubuntu2404-64" bundle exec rake beaker

Some softwares might also need to have an hostname set on the VM, to be used as FQDN, it is possible to set is through BEAKER_SETFILE:

BEAKER_SETFILE="almalinux9-64{hostname=almalinux9-64-puppet8.example.com}" bundle exec rake beaker

If you need to run tests against a different version or implementation of OpenVox (or Puppet) you can either export BEAKER_PUPPET_COLLECTION="openvox7" or add BEAKER_PUPPET_COLLECTION="openvox7" to your command line.

BEAKER_PUPPET_COLLECTION="openvox7" BEAKER_SETFILE="ubuntu2404-64" bundle exec rake beaker

Custom Facts

Projects might also have custom facts that can be set through environment variables, for example puppet-zabbix tests against a specific Zabbix version setting BEAKER_FACTER_zabbix_version which will pass a custom fact to the system of zabbix_version:

BEAKER_SETFILE="almalinux9-64" BEAKER_FACTER_zabbix_version="6.0" bundle exec rake beaker

For these cases refer to the module’s documentation and/or on GitHub CIs check the env for bundle exec rake beaker in tests.

For additional information, you can review voxpupuli-acceptance.

Run a specific test

As with unit tests, it is possible to run only a specific acceptance test, for example:

bundle exec rspec spec/acceptance/foo.rb

Running the tests in the VoxBox container

Struggling with Ruby installations or dependency issues? VoxBox simplifies your workflow by providing a ready-to-use container with all the Ruby tools you need. To guarantee a consistent rake environment, use -f /Rakefile to explicitly specify your Rakefile, rather than relying on potentially outdated versions in a repository. Learn more in the project’s README.

Installation

podman pull ghcr.io/voxpupuli/voxbox:8

Linting in VoxBox

cd my/module
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile lint
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile lint_fix

# lint with all lint plugins enabled, puppetlabs-spec_helper deactivates some of them
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile voxpupuli:custom:lint_all

Rubocop

podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile rubocop
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile rubocop:autocorrect

Unit tests in VoxBox

podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile spec
podman run -it --rm -e "SPEC_FACTS_OS=centos" -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile spec
podman run -it --rm -e "SPEC=spec/classes/myclass_spec.rb" -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile spec

REFERENCE.md update in VoxBox

podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile strings:generate:reference

Puppetfile

podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile r10k:syntax
podman run -it --rm -v $PWD:/repo:Z ghcr.io/voxpupuli/voxbox:8 -f /Rakefile r10k:dependencies