librarian & r10k

.
├── .git/
├── .gitmodules
└── modules
    ├── stdlib
    ├── apache
    ├── ...

Most modules these days are on

git & Forge

Ask yourself

• How do I get my modules?
  From git or forge? or custom?
• How about dependencies?
  Who takes care of missing links?
• How do I tackle changes?
  hotfixes, new featues, adjustments, ...
• One repo to rule them all?
  having all features/fixes tracked in one giant repo?

#1 Git Submodules

.
├── .git/
└── modules
    ├── a  (where is this module compared to upstream?)
    ├── b  (changes done here mix with...)
    └── c  (... changes here, which is difficult to re-use)

Giant-repo issues

• All change history is mixed together More difficult to separate and track what happened.
• Missing ACLs per folder Either your users get too many permissions or too few.
  The first may result in GAUs/catastrophy, the second in a tedious update process.
• You have to check upstream modules into your repo This may or may not be something you want.
  Checking it in offers stability at the cost of flexibility
• Additional work to publish a module At least if you want to offer the DVCS history to contributors to join.

.
├── .git/
├── .gitmodules
└── modules
    ├── a (via submodule)
    ├── b (via submodule)
    └── c (via submodule)

Great because...

• No extra tools to install; just git

  Though just a minor bonus, as Puppet requires Ruby

• Easy to move pinned commit

  Just change into the folder, look at git tracks, check out what you need

• Simple

  Get all modules with a simple command

Limitations

• Static

  (only commit-pinning; not branch- or ref-based)

• Complex process to create new features

  I.e. create feature at root repo, create feature at submodule, add contents, commit to submodule, commit to root

• Only works with git

  (which is pretty much the default)

• No dependency resolution

  you have to track it manually

#2 librarian-puppet

forge "https://forgeapi.puppetlabs.com"

metadata

mod 'puppetlabs-stdlib', '4.3.2'
mod 'puppetlabs-apache',
  git: 'git://github.com/puppetlabs/puppetlabs-apache.git'
		

librarian-puppet install

Librarian will take care of pulling all required modules

mod 'apache',
  git: 'https://github.com/puppetlabs/puppetlabs-apache',
  ref: '1.1.x'

mod 'stdlib',
  git: 'https://github.com/puppetlabs/puppetlabs-stdlib',
  ref: '4.3.2'

mod 'mysql',
  git: 'https://github.com/puppetlabs/puppetlabs-mysql',
  ref: 'f9dcac055baef105f57fdf63e26a3268a1d77cfd'
		

.
├── modules
│   ├── apache/
│   ├── concat/
│   ├── mysql/
│   └── stdlib/
├── .librarian        (local librarian config)
├── .tmp              (local cache)
├── Puppetfile
└── Puppetfile.lock   (pinned configuration)
		

FORGE
  remote: https://forgeapi.puppetlabs.com
  specs:
    puppetlabs-concat (1.1.0)
      puppetlabs-stdlib (>= 4.0.0)
    puppetlabs-stdlib (4.3.2)

GIT
  remote: https://github.com/puppetlabs/puppetlabs-apache
  ref: 1.1.x
  sha: 212e09d383c7382aa269e8c00e5c20c1c3808b2d
  specs:
    apache (1.1.1)
      puppetlabs-concat (>= 1.0.0)
      puppetlabs-stdlib (>= 2.4.0)
...

{
  "name": "hardening/ssh_hardening",
  "version": "1.0.2",
...
  "dependencies": [
    {
      "name": "saz/ssh",
      "version_requirement": ">= 2.3.6"
    },
    {
      "name": "puppetlabs/stdlib",
      "version_requirement": ">= 4.2.0"
    }
  ]
}

Commands:
  librarian-puppet clean           # Cleans out the cache
  librarian-puppet config          # Show or edit the config.
  librarian-puppet help [COMMAND]  # Describe commands
  librarian-puppet init            # Initializes the directory.
  librarian-puppet install         # Install modules
  librarian-puppet outdated        # Lists outdated dependencies.
  librarian-puppet package         # Cache to vendor/puppet/cache.
  librarian-puppet show            # Shows dependencies
  librarian-puppet update          # Updates and install
  librarian-puppet version         # Displays the version.

Great because...

• Supports Git and Puppet Forge

  also supports internal Forges like Pulp

• Track via Commit, Tag, Branch

  gives more flexibility and removes a few error-cases

• Dependency resolution

  so you don't forget new modules and versions

• Version pinning

  Your current installation is written to "Puppetfile.lock"

Limitations

• Caching only once per instance

  Which is tricky if you deploy from git for different environments and commits

• Doesn't handle failures gracefully

  It may remove local modules and doesn't load from cache

#3 r10k

mod 'apache',
  git: 'https://github.com/puppetlabs/puppetlabs-apache',
  branch: '1.1.x'

mod 'stdlib',
  git: 'https://github.com/puppetlabs/puppetlabs-stdlib',
  tag: '4.3.2'

mod 'mysql',
  git: 'https://github.com/puppetlabs/puppetlabs-mysql',
  commit: '4890ddb2f0c65c40101683c5577f943a7594b509'
		

Puppet originally introduced static environments

[master]
# Environment independent settings
vardir = '/var/lib/puppet'

[production]
modulepath = '/etc/puppet/environments/production/modules'

[testing]
modulepath = '/etc/puppet/environments/testing/modules'

This was troublesome for development Imagine 2+ features being developed at the same time
=> code pollution in development environment

Use dynamic environments

[master]
vardir = '/var/lib/puppet'
modulepath = '/etc/puppet/environments/$environment/modules'

$environment will get resolved once you run

puppet agent -t --environment myenv

r10k creates environments from branch names
E.g. a production branch will result in an environment by that name.

Configure r10k
Point it to your repository (and included branches) and environment folder:

---
sources:
  operations:
    remote: 'git://git-server.site/my-org/org-modules'
    basedir: '/etc/puppet/environments'

To deploy
To run the deployment with Puppetfile synchronized and verbose output

r10k deploy environment -pv

Which will create

/etc/puppetlabs/puppet/environments/production/modules
├── puppetlabs-apache (v1.1.1)
├── zack-r10k (v2.2.8)
/opt/puppet/share/puppet/modules
├── puppetlabs-apt (v1.1.0)
├── puppetlabs-stdlib (v4.3.2)
└── ...

Great because...

• Optimized caching of Git checkouts

  With cache reuse when possible

• Supports Puppetfile-Syntax

  with some adjustments

• Graceful with failures

  Uses cache whenever possible

• Conceived for changing environments

  Especially well fit for large projects and quick development cycles

Limitations

• No dependency resolution

  At the moment you have to do it manually

• Some hidden failures

  This is rare but may happen with some Puppetfile configurations

Traps

karabos.files.wordpress.com

Hidden failures

Modules may not get installed. In most cases you will get an error or at least warning when something went wrong. In some rare cases this also happens without any errors due to to internal inconsistencies.

Check your modules/ folder to verify.

Flexibility at a price

With git branch dependencies, commits may vary New commits to the branch will move it along. Two deployments may not be on the same commit-ID.

Librarian-puppet provides Puppetfile.lock In this file commit-IDs are pinned for all references. Use it to gain consistency. Alternatively pin to tags or IDs.

Missing commit

When pinning commits, they may be missing e.g. if they haven't been pushed yet
git history rewrites may alltogether wreck your debugging history.