I started using Chef in March 2015; before then, I had used Fabric to attempt to do a lot of similar things.

My initial workflow for Chef was pre-determined when I started at $DAYJOB. This involved keeping environment files within Git and using the knife spork omni command to bump, upload and promote new cookbook versions on Chef Server.

That’s all fine when you’re working alone or with one or two others on Chef, but problems start to emerge once more people start to get involved.

This led me to think about the way we use Chef and what we are trying to achieve.

My objectives

1. Avoid merge conflicts and confusion with environment files
2. Have more control over Chef environments and their versions
3. Allow the potential for using separate Git branches per environment

To expand on these points:

Avoid merge conflicts and confusion with environment files

As more people started to work on our (centralised) Chef repository, we began to experience more and more merge conflicts and confusion around which version of a cookbook was actually deployed on the Chef Server, because one person’s local environment file would differ from that of Chef Server, and even other people.

Have more control over Chef environments and their versions

I wanted us to more easily edit cookbook versions on our two environments, testing and stable. For a long time, we had neglected to make use of these properly, subsequently leading to most servers being associated witht he _default environment.

Allow the potential for using separate Git branches per environment

As part of the second objective, I wanted us to potentially begin to use separate Git branches for changes that should be rolled-out to the testing environment separately (and cleanly) from those destined for stable. In most cases, we would want to branch from testing to work on features and bug fixes without these changes being inadvertently included in subsequent feature/bug branches.

Implementing these objectives

Chef doesn’t seem to have any definitive, clear-cut best practices for what we wanted to do, so our implementation is a combination of a few concepts, and actually probably similar to what a lot of other organisations have done as well.

Avoid merge conflicts and confusion with environment files

We started by deleting our local environment files and changed from using knife spork omni to a more traditional (and manual) process. Our typical workflow now looks like this:

# hack
# increment version in <cookbook>/metadata.rb
# git commit and create pull request
knife cookbook upload -E testing <cookbook> --freeze
# pull request approved and merged
knife environment edit stable # then edit the version to match that of testing

This is slightly longer than our previous workflow, where knife spork omni would combine the version bump, cookbook upload and environment edit in one step, but allows us more control over the process.

Have more control over Chef environments and their versions

Now that our workflow allows us more control over which environment gets which version of a cookbook, we could start to address how we rollout these changes.

As shown in the command example above, we upload the cookbook to the testing environment and freeze it. Freezing a cookbook version means that Chef Server will complain if you try to upload the same cookbook and version again. This is really useful when multiple people are working on their own branches of the same cookbook, as it ensures that only one of those branches will be deployed on the testing environment.

To demonstrate this more, here’s a simple diagram.

In this example, you can see the original base cookbook version is 1.0.0. The first person creates a feature branch based on this, makes their changes, commits, creates a pull request and then uploads their new cookbook version (1.1.0).

When the second person does the same thing, they may not have seen the existing pull request (and its new version), so naturally, they increment their branch version to 1.1.0, commit, create a pull request and then upload their cookbook version. Because this is numbered the same as the first pull request, it overwrites the first version on the Chef Server. This is highly confusing.

The takeaway: Always freeze your new cookbook versions when uploading to Chef Server!

Getting back on track… Our general workflow is to freeze and upload a new cookbook version to our testing environment while a pull request is open. This allows us time to have it rolled-out to our test servers and soak in for 24 hours or so to allow for any problems to surface. After that time, we set the new version in the stable environment with knife environment edit stable, after which it will roll-out to our production servers.

Allow the potential for using separate Git branches per environment

Lastly, this workflow gives us more flexibility with Git branching. If we want to (and we will probably need to), we will be able to branch each new feature or fix from the master branch, follow our current workflow and then pull request against the testing branch. This means that people creating branches in quick succession will always branch from the known stable version, rather than (potentially) a pull request that was only merged minutes before.

Conclusion

I’ve brain-dumped a lot of information in this post, so I accept that some of it may be confusing. Please feel free to get in touch if you have any questions about this, or anything else I’ve mentioned here!