Berkshelf integration (yes, again)

[RSO]

At Factlink we're in the process of moving our setup from a single chef-repo with librarian-chef towards a repo-per-cookbook with Berkshelf setup. During this move we're taking on bugs as they show up, as I did with knife-spork's Berkshelf integration, only to later find out that knife-spork might not be so suiting for Berkshelf integration at all.

First I will describe our current setup/folder structure:

~/dev/
    chef-repo/
    chef-factlink-web/
    chef-factlink-database/
    chef-factlink-otherstuff/

Originally we packed a Berksfile in our chef-repo in which we'd state all the direct dependencies:

cookbook 'chef-factlink-web'
cookbook 'chef-factlink-database'
cookbook 'chef-factlink-otherstuff'

This caused a lot of overhead in the update/upload process with Berkshelf, which made me do some investigating on a fluid Berkshelf flow. In berkshelf/berkshelf#535 @reset states that the chef-repo should NOT contain a Berksfile and that all application cookbooks should manage their own dependencies, which kinda makes sense to me, although it introduces a new problem with setting up knife-spork integration with Berkshelf.

In #74 @sethvargo introduced a way to resolve cookbooks through Berkshelf, but this doesn't work completely with the flow that is proposed in berkshelf/berkshelf#535 since I'm no longer able to promote chef-factlink-web directly, since there is no place to resolve this.

As stated we just started using Berkshelf, so maybe I'm missing something in this flow, so I am interested in learning what the correct knife-spork/berkshelf flow looks like.

If we're doing everything correctly, but something is indeed still missing in the knife-spork/berkshelf integration, I would like to propose support for running knife-spork from out of a cookbook folder itself, something along the likes of:

knife spork promote ENVIRONMENT .

What do you think?

[poliva83]

Did you find any documentation about what correct knife-spork/berkshelf workflow looks like? I am interested in moving my team to knife-spork but need to know berkshelf integration works properly.

[sethvargo]

@jonlives @RSO I will work on this once we get Berkshelf 3 out the door.

[rafaelmagu]

Hey @sethvargo, any updates on this? I am trying to setup a CI pipeline for our in-house cookbooks and would like the CI server to automatically bump the versions for staging environment, but am also getting the errors described in #109. Can we get the gem pushed out to RubyGems?

UPDATE: I realised this change has already made it into 1.3.2, but for some reason Berkshelf isn't being called properly when I run knife-spork commands inside my cookbook folder.

/Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/berkshelf-3.1.1/lib/berkshelf/berksfile.rb:20:in `exist?': no implicit conversion of nil into String (TypeError)
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/berkshelf-3.1.1/lib/berkshelf/berksfile.rb:20:in `from_file'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/bundler/gems/knife-spork-15d5e2f45e16/lib/knife-spork/runner.rb:147:in `load_from_berkshelf'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/bundler/gems/knife-spork-15d5e2f45e16/lib/knife-spork/runner.rb:132:in `load_cookbook'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/bundler/gems/knife-spork-15d5e2f45e16/lib/chef/knife/spork-bump.rb:43:in `run'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/chef-11.12.2/lib/chef/knife.rb:492:in `run_with_pretty_exceptions'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/chef-11.12.2/lib/chef/knife.rb:174:in `run'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/chef-11.12.2/lib/chef/application/knife.rb:135:in `run'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/gems/chef-11.12.2/bin/knife:25:in `<top (required)>'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/bin/knife:23:in `load'
    from /Users/rfonseca/.rbenv/versions/2.1.1/gemsets/global/bin/knife:23:in `<main>'

[rafaelmagu]

Ignore me. I was being a muppet. Was trying to bump the cookbook from inside the cookbook folder. Had to actually add -o .. to get it to work.

[rafaelmagu]

And I'm back 😄. I have now reached the same stage as @RSO, as in I can't promote the cookbook using the same folder structure/dev workflow. It seems knife-spork does not like path: . (which I believe is Ruby 1.9 syntax for hashes):

site-base$ knife spork promote dev site-base -b Berksfile.lock
ERROR: SyntaxError: Berksfile.lock:3: syntax error, unexpected ':', expecting end-of-input
    path: .

[sethvargo]

@rafaelmagu that error is because spork is loading berkshelf 2 and can't parse the lockfile. That's a Berksfile.lock from Berkshelf 3

[jperry]

I am also trying to get knife-spork working with berkshelf3 and right out of the gates including berkshelf3 and knife-spork in my Gemfile the "-b" option isn't showing up in the help options.

[ouranos]

Same here. Any progress on this issue? Or is there any documentation on a working workflow including spork and berkshelf somewhere?

I had a look at some of the pull requests without any success :/

[howdoicomputer]

Berkshelf is now at version 3. @sethvargo Are there plans to continue support for knife spork?

[reset]

@howdoicomputer we don't work on knife spork or directly support it, but we can provide the things necessary for the knife spork team to get things working

[howdoicomputer]

@reset Sorry, I was going off of this comment from earlier in the thread

"I will work on this once we get Berkshelf 3 out the door."

My apologizes for the misunderstanding; I thought you guys were directly contributing to the project.

[jonlives]

@howdoicomputer @sethvargo has contributed Berkshelf support in the past - it's definitely something I'd love to continue supporting in knife-spork, but as I don't use Berkshelf myself it's unfortunately not something I'm particularly qualified to add. With that said, I'd be more than happy to work with Berkshelf users & @reset et al. to figure out what support needs adding

[mjuarez]

I'm using knife-spork and Berkshelf in a continuous integration pipeline for internal wrapper cookbooks. After running tests, I'll use knife-spork to version bump, commit back, tag. I also use knife-spork to then promote the new passed build of the cookbook in an environment. Berkshelf is primarily used to handle the dependencies of that cookbook.

I had a pull request against jperry's fork that got knife-spork and Berkshelf 3 to play nicely for my use case, but I hadn't tested for other cases. Might be a good reference. jperry#1

[sethvargo]

My reason for not doing anything yet is that I want Berkshelf 2 to completely die and deprecate before I (re)add Berkshelf integration to knife-spork. The APIs are significantly different and I don't think we should support both.

[mjuarez]

Cool, I could see that as a valid reason. Adding Berkshelf 3 support while removing Berkshelf 2 support might help drive the nails into the coffin.

[jperry]

@sethvargo @jonlives we need to figure out a solution for this. Keep getting the failure below with any of my cookbooks that have Berksfile. Can we either disable it until it's compatible with Berkshelf3 or at least have a flag we can set in our spork-config.yml to disable berkshelf integration?

master [08:56:11] ~/workspace/bithub/chef > bin/knife spork promote qa base
ERROR: TypeError: no implicit conversion of nil into String

[jonlives]

@jperry yeah it should def be possible to add a flag to optionally skip using Berkshelf - at the moment if the gem is installed, it's used. I'll open a separate issue for that and add it to this release's milestone.

[jonlives]

Opened #138 for this

[jperry]

thanks! Yeah Berkshelf will def be in my load path making this unavoidable without a flag to disable it. Thanks for crating a separate issue.

[jonlives]

skip_berkshelf config option was released in 1.4.0 as a stop-gap until Berkshelf 3 is fully supported :)

https://github.com/jonlives/knife-spork/blob/master/README.md#skip-berkshelf

[jperry]

thanks!

[dwradcliffe]

@jperry I had the same error. I had to specify a cookbook_path in my knife.rb and then it worked.

[loupgaroublond]

Hi there,

--cookbook-path does not unfuck up the Berkshelf integration, it just adds a new feature. Here's why:

Redacted.local:redacted-dir yankee$ knife spork promote development,staging,production nagios -v 5.2.0 --cookbook-path ~/.berkshelf/cookbooks/
WARNING: Unloading Berkshelf as skip_berkshelf option found in config
ERROR: Errno::ENOENT: No such file or directory @ rb_sysopen - /Users/yankee/.berkshelf/environments/development.json

It treats the berkshelf directory as the Chef project directory, which is double plus ungood for using this orthogonally from other settings. Yes, I know i can manually set these things in the spork-config.yml file,w which I will do now, but i shouldn't have to.