new snapshots are not deployed

[nefilim]

not sure if i'm doing something wrong or if it's related to #75

i publish a tar.gz artifact to my snapshots repo, containing my project jar and it's dependent jar (assembled with the sbt-pack plugin). when i publish a new artifact with the same snapshot version (but the project jar internally has changed), the artifact is not pulled down again but the cached version is reused.

as an aside (not sure if it has any bearing here), doing a /artifact/maven/redirect&v=LATEST does not work on my snapshots repository for some reason (but does on my public proxied group), so i set the version explicitly.

[BarthV]

Hi !
Please test latest master branch and see if current cookbook state fix your issue.

[nefilim]

doesn't look like it I'm afraid :(

[BarthV]

I can probably explain what's going wrong in this case :
For me, Deploy lwrp won't check SHA signature if your current deployed artifact matches cached signature (into /var/chef/cache/artifact_**).
We nées to 'short-circuit' this and force Nexus metadata check in case of snapshot or latest version.

I'll check if this is rly this kind pb

[BarthV]

I tried to reproduce your issue using the current master branch (6410b2f) and following this simple recipe:

conf = Chef::Artifact::NexusConfiguration.new(
  'http://factory-server.mycompany.local/nexus', # URL
  'projects-java-snapshots', # Repo
  nil, # User
  nil, # Pass
  false # Strict SSL
)

artifact_deploy 'daemon-redoer' do
  version 'LATEST'
  artifact_location 'local.mycompany.team:daemon-redoer:tar.gz'
  nexus_configuration conf
  deploy_to '/tmp/redoer'
  owner 'root'
  group 'root'
end

First run was ok and it fetched the latest build ->

redoer/
|-- current -> /tmp/redoer/releases/2.0-20140415.151204-16
|-- releases
|   `-- 2.0-20140415.151204-16
`-- shared
    |-- log
    |-- pids
    `-- system

Then triggered a build on my Jenkins, and made a second run ->

redoer/
|-- current -> /tmp/redoer/releases/2.0-20140415.152416-17
|-- releases
|   |-- 2.0-20140415.151204-16
|   `-- 2.0-20140415.152416-17
`-- shared
    |-- log
    |-- pids
    `-- system

All is fine for me (it also worked with a snapshot), can you retest and give a bit more details ?
Thanks

[nefilim]

Thanks for looking! The one thing that immediately stands out to is the use of version "LATEST", while I am using a specific version eg. "0.1-SNAPSHOT". I was having trouble with "LATEST" not working on my nexus instance (I'll revisit this, it's been upgraded).
What happens if you set the version to a snapshot explicitly?

[BarthV]

It's identical with 2.0-SNAPSHOT (in my case) version, working well for me (maybe i'll retest that, i'm 99% sure about this).

[nefilim]

I checked LATEST again, running into a 500 error, which I've reported to sonatype, ok with the exact version. I'm wondering if there's something wrong with my method of publishing (using SBT) or my repository setup.
Is there debug logging I can enable? Or could you point me to the location in the cookbook code where it's determined if the repository artifact is newer that I can take a closer look/add some debug?

Another thing I've noticed, my deployed releases does not have the timestamp appended to the directory name:

├── current -> /usr/local/reporting/releases/0.1.2-SNAPSHOT
├── releases
│   └── 0.1.2-SNAPSHOT

[BarthV]

• First, check the Chef cookbook local cache directory and verify if cookbook metadatas are correct (must be 1.11.13 with get_artifact_filename(coordinates) method in libraries/chef_artifact_nexus.rb file).
• ls artifact_deploy in the Chef local cache directory to check how the server identify the remote file and match if its hash is correct.
• to follow deploy steps , look :deploy action and follow :
• - https://github.com/RiotGames/artifact-cookbook/blob/master/providers/deploy.rb#L357
• - https://github.com/RiotGames/artifact-cookbook/blob/master/providers/deploy.rb#L544
• - https://github.com/RiotGames/artifact-cookbook/blob/master/providers/deploy.rb#L581

[nefilim]

Thanks, I have some data.

[root@ip-10-0-20-18 artifact]# grep version metadata.rb
version "1.11.3"
[root@ip-10-0-20-18 artifact]# grep get_artifact_file libraries/chef_artifact_nexus.rb
def get_artifact_filename(coordinates)

I have also have some debug log excerpts and a timeline, all in a file, is there somewhere I can send or upload it?

We can see that the sha1 is updated in nexus when the new tgz is published and it's reflected in the REST call to maven/resolve. I haven't looked at detail at the code yet but it seems like the cookbook looks at the (generated) manifest for the contents, any reason not to just short circuit when the artifact shasum has changed?

I've seen some very strange things in artifacts_deploy, not sure why there are no files there now, I saw my artifact (and a very old artifact appear at some point) there and then it disappeared again, in between chef-client runs, very odd.

[BarthV]

I'll see if I can take some time to re-test and try to reproduce again.

[nefilim]

Thanks. Would the debug logs I captured be of any use? If you're interested I could probably give you access to one of my nodes to take a first hand look.

[BarthV]

can't even reproduce your issue with my Nexus :

conf = Chef::Artifact::NexusConfiguration.new(
  'http://factory-server.mycompany.local/nexus', # URL
  'projects-java-snapshots', # Repo
  nil, # User
  nil, # Pass
  false # Strict SSL
)

artifact_deploy 'daemon-redoer' do
  version '2.0-SNAPSHOT'
  artifact_location 'local.mycompany.project:daemon-redoer:tar.gz'
  nexus_configuration conf
  deploy_to '/tmp/redoer'
  owner 'root'
  group 'root'
end

redoer/
|-- current -> /tmp/redoer/releases/2.0-20140417.153450-20
|-- releases
|   `-- 2.0-20140417.153450-20
`-- shared
    |-- log
    |-- pids
    `-- system

and after a new build ->

redoer/
|-- current -> /tmp/redoer/releases/2.0-20140417.154757-21
|-- releases
|   |-- 2.0-20140417.153450-20
|   `-- 2.0-20140417.154757-21
`-- shared
    |-- log
    |-- pids
    `-- system

# ls /tmp/kitchen/cache/artifact_file/
local_mycompany_projects_daemon_redoer_tar_gz_2_0_20140417_153450_2-75a329c34048c75d7aadb6eb77f34119
local_mycompany_projects_daemon_redoer_tar_gz_2_0_20140417_154757_2-84b3b09b4656319f7e0a6e645360caaf

# cat /tmp/kitchen/cache/artifact_file/*
7edbcf3ed9c1c710d679ae8c8a3ba1b7d1bf0751a25f645c9b57e1485ff72c1f
0c78d66b4d2759677c7986a4b03636e5ec7c0ef28c7526655f0cd30a2fd43911

after another build :

redoer/
|-- current -> /tmp/redoer/releases/2.0-20140417.155645-22
|-- releases
|   |-- 2.0-20140417.153450-20
|   |-- 2.0-20140417.154757-21
|   `-- 2.0-20140417.155645-22
`-- shared
    |-- log
    |-- pids
    `-- system

# ls /tmp/kitchen/cache/artifact_file/
local_mycompany_projects_daemon_redoer_tar_gz_2_0_20140417_153450_2-75a329c34048c75d7aadb6eb77f34119
local_mycompany_projects_daemon_redoer_tar_gz_2_0_20140417_154757_2-84b3b09b4656319f7e0a6e645360caaf
local_mycompany_projects_daemon_redoer_tar_gz_2_0_20140417_155645_2-a5fd1814b94e2a949fff47a4e37dd29a

# cat /tmp/kitchen/cache/artifact_file/*
7edbcf3ed9c1c710d679ae8c8a3ba1b7d1bf0751a25f645c9b57e1485ff72c1f
0c78d66b4d2759677c7986a4b03636e5ec7c0ef28c7526655f0cd30a2fd43911
19cf9f4104b9fbebc5ade5a2d8489e66061bbb8ef78cc12d4a6a55f21e3eef68

Can't help you more with that.

[nefilim]

I looked into this a bit, the problem is that the deploy step does a simple version comparison in manifest_differences?, where version is from the version tag in the response from the nexus/service/local/artifact/maven/resolve? query.

Looks like publishing with maven has some trickery to replace SNAPSHOT with a timestamp & build number, this is why your releases directory entries looks like 2.0-20140417.153450-20 instead of 2.0-SNAPSHOT … which is how mine looks like when published with SBT, as I remarked earlier.

This lead me down a dark and scary path on publishing Maven style with SBT, unfortunately there's no clear & livable solution for my specific environment yet. I'll take another run at it tonight.

Either way, this seems like more maven silliness, I'd propose that the deploy action rather use Chef::Artifact::Nexus.get_artifact_sha and compare with a/the SHA1 checksum of the currently deployed artifact. This should be more accurate and a more portable solution to a wider variety of scenarios.

[BarthV]

I don't know what kind of artifact snapshot tagging is really expected, but in your case if your version is strictly the same on each build, manifest_differences? is clearly not updating the resource.

That's what I thought in my first comment ;) :
#117 (comment)

Now the question is : who's making it wrong ? Your maven job, mine, or the deploy LWRP ?

[tarrall]

@nefilim we've had good luck using arktekk's sbt-aether-deploy plugin to solve the "dark and scary path on publishing Maven style with SBT" problem. Works great for us but if that's too much magic, there's also a big ugly curl command you can use; let me know if you need that.

[nefilim]

@tarrall thanks, have had reasonable success with it, just one (nexus) issue: arktekk/sbt-aether-deploy#21