Implement reindex operation

[hypnoglow]

Due to many requests from users, it is necessary to implement this feature.

Refs:

• Feature Requests - AWS Credential profiles, Deleting charts, Reindexing repo #5 manual actions inside S3 bucket where charts are held may lead to broken index
• Race condition causes inconsistencies in chart repository's index.yaml #18 race conditions may lead to broken index

[Stono]

Hey,
Just adding my +1 to this, I was just about to raise an issue around concurrency issues we're having when multiple pipelines happen to index at the same time.

[hypnoglow]

Hey folks, I rolled out reindex command in 0.5.0

Let me know if it works well or not.

[markhu]

Got this error:

$ helm s3 reindex redacted
Warning: reindex feature is in beta. If you experience any issues,
please provide your feedback here: https://github.com/hypnoglow/helm-s3/issues/22

2018/01/09 12:43:22 traverse the chart repository: load archive from s3 object: gzip: invalid header
Error: plugin "s3" exited with error

[hypnoglow]

@markhu , seems like the file was processed is not a helm chart. Do you have any other files in your repository directory, besides index.yaml and chart files?

[markhu]

Um, you mean like a .DS_Store file?!? (I swear it wasn't me, it was my predecessor!) :)
But I also found sub-directories. Which is probably why I got these next errors:

2018/01/09 14:03:03 traverse the chart repository: load archive from s3 object: chart metadata (Chart.yaml) missing

[hypnoglow]

Directories should be ignored. I guess the plugin should print some additional debug info like a file name.

Also, this is also an interesting case - should the plugin fail on such error, or just output the warning and keep working? Maybe we can even control the behavior with some cli flag, so a user can choose.

[AndriiHryshaienko]

Hi, after reindex repo index.yaml is empty now:

apiVersion: v1
entries: {}
generated: 2018-03-09T14:59:17.746753+02:00

[Evesy]

Hi,

FWIW we've observed some timeouts with the reindex command after ~8 minutes:

upload index to the repository: upload index to S3 bucket: RequestCanceled: request context canceled
caused by: context deadline exceeded
Error: plugin "s3" exited with error

This is in a repo with ~5000 charts, so we may have just reached a limit on how many charts can be indexed in a reasonable amount of time.

[kfiles]

My experience is the same as @AndriiHryshaienko. No matter how many times I ran helm s3 reindex myrepo, I just had an empty index.yaml. I was forced to do an s3 cp of all my charts to a local dir, do helm repo index localdir and then fix the urls and upload the index.yaml back to s3. Ugh.

[marcin-je]

@AndriiHryshaienko I got same issue as @kfiles

Seems the issue happens when

helm s3 init s3://sample-bucket/charts
helm repo add myrepo s3://sample-bucket/charts

But not when:

helm s3 init s3://sample-bucket
helm repo add myrepo s3://sample-bucket

[hypnoglow]

@Evesy

I think the simplest workaround here is to add --timeout flag, so you can specify how long are you willing to wait for reindexing your repo.

I'm adding --timeout support right away: #39

[hypnoglow]

@AndriiHryshaienko @kfiles @marcin-je

This is definitely a bug. Thanks to @marcin-je pointed me out that it occurs when your repository resides within bucket directory, but not bucket root.

Fix: #40
Will be released soon in v0.6.1

[hypnoglow]

https://github.com/hypnoglow/helm-s3/blob/master/CHANGELOG.md#changed

Moved helm s3 reindex command out of beta, as it seems there are no more issues related to it.