Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add minio scheme for S3-style storage if you're not using Amazon S3 #118

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

PeterGrace
Copy link
Contributor

This PR adds a new scheme for the s3-bucket argument, minio://, which allows you to specify a custom endpoint for your s3 connection rather than using the AWS defaults. This allows you to use S3-compatible storage systems such as MinIO.

I am an extreme novice at Go programming, so if I've done anything weird here that might break other things, I apologize in advance. I wanted to provide this as a proof of concept to indicate how easy it is to support third-party S3-compatible endpoints.
If this is close to mergable, please feel free to suggest any amendments.

log.Fatal().Err(err).Msg("Create S3 Session")
}
// Create an uploader with the session and default options
if len(host) > 0 {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be simplified the config is the thing that varies based on host info. So something like this could be better...

cfg := aws.Config{LogLevel: aws.LogLevel(aws.LogDebugWithRequestErrors)})}
if host != "" {
  cfg.Endpoint = aws.String(host)
}
s, err := session.NewSession(&cfg)
if err != nil {
  log.Fatal().Err(err).Msg("Create S3 Session")
}

func parseBucket(bucketURI string) (string, string, error) {
var bucket, key string
func parseBucket(bucketURI string) (string, string, string, error) {
var host, bucket, key string
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A test should be provided here to make sure we're doing the right thing with minio urls

@OllowainT
Copy link

OllowainT commented Jun 15, 2022

Would be really nice, if this feature will be finalized and integrated
Edit: I just found out that minio is already working. I just had to use the parameters as follows (k8s):

  • --s3-bucket=""
  • --s3-endpoint={serveradress}/{bucketname}
  • --s3-region={regionname}

@rbjorklin
Copy link

@OllowainT did you manage to upload anything to a non s3 bucket?

@OllowainT
Copy link

OllowainT commented Feb 10, 2023

@OllowainT did you manage to upload anything to a non s3 bucket?

@rbjorklin: Yes, as I pointed out, I had to add the three args:

  • --s3-bucket=""
  • --s3-endpoint={serveradress}/{bucketname}
  • --s3-region={regionname}

Its just a workaround, but for me its ok for now. You need to add --s3-bucket option with quotes. Unfortunately it also creates the directory in the bucket with quotes...

Here the Spec part of my CronJob

        spec:
          serviceAccountName: {{ .Values.popeye.accountname }}
          restartPolicy: Never
          containers:
          - name: popeye
            image: "{{ .Values.popeye.image.name }}:{{ .Values.popeye.image.tag }}"
            ports:
            - containerPort: 9216
              protocol: TCP
            imagePullPolicy: IfNotPresent
            command: ["/bin/popeye"]
            args:
              - --cluster-name
              - {{.Values.global.clusterName}}
              - -f
              - /etc/config/popeye/spinach.yml
              - --force-exit-zero=true
              - --all-namespaces
              - -o
              - html
              - --pushgateway-address
              - prometheus-pushgateway.prometheus.svc.cluster.local:9091
              # because of a popeye bug, we cannot use --s3-bucket without quotations - in the bucket the directory has also quotations
              - --s3-bucket="{{.Values.global.squadKey}}"
              - --s3-endpoint={{ template "s3.endpoint" . }}
              - --s3-region={{.Values.popeye.cronjob.s3.region}}
            env:
              - name: AWS_SHARED_CREDENTIALS_FILE
                value: "/creds/credentials"
            resources:
              limits:
                memory: 96Mi
              requests:
                cpu: 100m
                memory: 32Mi
            volumeMounts:
              - name: spinach
                mountPath: /etc/config/popeye

Hope this helps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants