[git-sync] about maxFailures... (with http status 429 Too many Requests from bitbucket)

[seunggabi]

https://community.atlassian.com/t5/Bitbucket-questions/git-sync-Too-many-requests-nfatal-unable-to-access-https/qaq-p/2461183#M96947

airflow-helm/charts#776

INFO: detected pid 1, running init handler
"level"=0 "msg"="starting up" "pid"=12 "args"=["/git-sync"]
"msg"="too many failures, aborting" "error"="Run(git ls-remote -q https://bitbucket.org/asdf/qwer.git refs/heads/stg): exit status 128: { stdout: "", stderr: "remote: Too many requests\nfatal: unable to access 'https://bitbucket.org/asdf/qwer.git/': The requested URL returned error: 429\n" }" "failCount"=0

Maybe bitbucket launched new policy (http status 429 Too many Requests from bitbucket).
(2023-08-25 23:16 KST)

So, My team changed gitSync.syncWait 5 -> 60 and considering applying maxFailures.
Because git-sync failures affect every component failures.

What value should I use for maxFailures?
I hope that if bitbucket requests fail, the entire component will not be affected.

I found related issue. but I am not confidence. #30268

dags:
  gitSync:
    maxFailures: -1 # never die..

it is good?..?

[potiuk]

It's more of a question to git-sync repo https://github.com/kubernetes/git-sync you can see all the options they have there and what effect they might have - and they have all teh expert knowledge about it.

it is good?..?

No idea. Depending what you want to achieve. It very much depends on why you have the problem I think you need to solve the problem of too many request first. And also work with Bitbucket repo. If BitBucket does not allow you to run too many requests you should consider fine-tuning (as you did) or - if possible - lifting the limits you have imposed by BitBucket. Since (I preseume) you pay them for hosting your repo you can talk to their support on that.

Usually when you are syncing from any remote Git repo. they apply various kind of measures to limit overloading their system. If you are not authenticated to your account, they have very low limits based on public IP address. You have not explained what mechanism you have but if you are not authenticated, then you are likely to hit the limit because you are sharing the limit with every other bitbucket request from the same public IP (Which might be thousands of people and system depending on your networking setup). So if you are not authenticated - you should. Maybe also BitBucket has the optiont to increase your limits - generally speaking you should read their docs about it and contact their support to see what you can do to NOT hit the limits at all. But this is between you and BitBucket.

Finally if you cannot get the limits lifted, I can recommend you to setup pipeline where you can trigger syncing with git-sync via sending a signal to git-sync containers when the repository changes. Git-sync has such an option and you can connect webhook from BitBucket to trigger your git-sync containers when there is a change pushed to your repo. It would require a custom implementaiton of such a webhook and signal propagation, it would require an ingress in your K8S and something that would propagate such signal - but it is entirely possible via

    --sync-on-signal <string>, $GITSYNC_SYNC_ON_SIGNAL
            Indicates that a sync attempt should occur upon receipt of the
            specified signal name (e.g. SIGHUP) or number (e.g. 1). If a sync
            is already in progress, another sync will be triggered as soon as
            the current one completes. If not specified, signals will not
            trigger syncs.

[seunggabi]

Thank you for your wise answer.