[OKD 4.10] builder Service accounts seem to have lost the ability to write to local registry

[fortinj66]

Build process writing to the local OKD image registry are now getting permission denied messages writing with the builder service account. I've tried multiple builds in multiple projects. These worked fine in OKD 4.9...

from build logs:

Warning: Push failed, retrying in 5s ...
Getting image source signatures
Copying blob sha256:9ffaa45bf5e6266a04fd0689618322dab52cf9d8319d12a48841c29934387b26
Copying blob sha256:10231e77f89e7377b7253ce34c6decef817a2a2515b282b1acbb96d59970e714
Copying blob sha256:3cf11d9cb4970993e49b10f6450f99f858b1c794bbba1ca47291daf62ddcb626
Copying blob sha256:acc2461c71d34a368f915c0b352c6be549b320c7ad3dff9efbb7210cae4f39d3
Copying blob sha256:761b97ab1a7f433105e1ecdb4d51cf0a117facecae590abcb679c285bc8ab7e0
Copying blob sha256:be41b66cdf6346857fd0718fc076b40ad0d0f70e617971c859e39d0d5b5e5a28
Copying blob sha256:8b59f5c45c07ddea0229ea111d3f401728d3547179170e3084c75147fba96435
Copying blob sha256:26b183a2a5b5355f4fc04dffcbda29efeab1c15ce300add22510bd5821387552
Copying blob sha256:040d420a318c713ea0c95c23c0b526dab67e47c9db0e965ab0800a17b05f6078
Copying blob sha256:df962d62a357a8bb27ad8a896ad1e582cfe4fa992836ea44638fb37e97d90249
Copying blob sha256:19f65d057b806b3501ea05c15cabdbb2287dff8a8654f1cb05909818803f5466
Copying blob sha256:a927642233452ff5e4bb2ab7b99129d871229ea4bcfc36126a3413ff04268069
Warning: Push failed, retrying in 5s ...
Registry server Address:
Registry server User Name: serviceaccount
Registry server Email: [email protected]
Registry server Password: <<non-empty>>
error: build error: Failed to push image: writing blob: initiating layer upload to /v2/dev-shop-micro/artifact-micro-addressbook-jdk11/blobs/uploads/ in 
image-registry.openshift-image-registry.svc:5000: received unexpected HTTP status: 500 Internal Server Error

From registry logs:

time="2022-03-14T15:44:27.02764624Z" 
level=error 
msg="response completed with error" 
err.code=unknown 
err.detail="filesystem: open /registry/docker/registry/v2/repositories/dev-shop-micro/artifact-micro-addressbook-jdk11/_uploads/16726db4-2946-4ce6-9e83-28ce1980fcd0/data: permission denied" 
err.message="unknown error" 
go.version=go1.17.5 
http.request.host="image-registry.openshift-image-registry.svc:5000" 
http.request.id=a3d79156-6c59-4355-b54f-1b30d409bba1 
http.request.method=POST 
http.request.remoteaddr="10.131.0.142:41420" 
http.request.uri=/v2/dev-shop-micro/artifact-micro-addressbook-jdk11/blobs/uploads/ 
http.request.useragent="containers/5.16.1 (github.com/containers/image)" 
http.response.contenttype="application/json; charset=utf-8" 
http.response.duration=21.812669ms 
http.response.status=500 
http.response.written=279 
openshift.auth.user="system:serviceaccount:dev-shop-micro:builder" 
vars.name=dev-shop-micro/artifact-micro-addressbook-jdk11

[vrutkovs]

Seems permission denied comes from storage FS, not due serviceaccount RBAC

[mschaefers]

My workaround is the same - using RBD block device with a single registry pod have a "Recreate" strategy instead of "RollingUpdate". Far from optimum.
Bear with me - I will be unavailable for about 2 weeks now and unable to respond in time

[fortinj66]

No worries

[vrutkovs]

Lets convert this into an okd issue so that we could track it?

It definitely looks like F35 change has caused this

[fortinj66]

I'll create it...

[jngrb]

As @schuemann wrote in the created ticket #1160 we see general issues with write access to cephfs volumes with OKD 4.10 and FCOS 35. Usin a ceph block storage still works. Thanks for bringing this forward.

[bdlink]

John Have you tried using an NFS mount from a services server (running load balancer etc) rather than ceph? I had long ago read somewhere that it was problematic using ceph for the registry Bruce

…

Sent from my iPhone On Mar 14, 2022, at 4:48 PM, John Fortin ***@***.***> wrote:  CAUTION: This email originated from outside of BCIT. Do not click links or open attachments unless you recognize the sender and know the content is safe. So this cluster is using ceph-rook and the /registry mount itself is using a rook shared filesystem. I tried creating a new shared filesystem and it also failed in the same manner. I created/used a new ceph block storage filesystem and it works... Very odd... I'm going to have to dig into the ceph-fs logs some I guess — Reply to this email directly, view it on GitHub<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenshift%2Fokd%2Fdiscussions%2F1153%23discussioncomment-2361335&data=04%7C01%7Cbruce_link%40bcit.ca%7Ce42805ff1c7749b44d9308da06151cb6%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637828984983755995%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=U7zYZxBOvrQpFmfMXYc0HzP3hl6VB835sdpHkxA2Luk%3D&reserved=0>, or unsubscribe<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAEBGEZLY2UHTBRK4T44ER3U77F4BANCNFSM5QV7HDXQ&data=04%7C01%7Cbruce_link%40bcit.ca%7Ce42805ff1c7749b44d9308da06151cb6%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637828984983755995%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ctx5hFLSeaEAG7M4ZqHwc6cBplEifPdAM8cE7MeF5vI%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cbruce_link%40bcit.ca%7Ce42805ff1c7749b44d9308da06151cb6%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637828984983755995%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0KfanC7oF3nZHRZJvsEQCedTKzh6nzbwRyG3opA%2BjwM%3D&reserved=0> or Android<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cbruce_link%40bcit.ca%7Ce42805ff1c7749b44d9308da06151cb6%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637828984983755995%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=o%2BQ8ZMk%2F9DQP6FsOI9t6pn%2FpEi%2BVMpnPgN7PCzBGjlI%3D&reserved=0>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[fortinj66]

Hi Bruce,

We are trying to stay away from NFS as much as possible... This is the first time I've had an issue with Ceph and I've been using it for image registry since OKD 4.6.

This is just very weird..

[bdlink]

John I understand. I used ceph from the start (after prototyping with NFS), 4.4 I believe. One upgrade corrupted ceph and I couldn’t recover. Upon rebuild I switched back. Bruce

…

Sent from my iPhone On Mar 15, 2022, at 7:24 AM, John Fortin ***@***.***> wrote:  CAUTION: This email originated from outside of BCIT. Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi Bruce, We are trying to stay away from NFS as much as possible... This is the first time I've had an issue with Ceph and I've been using it for image registry since OKD 4.6. This is just very weird.. — Reply to this email directly, view it on GitHub<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fopenshift%2Fokd%2Fdiscussions%2F1153%23discussioncomment-2365483&data=04%7C01%7Cbruce_link%40bcit.ca%7C03852c1eddac430b0ce508da068f852a%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637829510718488815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=aUHceGGsBd7ltuS4o4qLlzuAArXsRmG2izqvcg5uRRQ%3D&reserved=0>, or unsubscribe<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAEBGE4JAAJQTXCH46726CTVACMR5ANCNFSM5QV7HDXQ&data=04%7C01%7Cbruce_link%40bcit.ca%7C03852c1eddac430b0ce508da068f852a%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637829510718488815%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8d1JgP9yH%2BpgoLl8N4en1mVLMI8VWjGuVQFiM4pbjnc%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cbruce_link%40bcit.ca%7C03852c1eddac430b0ce508da068f852a%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637829510718645023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=dl5GL83URz6AauXxY2ZJ%2F2gA2Mg6xWk7TwkTalr%2FbCQ%3D&reserved=0> or Android<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cbruce_link%40bcit.ca%7C03852c1eddac430b0ce508da068f852a%7C8322cefd0a4c4e2cbde5b17933e7b00f%7C0%7C0%7C637829510718645023%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7O6KS0ngcBg7C3djH1eBWnys%2BRP5ZmPrwbNfNOtzAvo%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***>