Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dash build bmv2 container permission error #143

Closed
mukeshmv opened this issue Jul 14, 2022 · 2 comments · Fixed by #155
Closed

Dash build bmv2 container permission error #143

mukeshmv opened this issue Jul 14, 2022 · 2 comments · Fixed by #155
Assignees
@chrispsommers

Comments

@mukeshmv
Copy link
Collaborator

The bmv2 container image used in the Dash build requires write access to Volumes mounted on the host machine.
However this docker image comes with hard-coded uid/gid assigned in the Dockerfile when building the image.
The build workflow involves using a pre-built bmv2 image pulled from the registry.
When running a container from this docker image, the build commands run from within the container using this uid try to write to the mounted volumes in the host machine resulting in permission issues.

# make sai
Generate SAI library headers and implementation...
docker run -v /home/mukesh/dash/dash-pipeline/bmv2:/bmv2 -v /home/mukesh/dash/dash-pipeline/SAI:/SAI -v /home/mukesh/dash/dash-pipeline/tests:/tests --network=host --rm -it \
        --name build_sai-mukesh \
        -w /SAI chrissommers/dash-bmv2:pr127-220623 \
    ./generate_dash_api.sh
Directory ./lib will be deleted...
Traceback (most recent call last):
  File "./sai_api_gen.py", line 357, in <module>
    shutil.rmtree('./lib')
  File "/usr/lib/python3.8/shutil.py", line 718, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.8/shutil.py", line 675, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.8/shutil.py", line 673, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'utils.cpp'
Makefile:85: recipe for target 'sai' failed
@mukeshmv
Copy link
Collaborator Author

Workaround -

cd dash-pipeline
chmod -R o+rw SAI/ bmv2/ tests/

@chrispsommers
Copy link
Collaborator

chrispsommers commented Jul 14, 2022 •
edited
Loading

NOTE: You may need to use sudo chmod -R o+rw SAI/ bmv2/ tests/
Thanks for filing this! I'll work on a permanent fix.

chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 17, 2022
@chrispsommers
Temporary workaround for container permissions issues per sonic-net#143…
5928f61 
…. Executes chmod as required. Permanent fix will require some Docker mods.
@chrispsommers chrispsommers mentioned this issue Jul 17, 2022
Merged
chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 20, 2022
@chrispsommers
Temporary workaround for container permissions issues per sonic-net#143…
ad1c0ae 
…. Executes chmod as required. Permanent fix will require some Docker mods.
chrispsommers added a commit to chrispsommers/DASH that referenced this issue Jul 20, 2022
@chrispsommers
Temporary workaround for container permissions issues per sonic-net#143…
e77e1f9 
…. Executes chmod as required. Permanent fix will require some Docker mods.
chrispsommers added a commit that referenced this issue Jul 20, 2022
@chrispsommers
Temporary workaround for container permissions issues per #143. Execu…
8c6a9ce 
…tes chmod as required. Permanent fix will require some Docker mods. (#155)

Co-authored-by: Chris Sommers <[email protected]>
chrispsommers added a commit that referenced this issue Jul 21, 2022
@chrispsommers
Docker Diet: shrink images 80% and CI runtimes 50% (#140)
3c1d634 
* Build sai-thrift server (not working yet). New dockerfile.

* Removed extraneous step.

* README

* Move Dockerfile under dockerfiles/

* Pass flags to saithrift-build, expecting opencomputeproject/SAI#1514 to merge eventually. Meanwhile can locally modify SAI makefiles.

* Fix CI files for new location of Docekrfile.bmv2, add CI for dash-saithrift docker.

* Fix CI workflow for new dockerfiles.

* Add line to trigger CI.

* Restore CI filename, try to get it to trigger.

* Rename CI file; modify DOckerfile to trigger; add docker targets to Makefile.

* Modify dockerfiles to trigger CI.

* Fix make target names in CI and Makefile

* Update CI to trigger on CI file changes itself.

* Trigger on CI file change.

* Remove thrift source from dash-saithrift-bldr docker image. New tag 220625.

* Enhance saithrift docker to handle dash sai header generation and libsai compilation.

* Remove thrift source and source tarball from docker image; copy python dist tarball to /usr/lib inside container so can be extracted for deployment to a host.

* Experimenting with reduced Dockerfiles.

* Makefile: p4 source dependencies. dash_pipeline.p4: egress_spec=ingress_spec, comments.

* Fix stray merge conflict.

* Fix merge conflict in CI file.

* Optimized docker build and run images by creating a dedicated gRPC library image and selectively copying to BMV2 builder/runtime image based on available p4lang docker image.

* FIx CI, Makefile for new docker images.

* Optimize dockerfiles, reduced to 3 images totalling ~2.7G, down from ~10G.

* Fix CI files, add Make targets; cleanup.

* CI files, badge

* CI files.

* Split up READMEs into smaller files. Diagram.

* READMEs, diagram, SHA version of p4c

* Add missing CI workflow (p4c); update main CI workflow to trigger on any dockerfile change.

* Fix make command in CI file.

* Pin the SHA version of p4lang/behavioral-model:latest.

* Ping SHA versions of Ubuntu used in base images.

* READMEs - workflows etc.

* README typos, clarifications.

* README

* Use dev fork of SAI which passes flags to gensairpc.pl per opencomputeproject/SAI#1514.
Once merged, we can change back to upstream branch of SAI.

* Use fork of SAI with fixes for sairpcgen Make flags, and sairpcgen templates which add static_cast to avoid type mismatches due to sai extensions.

* Add placeholder code for SAI fixed functions, -fpermissive to avoid enum Long error. TODO: P4RT libs missing, otherwise sai-thrift server builds.

* Created init_switch test program and make target to initialize bmv2 model (P4RUntime SetForwardingPipelineRequest) to allow explicit switch intialization via P4Runtime. Removed sudo from most Make commands. READMEs.

* Remove dependency to avoid rebuilding files while running test in CI. Delete stray file.

* README

* Restore dash_pipeline.p4 to same as upstream. Had temporary changes (egress port = ingress port; comments).

* Add CI triggers for all .py, .sh, .yml and requirements.txt. under test/ dir

* Update to snappi-0.7.38 for fix #153

* Temporary workaround for container permissions issues per #143. Executes chmod as required. Permanent fix will require some Docker mods.

* Delete obxolete README, comment out WIP sai-thrift server make targets

Co-authored-by: Chris Sommers <[email protected]>
KrisNey-MSFT pushed a commit that referenced this issue Aug 2, 2022
@chrispsommers
saithrift client & server integration with DASH bmv2 (Fix #131,158) (#…
263ef61 
…164)

* Build sai-thrift server (not working yet). New dockerfile.

* Removed extraneous step.

* README

* Move Dockerfile under dockerfiles/

* Pass flags to saithrift-build, expecting opencomputeproject/SAI#1514 to merge eventually. Meanwhile can locally modify SAI makefiles.

* Fix CI files for new location of Docekrfile.bmv2, add CI for dash-saithrift docker.

* Fix CI workflow for new dockerfiles.

* Add line to trigger CI.

* Restore CI filename, try to get it to trigger.

* Rename CI file; modify DOckerfile to trigger; add docker targets to Makefile.

* Modify dockerfiles to trigger CI.

* Fix make target names in CI and Makefile

* Update CI to trigger on CI file changes itself.

* Trigger on CI file change.

* Remove thrift source from dash-saithrift-bldr docker image. New tag 220625.

* Enhance saithrift docker to handle dash sai header generation and libsai compilation.

* Remove thrift source and source tarball from docker image; copy python dist tarball to /usr/lib inside container so can be extracted for deployment to a host.

* Experimenting with reduced Dockerfiles.

* Makefile: p4 source dependencies. dash_pipeline.p4: egress_spec=ingress_spec, comments.

* Fix stray merge conflict.

* Fix merge conflict in CI file.

* Optimized docker build and run images by creating a dedicated gRPC library image and selectively copying to BMV2 builder/runtime image based on available p4lang docker image.

* FIx CI, Makefile for new docker images.

* Optimize dockerfiles, reduced to 3 images totalling ~2.7G, down from ~10G.

* Fix CI files, add Make targets; cleanup.

* CI files, badge

* CI files.

* Split up READMEs into smaller files. Diagram.

* READMEs, diagram, SHA version of p4c

* Add missing CI workflow (p4c); update main CI workflow to trigger on any dockerfile change.

* Fix make command in CI file.

* Pin the SHA version of p4lang/behavioral-model:latest.

* Ping SHA versions of Ubuntu used in base images.

* READMEs - workflows etc.

* README typos, clarifications.

* README

* Use dev fork of SAI which passes flags to gensairpc.pl per opencomputeproject/SAI#1514.
Once merged, we can change back to upstream branch of SAI.

* Use fork of SAI with fixes for sairpcgen Make flags, and sairpcgen templates which add static_cast to avoid type mismatches due to sai extensions.

* Add placeholder code for SAI fixed functions, -fpermissive to avoid enum Long error. TODO: P4RT libs missing, otherwise sai-thrift server builds.

* Progress on sai-thrift server, final saiserver linkage still has errors but was able to pass grpc/proto/pi libs into make.

* Created init_switch test program and make target to initialize bmv2 model (P4RUntime SetForwardingPipelineRequest) to allow explicit switch intialization via P4Runtime. Removed sudo from most Make commands. READMEs.

* Remove dependency to avoid rebuilding files while running test in CI. Delete stray file.

* README

* Build saithrift-bldr in Ubuntu-20.04 to avoid glibcc conflicts wth libgrpc etc. Modify sai-thrift Maefile to accomodate Python3 vs Python2.
TODO - saiserver builds but get errors in later stages of make saithrift-build target (ctypesgen...)

* Can build and run saithrift server. Fixed makefiles to call SAI makes with proper env vars etc., fixed install steps.

* Add sai-thrift-server to CI job.

* saithrift server starts up properly - added volume mounts to p4info

* Add DOCKER_FLAGS for sai targets

* Add sai-thrift-server CI test.

* Workaround for git repo ownership in CI.

* Correct trigger path for CI.

* Workaround for SAI submodule - fixed step name.

* Change dir into SAI/SAI to make safe directory

* Try safe directory from /dash

* Remove -u root from CI flags, remove git safe directory.

* ove lib dir creation to Makefile, remove from .pyu script - permission errors during CI.

* Add back -privileged flag

* ADd -u root to make sai in CI

* Change CI docker flags, use sudo for python script to overcome perms.

* Update SAI branch which fixes sairpcgen Makefiles (sai header include path).

* Remove experimental sudo from bash script, clean up overwrite options from python script (leave such things to makefiles.) Still struggling with perms issues in CI script.

* Try sudo in CI make sai step.

* Try -u root in CI file and git config --global --add safe.directory in Makefile

* Use "git config --global --add safe.directory /dash/dash-pipeline/SAI/SAI" per error message, even though it doesn't match actual path IMO.

* Try git config --global --add safe.directory '*' in Makefile.

* Temporarily skip make sai step, see if rest of pipeline runs.

* REstore CI script, try sudo workarounds in Makefile.

* fix chmod command

* Remove -u root, use sudo calling python script.

* fix chmod to a+rw and also commit bash script.

* chown and git safe dirs script attempt

* pass user to sudo

* export user

* FIx chown

* More wrappers and scripts to solve CI git ownership issues.

* REmove chown to try to fix "./checkenumlock.sh ...fatal: Unable to create '/dash/.git/modules/SAI/index.lock': Permission denied

* remove git safe dirs, getting lockfile error in CI

* REmove sudo calling python script; shotgun sudo for all CI steps.

* REstore sudo for py script; add -u root, --privileged to flags in CI

* Workaround, delete scripts in SAI/meta/Makefile which throw Git permission errors.

* Add sudo for sai-thrift-server target due to -u roots above. Got past git repo errors via previous commit, yay!

* Use sudo to avoid permissions problem in CI pipeline for SAI/rpc dir

* chown in sai-thrift Makefile to overcome CI perms issues.

* Change perms instead of chown in saithrift build.

* Add missing DOCKER_FLAGS to sai-thrift-server target

* Add DOCKER_FLAGS to CI step; remove obsolete wrapper scripts.

* Pass DOCKER_FLAGS to make target for sai-thrift-server test

* Use different SAI submodule branch (better name, synced to master). Add saithrift clean target to clean.

* Remove extraneous permissions workarounds.

* Update SAI submodule.

* Dockerfile for saithrift test clients.

* ADd CI for saithrift client docker

* Add prerequisite builds steps for sai-thrift-client. Update drawing.

* Build prerequisites first to make docker-saithrift-client

* Add missing requirements.txt, make targets.

* Progress with saithrift-client docker image, pytest, ptf test installation and skeletons.

* Refactor pytest to use common theift Client class; run sai client tests in CI. NOte - need sai_api_query impl to avoid sai thrift server crashes.

* Set docker flags for saithrift client test in CI

* Docs, diagrams. saithrift workflows.

* Use updated SAI dev branch to insert checks for NULL sai API funcs before calling them. Add simple test for switch attribute to prove thrift API working.

* Hand-modified sai_api_query to return SAI_API_DASH_VNET apis. Should be auto-generated.

* Use workaround for saithrift builder per opencomputeproject/SAI#1537. Reorganize nacent pytests to use fixtures to get saithrift client, put in directories, add custom markers, new README.

* mark vnet test x-fail for CI

* Update saithrift client docker image.

* Add empty README for saithrift/ptf

* Clean up make targets.

* Permissions fixes (manually, not merged). Add gdb to sai-thrift-server docker.

* Fix sai-thrift-server; got first RPC working (vnet). Slight refactoring of pytests (imports); fixed sai_api_query & made _impl structs non-static.

* Fix path for sai-thrift-server according to new -w path in Makefile

* Add CI triggers for all .py, .sh, .yml and requirements.txt. under test/ dir

* Update to snappi-0.7.38 for fix #153

* Temporary workaround for container permissions issues per #143. Executes chmod as required. Permanent fix will require some Docker mods.

* Update snappi to 0.7.38 in saithrift tests

* FIx dangling merge conflicts in README.

* s/sai-thrift/saithrift to for consistency.

* Delete sai_test which was accidentally copied from SAI repo.

* Fix #158 - libsai delete operation failure and log msg. THe test for error code was backwards and log message used same write updateType string for all operations. I fixed the compare and used a dynamic enum print method.

* Modify vnet pytest case to create/delete. README. Makefile polishing.

* Add more table create/deletes + cleanup to saithrift vnet test case

* Assign container names in ixia-c deployments (append username)

* Got preliminary PTF and Pytests including saithrift table accessors, packet echo tests. Removed ixia-c test as standalone test; now it runs as a pytest inside saithrift-client container so host tool installs are reduced (snappi, python, pip).

* Add back missing line to build saithrift server.

* Fix CI trigger, a prior merge messed this up.

* Add deploy-ixiac to Makefile deps and CI steps to support pytests.

* Refactor saithrift-client docker image into builder and runtime images; runtime client is always built locally, not pulled.
Change python requirements to hardlink to DASH/test/requirements.txt. Change DOckerfile name for consistency.

* Create tests/libsai dir and move "C++" tests there; fix typo in CI file.

* Defer deploying ixia-c until PTF tests complete to see if CI tests fail less often, suspect CI runner limits are being strained. Tests fail sometimes w/o reason.

* Moved saithrift tests under tests/ for consistency.

* FIx path, paste errors which run tests.

* Move requirements.txt to hardlink under tests (was under saithrift). Delete obs file.

* READMEs (saithrift test framework), Makefile (fix dev tests).

* READMEs - test workflows.

* Fix paste error - README

* READMEs - workflows. Diagram. Remove "make run-test," replaced by "make libsai-test."

* Fix CI action; READMEs polish.

* Minor README improvements; fix URL, diagrams (sirius->dash); improve Quick-start instructions.

* Add back URL.

* REmove python, pip install deps.

* READMEs; move saithrift test scripts into subdirs; change verify_packets() to verify_packet() to avoid failing due to junk entering veths from host network.

* Change CI job title Sirius->DASH

* Update SAI submodule to rescind changes to SAI include paths for sairpcgen, wasn't needed after all.

Co-authored-by: Chris Sommers <[email protected]>
@chrispsommers chrispsommers mentioned this issue Aug 3, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chrispsommers chrispsommers

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Temporary workaround for container permissions issues chrispsommers/DASH
2 participants
@mukeshmv @chrispsommers