Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#12775: Cleanup docker run action #12777

Merged
merged 1 commit into from
Sep 24, 2024
Merged

#12775: Cleanup docker run action #12777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 28 additions & 5 deletions .github/actions/docker-run/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ inputs:
docker_username:
description: docker login username
required: true
default: ${{ github.actor }}
docker_password:
description: docker login password
required: true
Expand All @@ -25,13 +26,23 @@ inputs:
default: |
-v /dev/hugepages-1G:/dev/hugepages-1G
--device /dev/tenstorrent
install_wheel:
description: "Install the wheel that contains all of the Python environment. The artifact needs to be present."
type: boolean
required: false
default: false
runs:
using: "composite"
steps:
- name: Determine docker image tag
uses: ./.github/actions/generate-docker-tag
with:
image: ${{ inputs.docker_os_arch }}
- name: Set
shell: bash
run: |
echo "RUNNER_UID=$(id -u)" >> $GITHUB_ENV
echo "RUNNER_GID=$(id -g)" >> $GITHUB_ENV
- name: Docker login
uses: docker/login-action@v3
with:
Expand All @@ -49,16 +60,28 @@ runs:
password: ${{ inputs.docker_password }}
registry: ghcr.io
image: ${{ env.TT_METAL_DOCKER_IMAGE_TAG }}
# The most important option below is `--rm`. The machines will fill up with containers.
# The most important option below is `--rm`. Otherwise, the machines will fill up with undeleted containers.
# The mounting of /etc/passwd, /etc/shadow, and /etc/bashrc is required in order for the correct file permissions
# for newly created files.
# Passing HOME variable is necessary to avoid Python lib installation into /home/ubuntu/.local folder which
# may not be writable by the RUNNER_UID user.
options: |
-u ${{ env.RUNNER_UID }}:${{ env.RUNNER_GID }}
--rm
-v ${{ github.workspace }}:/github_workspace:ro
-v /etc/passwd:/etc/passwd:ro
-v /etc/shadow:/etc/shadow:ro
-v /etc/bashrc:/etc/bashrc:ro
-v ${{ github.workspace }}:${{ github.workspace }}
--net=host
${{ inputs.docker_opts }}
-e LOGURU_LEVEL=${{ env.LOGURU_LEVEL }}
-e PYTHONPATH=/usr/app
-e PYTHONPATH=${{ github.workspace }}
-e HOME=${{ github.workspace }}
${{ inputs.device }}
-w ${{ github.workspace }}
run: |
cp -r /github_workspace/* /usr/app/
cd /usr/app/
if [ ${{ inputs.install_wheel }} ]; then
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we mention in the meeting if we wanted to move the artifact downloading into the docker run action?

I do not know if that makes sense but I also do not like the dependency on the wheel presence that is implicit.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what artifact?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wheel artifact. This action relies on its presence but there is no explicit dependency defined in the Github Acton jobs.

WHEEL_FILENAME=$(ls -1 *.whl)
pip3 install $WHEEL_FILENAME
fi
${{ inputs.run_args }}
7 changes: 1 addition & 6 deletions .github/workflows/fast-dispatch-build-and-unit-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,8 +57,6 @@ jobs:
name: ${{ matrix.test-group.name }} ${{ inputs.arch }} ${{ inputs.runner-label }}
env:
LOGURU_LEVEL: INFO
# may not need this
LD_LIBRARY_PATH: ${{ github.workspace }}/build/lib
runs-on:
- ${{ inputs.runner-label }}
- cloud-virtual-machine
Expand All @@ -72,12 +70,9 @@ jobs:
timeout-minutes: ${{ inputs.timeout }}
uses: ./.github/actions/docker-run
with:
docker_username: ${{ github.actor }}
install_wheel: true
docker_password: ${{ secrets.GITHUB_TOKEN }}
docker_image_arch: ${{ inputs.arch }}
run_args: |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could combine this into just one line so it's pretty: run_args: ${{ matrix.test-group.cmd }}

WHEEL_FILENAME=$(ls -1 *.whl)
pip3 install --user $WHEEL_FILENAME
${{ matrix.test-group.cmd }}
- uses: ./.github/actions/slack-report
if: ${{ failure() }}
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/models-post-commit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,6 @@ jobs:
name: ${{ matrix.test-group.name }} ${{ inputs.arch }} ${{ inputs.runner-label }}
env:
LOGURU_LEVEL: INFO
# may not need this
LD_LIBRARY_PATH: ${{ github.workspace }}/build/lib
runs-on:
- ${{ inputs.runner-label }}
- in-service
Expand All @@ -64,12 +62,9 @@ jobs:
timeout-minutes: ${{ inputs.timeout }}
uses: ./.github/actions/docker-run
with:
docker_username: ${{ github.actor }}
install_wheel: true
docker_password: ${{ secrets.GITHUB_TOKEN }}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not move the docker_password parameter to be filled by default since secrets are not available inside of Github custom Actions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have we tried using
secrets: inherit?
https://docs.github.com/en/actions/sharing-automations/reusing-workflows#using-inputs-and-secrets-in-a-reusable-workflow

Seem strange that a github action can't get access to custom GH actions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I have tried and the secrets are inaccessible in the Github actions - see here

https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions#accessing-your-secrets

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is true - the secrets context in general is not available in a composite action. You explictly must pass it

docker_image_arch: ${{ inputs.arch }}
run_args: |
WHEEL_FILENAME=$(ls -1 *.whl)
pip3 install --user $WHEEL_FILENAME
source tests/scripts/run_python_model_tests.sh && run_python_model_tests_${{ inputs.arch }}
- uses: ./.github/actions/slack-report
if: ${{ failure() }}
Expand Down
2 changes: 0 additions & 2 deletions .github/workflows/ttnn-post-commit.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,6 @@ jobs:
name: ${{ matrix.test-group.name }} ${{ inputs.arch }} ${{ inputs.runner-label }}
env:
LOGURU_LEVEL: INFO
# may not need this
LD_LIBRARY_PATH: ${{ github.workspace }}/build/lib
runs-on:
- ${{ inputs.runner-label }}
- "in-service"
Expand Down
Loading