Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pytest runs out of memory when coverage is enabled #2330

Closed
mint570 opened this issue Jun 14, 2022 · 2 comments · Fixed by #2350
Closed

Pytest runs out of memory when coverage is enabled #2330

mint570 opened this issue Jun 14, 2022 · 2 comments · Fixed by #2350

Comments

@mint570
Copy link
Contributor

mint570 commented Jun 14, 2022

This is found in #2237. The last few pytest failed.
We suspect that the cause is the coverage enabling: https://github.com/Azure/sonic-swss/blob/bf4d89082ad413ef71eb40a8ff2f279b87f1b002/.azure-pipelines/test-docker-sonic-vs-template.yml#L72

The "--keeptb" flag in pytest will keep the docker containers alive after the test finishes. In local experiments, each test will open two containers. After all the tests finish, there will be hundreds of containers open. If we add more pytest tests, the workflow will run out of memory.

We should try to run one test at a time. Collect the coverage artifacts, kill the containers. And then begin the next test.

@mint570
Copy link
Contributor Author

mint570 commented Jun 14, 2022

@mint570
Copy link
Contributor Author

mint570 commented Jun 22, 2022

@bocon13 had mentioned some good approaches to consider:

  1. Mount the file system instead of keeping the container running. So that we can collect the code coverage files even after the container finishes.
  2. Keep the container file system but stop the compute part of the vs docker to reduce resource usage when pytest is running.

But in both the approaches, we will still need to do something at the end of each test (send a kill signal, collect/rename the files etc). Maybe that can be done as part of the pytest workflow?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant