Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TEST] test_lstms taking 20+ mins on CI runs #8608

Closed
denise-k opened this issue Jul 30, 2021 · 3 comments
Closed

[TEST] test_lstms taking 20+ mins on CI runs #8608

denise-k opened this issue Jul 30, 2021 · 3 comments

Comments

@denise-k
Copy link
Collaborator

Context

CI is taking a while (see #8552 and build#1384), and I am tracking short-term fixes to alleviate this.

Summary of the issue

test_lstms.py was recently added in PR #8447. It takes 20+ minutes to run 8 network-sized tests.

The tests uses the following parameters, which don't seem minimal:

## Model parameters
model_feature_size = 5
model_hidden_size = 10
model_num_layers = 2
seqs_length = 15
projection_size = 7
batch_size = 3

Also, the In the test console, there are 64 of the following warnings. It seems like there are issues exporting the pytorch frontend to ONNX:

WARNING: projection is not supported for torch version less than 1.8.0!  LSTM was constructed without projection!
WARNING: torch.onnx.export does not support conversion LSTM with projection from pytorch! TODO: waiting for the support and correct test after that.

Suggested actions

  • Try to minimize the size of the model testcases even further.
  • Consider minimizing the number of calls to torch.onnx.export until we get a nightly CI in place.
@vvchernov
Copy link
Contributor

Hello! It was resolved in PR #8583

@vvchernov
Copy link
Contributor

vvchernov commented Jul 31, 2021

Warnings are a part of the code in TVM. The first one is related to that LSTM with projection supported by pytorch starting from 1.8.0 version. But TVM works stably with pytorch 1.7.0. The second warning is related to difference between modifications of LSTM supported in onnx and pytorch. It means that in some cases the conversion from pytorch LSTM to ONNX one fails. But it is external issue, it can not be solved on TVM side. Therefore I did not raise issue ticket.

@denise-k
Copy link
Collaborator Author

denise-k commented Aug 2, 2021

thank you for addressing this so quickly @vvchernov! the test times on test_lstms are looking much better now so I'm closing this issue.

@denise-k denise-k closed this as completed Aug 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants