Fix conda build #2458
Fix conda build #2458
Conversation
@pbchekin the test was failing in a normal run due to psutil missed in pip cache; I have restored the way it worked before (without caching) |
| - name: Install transformers | ||
| if: (inputs.env_manager != 'base') && (steps.conda-cache.outputs.status == 'miss') | ||
| run: | | ||
| git clone --recursive https://github.com/huggingface/transformers |
There was a problem hiding this comment.
Pavel @pbchekin , what do you think if I add e2e test (including installation of required parts) to the test-triton.sh script, so this would work uniformly with other tests? Do developers run this test?
There was a problem hiding this comment.
It could work, but what's your idea for installing transformers, torch{vision,audio,text} from cache? Compiling them every time from sources will be time consuming.
There was a problem hiding this comment.
I can use gh to download the pre-built artifacts. That way this will work for developers as well. Aren't these artifacts already available in pip? If yes, I can install them using pip
There was a problem hiding this comment.
They are available as artifacts in our nightly wheels, but the workflow should also work when we change the pytorch pin (pytorch has pins for all these dependencies).
There was a problem hiding this comment.
ok, I see; this definitely requires some rework (note, your existing actions do not work for conda workflow, I cannot just reuse them); I suggest splitting the task into two - first I do the "fixing", than I can implement caching, or rebuilt wheels if they are not found
There was a problem hiding this comment.
why do we need both wheels and a cache?
|
I've tried using pre-built wheels, it does not work (jit compiler cannot find oneapi) |
No-basekit requires more work