Skip to content

Commit

Permalink
Improve inference batch manager (#47)
Browse files Browse the repository at this point in the history 
* fix prefill CUDA mem leakage

* update batch manager

* fix logic

* switch back to spawn and add cli arg
  • Loading branch information
@loubbrad
loubbrad authored Jul 17, 2024
1 parent e0f66da commit d481680
Show file tree
Hide file tree
Showing 3 changed files with 175 additions and 71 deletions.
Loading

0 comments on commit d481680

Please sign in to comment.