Explicit shared memory registry when Instantiating Run(..., shm=True) or InferenceModule(..., shm=true)
#369
Comments
4 tasks
outtanames
pushed a commit
that referenced
this issue
Oct 11, 2023
…#405) ## Summary - updated API to `client.Run(inputs: Dict[str, Any])` - new usage will be `client.Run(inputs={'image': [image]})` Resolves #367 #369 <!-- Thank you for your contribution! Please review https://github.com/autonomi-ai/nos/blob/main/docs/CONTRIBUTING.md before opening a pull request. --> <!-- Please add a reviewer to the assignee section when you create a PR. If you don't have the access to it, we will shortly find a reviewer and assign them to your PR. --> <!-- Please give a short summary of the change and the problem this solves. --> ## Related issues <!-- For example: "Closes #1234" --> ## Checks - [ ] `make lint`: I've run `make lint` to lint the changes in this PR. - [ ] `make test`: I've made sure the tests (`make test-cpu` or `make test`) are passing. - Additional tests: - [ ] Benchmark tests (when contributing new models) - [ ] GPU/HW tests
4 tasks
spillai
added a commit
that referenced
this issue
Oct 16, 2023
## Summary
This PR makes a significant update to the core API that simplifies the
need to specify TaskType's in every `Run()` call and also supports
multiple methods for any wrapped Model.
**Key Features:**
- Updated `Run()` to remove the need to specify `TaskType`s
```python
# Old `Run()` implementation
client.Run(task=TaskType.IMAGE_EMBEDDING, model_name="openai/clip")
# New `Run()` implementation (wraps `__call__`)
client.Run("yolox/yolox-medium", inputs={...})
# New `Run()` implementation with custom methods (wraps `encode_image`)
client.Run("openai/clip", inputs={...}, method="encode_image")
```
- Patch `ModelHandle` with additional methods that may be implemented
(`forward`, `encode_text`, `encode_image` ). This allows us remotely
execute model methods with a familiar API. In addition, we support
`ModelHandlePartial` objects to also allow for async submissions of
specific methods.
submisions for a specific method.
```python
spec: ModelSpec = hub.load_spec("openai/clip")
handle: ModelHandle = manager.load(spec)
# New methods automatically patched
img_embeddings = handle.encode_image(...)
txt_embeddings = handle.encode_image(...)
# ModelHandlePartials allow us to make async `submit()` calls via:
# Async submission of image encoding
handle.encode_image.submit(...)
```
- `ModelHandlePartial` now enables the following
- `handle.encode_image` is actually a `ModelHandlePartial` that wraps
`handle.__call__(..., _method="encode_image")` through it's `__call__`
method.
- `handle.encode_image.submit` wraps the `handle.submit(...,
_method="encode_image")` method.
- `Module` interface now supports multiple methods that are dynamically
patched that is automatically inspected.
- `FunctionSignature` now supports multiple signature methods and
metadata (tasks, model resources now are defined here).
**Tests**
- added checks in `test_hub.py` to test for model info and metadata
Fully functional ModelHandlePartial impl with multiple method signatures
- adds tests for `ModelHandle` and `ModelHandlePartial`s to call custom
methods including `__call__` and `submit()`
- Added tests for multiple signatures and default signatures/methods.
- Added / updated and working tests for `hub`, `common`, `executor` and
`manager`
## Related issues
Resolves #366 #368 #369 #370 #362
## Checks
- [x] `make lint`: I've run `make lint` to lint the changes in this PR.
- [x] `make test`: I've made sure the tests (`make test-cpu` or `make
test`) are passing.
- Additional tests:
- [ ] Benchmark tests (when contributing new models)
- [ ] GPU/HW tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: