DJL BlockFactory #697
DJL BlockFactory #697
Conversation
|
If the BlockFactory is for DJL serving, does it integrate with the existing ModelZoo or ModelLoader concepts? |
| .optEngine(name) | ||
| .optGroupId("ai.djl." + name.toLowerCase()) | ||
| .optFilter("layers", "50"); | ||
| Model model = ModelZoo.loadModel(builder.build()); |
There was a problem hiding this comment.
In this case it loads the model with just name, since you don't specify any url/file?
There was a problem hiding this comment.
No, it does looking for the model stored in DJL model zoo.
integration/src/main/java/ai/djl/integration/tests/nn/BlockFactoryTest.java
Outdated
Show resolved
Hide resolved
The blockfactory will be both used in ModelZoo and DJLServing. It can be the way for ModelLoader to load customized DJL trained model |
Is the use of the BlockFactory mandatory or optional? It seems like it is a possible way for implementing the modelLoader, but doesn't really ease the process. It just adds an additional class and moves the work around. I am trying to understand how the BlockFactory will be used and how it integrates with the rest of the DJL API:
|
@zachgk Here is my thoughts: This will only be mandatory for any DJL based model. If any model contains DJL imperative blocks should implement the
Ideally, All DJL block that can be used directly should implement the BlockFactory. Here is an example of the case when we use it: Assume you have a symbolic model (e.g MXNet Symbol, TorchScript) and try to use it along with Imperative part (e.g fullyConnected). The model loading part is quite complicated: You have to Load the Symbolic part and then the imperative parts. If we implement the block factory, it works like the The imperative model stored in the model zoo will be like the followings:
These two files will be sufficient for us to serve the model in anywhere. Since the serialization steps is complete, all model information will be stored into parameters.param. |
|
Make factory serializable |
|
maybe add an example use case and reason on the BlockFactory javadoc |
* This creates the component which will populate the Download Tab with Download Buttons. * Making a place for the download buttons. * Adding the Model Download Handler allowing the backend to feed the links into the Model View and making slight changes for readablity. * Getting rid of some of the test code. * Improve Block usability (#712) * Use builder pattern for Parameter (#661) * Make XavierInitializer default value & Improve setInitializer (#664) * Refactor initialize (#675) * Remove NDManager on getOutputShapes (#710) * Removing unnecessary logging messages. * block factory init commit (#697) * [DOCS] Fixing TrainingListener documentation (#718) * Fixing TrainingListener documentation * Fixing PR reviews * Fix DJL serving flaky test for mac (#721) Change-Id: I9eccc84b0c34652e50c5fe5a4fe42f2b82d65a3d * Fixing all of the nits. * Getting rid of unnecessary methods. * update onnxruntime along with String tensor (#724) * Add profiler doc (#722) * Resolving some comments. * Using a better criteria incase multiple models have the same name. * Fixing the java doc. * Configure verbose of mxnet extra libraries (#728) Change-Id: I66d54aa496cccbb9e8c0a89eeaa458605958d9c6 * Added a TODO for using the artifact repo to get the base uri. * paddlepaddle CN notebook (#730) * paddlepaddle CN notebook * install font Change-Id: I2d749e617b0bf78ecbcd168b82c53a1fab49a2c0 * refactor on name Change-Id: I9e379eee51ceae16391850b3ba9782acb04c4021 * Refine the text Co-authored-by: gstu1130 <[email protected]> * add EI documentation (#733) * add EI documentation * fix pmd rules Change-Id: Ieee5577c26f6df2843781f8f9180de35069a5de3 * allow pytorch stream model loading (#729) * allow pytorch stream model loading * updates Change-Id: Ibc26261b90de673712e90de0d640a8f32f23763e * add NDList decode from inputStream (#734) Change-Id: I6a31d8b0b955f2dbb762220b101e3928a34699c1 * Remove memory scope and improve memory management (#695) The MemoryScope reveals a number of shortcomings within the DJL memory management. While the MemoryScope is deleted, many of them are fixed as part of this PR. First, the NDManager.{attach, detach} were renamed to xxxInternal. This is to differentiate them from the attach and detach methods that are intended to be used. There are two new concepts in memory management. An NDResource interface was created to combine the concepts of managed memory that was used in NDArray and NDList. It could also be used in more classes in the future. This includes the getManager, attach, and detach. Within the NDManager, it gains a second "management convention". The first convention of normal resources are added to the manager and then closed when the manager closes. This works for small numbers of things on the NDArray, but not when operations transitively create. So, the second convention is a tempResource. Instead of freeing them when the manager is closed, they are returned to their original manager. This is used to create a temporary scope, do operations within it, and then the inputs and return value are returned to the parent while the intermediate work is cleaned. This also matches the concepts of ownership/borrowing as well. Using these, a few additional helper methods were created. There is `NDManager.from(resource)` to ease creation of managers based on a resource. There is also `scopeManager.ret(returnValue)` to help with returning values outside of the scopeManager. Lastly, there is a `scopeManager.{temp,}AttachAll` to attach a number of resources to a manager within a single call. Using these improvements, the new method were applied to the old locations where MemoryScope was used as well as an additional case in NDManagerEx. Also, the old attach methods were altered to be `void`. Because the return values are no longer used anywhere and are not as necessary in the current scheme, I figured it would simplify things. It also helps for things like `NDList.attach` which does not have a single original NDManager when attaching. Change-Id: I91d109cd14d70fa64fd8fffa0b50d88ab053013e * Remove erroneous random forest application (#726) The application was changed to the more accurate softmax_regression (matching the terminology from the D2L book). Change-Id: I1f69f005bbe38b125f2709c2988d06c14eebb765 * Minor fixes on duplicated code (#736) * remove methods that already defined in the NDArrayAdapter Change-Id: I01cc03a7f5b427bf31c6b3fd8d2136f2a27fe93b * refactor toString Change-Id: Iea22b16e1daa9f759b55c1a8b8b85536482e551a * remove sparse NDArray Change-Id: Icb44096519775f54cb32cc768c14f49e33dc7ea5 * fix test Change-Id: Icef580ed77e7bba22864ce44577de3cba51e3e41 Co-authored-by: Jake Lee <[email protected]> Co-authored-by: Lanking <[email protected]> Co-authored-by: aksrajvanshi <[email protected]> Co-authored-by: Frank Liu <[email protected]> Co-authored-by: Zach Kimberg <[email protected]>
Description
The block factory is a class that provides easy access to recover DJL based model for inference with DJLServing. It contains a single function
getBlock()and essentially just recover the block.Apart from that, there is a new
SymbolBlock.newInstance()method that only create a empty SymbolBlock holder. It can now be used in PyTorch and MXNet to recover the Symbolic model information from combined params file. In this case, we no longer needs to load the SymbolBlock from the model, instead, everything is loaded from 1 file.