Multiple serialized models in one .mar file?

[alexdmiller]

Is your feature request related to a problem? Please describe.

We have multiple serialized models that we would like to use for a single TorchServe endpoint.

Describe the solution

Right now, I'm not aware of a way to achieve this with TorchServe. When creating the .mar archive, only a single serialized model can be passed in.

Describe alternatives solution

If we could store multiple models in the .mar archive, then we could write a custom handler to load those models in and chain them together.

[harshbafna]

@alexdmiller: Currently TorchModelArchiver allows supplying only one serialized file with --serialized-file parameter while creating the mar. However, you can supply any number and any type of file with --extra-files flag. All the files supplied in the mar file are available in model_dir location which can be accessed through the context object supplied to the handler's entry point.

Sample code snippet:

properties = context.system_properties
model_dir = properties.get("model_dir")

[alexdmiller]

I see, that makes sense. That solves my problem, thanks!