Torchserve REST API vs Flask REST API

[MichaelMMeskhi]

I am trying to serve a model on a server. Apart from doing inference, I want to be able to send inference results to a SQL database as well as back to the request. Basically, I want to add customer API methods. Is this done via handler.py or should I just use Flask to get this done? What are the major differences between TS and Flask?

[harshbafna]

@MichaelMMeskhi: There are a couple of ways with which you can handle this scenario

• Handle at the client end: You can write a simple script to collect the inference response returned by TorchServe and execute the required SQL query.

• Handle at the server end: Execute the required SQL query in the post process method of your handler.

  • This will require your database-specific python module to be installed on the same machine where TorchServe is running. You can use the new feature of TorchServe which allows users to supply a model-specific requirements.txt file containing a list of 3rd-party python dependency packages. Refer documentation for more details. Note that this feature is only available in the latest master as of now and will be available in the next release.

[MichaelMMeskhi]

@harshbafna Thank you for your feedback! I am looking to handle this on the server end. So basically, customizing the post process method to whatever I need to get done after prediction is the best option I see.

[harshbafna]

Closing as the question has been answered.