Add Stream API Example

README.md

@@ -8,9 +8,9 @@
 **High-throughput serving engine for AI models**
 
 <pre>
-✅ Batching       ✅ Streaming          ✅ Auto-GPU, multi-GPU 
-✅ Multi-modal    ✅ PyTorch/JAX/TF     ✅ Full control        
-✅ Auth           ✅ Built on Fast API                         
+✅ Batching       ✅ Streaming          ✅ Auto-GPU, multi-GPU
+✅ Multi-modal    ✅ PyTorch/JAX/TF     ✅ Full control
+✅ Auth           ✅ Built on Fast API
 </pre>
 
 

src/litserve/examples/simple_example.py

@@ -68,3 +68,36 @@ def predict(self, x):
     def encode_response(self, output):
         # float will take the output value directly onto CPU memory
         return {"output": float(output)}
+
+
+class SimpleStreamAPI(ls.LitAPI):
+    """
+    Run as:
+        ```
+        server = ls.LitServer(SimpleStreamAPI(), stream=True)
+        server.run(port=8000)
+        ```
+    Then, in a new Python session, retrieve the responses as follows:
+        ```
+        import requests
+        url = "http://127.0.0.1:8000/predict"
+        resp = requests.post(url, json={"input": "Hello world"}, headers=None, stream=True)
+        for line in resp.iter_content(5000):
+        if line:
+            print(line.decode("utf-8"))
+        ```
+    """
+
+    def setup(self, device) -> None:
+        self.model = lambda x, y: f"{x}: {y}"
+
+    def decode_request(self, request):
+        return request["input"]
+
+    def predict(self, x):
+        for i in range(3):
+            yield self.model(i, x.encode("utf-8").decode())
+
+    def encode_response(self, output_stream):
+        for output in output_stream:
+            yield {"output": output}