Streaming for serving with chat's generate function

[rasbt]

This is an alternative to #1424 using the generate function from chat.

[rasbt]

This now also works:

litgpt  serve --checkpoint_dir checkpoints/EleutherAI/pythia-1b --stream true

import requests
import litserve

print("LitServe:", litserve.__version__)
url = "http://127.0.0.1:8000/predict"

resp = requests.post(url, json={"prompt": "Hello world"}, headers=None, stream=True)
for line in resp.iter_lines():
    if line:
        print(line.decode("utf-8"))

LitServe: 0.1.1.dev0
{"output": "!\""}{"output": "\n"}{"output": "\n"}{"output": "After"}{"output": " my"}{"output": " visit"}{"output": " to"}{"output": " the"}{"output": " United"}{"output": " Kingdom"}{"output": ","}{"output": " I"}{"output": " found"}{"output": " a"}{"output": " new"}{"output": " favorite"}{"output": " place"}{"output": " to"}{"output": " visit"}{"output": "."}{"output": " I"}{"output": " was"}{"output": " excited"}{"output": " to"}{"output": " go"}{"output": " back"}{"output": " to"}{"output": " the"}{"output": " World"}{"output": "'s"}{"output": " End"}{"output": " to"}{"output": " experience"}{"output": " more"}{"output": " of"}{"output": " the"}{"output": " area"}{"output": "."}{"output": "\n"}{"output": "\n"}{"output": "The"}{"output": " World"}{"output": "'s"}{"output": " End"}{"output": " is"}{"output": " situated"}{"output": " in"}{"output": " the"}{"output": " f"}{"output": "ooth"}

The only remaining issue is that the stop token terminates everything. Otherwise, it works fine.

[rasbt]

@aniketmaurya Are there any best practices / examples for how to piece the outputs together as a string? It looks very obvious at first glance, and I have tried many things, but it's harder than I thought and requires lots of lines of code because I sometimes get invalid JSON errors.

[aniketmaurya]

@rasbt opened an issue for you here. The PR should make it easier for clients to iterate over the chunks and decode the JSON response.

[rasbt]

Thanks @aniketmaurya , I can now do:

import requests
import litserve
import json


print("LitServe:", litserve.__version__)
url = "http://127.0.0.1:8000/predict"

resp = requests.post(url, json={"prompt": "Hello world"}, headers=None, stream=True)
for line in resp.iter_lines():
    if line:
        print(json.loads(line)["output"], end="")

and it works perfectly!

[rasbt]

The tests are failing because there hasn't been a litserve 0.1.1 release yet. It's fine (no rush), I just think we should wait until there's been a new release.

[aniketmaurya]

The tests are failing because there hasn't been a litserve 0.1.1 release yet. It's fine (no rush), I just think we should wait until there's been a new release.

@lantiga do we want to make a release anytime soon? LitServe recently had a lot of bug fixes in addition to OpenAI spec too since the last release.

[aniketmaurya]

hi @rasbt, can we add a __init__.py to the deploy module so that the BaseLitAPI is importable in case user want to override the default behaviour or customize it?

EDIT: Actually, I was able to import it even without that. I got an import error when I did an editable install of LitGPT.

[rasbt]

Thanks for the suggestion, it absolutely makes sense to add it, @aniketmaurya . Just did.