Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add API for simultaneously streaming and iterating #360

Open
elijahbenizzy opened this issue Sep 8, 2024 · 0 comments
Open

Add API for simultaneously streaming and iterating #360

elijahbenizzy opened this issue Sep 8, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@elijahbenizzy
Copy link
Contributor

elijahbenizzy commented Sep 8, 2024

Is your feature request related to a problem? Please describe.
See this discussion: #358.

The problem is this:

  1. You have streaming results
  2. You want to still run a chain/state machine
  3. You want to stream back intermediate results

For instance, say you have a voice call -- you want to move from one state to another. You need to stream it in on each step.
Or, say you have an agent that does some complicated stuff. You want to stream back intermediate results from each step to show how the agent is currently thinking. You'll want to stream, while iterating through...

Describe the solution you'd like

Sync

for action, streaming_container in app.iterate_streams(
    halt_after=[...], 
    halt_before=[...], 
    inputs={...}
):
    do_something_with(streaming_container) # stream back in your fastAPI endpoint, say

Async

async for action, streaming_container in app.iterate_streams(
    halt_after=[...], 
    halt_before=[...], 
    inputs={...}
):
    await do_something_with(streaming_container) # stream back in your fastAPI endpoint, say

The nice thing is we should be able to call stream_results in a loop. The tough thing is it's a bit complex -- we could stream back all the individual results at once, but there's no guarentee the shape is the same.

Other things to think about:

  • should we stop at all results? Or just the streaming ones (and walk through the non-streaming ones...). Maybe have this be an option?
  • Should we automatically chain it? Or not? My guess is not.

Describe alternatives you've considered

See the possibility here: #358. It's all doable, but not particularly ergonomic. So the alternative is potentially to have this workaround. We should add this to the documentation too.

@elijahbenizzy elijahbenizzy added the enhancement New feature or request label Sep 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant