Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use HTTP Livestreaming for audio/video streaming out #8906

Merged
merged 30 commits into from
Jul 31, 2024

Conversation

freddyaboulton
Copy link
Collaborator

@freddyaboulton freddyaboulton commented Jul 25, 2024

Description

Reworks our approach for streaming media to use http live streaming. This is widely supported across browsers, mobile devices and gives us the following benefits:

  • More control of when playback starts via "low latency mode". This lets the playback for audio start as soon as the first chunk is received
  • Can playback/rewind streams

Additional changes:

  • Added ability to stream video in chunks. Each chunk should be a .ts file with h.264 codec. The strictness of the encoding is set by http live streaming. The h.264 codec is pretty standard from what I read but I added the ability to convert .mp4 files into the expect file type for streaming to make a bit easier for developers.
  • Ability to download generated audio/video streams

🎯 PRs Should Target Issues

Closes #8185
Closes #6570
Closes: #5187
Closes #8177

Tests

  1. PRs will only be merged if tests pass on CI. To run the tests locally, please set up your Gradio environment locally and run the tests: bash scripts/run_all_tests.sh

  2. You may need to run the linters: bash scripts/format_backend.sh and bash scripts/format_frontend.sh

@gradio-pr-bot
Copy link
Collaborator

gradio-pr-bot commented Jul 25, 2024

🪼 branch checks and previews

Name Status URL
Spaces ready! Spaces preview
Website ready! Website preview
Storybook ready! Storybook preview

Install Gradio from this PR

pip install https://gradio-builds.s3.amazonaws.com/bed454c3d22cfacedc047eb3b0ba987b485ac3fd/gradio-4.40.0-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@bed454c3d22cfacedc047eb3b0ba987b485ac3fd#subdirectory=client/python"

Install Gradio JS Client from this PR

npm install https://gradio-builds.s3.amazonaws.com/bed454c3d22cfacedc047eb3b0ba987b485ac3fd/gradio-client-1.4.0.tgz

@@ -238,6 +238,30 @@ def show(n):
return demo


@pytest.fixture
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This got deleted somewhere along the line

@freddyaboulton freddyaboulton marked this pull request as ready for review July 26, 2024 19:08
@ylacombe
Copy link

Hey @freddyaboulton, thanks for taking care of this PR!

Just a quick question from my side, since I'm following this feature very closely. When testing the stream_audio_out demo, if I click again Stream as File (resp. Stream as Bytes) after the audio has been fully streamed, it doesn't work / restart again. So the audio streaming seems to only work once, is this expected?

@freddyaboulton
Copy link
Collaborator Author

@ylacombe - might be the same bug as @abidlabs. Will take a look.

@freddyaboulton
Copy link
Collaborator Author

That issue should be fixed now @ylacombe . Will work on the remaining tweaks/guides now!

@freddyaboulton
Copy link
Collaborator Author

Going to merge and will add a guide in a follow up pr. e2e tests are being funky s skipping for now since they pass locally.

@freddyaboulton freddyaboulton merged commit 51b7a8b into 5.0-dev Jul 31, 2024
7 checks passed
@freddyaboulton freddyaboulton deleted the http-livestreaming branch July 31, 2024 22:12
@abidlabs
Copy link
Member

Thanks so much @freddyaboulton! Tested once more and the issue I found earlier is resolved.

@shaoqingyang
Copy link

Using this bugfix tag,the first time I click the button, there will be no voice on the webUI, I need to click again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants