Have trouble streaming the video with sound.

[Henry-23]

Describe the bug

Hi, thanks for this great work. I was trying to use the new video real-time streaming output feature that is enabled by this (currently under the 5.0-dev branch).

I want to know whether it supports streaming of videos with sound? I wrote the code and yielded several videos with sound, but gradio can only play the first video. Subsequent videos cannot be played on the web page, and the progress bar cannot be dragged backward. After I downloaded the ts video to the local computer, I found that the video is complete. Is this a problem that the gradio front-end does not support it? Or is it a problem with my video encoding? The ffmpeg command I used is shown below.

Have you searched existing issues? 🔎

• I have searched and found no existing issues

Reproduction

    ffmpeg_command = [
        'ffmpeg',
        '-framerate', str(fps),
        '-i', str(image_folder / '%08d.png'),
        '-i', audio_path,
        '-c:v', 'libx264',     
        '-c:a', 'aac',          
        '-strict', 'experimental',
        '-shortest',
        '-f', 'mpegts',         
        output_file
    ]

Screenshot

No response

Logs

No response

System Info

pip install with: https://gradio-builds.s3.amazonaws.com/bed454c3d22cfacedc047eb3b0ba987b485ac3fd/gradio-4.40.0-py3-none-any.whl

Severity

Blocking usage of gradio

[abidlabs]

cc @freddyaboulton

[freddyaboulton]

Hi @Henry-23 ! Happy to look into this, if you can share the full application code it might be a bit easier for me.

[Henry-23]

Hi @Henry-23 ! Happy to look into this, if you can share the full application code it might be a bit easier for me.

Hi @freddyaboulton Thanks for your reply!

My code uses the ffmpeg command to merge all the pictures and corresponding audio in a folder into a video with sound. Merging to .ts or merging to .mp4 has a similar problem, only the first video of yield and the end part of the last video of yield can be played (for example, the first video has 4s, and the last The video lasts 5s, so the effect of playing it on the webpage is: the first 4s is the content of the first video, and 4-5s is the last second of the last video). However, after downloading the ts file, the local player can play the transmitted complete video.

Is it a video encoding/decoding issue？

def gen_video_by_index(audio_path, index, fps=25):
    print(f"Processing {index}...")
    image_folder = IMAGES_DIR / index
    image_paths = sorted([f for f in os.listdir(image_folder) if f.endswith('.png')])
    output_file = str(current_dir / f"output_{index}.ts")

    ffmpeg_command = [
        'ffmpeg',
        '-framerate', str(fps),
        '-i', str(image_folder / '%08d.png'),
        '-i', audio_path,
        '-c:v', 'libx264',     
        '-c:a', 'aac', 
        '-shortest',
        '-f', 'mpegts',         
        output_file
    ]

    subprocess.run(ffmpeg_command, check=True)

    return output_file

[Henry-23]

Hi @freddyaboulton ! When I was developing my project, I also encountered the problem of out-of-sync and repeated playback of audio and video. You can try my project here. The video stream box on the right uses the video streaming of Gradio to continuously yield multiple video clips. When playing, it is easy to cause lagging and playback problems. The relevant code is in app.py and src/pipeline.py, if you could help me figure this out I would really appreciate it!

[Henry-23]

@freddyaboulton, I suspect that this issue occurs when the previous video segment has just finished playing and the next video segment is being yielded, which may lead to some conflicts.

And in my tests, the FFmpeg command in async_convert_mp4_to_ts:

ff = FFmpeg(  # type: ignore
    inputs={mp4_file: None},
    outputs={
        ts_file: "-c:v libx264 -c:a aac -f mpegts -bsf:v h264_mp4toannexb -bsf:a aac_adtstoasc"
    },
    global_options=["-y"],
)

results in audio being present only for the first streaming video segment, while subsequent segments have no audio. However, when I remove the -c:a aac and -bsf:a aac_adtstoasc parameters, the audio plays correctly. I would greatly appreciate your help with this. Thank you!