The context/modality separation issues due to independent calls to gpt-4-turbo

[XinshaoAmosWang]

Hi @yongliang-wu ,

Thanks for providing the details about how to reproduce your great work, I followed your guide to run the multiple steps.

After reading the steps in detail, I noted some unintuitive issues, which I think may affect the effectiveness of this end2end pipeline for long-video understanding, and video-based Q&A.

1. The description generation per clip is separate and independent call to gpt-4-turbo.
2. After concatenate clips' descriptions and ASR transcript, the final script generation is another independent call to gpt-4-turbo.

I have the following questions to ask for your kind help:

1. Had you tried Multi-turn conversation considering the previous context of images sequence per clip? If so, do you have a workable example? Or due to the large size of context window of images sequences, did you find it is impossible?
2. Could you please provide an example of asking questions about the input video based on the generated final script?

Thanks very much in advance.
Best regards,
Amos