RAG with vision models: Session Resources

[multispark]

This is the resources and discussion thread for this live stream: https://aka.ms/raghack/vision
It will be updated with links after the session.

Recording: https://www.youtube.com/watch?v=C3Zq3z4UQm4
Slides: PPT
Slide links:

• https://github.com/Azure-Samples/openai-chat-vision-quickstart/blob/main/notebooks/chat_pdf_images.ipynb
• https://github.com/Azure-Samples/openai-chat-vision-quickstart/
• https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/gpt4v.md
• https://github.com/Azure-Samples/rag-with-azure-ai-search-notebooks/blob/main/image_search.ipynb
• https://blog.pamelafox.org/2024/09/integrating-vision-into-rag-applications.html

Please ask any follow-up questions here, and the speakers will see them.

[uquad]

@pamelafox - maybe I am missing something here, instead of inserting the image source into the top left corner of the image ,is it not possible to pass the values of the "sourcefile" and "sourcepage" that are returned from the Search index , to the GPT4o model as text (along with the image) ,we can then update the system prompt to use take the values passed in the text field to the GPTo model as the source/

Video context: https://www.youtube.com/live/C3Zq3z4UQm4?si=A83y8lkqa9mSLdAG&t=2828

[pamelafox]

Good question! As far as I can tell from the OpenAI API, we can only send image data with an image-type message. We can also send text, but when we send multiple images, there's no way for the text to be tied with each image. So even if we said "we're sending page1.png and page2.png" and then include page1.png bytes and page2.png bytes, it wouldn't know which was which.
See example at bottom of https://github.com/Azure-Samples/openai-chat-vision-quickstart/blob/main/notebooks/chat_pdf_images.ipynb to see what I mean.

[uquad]

ah!! Makes sense . Thank you for clarifying :)