Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run VizWiz captioning baseline and ALT prompts and scoring on larger (~5K images) dataset #25

Open
danielnapierski opened this issue Mar 29, 2023 · 3 comments
Assignees

Comments

@danielnapierski
Copy link

I have run the baseline captioning VizWiz on the standard test 8000 image leaderboard.

@danielnapierski danielnapierski self-assigned this Mar 29, 2023
@danielnapierski
Copy link
Author

Baseline results on standard 8000 test:

[{"test": {"B1": 55.59, "B2": 37.76, "B3": 25.32, "B4": 16.71, "METEOR": 17.56, "ROUGE-L": 38.71, "CIDEr": 48.25, "SPICE": 13.13}}]

vizwiz_joined_captions.json.txt

I split the execution across 10 GPUs by splitting the vizwiz test.json annotation file into 10 files. I ran 10 docker processes (each using a single GPU on 1 of 3 gaia-lg machines). I then combined all the results and submitted one json file.

@danielnapierski
Copy link
Author

Team B1 B2 B3 B4 ROUGE-L METEOR CIDEr SPICE
Unified-io-inference docker 55.59 37.76 25.32 16.71 38.71 17.56 48.25 13.13

@danielnapierski
Copy link
Author

The baseline captioning implementation would place 9th on the VizWiz Captioning 2021 Leaderboard: https://eval.ai/web/challenges/challenge-page/739/leaderboard/2006

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant