When inferring with “infer_A_md_details.sh” provided by the project, there are a large number of errors in the inference results as shown in the following example. Can this kind of problem be solved？

[StarrySkyGao]

eg:
{
"id": "st-catherines-loch-fyne-1256364_0",
"text": "The image depicts a person standing on a body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body",
"output_id": null,
"prob": null,
"confidence": 0.28732764565010416
},
A similar situation occurred when we tested the gradio demo.

[zhiyuanyou]

Hello, which model are you using? Can you provide the link?

[StarrySkyGao]

The pretrained delta checkpoin is: https://huggingface.co/zhiyuanyou/DepictQA2-DQ495K/blob/main/ckpt.pt

[zhiyuanyou]

I have a test, and the results are normal.

    {
        "id": "st-catherines-loch-fyne-1256364_0",
        "text": "The image depicts a serene seascape with a boat on the water and a hill in the background. The evaluated image suffers from serious noise distortion, which significantly degrades the visual clarity, obscuring details and reducing the overall sharpness. The distortion creates a grainy texture throughout, particularly noticeable in the sky and water, which diminishes the aesthetic appeal. The overall quality of the evaluated image is poor, with the noise greatly affecting the image's fidelity and making it less suitable for purposes requiring clear visual information.\n ",
        "output_id": null,
        "prob": null,
        "confidence": 0.7676032081398855
    }

Please check whether the ckpt.pt is loaded, or, revise config.model.delta_path in config.yaml to absolute path.

[StarrySkyGao]

I revised config.model.delta_path in config.yaml to absolute path and tried angin.The result is still the same.
{
"id": "st-catherines-loch-fyne-1256364_0",
"text": "The image depicts a person standing on a body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body body",
"output_id": null,
"prob": null,
"confidence": 0.28732764565010416
},

If I don't load the pretrained delta checkpoin, the result is：
{
"id": "st-catherines-loch-fyne-1256364_0",
"text": "",
"output_id": null,
"prob": null,
"confidence": null
}
And when inferring, it prints the model structure as follows:

What else do you think could be causing the problem?

[zhiyuanyou]

Have you ever revised some codes? And what is your environment, especially the version of torch and transformers.

[StarrySkyGao]

torch--1.13.1
torchvison--0.14.1
transformers--4.43.3

[zhiyuanyou]

Try:
torch == 1.13.1
torchvison == 0.14.1
transformers == 4.35.0

[StarrySkyGao]

Thanks for your patient help, I re-cloned the code without any revise, and then configured the environment as the requirements in your project (transformers==4.35.0,torch==1.31.1). But the result was the same.
{
"id": "table-709710_0",
"text": "The image depicts a well-set table with a variety of dishes, including a salad, a plate, and a bowl of soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup soup",
"output_id": null,
"prob": null,
"confidence": 0.6697671168691698
},

Are there any other possible reasons?

[StarrySkyGao]

torch==1.13.1

[zhiyuanyou]

If nothing is changed, the most possible reasons are the pre-trained weights.

You can email to me ([email protected]) to get my WeChat account.

[zhiyuanyou]

After checking, our delta weights are trained on vicuna-v1.5-7b, this error is caused by wrongly downloading vicuna-v1.5-7b-16k weights.

[StarrySkyGao]

Thanks for your help! The problem has already been solved.