How to get coordinates from image

[cmahnke]

Is there a way to get coordinates from the image for recognized objects?

[liebharc]

The short answer is, unfortunately, "no, it's not implemented yet."

However, if we extend the current implementation, it's possible to approximate object coordinates. Here's how it should work (run homr with --debug to see the intermediate steps):

1. homr scales any input image to a resolution between 3-4 MP, so all obtained coordinates need to be translated - this is simple.
2. Staffs are detected using classical image recognition, so their coordinates are easy to obtain:
   
3. Other objects are detected by a transformer model, where we lose a direct pixel-to-object relationship. However, by looking at the transformer's attention map, we can assume that the most focused pixels represent the detected object. While not guaranteed, this works in most cases:
   

In summary, by using the center of attention for each object, you can calculate image coordinates for each recognized object.

It would be helpful to know your use case for a deeper understanding of your question.

[cmahnke]

Thanks for your fast and extensive answer! I forgot to mention it in my initial question, but it's a very impressive piece of software you created! Very good work!

I Try to display the progress of playing the recognized music on top of the digitised image. The first step would be to have the bounding boxes as you wrote in the second item, given that the speed is known, it should be possible to do a rough estimation...

And a minor nitpick, I can create an issue if you like: Python 3.13 isn't supported yet, since PyO3 complains during installation of required dependencies.

[liebharc]

Thanks! For your use case, the steps I mentioned should be sufficient. However, I’m not entirely confident that you can skip step 3. In my experience from implementing a position display for SVG sheet music, I found that neglecting certain details can easily lead to timing errors of 1, 2, or even 3 seconds. Here are a few considerations:

• Number of Measures and Their Size: The method I outlined in point 2 theoretically provides this information, but it wasn’t optimized for accuracy. As a result, it may overlook some bars and give you an incorrect measure count. While you can retrieve the number of measures from the transformer output, keep in mind that measures often vary in width, so you’ll likely need to reference the attention map at some point.
• Pickup Notes: It’s common for the first and last measures to be shorter than the others due to pickup notes. To handle this accurately, you'll need to extract rhythm information from the transformer model.

Regarding Python 3.13, it would be good to have an issue for that. I may not address it myself in the near future, but documenting it this way increases the chances that someone else will take it on.

[cmahnke]

I've added #10 - as I already wrote, it's caused by a depedency, so no need to fix for now, just something to document.
I'll check your hints in the next weeks

[cmahnke]

I had a first look and would like to ask for some feedback. I'm referring to point 1 from your reply above. This can be added in main.py as part of load_and_preprocess_predictions, just measure the input and output resolution and compute a scale factor from them.

On the other hand it could also be done in InputPredictions (from model.py) since both images are available there as well.

Since the InputPredictions object is passed into the next step I would add the handling / calculation there even though the current class don't have methods yet.

What would you propose?

[liebharc]

I right now tend towards adding it to main.py. There isn't a strong argument behind this. The main thought is to keep this information local to where it is needed. The decision might be clearer if we get an overview over all changes. I added some notes to this branch https://github.com/liebharc/homr/tree/feature/position-info . The transformer code can be a hard to navigate, so hopefully this helps. Of course, the branch is at best a starting point to what you want to do. The flow could be to add the initial position to the result objects by using the attention map. And then as the result is passed back as part of the method call stack, we transform the coordinates based on what transformations have been done to the image in the related code parts. Hope that wasn't too cryptic - it's getting late :).

[liebharc]

I've updated the branch. It's still a draft, and it would be a miracle if the calculations would already be correct, Hopefully it still makes things clearer.

[cmahnke]

Thank you very much!
I'm sorry, I didn't find the time to have a deeper look yet.