kornia-rs integration for cv and dnn

[edgarriba]

hi! I recently learned about Dora and I wanted to share about kornia-rs, a project which we have been working on since last year based on production use cases at our company for the self-driving / perception / agtech robotics areas.

For context, kornia-rs is an extension of the kornia pytorch library that we have been curating for more thant 5 years that includes more than 500 geometric computer vision operators in pytorch very inspired by Opencv (we are friends orgs). We started the Rust flavour with the I/O module to load/write images and read from cameras in Python using Rust as backend but the project evolved more to a Rust production library and less Python support.

Now we are growing the examples and I wanted to make some integrations to show how to build more complex pipelines (initially with our Nodo framework) but open to others like Dora and soon adding a dnn module (possibly based out of onnxruntime as it's the most performant crate I found to production today (sad is still c++ written only). Recently also adopted Arrow to base our Tensor structs instead of ndarray to be more aligned with tools like rerun.io.

Would be great to find some collaboration points between the two projects. Happy to arrange a call and discuss.

[haixuanTao]

That sounds awesome!

We want to do something a bit similar called fastformat maily led by @Hennzau, but not necessarely limited to images, so it can handle things like pointclouds, bounding boxes, ... Wonder if maybe there is a way to either integrate kornia within fastformat.

I think the problem that we have is that we would want to have a generic way to handle images so that you can just focus on other things than the shape or encoding, which can be particularly tricky when dealing with multi-language, multiple-libraries...

It seems to me like performance on image processing can be heavily optimised, from trying to avoid copies to using SIMD or GPU computation. Is it performance something you guys work on? Or are you guys maybe more focus on other things like differentiation?

[Hennzau]

Hi! That's awesome! 🔥🔥

I'd be very happy to integrate Kornia into our Image Datatype within FastFormat 😁

Currently, we only have image representation from a sequence of bytes, and conversion to and from different libraries like ndarray and Arrow without copying. It would be great to add Kornia for image processing!

[edgarriba]

this sounds good to me

Is it performance something you guys work on? Or are you guys maybe more focus on other things like differentiation?

kornia / pytorch is meant for research / prototyping and we are using it only as reference to implement the algorithms in kornia_rs which is targeting performance and production stability. Differentation is for the research domain, and it would complicate things in the rust side.

I understand fastformat as a user "messages" package for dora-rs ? If you see the structure of kornia-rs, kornia-core is a light arrow buffer based Tensor with the mere purpose interoperability (e.g Rerun.io, or Nodo concurrency). Now i'm in the middle of migrating Tensor to the rest of the library in order to remove ndarray as dependency. Possibly I'll keep ndarray for a bit to keep the baseline, but the idea is to rely on accelerated image kernels via the Tensor Allocator api and curate SIMD or Cuda kernels (if needed, and if financial support) that will go to the kornia-imgproc crate.

The idea of kornia-image then I think is a bit what you are doing in FastFormat for the image flavour to standardise types, which is what we have been validating internally at the company with Nodo for real time performance pipes. Other types as bounding boxes, nms ops, and needed things will come in too as soon as we add models with that need (see prototype here for face detection from kornia torch). Pointclouds possibly too as we are drifting to spatial ai area to support more 3d stuff.

We have also integrated gstreamer into kornia-io which works quite well to capture image data from different types of cameras. I added examples for v4l and rtps and tested also with http streams too.

In summary, I see dora-rs as a good candidate to compete with ROS, and the same as opencv was the horse-power to vision_opencv and cv in general, would be great to attempt kornia-rs to be that actor. Maybe a good place to start is to build a simple camera capture example using dora-rs and kornia-rs and shape more collabs from that.