InFur showcases ONNX dense model inference on videos (and images).
Infur should compile on any Tier 1 platform with a recent Rust toolchain.
You must also have an ffmpeg executable in your PATH (>=4.3 recommended).
cargo test
cargo test will ensure synthetic test videos exist in ./media
and download a quantized segmentation model
to ./models.
The first test run will fail if no onnxruntime with Opset support >= 8 is
on the system path.
One fix is to copy the .dll downloaded by onnxruntime-sys next to the target .exe:
find target -name onnxruntime.dll -exec cp "{}" target/debug/deps \;
cargo testCopy it also to debug and release to run the main application
(e.g. cp target/debug/deps/onnxrutime.dll target/release).
You can provide a video URL to start with:
cargo run --release -- http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/SubaruOutbackOnStreetAndDirt.mp4
Next, paste the path of the segmentation model from the test fixture into the Inference text box:
models/fcn-resnet50-12-int8.onnx
The model's dense multi-class prediction, i.e. a segmentation mask is color-coded (argmax) and shaded (by confidence):
A model's output often varies greatly with the scale of the input image. Thus, you can
tune its scale factor on Pause:
By default, the app's settings are persisted after closing.
The purpose of this crate is to study tradeoffs regarding model inference, native GUIs and video decoding approaches, in Rust 🦀.
There are a couple of Todos will make InFur more interesting beyond exploring
production-readiness as now:
- GATify
type Outputintrait Processor - bi-linear image scaling
- meta-data aware image pre-processing choices
- softmax if model predictions are logits (and/or clamp confidence shading)
- class label captions
- file-picker for model and video input
- video fast-forward/backward
- video seeking