Skip to content

Latest commit

 

History

History
106 lines (65 loc) · 6.42 KB

README.md

File metadata and controls

106 lines (65 loc) · 6.42 KB
Raw

PreSight

[ECCV 2024] PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors

arXiv

Introduction

This repository is an official implementation of PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors.

News

  • [2024/07/20]: 🎉 We have released the code of PreSight!

  • [2024/07/09]: 🎊 Our paper has been accepted by the The 18th European Conference on Computer Vision (ECCV 2024)! Our code will be release this month. Stay tuned!

Main Results

Online HD Mapping

Model Metric w. Prior Ped Crossing Divider Boundary All Runtime (FPS)
StreamMapNet AP × 10.19 11.26 11.87 11.10 22.4
StreamMapNet AP ✓ 21.11 23.73 32.31 25.72 (+14.62) 21.9
MapTR AP × 4.97 8.20 9.83 7.67 25.2
MapTR AP ✓ 16.18 19.04 34.14 23.12 (+15.45) 23.2
BEVFormer IoU × 14.90 29.88 32.74 25.84 15.5
BEVFormer IoU ✓ 16.37 34.82 51.66 34.28 (+8.44) 14.3

Occupancy

Method w. Priors mIoU Dynamic Static others barrier bicycle bus car constr. vehicle motorcycle pedestrian traffic cone truck drive surface other flat sidewalk terrain manmade vegetation Runtime (FPS)
BEVDet × 29.3 24.4 38.2 1.5 42.4 11.0 43.0 47.1 19.1 23.3 23.4 19.5 37.8 72.9 11.6 30.9 48.6 32.7 32.5 5.1
BEVDet ✓ 33.7 (+4.4) 24.4 50.5 (+12.3) 1.2 40.1 14.8 42.1 48.3 15.7 26.4 24.4 18.7 37.2 81.8 15.2 40.3 60.5 50.4 54.9 4.9
FB-Occ × 30.0 25.1 39.2 9.2 37.2 21.8 41.6 43.4 15.8 27.3 25.4 23.8 30.3 74.7 17.3 33.0 50.6 28.2 31.1 9.1
FB-Occ ✓ 34.3 (+4.3) 25.4 50.7 (+11.5) 9.3 38.3 21.0 40.3 45.0 15.9 29.9 26.0 23.8 30.2 82.3 18.5 39.1 61.2 48.0 54.7 8.6

Getting Started

To get started, please follow the instructions below step-by-step.

Pretrained Weights

Extracted Priors

Boston-Seaport Singapore-Onenorth Singapore-Queenstown Singapore-Hollandvillage
Google Drive Download Download Download Download

Perception Models

Vectorized Online Mapping Occupancy Prediction
Google Drive Download Download

TODO

  • Add scripts to inference per-image monocular depth using Depth-Anything to enable training NeRFs with monocular-depth loss. Monocular-depth loss improves visualization quality but do not improve downstream perception metrics.

Acknowledgement

This project builds upon the outstanding work of several open-source projects. We extend our sincere thanks to the following codebases:

Citation

If you find our work useful in your research, please consider citing:

@article{yuan2024presight,
  title={PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors},
  author={Yuan, Tianyuan and Mao, Yucheng and Yang, Jiawei and Liu, Yicheng and Wang, Yue and Zhao, Hang},
  journal={arXiv preprint arXiv:2403.09079},
  year={2024}
}