A TensorFlow implementation of the paper Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car by NVIDIA, Google Research & New York University.
Follow the steps to install and try this locally in your system
- Clone the repository
https://github.com/vishalkrishnads/pilotnet.git
- Change working directory
cd pilotnet
- Create a conda environment specificially with Python 3.8 and activate it
conda create -n "pilotnet" python=3.8.0 conda activate pilotnet
- Install tensorflow into the environment by following the instructions in the official documentation. If you're not using a CUDA capable GPU or don't have a GPU, skip to the next step.
- Install other required modules
pip install -r requirements.txt
- Run the app
python app.py
- Run the
app.py
file. It is the entry point. You will be presented with a menu$ python app.py # a banner 1. Train using generated data 2. Generate new data 3. Predict on a single video frame 4. Predict on live video feed 5. Wrap up. I wanna quit. Enter your choice >>
- Enter your choice and follow the prompts. This is an infinitely looping menu. Choose 5 to leave.
-
Training
- If your machine is not that compute intensive, this model can't be forced to take in high quality images, say like 1920x1080. If you try to do this, it will raise an exception that the resources have finished and quit. Try scaling the quality up slowly from the default values and see where it sticks for your system.
- Sometimes, you might get too excited and try to train 100 epochs on 2 minutes of recordings. This will obviously end with an error saying that the generator can't produce enough data for all these epochs. Either record more data or reduce epochs.
-
Data Generator
- In WSL, the data generator has a fallback system for WSL connections. But just in case yours is failing, try using the
ping $(hostname).local
command to find the host IP address. Now, openapp.py
and change the IP from127.27.144.1
to your IP address inCollector.run2()
. Restart.# ... warn('There seems to be a problem with your CARLA server. Retrying with WSL address...') # change here # client = carla.Client('172.27.144.1', 2000) client = carla.Client('<your IP>', 2000) world = client.get_world() # ...
- There have been many reports online of not being able to connect to CARLA server but most of these relate to blocked ports or network issues. Check your computer by following code. If this crashes, then try fixing this first and then the generator should work
import carla client = carla.Client('localhost', 2000) world = client.get_world()
- If your disk doesn't have enough empty space, then it's obvious why the generator crashses. It stores recordings in the
recordings/
directory. Try clearing space.
- In WSL, the data generator has a fallback system for WSL connections. But just in case yours is failing, try using the
-
Predicting on frame
- The only possible issue here would be to enter paths of stuff that doesn't exist. Maybe a test image or saved model. The script will crash if you enter something wrong. Try double checking the paths.
pilotnet
|
|-pilotnet
| |
| |- data.py (a custom datatype for training)
| |- model.py (the actual model cum helpers)
|
|-utils
| |
| |-collect.py (data collector)
| |-screen.py (screen utilities)
|
|-app.py (entry point for running app)
|-requirements.txt (python requirements file)
|-README.md (this documentation)