Traffic Light Detection
Once the vehicle is able to process waypoints, generate steering and throttle commands, and traverse the course, it will also need stop for obstacles. Traffic lights are the first obstacle that we'll focus on.
The traffic light detection node subscribes to the following topics:
- image_color provides an image stream from the car's camera. These images are used to determine the color of upcoming traffic lights
- current_pose provides the vehicle's current position
- base_waypoints provides a complete list of waypoints of the car's course
- vehicle/traffic_lights provides the (x, y, z) coordinates of all traffic lights. This helps to acquire an accurate ground truth data source for the traffic light classifier by sending the current color state of all traffic lights in the simulator. When testing on the vehicle, the color state will not be available. You'll need to rely on the position of the light and the camera image to predict it
It publishes the index of the waypoint for the nearest upcoming red traffic light's stop line to the traffic_waypoint topic.
The camera image data is used to classify the color of the upcoming traffic light using a classifier.
- We took the approach of classifying the entire image.
- We used the transfer learning technique on the ResNet50 architecture
- We used Keras pretrained model trained on Imagenet dataset
- We freezed the first 150 layers and removed the top layers
- We added one GlobalAveragePooling2D() and one Dense layer with relu activation (size = 1024) and the final softmax layer with 4 classes (Green, None, Red, Yellow). We used two dropout layers.
| Layer (type) | Output Shape | Param # |
|---|---|---|
| image_input (InputLayer) | (None, 224, 224, 3) | 0 |
| resnet50 (Model) | multiple | 23587712 |
| global_average_pooling2d_1 | (None, 2048) | 0 |
| dropout_1 (Dropout) | (None, 2048) | 0 |
| fc1 (Dense) | (None, 1024) | 2098176 |
| dropout_2 (Dropout) | (None, 1024) | 0 |
| predictions (Dense) | (None, 4) | 4100 |
- Total params: 25,689,988, Trainable params: 11,037,700, Non-trainable params: 14,652,288
- We used Adam optimizer with a low learning rate (0.00001)
- We prepared dataset by combining Simulator data provided by Udacity and real world samples of Rosbag data from test site. It is saved [here]
- Training Images were resized to (224,224) which is the input to ResNet50 architechture
- We used the preprocessing function used for training the original ResNet50 provided in the keras
- In training we used random cropping of the image;
- rotation on the random angle (+/- 5 degrees)
- random horizontal flipping and
- random zooming (+/- 0.3)
Training was done on Flyodhub.com The code, graphs and results for the final model are available [here]
The final trained model file is included in the repository as 4 split up parts. On first launch of the ROS program, the split parts will automatically be joined into the complete model file, stored in the /ros/src/tl_detector/light_classification/models/ folder.
-
ResNet50-UdacityRealandSimMix-Best-val_acc-1.0.hdf5
- Trained on simulator data + Udacity data together.
- Works best on rosbag file as well as on highway track
-
ResNet50-SimData-tl-weights-Best-val_acc-1.0.hdf5
- Trained on simulator data from highway track
- Works best only on highway track
-
ResNet50-UdacityMix-tl-weights-Best-val_acc-1.0.hdf5
- Trained on traffic_light_bag_file.zip (Trained with Images of both glare and non-glare rosbag data)
- Works best only on rosbag file
Each predicted state has to occur STATE_COUNT_THRESHOLD number of times till we start using it. Otherwise the previous stable state is used. The traffic light judgement is made based on the state machine which uses the previous light state information to determine the allowed light states and compare it with the classified light state. If the classified light state doesnt match with any of the allowed light state then the previous light state is used.
The base waypoints provides the complete list of waypoints for the course. Using the car's current location and the coordinates for the traffic lights, the nearest visible traffic light ahead of the vehicle is identified. The closest waypoint index is published to the waypoint updater node if the car is approaching nearest light i.e when the nearest traffic light is red or yellow.