You can find videos regarding the project here!
- ROS Noetic
- Computer vision (OpenCV)
- Python 3
- Numpy
This repository has one workspace namely, sdp_pouring_ws. It consists of further three packages:
- hsr_vision
- hsr_force
- hsr_joint_controller
[NOTE] The code is to be executed from within Lucy after login using SSH.
ssh -X [email protected]
Following sequence of commands to execute
1. Sourcing the workspace
source sdp_pouring_ws/devel/setup.bash
2. Feedback from force sensor
Run the force_sensor.py file to obtain the current weight in the grasped container.
rosrun hsr_force force_sensor.py
(i) Action performed:
a. This command will bring the Lucy to its initial pouring position. Make sure Lucy is far away from table to prevent collision.
b. Lucy will ask to place the container between the gripper. Grasp at a position between mid to top region of the container. Selection of mid-to-top position for grasping in order to avoid collision with the cup or table.
c. Weight of the grasped object would be announced as well as published (topic: \grams).
3. Feedback from vision sensor
For this project HSR head right stereo camera is used.
3.1 Colour independent
Run the edge detection file, if edge based detection algorithm (colour-independent) is to be performed)
rosrun hsr_vision edge_detection.py
(i) Bounding box generation:
a. This command will pop up the window of the current image.
b. Draw bounding box around the cup by dragging the mouse around the cup (Eventhough the width of the glass is not entirely covered by the bounding box, make sure the entire height is covered for better accuracy). Press “r” to redraw the bounding box. Press “c” to confirm the bounding box and start publishing percentage.
!!! Make sure the glass position is not changed after the bounding box is set because the initial background image is subtracted for noise suppression.
(ii) Following pop up windows appear:
a. Cropped image of the selected region.
b. Sobel-y filter output subtracted with initial reference frame.
c. Thresholded and noise suppressed image.
d. A relative representation of the content within the glass.
(iii) Action performed:
a. Current liquid/cereal level in the cup is published (topic: \percent).
3.2 Colour dependent
Run the colour detection file, if colour-mask based detection algorithm is to be performed.
rosrun hsr_vision colour_detection.py
(i) Bounding box generation:
a. The above command will pop up the window of the current image as well as the track bar to adjust the bounding box.
b. Adjust the bounding box height and width from the track bar based on the cup selected.
c. Once the bounding box is generated, it starts publishing percentage of filled cup.
(ii) Following pop up windows appear:
a. Track bar for selecting HSV parameters to detect appropriate color.
b. Track bar for selecting height and width of the cup to detect level of the cereals/ liquid during pouring.
c. Real time view of the camera.
d. Mask would be displayed when lucy starts pouring liquid/cereal.
(iii) Action performed:
a. Current liquid/cereal level in the cup is published (topic: \percent).
4. Controlling pouring action based on the feedback received from different sensors.
Run the pouring_action.py file along with the target level
rosrun hsr_joint_controller pouring_action.py --target user_input_level
This code controls the motion of wrist for pouring as well as desired level of liquid required in the cup.
This subscribe to two topics:
a. \percent published by vision .
b. \grams published by force.
(i) Action performed:
a. The forward movement and the task completion condition is being handled by vision feedback.
b. The intermediate stop and go-back-a-bit action when weight changes beyond a level is being handled by force feedback.