ping pong marker not being detected

[jorelllinsangan]

Hi Carlos!

I've been trying to work with your code for a while now to get my Crazyflie to hover. Great work btw! The videos are really cool.

Anyway, I'm just not sure whats wrong with what I'm doing. I think I'm doing it right when I run adjust_cam_settings.py. I tweak my settings until my ping pong ball marker is highlighted by the program. I then save the settings. But when I run calibrate_2d_3d.py the ping pong ball marker is not detected. Would you have a general idea on why this is happening?

Thanks!

[CarlosRDomin]

Hi @jorelllinsangan,

Thanks! Glad you like my work :)
The code in calibrate_2d_to_3d.py uses Spotter.CAMERA_SETTINGS_FILE and Spotter. COLOR_THRESH_SETTINGS_FILE to load the camera and color threshold settings respectively. This means that you should go to the file full_control_with_cam.py and edit the variables CAMERA_SETTINGS_FILE and COLOR_THRESH_SETTINGS_FILE in the class Spotter (lines 517-518) to point to the right path where you saved your settings. E.g.:

class Spotter:
	# .....
	CAMERA_SETTINGS_FILE = "config/cam_settings/MyCamSettings.txt"
	COLOR_THRESH_SETTINGS_FILE = "config/color_thresh/MyColorSettings.txt"

Hope this helps!

[jorelllinsangan]

Hey @CarlosRDomin,

Thanks for responding! I did modify that.

SETTINGS_ENVIRONMENT = "CSSA3"
CAMERA_SETTINGS_FILE = "config/cam_settings/Camera settings - USB 2.0 Camera -{}.txt".format(SETTINGS_ENVIRONMENT)
COLOR_THRESH_SETTINGS_FILE = "config/color_thresh/Color threshold settings - {}.txt".format(SETTINGS_ENVIRONMENT)
BLOB_DETECTOR_SETTINGS_FILE = "config/blob_detector/Blob detector settings - {}.txt".format(SETTINGS_ENVIRONMENT)

Edit: So basically my files name is Camera settings - USB 2.0 Camera - CSSA3.txt

When I run the calibrate_2d_to_3d.py it does indicate that it finds the files and that it sets the HSV values accordingly.

I'm also not quite sure what cam_calibration_output.npz does. Is there a specific one that I should be choosing?

[jorelllinsangan]

Adding images might provide a bit more clarification:

from adjust_cam_settings.py

from calibrate_2d_to_3d.py

[CarlosRDomin]

Ooooh, I think I know what's going on!!!
In the Spotter's initialization routine, the first step after opening the camera is to learn how the camera's frame of reference is aligned with respect to the world. We do this by looking for the chessboard pattern (e.g.: the bottom right corner of the chessboard will correspond to point {0,0,0} in the "world"). You have four options while this step is executing:

• Press "b" to bypass this step, creating a world coordinate frame centered at the camera's origin, and with the axes aligned with the camera's pose (x is depth, y is horizontal, z is vertical)
• Press "y" (yes) once a chessboard pattern has been found (like in the image you attached) to save the current extrinsic matrix (the transformation from camera coordinates to world coordinates, you may read more details at vision_aux_functions.py). Once you press "y" (or "b" in the case above) the rest of the code will execute (so if you were running calibrate_2d_to_3d.py, you should now see how it finds the drone and estimates the distance to the camera).
• Press "n" (no) once a chessboard pattern has been found to reject the computed extrinsic matrix (e.g.: you may want to do this if you notice it didn't correctly find the chessboard pattern, or you want to move the camera before starting to fly, etc.). If you press "n" you will go back to seeing a live feed of the camera until a chessboard pattern is found again, then the image will freeze and you can press any of the keys described here.
• Press any other key to stop execution of the code (e.g.: say you want to change something in the code, or you accidentally started running but don't want to fly, etc.)

Side note: from the image you attached, it looks like your camera is not calibrated (or the correct calibration file wasn't loaded), because it's estimating that the chessboard pattern was 13.47m away from the camera, which from the image is clearly wrong. Did you calibrate the camera using calibrate_cam_params.py? If not, you should do that (once) before anything else.

[jorelllinsangan]

I figured out the "b", "y", "and "n" thing.

When I'm running calibrate_2d_to_3d.py and I press y the program exits. Is that expected behaviour? And should the program be "highlighting" the pingpong ball while running calibrate_2d_to_3d.py?

I did use calibrate_cam_params.py. What loss value should I be getting for good results?

When I press y when I run full_control_with_cam.py the drone just shoots up in the air really quickly even when after I press e

By the way, thanks a lot for the detailed responses to my questions! :)

[CarlosRDomin]

Could you include the log/output of the program when you press "y"? Otherwise it's hard to know why the program exits in that case (it shouldn't).

On calibrate_2d_to_3d.py the ball should be highlighted, yes (with a circle outline, not a filled one, but still).

For full_control_with_cam.py a couple things could be the cause:

• First, did you flash my custom firmware (https://github.com/CarlosRDomin/crazyflie-firmware) on the CrazyFlie? Otherwise the drone will be misinterpreting the radio commands.
• If you've already done that, then lower the TAKEOFF_THRUST (line 185 on full_control_with_cam.py), try values from 35000 to 45000, maybe increasing 1000 at a time, until you see it goes up slowly, then leave that value.
• Last thing, look at the output images saved in the img/ folder and see if the drone can be found, etc.

[jorelllinsangan]

After pressing y in calibrate_2d_to_3d.py this is what I get.

DEBUG:uvc:Stream stopped
DEBUG:uvc:Stream closed
DEBUG:uvc:Stream stop.
DEBUG:uvc:UVC device closed.

Yeah, I'm not sure why the ball isn't highlighted in calibrate_2d_to_3d.py.

I did flash the firmware. I was getting something like "value posSet not found". After flashing it, the problem went away.

I changed the thirst just now and that fixed its shooting up problem.

It does look like its seeing the marker in the images in the img/ folder.

That's after me pressing e its weird that the drone didn't fly up.

Here's an image after I kept pressing u

[jorelllinsangan]

Here's a gif of what happens after I press 'e'.

Thrust is set to 35000.

https://imgflip.com/gif/1m7ac2

I just took the images from the img/ folder and made a gif out of it.

[CarlosRDomin]

Hmmm that's weird. It seems like it could be a misalignment of the coordinate axes.

• First of all, try to figure out why the camera stream is closed after pressing "y" in calibrate_2d_to_3d.py. Could you copy the whole output? As in, the messages before DEBUG:uvc:Stream stopped.
• Another thing I'd try is: keep the camera as horizontal as you can, and bypass ("b") the extrinsic initialization of the Spotter. Make sure the Crazyflie is pointing towards the camera (with the nose/forward closer to the camera than the back (otherwise left-right and backward-forward will be reversed).
• Also, the Crazyflie doesn't have a magnetometer, so yaw drifts over time (if you leave it steady and log the yaw, you'll see it keeps increasing/decreasing). The way around this is: turn it off, and back on every time before flying, so yaw is reset at 0º.
• I don't remember 100%, but I believe my code expects the chessboard to be aligned with the Crazyflie's frame of reference, meaning, you should place it flat on the ground, with the shorter side of the chessboard pointing in the same direction as the nose of the CF:

• Last, make sure you're loading the right calibration file for your camera (how many images did you collect to calibrate it? For best results you should get 25-30 in different positions and orientations on the image). Could you share your camera extrinsics in the initialization of the Spotter in calibrate_2d_to_3d.py after doing that?

[jorelllinsangan]

DEBUG:uvc:Found device that mached uid:'20:17'
DEBUG:uvc:Device '20:17' opended.
DEBUG:uvc:avaible video modes: [{'rates': [30], 'size': (1920, 1080)}, {'rates': [60], 'size': (1280, 720)}, {'rates': [30], 'size': (1024, 768)}, {'rates': [120], 'size': (640, 480)}, {'rates': [60], 'size': (800, 600)}, {'rates': [30], 'size': (1280, 1024)}, {'rates': [120], 'size': (320, 240)}]
DEBUG:uvc:Adding "Auto Exposure Mode" control.
DEBUG:uvc:Adding "Auto Exposure Priority" control.
DEBUG:uvc:Adding "Absolute Exposure Time" control.
DEBUG:uvc:Adding "Backlight Compensation" control.
DEBUG:uvc:Adding "Brightness" control.
DEBUG:uvc:Adding "Contrast" control.
DEBUG:uvc:Adding "Gain" control.
DEBUG:uvc:Adding "Power Line frequency" control.
DEBUG:uvc:Adding "Hue" control.
DEBUG:uvc:Adding "Saturation" control.
DEBUG:uvc:Adding "Sharpness" control.
DEBUG:uvc:Adding "Gamma" control.
DEBUG:uvc:Adding "White Balance temperature" control.
DEBUG:uvc:Adding "White Balance temperature,Auto" control.
File name config/cam_settings/Camera settings - USB 2.0 Camera - LivingRoom2.txt
DEBUG:root:Loading UVC camera settings from file 'config/cam_settings/Camera settings - USB 2.0 Camera - LivingRoom2.txt'
DEBUG:uvc:Setting mode: 1280,720,60
DEBUG:root:	Loaded camera settings: frame_size = 1280x720, frame_rate = 60 fps
DEBUG:root:	Successfully initialized undistortion maps from '/Users/School/SchoolStuff/Industrial/SensorFlySwarm/MyCode/cam_calibration/cam_calibration_output.npz'.
DEBUG:root:	Loaded control setting: 'Auto Exposure Mode' = 1
DEBUG:root:	Loaded control setting: 'Auto Exposure Priority' = 1
DEBUG:root:	Loaded control setting: 'Absolute Exposure Time' = 58
DEBUG:root:	Loaded control setting: 'Backlight Compensation' = 0
DEBUG:root:	Loaded control setting: 'Brightness' = 45
DEBUG:root:	Loaded control setting: 'Contrast' = 36
DEBUG:root:	Loaded control setting: 'Gain' = 0
DEBUG:root:	Loaded control setting: 'Power Line frequency' = 1
DEBUG:root:	Loaded control setting: 'Hue' = 0
DEBUG:root:	Loaded control setting: 'Saturation' = 60
DEBUG:root:	Loaded control setting: 'Sharpness' = 0
DEBUG:root:	Loaded control setting: 'Gamma' = 125
DEBUG:root:	Loaded control setting: 'White Balance temperature' = 3836
DEBUG:root:	Loaded control setting: 'White Balance temperature,Auto' = 1
INFO:root:Camera opened! :)
Color Thresh settings
DEBUG:root:Loading color threshold settings from file 'config/color_thresh/Color threshold settings - LivingRoom2.txt'
DEBUG:root:	Loaded color threshold settings: HSV_min=[121  78 100]; HSV_max=[179 255 255]
DEBUG:root:Loading blob detector settings from file 'config/blob_detector/Blob detector settings - LivingRoom2.txt'
DEBUG:root:	Loaded blob detector setting: 'blobColor' = 255
DEBUG:root:	Loaded blob detector setting: 'filterByArea' = True
DEBUG:root:	Loaded blob detector setting: 'filterByCircularity' = True
DEBUG:root:	Loaded blob detector setting: 'filterByColor' = True
DEBUG:root:	Loaded blob detector setting: 'filterByConvexity' = False
DEBUG:root:	Loaded blob detector setting: 'filterByInertia' = False
DEBUG:root:	Loaded blob detector setting: 'maxArea' = 40000.0
DEBUG:root:	Loaded blob detector setting: 'maxCircularity' = 1.0
DEBUG:root:	Loaded blob detector setting: 'maxConvexity' = 1.0
DEBUG:root:	Loaded blob detector setting: 'maxInertiaRatio' = 1.0
DEBUG:root:	Loaded blob detector setting: 'maxThreshold' = 255.0
DEBUG:root:	Loaded blob detector setting: 'minArea' = 15.0
DEBUG:root:	Loaded blob detector setting: 'minCircularity' = 0.330000013113
DEBUG:root:	Loaded blob detector setting: 'minConvexity' = 0.699999988079
DEBUG:root:	Loaded blob detector setting: 'minDistBetweenBlobs' = 3000.0
DEBUG:root:	Loaded blob detector setting: 'minInertiaRatio' = 0.5
DEBUG:root:	Loaded blob detector setting: 'minRepeatability' = 1
DEBUG:root:	Loaded blob detector setting: 'minThreshold' = 254.0
DEBUG:root:	Loaded blob detector setting: 'thresholdStep' = 10.0
Estimated / selected altsetting bandwith : Iu / Iu.
DEBUG:uvc:Stream start.
DEBUG:uvc:Could not get Frame of first try: "Operation timed out.". Attempt:1/3
Camera is at position [x=12.39cm, y=-104.00cm, z=-326.78cm]; F=1250.00px (length of 1m segment, 1m away from the camera)
DEBUG:uvc:Stream stopped
DEBUG:uvc:Stream closed
DEBUG:uvc:Stream stop.
DEBUG:uvc:UVC device closed.

Would you be able to post what I should be seeing after I press 'y' in calibrate_2d_to_3d.py?

I'll try your other suggestions.

[CarlosRDomin]

Try changing line 62 in calibrate_2d_to_3d.py from

while cv2.waitKey(1) == 255:

to

while cv2.waitKey(1) < 0:

Hopefully that'll do :)

[jorelllinsangan]

The change in calibrate_2d_to_3d.py helped me fix the problem where it just exits.

I'll be trying your frame of reference suggestion next. The drone still overcompensates as soon as I press e.

[jorelllinsangan]

Bypassing the extrinsic by pressing b allowed the Crazyflie to hover. This is super cool.

[CarlosRDomin]

Yaaay, glad it worked!
I'm a little busy now, but I remember that in the file full_control_with_cam.py there were two functions to deal with the frame of references:

• img_to_cf_world_coords (line 888) converts world coordinates to crazyflie coordinates. If you want to have your chessboard vertical (you'll have to make sure it's at 90º for best results, or else the PID loop for height control will be affected by sin(vertical angle)), then you would want to change line 888 to something like (untested):

return np.array([-1, 1, -1]) * world_coords[[2,0,1]]  # This means: CF_x=-world_z, CF_y=world_x, CF_z=-world_y
# Note that this also assumes the chessboard's short side (world_x) goes in the left-right direction, and the long side (world_y) goes up-down.

• cf_world_to_img_coords (line 900) -> Using the same logic, we just have to express world_x, world_y and world_z in terms of CF_x, CF_y, CF_z. So:

world_coords = np.array([1, -1, -1]) * cf_world_coords[[1,2,0]]  # This means: world_x=CF_y, world_y=-CF_z, world_z=-CF_x

[jorelllinsangan]

Hey Carlos!

Any idea why the drone is doing this?

Thanks!

[jorelllinsangan]

Also, I noticed that in the videos you've posted, the drone seems like to know to just hover at the point where you press "e". In my experience where the drone actually hovered, it took a while to find the right values so it can hover. I hope this makes sense.