Getting frames from multiple Kinects within a single script

[negvet]

Hey!
I have connected two kinects and both of them are working just fine if I start each of them with a separate Protonect or multiframe_listener (each of them is connected to a separate USB controller and libfreenect2/issues/807 is implemented)

But I am trying to get frames from both kinects within a single script.
Here is the code:

from scipy import ndimage
import numpy as np
import cv2
import sys
from pylibfreenect2 import Freenect2, SyncMultiFrameListener
from pylibfreenect2 import FrameType, Registration, Frame
from pylibfreenect2 import createConsoleLogger, setGlobalLogger
from pylibfreenect2 import LoggerLevel
import os
from pylibfreenect2 import OpenGLPacketPipeline
pipeline = OpenGLPacketPipeline()
print("Packet pipeline:", type(pipeline).__name__)

enable_rgb = True
enable_depth = True

fn = Freenect2()

device = fn.openDevice("025159651347", pipeline=pipeline)
device2 = fn.openDevice("004404264147", pipeline=pipeline)

types = 0
if enable_rgb:
    types |= FrameType.Color
if enable_depth:
    types |= (FrameType.Ir | FrameType.Depth)
listener = SyncMultiFrameListener(types)
listener2 = SyncMultiFrameListener(types)

# Register listeners
device.setColorFrameListener(listener)
device.setIrAndDepthFrameListener(listener)

# Here problems starts...
device2.setColorFrameListener(listener2)
device2.setIrAndDepthFrameListener(listener2)

I register listeners for the second device, but it does not work after that. It goes further to a while loop but it does not iterate.

Here is the remaining part of my code:

device.startStreams(rgb=enable_rgb, depth=enable_depth)
# device2.startStreams(rgb=enable_rgb, depth=enable_depth)

if enable_depth:
    registration = Registration(device.getIrCameraParams(),
                                device.getColorCameraParams())
    # registration2 = Registration(device2.getIrCameraParams(),
    #                             device2.getColorCameraParams())

undistorted = Frame(512, 424, 4)
registered = Frame(512, 424, 4)
# undistorted2 = Frame(512, 424, 4)
# registered2 = Frame(512, 424, 4)

while True:
    frames = listener.waitForNewFrame()
    # frames2 = listener2.waitForNewFrame()

    if enable_rgb:
        color = frames["color"]
        # color2 = frames2["color"]
    if enable_depth:
        ir = frames["ir"]
        depth = frames["depth"]
        # ir2 = frames2["ir"]
        # depth2 = frames2["depth"]

    if enable_rgb and enable_depth:
        registration.apply(color, depth, undistorted, registered)
        # registration2.apply(color2, depth2, undistorted2, registered2)
    elif enable_depth:
        registration.undistortDepth(depth, undistorted)
        # registration2.undistortDepth(depth2, undistorted2)

    depth_img = depth.asarray() / 4500.
    # depth_img = ndimage.rotate(depth_img, -90)
    cv2.imshow("depth", depth_img)

    listener.release(frames)
    # listener2.release(frames2)

    key = cv2.waitKey(delay=1)
    if key == ord('q'):
        break

device.stop()
device.close()
# device2.stop()
# device2.close()

sys.exit(0)

Is it correctly to do like this by the way:

while(1):
    frames = listener.waitForNewFrame()
    frames2 = listener2.waitForNewFrame()
    # frames and frames2 processing
    listener.release(frames)
    listener2.release(frames2)

Is there a way to sync kinects (not exactly, but at least somehow)?

I am using Ubuntu 16.04
Thanks a lot!

[r9y9]

I haven't tried to run two kinects in a single process and I'm not sure if it's possible, but I think you should create packet pipeline for each device.

pipeline1 = OpenGLPacketPipeline()
pipeline2 = OpenGLPacketPipeline()
.
..

device = fn.openDevice("025159651347", pipeline=pipeline1)
device2 = fn.openDevice("004404264147", pipeline=pipeline2)

[negvet]

Yes, you are right. All of the code that I posted now works.
Thanks a lot!!

[r9y9]

I'm glad I could help:)