Merge pull request #1066 from luxonis/refactor_xout

Refactor xout

depthai_sdk/docs/source/features/ai_models.rst

@@ -53,41 +53,53 @@ The following table lists all the models supported by the SDK. The model name is
    * - ``face-detection-retail-0004``
      - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_face_detection_retail_0004.html>`__
      - 33
+   * - ``facemesh_192x192``
+     - `DMZ <https://github.com/luxonis/depthai-model-zoo/tree/main/models/facemesh_192x192>`__
+     - 32
+   * - ``facial_landmarks_68_160x160``
+     - 32
+     - `DMZ <https://github.com/luxonis/depthai-model-zoo/blob/main/models/facial_landmarks_68_160x160>`__
+   * - ``human-pose-estimation-0001``
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_human_pose_estimation_0001.html>`__
+     - 8
    * - ``mobilenet-ssd``
      - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_mobilenet_ssd.html>`__
      - 31
    * - ``mobilenetv2_imagenet_embedder_224x224``
      - `DMZ <https://github.com/luxonis/depthai-model-zoo/tree/main/models/mobilenetv2_imagenet_embedder_224x224>`__
      - /
    * - ``pedestrian-detection-adas-0002``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_pedestrian_detection_adas_0002.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_pedestrian_detection_adas_0002.html>`__
      - 19
    * - ``person-detection-0200``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_person_detection_0200.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_person_detection_0200.html>`__
      - 14
    * - ``person-detection-retail-0013``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_person_detection_retail_0013.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_person_detection_retail_0013.html>`__
      - 15
    * - ``person-reidentification-retail-0288``
-     - `OMZ <https://docs.openvino.ai/cn/latest/omz_models_model_person_reidentification_retail_0288.html>`__
+     - `OMZ <https://docs.openvino.ai/cn/2021.4/omz_models_model_person_reidentification_retail_0288.html>`__
      - 33
    * - ``person-vehicle-bike-detection-crossroad-1016``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_person_vehicle_bike_detection_crossroad_1016.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_person_vehicle_bike_detection_crossroad_1016.html>`__
      - 12
+   * - ``sbd_mask_classification_224x224``
+     - `DMZ <https://github.com/luxonis/depthai-model-zoo/tree/main/models/sbd_mask_classification_224x224>`__
+     - 64+
    * - ``vehicle-detection-0202``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_vehicle_detection_0202.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_vehicle_detection_0202.html>`__
      - 14
    * - ``vehicle-detection-adas-0002``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_vehicle_detection_adas_0002.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_vehicle_detection_adas_0002.html>`__
      - 14
    * - ``vehicle-license-plate-detection-barrier-0106``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_vehicle_license_plate_detection_barrier_0106.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_vehicle_license_plate_detection_barrier_0106.html>`__
      - 29
    * - ``yolo-v3-tf``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_yolo_v3_tf.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_yolo_v3_tf.html>`__
      - 3.5
    * - ``yolo-v3-tiny-tf``
-     - `OMZ <https://docs.openvino.ai/latest/omz_models_model_yolo_v3_tiny_tf.html>`__
+     - `OMZ <https://docs.openvino.ai/2022.1/omz_models_model_yolo_v3_tiny_tf.html>`__
      - 33
    * - ``yolov4_coco_608x608``
      - `DMZ <https://github.com/luxonis/depthai-model-zoo/tree/main/models/yolov4_coco_608x608>`__

depthai_sdk/docs/source/fundamentals/visualizer.rst

@@ -200,10 +200,6 @@ Visualizer
     :members:
     :undoc-members:
 
-.. autoclass:: depthai_sdk.visualize.visualizer.Platform
-    :members:
-    :undoc-members:
-
 Objects
 -------
 

depthai_sdk/docs/source/oak-camera.rst

@@ -17,8 +17,8 @@ Interoperability with DepthAI API
 ---------------------------------
 
 DepthAI SDK was developed with `DepthAI API <https://docs.luxonis.com/projects/api/en/latest/>`__ interoperability in mind.
-Users can access all depthai API nodes inside components, and after ``oak.build()`` also the `dai.Pipeline <https://docs.luxonis.com/projects/api/en/latest/components/pipeline/>`__
-and `dai.Device <https://docs.luxonis.com/projects/api/en/latest/components/device/>`__ objects.
+Users can access all depthai API nodes inside components, along with the `dai.Pipeline <https://docs.luxonis.com/projects/api/en/latest/components/pipeline/>`__ (``oak.pipeline``)
+and `dai.Device <https://docs.luxonis.com/projects/api/en/latest/components/device/>`__ (``oak.device``) objects.
 
 .. literalinclude:: ../../examples/mixed/api_interop.py
    :language: python

depthai_sdk/docs/source/visualizer_formats/example.json

@@ -1,5 +1,4 @@
 {
-  "platform": "pc",
   "frame_shape": [720, 1280],
   "config": {
     "output": {

depthai_sdk/docs/source/visualizer_formats/format.json

@@ -1,11 +1,4 @@
 {
-  "platform": {
-    "type": "string",
-    "enum": [
-      "pc",
-      "robothub"
-    ]
-  },
   "frame_shape": {
     "type": "array",
     "items": {

depthai_sdk/examples/CameraComponent/camera_encode.py

@@ -0,0 +1,9 @@
+from depthai_sdk import OakCamera
+
+with OakCamera() as oak:
+    color = oak.create_camera('color', encode='h265')
+
+    oak.visualize(color.out.encoded, fps=True, scale=2/3)
+    # By default, it will stream non-encoded frames
+    oak.visualize(color, fps=True, scale=2/3)
+    oak.start(blocking=True)

depthai_sdk/examples/CameraComponent/preview_all_cameras.py

@@ -2,5 +2,5 @@
 
 with OakCamera() as oak:
     cams = oak.create_all_cameras(resolution='max')
-    oak.visualize(cams)
+    oak.visualize(cams, fps=True)
     oak.start(blocking=True)

depthai_sdk/examples/CameraComponent/rotated.py

@@ -1,8 +1,6 @@
 from depthai_sdk import OakCamera
 
 with OakCamera(rotation=90) as oak:
-    color = oak.create_camera('color', resolution='1080p')
-    left = oak.create_camera('left', resolution='400p')
-    right = oak.create_camera('right', resolution='400p')
-    oak.visualize([color, left, right], fps=True)
+    all_cams = oak.create_all_cameras()
+    oak.visualize(all_cams, fps=True)
     oak.start(blocking=True)

depthai_sdk/examples/IMUComponent/imu.py

@@ -3,5 +3,7 @@
 with OakCamera() as oak:
     imu = oak.create_imu()
     imu.config_imu(report_rate=400, batch_report_threshold=5)
-    oak.visualize(imu.out.main)
+    # DepthAI viewer should open, and IMU data can be viewed on the right-side panel,
+    # under "Stats" tab (right of the "Device Settings" tab).
+    oak.visualize(imu.out.main, visualizer='viewer')
     oak.start(blocking=True)

depthai_sdk/examples/NNComponent/age-gender.py

@@ -14,7 +14,7 @@ def callback(packet: TwoStagePacket):
         gender_str = "Woman" if gender[0] > gender[1] else "Man"
 
         visualizer.add_text(f'{gender_str}\nAge: {age}',
-                            bbox=(*det.top_left, *det.bottom_right),
+                            bbox=packet.bbox.get_relative_bbox(det.bbox),
                             position=TextPosition.BOTTOM_RIGHT)
 
     frame = visualizer.draw(packet.frame)

depthai_sdk/examples/NNComponent/custom_decode.py

@@ -1,13 +1,11 @@
 import blobconverter
-import cv2
 import numpy as np
-from depthai import NNData
-
+import depthai as dai
 from depthai_sdk import OakCamera
-from depthai_sdk.classes import Detections, DetectionPacket
+from depthai_sdk.classes import Detections
 
 
-def decode(nn_data: NNData) -> Detections:
+def decode(nn_data: dai.NNData) -> Detections:
     """
     Custom decode function for the NN component. Decode function has to accept NNData argument.
     The return type should preferably be a class that inherits from depthai_sdk.classes.GenericNNOutput,
@@ -18,29 +16,27 @@ def decode(nn_data: NNData) -> Detections:
     layer = nn_data.getFirstLayerFp16()
     results = np.array(layer).reshape((1, 1, -1, 7))
     dets = Detections(nn_data)
-
     for result in results[0][0]:
-        if result[2] > 0.5:
+        if result[2] > 0.3:
             label = int(result[1])
             conf = result[2]
             bbox = result[3:]
-            dets.add(label, conf, bbox)
+            det = dai.ImgDetection()
+            det.confidence = conf
+            det.label = label
+            det.xmin = bbox[0]
+            det.ymin = bbox[1]
+            det.xmax = bbox[2]
+            det.ymax = bbox[3]
+            dets.detections.append(det)
 
     return dets
 
-
-def callback(packet: DetectionPacket):
-    visualizer = packet.visualizer
-    frame = packet.frame
-    frame = visualizer.draw(frame)
-    cv2.imshow('Custom decode function', frame)
-
-
 with OakCamera() as oak:
     color = oak.create_camera('color')
 
-    nn_path = blobconverter.from_zoo(name='person-detection-0200', version='2021.4')
+    nn_path = blobconverter.from_zoo(name='person-detection-0200', version='2021.4', shaves=6)
     nn = oak.create_nn(nn_path, color, decode_fn=decode)
 
-    oak.visualize(nn, callback=callback)
+    oak.visualize(nn)
     oak.start(blocking=True)

depthai_sdk/examples/NNComponent/emotion-recognition.py

@@ -16,7 +16,7 @@ def callback(packet: TwoStagePacket):
         emotion_name = emotions[np.argmax(emotion_results)]
 
         visualizer.add_text(emotion_name,
-                            bbox=(*det.top_left, *det.bottom_right),
+                            bbox=packet.bbox.get_relative_bbox(det.bbox),
                             position=TextPosition.BOTTOM_RIGHT)
 
     visualizer.draw(packet.frame)

depthai_sdk/examples/NNComponent/nn_component.py

@@ -4,7 +4,8 @@
     color = oak.create_camera('color')
     # List of models that are supported out-of-the-box by the SDK:
     # https://docs.luxonis.com/projects/sdk/en/latest/features/ai_models/#sdk-supported-models
-    nn = oak.create_nn('yolov7tiny_coco_640x352', color)
+    nn = oak.create_nn('yolov5n_coco_416x416', color)
+    nn.config_nn(resize_mode='stretch')
     oak.visualize([nn.out.main], fps=True)
     oak.visualize(nn.out.passthrough)
     oak.start(blocking=True)

depthai_sdk/examples/NNComponent/spatial_detection.py

@@ -15,6 +15,6 @@
         calc_algo=dai.SpatialLocationCalculatorAlgorithm.AVERAGE
     )
 
-    oak.visualize([nn.out.main], fps=True)
-    oak.visualize(nn.out.passthrough)
+    oak.visualize(nn.out.main, fps=True)
+    oak.visualize([nn.out.passthrough, nn.out.spatials])
     oak.start(blocking=True)

depthai_sdk/examples/PointcloudComponent/pointcloud.py

@@ -1,23 +1,9 @@
-import cv2
 from depthai_sdk import OakCamera
-from depthai_sdk.classes.packets import PointcloudPacket, FramePacket
-import rerun as rr
-import subprocess
-import time
-
-subprocess.Popen(["rerun", "--memory-limit", "200MB"])
-time.sleep(1)  # Wait til rerun spins up
-rr.init("Rerun ", spawn=False)
-rr.connect()
-
-def callback(packet: PointcloudPacket):
-    colors = packet.color_frame.getCvFrame()[..., ::-1] # BGR to RGB
-    rr.log_image('Color Image', colors)
-    points = packet.points.reshape(-1, 3)
-    rr.log_points("Pointcloud", points, colors=colors.reshape(-1, 3))
-
 
 with OakCamera() as oak:
-    pcl = oak.create_pointcloud()
-    oak.callback(pcl, callback=callback)
+    color = oak.camera('color')
+    stereo = oak.create_stereo()
+    stereo.config_stereo(align=color)
+    pcl = oak.create_pointcloud(stereo=stereo, colorize=color)
+    oak.visualize(pcl, visualizer='depthai-viewer')
     oak.start(blocking=True)

depthai_sdk/examples/mixed/api_interop.py

@@ -6,15 +6,12 @@
     nn = oak.create_nn('mobilenet-ssd', color)
     oak.visualize([nn.out.passthrough, nn], fps=True)
 
-    # Build the pipeline, connect to the oak, update components. Place interop logic AFTER oak.build()
-    pipeline = oak.build()
-
     nn.node.setNumInferenceThreads(2) # Configure components' nodes
 
-    features = pipeline.create(dai.node.FeatureTracker) # Create new pipeline nodes
+    features = oak.pipeline.create(dai.node.FeatureTracker) # Create new pipeline nodes
     color.node.video.link(features.inputImage)
 
-    out = pipeline.create(dai.node.XLinkOut)
+    out = oak.pipeline.create(dai.node.XLinkOut)
     out.setStreamName('features')
     features.outputFeatures.link(out.input)
 

depthai_sdk/examples/mixed/collision_avoidance.py

@@ -1,6 +1,6 @@
 from depthai_sdk import OakCamera
 from depthai_sdk.visualize.configs import StereoColor
-from depthai_sdk.classes.packets import DepthPacket
+from depthai_sdk.classes.packets import DisparityDepthPacket
 import math
 import depthai as dai
 import cv2
@@ -11,7 +11,7 @@
 
 slc_data = []
 
-def cb(packet: DepthPacket):
+def cb(packet: DisparityDepthPacket):
     global slc_data
     fontType = cv2.FONT_HERSHEY_TRIPLEX
 
@@ -56,8 +56,6 @@ def cb(packet: DepthPacket):
 
     oak.visualize([stereo], fps=True, callback=cb)
 
-    oak.build()
-
     slc = oak.pipeline.create(dai.node.SpatialLocationCalculator)
     for x in range(15):
         for y in range(9):

depthai_sdk/examples/mixed/packet_callback.py

@@ -0,0 +1,45 @@
+from depthai_sdk import OakCamera
+from depthai_sdk.classes.packets import FramePacket
+from datetime import timedelta
+from typing import Dict
+import cv2
+
+def cb_1(packet: FramePacket):
+    # Called from main thread, so we can call cv2.imshow
+    cv2.imshow('Color frames from cb', packet.frame)
+
+def cb_2(packets: Dict[str, FramePacket]):
+    print(packets)
+    # Sycned packets.
+    ts_color = packets['color'].get_timestamp()
+    ts_left = packets['left'].get_timestamp()
+    ts_imu = packets['imu'].get_timestamp()
+    print(f"---- New synced packets. Diff between color and left: {abs(ts_color-ts_left) / timedelta(milliseconds=1)} ms, color and IMU: {abs(ts_imu-ts_color) / timedelta(milliseconds=1)} ms")
+
+    for name, packet in packets.items():
+        print(f'Packet {name}, timestamp: {packet.get_timestamp()}, Seq number: {packet.get_sequence_num()}')
+
+with OakCamera() as oak:
+    color = oak.create_camera('color', fps=32, name='color')
+    left = oak.create_camera('left', fps=30, name='left')
+    right = oak.create_camera('right', fps=30, name='right')
+    imu = oak.create_imu()
+
+    oak.callback(
+        color, # Outputs whose packets we want to receive via callback
+        callback=cb_1, # Callback function
+        main_thread=True # Whether to call the callback in the main thread. For OpenCV's imshow to work, it must be called in the main thread.
+    )
+
+    cb_handler = oak.callback(
+        [left, right, color, imu],
+        callback=cb_2,
+        main_thread=False # Will be called from a different thread, instead of putting packets into queue and waiting for main thread to pick it up.
+    )
+    # Timestamp syncing all 3 streams. We selected (1000/30) / 2 as threshold_ms, because
+    # left/right are slower (30FPS), so threshold should be about 16ms. This means SDK will discard some
+    # color packets (2 per second), but we will have synced frames.
+    cb_handler.configure_syncing(threshold_ms=int((1000/30) / 2))
+
+    # oak.show_graph()
+    oak.start(blocking=True)