Support 3D pose estimation

[tucan9389]

lightweight-human-pose-estimation-3d-demo.pytorch

repo: https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch

Models

• pytorch model: human-pose-estimation-3d.pth.zip
• onnx model: human-pose-estimation-3d.onnx.zip
• mlmodel: human-pose-estimation-3d.mlmodel.zip

Metadata

• input: a 256x448 color image
• outputs:
  • features:
  • heatmaps:
  • pafs:

Parsing Reference

• https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch/blob/master/modules/parse_poses.py
• https://github.com/Daniil-Osokin/lightweight-human-pose-estimation-3d-demo.pytorch/blob/master/modules/legacy_pose_extractor.py

[sebo361]

Hi @tucan9389 how about using ARKit's 3D motion tracking: https://developer.apple.com/documentation/arkit/capturing_body_motion_in_3d ? Or do you think your proposed 3D pose estimation is more accurate?

[tucan9389]

@sebo361
Thanks for your suggestion. I agree with that If the case prefers more.
But I think there is some other side of benefits when using Core ML rather than AR Kit.

1. When you want to inference person's 3D keypoints with a single image
2. When you want to inference not only a person's but also an object's 3D keypoints which was trained by yourself

[sebo361]

@tucan9389
True, these are interesting benefits to find out! However I am more interested on comparing both approaches regarding the inference speed and accuracy of 3D human pose keypoints. Now as the latest iPad Pro (and iPhone 12) have the LIDAR sensor, i expect to have more precision in 3D human motion tracking, but I am note sure if ARKit's 3D motion tracking uses LIDAR Depth API by default - do you have any insights of that?

Maybe I should setup a project to compare ARKit's 3D motion tracking with your proposed CoreML 3D pose estimation.

[jookovjook]

@sebo361
Also, Apple's 3D body tracking ARBodyTrackingConfiguration can't work simultaneously with the world tracking ARWorldTrackingConfiguration. So, unfortunately, you can't use Apple's 3D body tracking if you also need world tracking

[tucan9389]

https://github.com/tucan9389/PoseEstimation-TFLiteSwift

Here is 3D pose estimation demo by using TFLiteSwift. I implemented softargmax with pure Swift and Accelerate framework. In Numpy or Pytorch(or TF), there support dimension summation to use easily, but in swift there is no softargmax function in multi-dimension matrix(tensor). So the implementation of softargmax is a little bit hard.