Add support for rendering attributes of detections in expanded sample view

[brimoor]

Object detections can contain custom attributes in two ways:

• as dynamically added fields (like age, happiness, breed, cute below)
• in an attributes field, which contains a dict mapping attribute names to fiftyone.core.label.Attribute instances, which can be CategoricalAttribute, NumericAttribute, or BooleanAttribute instances

If the user provides either of these types of data on their detections, they should be able to see these fields rendered in Player51, using its attribute panel feature:

For example, the data below should be rendered as:

age: 5
happiness: 99.99
breed: tabby
cute: true
other_age: 5
other_happiness: 99.99
other_breed: tabby
other_cute: true

Note that detections could have other fields with unsupported types like dict (other than attributes, which specifically is supported because its contents are known to be Attribute instances), list, numeric arrays, etc. Such fields should be ignored for visualization purposes.

We should include a reasonable overflow behavior, such as replace attributes 5+ with ...

Also, since these attributes can take up a lot of space, we should expose an option in the Player51 controls to toggle whether the attributes are displayed or not. We don't need the extra options related to attr: value vs list, of, attrs formatting. Either they render in attr: value syntax or not at all

import fiftyone as fo

sample = fo.Sample(
    filepath="test.png",
    ground_truth=fo.Detections(
        detections=[
            fo.Detection(
                label="cat",
                confidence=0.9,
                bounding_box=[0.33, 0.33, 0.2, 0.2],
                age=5,
                happiness=99.99,
                breed="tabby",
                cute=True,
                attributes={
                    "other_age": fo.NumericAttribute(value=5),
                    "other_happiness": fo.NumericAttribute(value=99.99),
                    "other_breed": fo.CategoricalAttribute(value="tabby"),
                    "other_cute": fo.BooleanAttribute(value=True),
                }
            )
        ]
    )
)

print(sample)

<Sample: {
    'id': None,
    'filepath': '/Users/Brian/dev/fiftyone/test.png',
    'tags': [],
    'metadata': None,
    'ground_truth': <Detections: {
        'detections': BaseList([
            <Detection: {
                'id': '5f4588a4fa79d142ebc3b6cf',
                'label': 'cat',
                'bounding_box': BaseList([0.33, 0.33, 0.2, 0.2]),
                'confidence': 0.9,
                'attributes': BaseDict({
                    'other_age': <NumericAttribute: {'value': 5}>,
                    'other_happiness': <NumericAttribute: {'value': 99.99}>,
                    'other_breed': <CategoricalAttribute: {'value': 'tabby', 'confidence': None, 'logits': None}>,
                    'other_cute': <BooleanAttribute: {'value': True}>,
                }),
                'age': 5,
                'happiness': 99.99,
                'breed': 'tabby',
                'cute': True,
            }>,
        ]),
    }>,
}>

[jguo1002]

Is there any way to add these attributes to sample fields so that ground_truth have subfields besides the default ones (attributes, id, label, tags, bounding_box, mask, confidence, index...)

Like this in doc sample, age and cute are included:

{
 ...
"sample_fields": [
    {
      "_cls": "SampleFieldDocument",
      "name": "ground_truth",
      "ftype": "fiftyone.core.fields.EmbeddedDocumentField",
      "subfield": null,
      "embedded_doc_type": "fiftyone.core.labels.Detections",
      "db_field": "ground_truth",
      "fields": [
        {
          "_cls": "SampleFieldDocument",
          "name": "detections",
          "ftype": "fiftyone.core.fields.ListField",
          "subfield": "fiftyone.core.fields.EmbeddedDocumentField",
          "embedded_doc_type": "fiftyone.core.labels.Detection",
          "db_field": "detections",
          "fields": [
            {
              "_cls": "SampleFieldDocument",
              "name": "attributes",
              "ftype": "fiftyone.core.fields.DictField",
              "subfield": "fiftyone.core.fields.EmbeddedDocumentField",
              "embedded_doc_type": "fiftyone.core.labels.Attribute",
              "db_field": "attributes",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "id",
              "ftype": "fiftyone.core.fields.ObjectIdField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "_id",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "tags",
              "ftype": "fiftyone.core.fields.ListField",
              "subfield": "fiftyone.core.fields.StringField",
              "embedded_doc_type": null,
              "db_field": "tags",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "label",
              "ftype": "fiftyone.core.fields.StringField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "label",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "bounding_box",
              "ftype": "fiftyone.core.fields.ListField",
              "subfield": "fiftyone.core.fields.FloatField",
              "embedded_doc_type": null,
              "db_field": "bounding_box",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "mask",
              "ftype": "fiftyone.core.fields.ArrayField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "mask",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "confidence",
              "ftype": "fiftyone.core.fields.FloatField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "confidence",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "index",
              "ftype": "fiftyone.core.fields.IntField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "index",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "age",
              "ftype": "fiftyone.core.fields.IntField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "age",
              "fields": []
            },
            {
              "_cls": "SampleFieldDocument",
              "name": "cute",
              "ftype": "fiftyone.core.fields.BooleanField",
              "subfield": null,
              "embedded_doc_type": null,
              "db_field": "cute",
              "fields": []
            },
          ]
        }
      ]
    },
  ],
...
"

[ehofesmann]

@jguo1002 Yes, this is best done through the Python API. When creating a label, you can add custom attributes to it like so:

# Provide some default fields
label = fo.Classification(label="cat", confidence=0.98)

# Add custom label subfields
label["int"] = 5
label["float"] = 51.0
label["list"] = [1, 2, 3]
label["bool"] = True
label["dict"] = {"key": ["list", "of", "values"]}

[jguo1002]

@ehofesmann Thanks for the follow-up!

Two questions. First, these are examples with fo.Classification.

sample["prediction"] = fo.Classification(value=42.0, confidence=0.9)

I added the label attributes but the custom fields are not added to the sample fields, the subfields of test_ground_truth are the default ones (e.g., id, tags, logits...).

Sample:

<SampleView: {                                                               
    'id': ...,
    'media_type': 'image',
    'filepath': ...,
    'tags': BaseList([]),
    'metadata': ...,
    'test_ground_truth': <Classification: {
        'id': '62fbe3f93542d15a3e16d7e2',
        'tags': BaseList([]),
        'label': 'ColorOff',
        'confidence': None,
        'logits': None,
        'cute': True,
        'age': 9.0,
        'happiness': 99.99,
        ...
    }>,
}>

Sample field:

    {
      "_cls": "SampleFieldDocument",
      "name": "test_ground_truth",
      "ftype": "fiftyone.core.fields.EmbeddedDocumentField",
      "subfield": null,
      "embedded_doc_type": "fiftyone.core.labels.Classification",
      "db_field": "test_ground_truth",
      "fields": [
        {
          "_cls": "SampleFieldDocument",
          "name": "id",
          "ftype": "fiftyone.core.fields.ObjectIdField",
          "subfield": null,
          "embedded_doc_type": null,
          "db_field": "_id",
          "fields": []
        },
        {
          "_cls": "SampleFieldDocument",
          "name": "tags",
          "ftype": "fiftyone.core.fields.ListField",
          "subfield": "fiftyone.core.fields.StringField",
          "embedded_doc_type": null,
          "db_field": "tags",
          "fields": []
        },
        {
          "_cls": "SampleFieldDocument",
          "name": "label",
          "ftype": "fiftyone.core.fields.StringField",
          "subfield": null,
          "embedded_doc_type": null,
          "db_field": "label",
          "fields": []
        },
        {
          "_cls": "SampleFieldDocument",
          "name": "confidence",
          "ftype": "fiftyone.core.fields.FloatField",
          "subfield": null,
          "embedded_doc_type": null,
          "db_field": "confidence",
          "fields": []
        },
        {
          "_cls": "SampleFieldDocument",
          "name": "logits",
          "ftype": "fiftyone.core.fields.VectorField",
          "subfield": null,
          "embedded_doc_type": null,
          "db_field": "logits",
          "fields": []
        }
      ]
    }

Second, I want to add custom attributes as object detections not classifications. Here the label tag in fo.Detection only accepts strings.

sample["ground_truth"] = fo.Detections(
        detections=[
            fo.Detection(
                label="cat",
                confidence=0.9,
                bounding_box=[0.33, 0.33, 0.2, 0.2],
                age=5,
                happiness=99.99,
                breed="tabby",
                cute=True,
            )
        ]
    )

Is there any way to work it out? Or did I miss anything? Thank you!

[jguo1002]

I found a temporary solution that is to create a custom Detection class.

# fiftytwo/labels.py
import fiftyone as fo
import fiftyone.core.fields as fof

class CustomDetection(fo.Detection):
    age= fof.IntField()
    cute= fof.BooleanField()

class CustomDetections(fo.Detections):
    detections = fof.ListField(fof.EmbeddedDocumentField(CustomDetection))

It added age and cute in sample_fields schema under ground_truth.

print(dataset.view)

...
Sample fields:
    ...
    ground_truth:        fiftyone.core.fields.EmbeddedDocumentField(fiftytwo.labels.CustomDetections)

But now I still need to change a lot in both frontend and backend since the label path is hardcoded. For example:

// @fiftyone/utilities
export const LABELS_PATH = "fiftyone.core.labels";

# fiftyone/core/patches.py
_SINGLE_TYPES_MAP = {
    fol.Detections: fol.Detection,
    fol.Polylines: fol.Polyline,
}

Any other ways?

[brimoor]

Hey @jguo1002 just confirming that we are working on a feature that will allow custom attributes of Detection and other Label classes to be added to the dataset's schema and thus available for filtering in the App.

We're considering two possible approaches to this:

• Adding support for dynamic embedded document field schemas #1825
• [WIP] Adding dynamic fields to schema #2 #1826

This feature is currently blocked by the need to improve the App's performance a bit so that it scale to dataset's with more embedded fields in their schemas (but that's being actively worked on and this is next!)

[jguo1002]

@brimoor Thanks for the follow-up! Looking forward to this feature!