[FEATURE] Review record flatten process

[frascuchon]

notes:

• user id -> username
• response status?
• "text" -> "fields.text" ?
• "metadata_key" -> "metadata.metadata_key"?

[frascuchon]

I think we should double-check the generated structure of to_json or to_list so we're not missing anything.

[burtenshaw]

Current nested structure

next(iter(dataset.records(with_responses=True))).to_dict()

{'id': UUID('25b8ca21-d0fb-4135-815d-0887393007b8'),
 'fields': {'post': 'Another clown in favour of more tax in this country. Blows my mind people can be this stupid.'},
 'metadata': {},
 'suggestions': {'is_toxic': {'value': '1', 'score': None, 'agent': None},
  'toxic_spans': {'value': [{'label': 'insult',
     'start': 86,
     'end': 92,
     'score': 0.6666666666666666},
    {'label': 'insult', 'start': 8, 'end': 13, 'score': 0.6666666666666666}],
   'score': None,
   'agent': None}},
 'responses': defaultdict(list, {}),
 'vectors': {},
 '_server_id': '4836f966-0a7f-4d6e-b017-398270265c95'}

Current Flattened structure

from pprint import pprint

pprint(dataset.records.to_list(flatten=False))

[{'_server_id': '0f3fd360-7776-464e-82a2-9654da527212',
  'fields': {'question': 'What is the capital of France?'},
  'id': '1',
  'metadata': {},
  'responses': defaultdict(<class 'list'>, {}),
  'suggestions': {'answer': {'agent': None, 'score': None, 'value': 'F'}},
  'vectors': {}},
 {'_server_id': '5aa81ed0-3cd2-4208-bae0-b8fb0ba8fd1d',
  'fields': {'question': 'What is the capital of Germany?'},
  'id': '2',
  'metadata': {},
  'responses': defaultdict(<class 'list'>, {}),
  'suggestions': {'answer': {'agent': None, 'score': None, 'value': 'Berlin'}},
  'vectors': {}}]

[sdiazlor]

@frascuchon @burtenshaw This is the issue I mentioned regarding the to_list flattening and the responses not being added. #5042

[frascuchon]

Also, @MoritzLaurer found errors when creating HF datasets with records partially annotated. We need to review this to:

1. Support export record list partially annotated
2. Simplify how response values would be read (the generated columns include the user.id)