[Task] SQL generator: Handling Collections of Entities #622
Labels
acknowledged
Has been viewed by one of the maintainers and is ready for further work, discussion or other steps.
task
Is your task related to a problem? Please describe.
Currently, for a Property that has a Collection/List/Set/SortedSet Characteristic, the Databricks SQL generator will create one column with an
ARRAY<>
datatype, with aSTRUCT
for the nested Entity type. This makes querying difficult and the approach is not applicable to other SQL dialects.Instead, such collections of Entities should be flattened into columns for the Entity's type, with additional discriminator columns that allow retrieving the original dataset. Additionally,
for Aspects with one Property called
items
(a "collection Aspect"), flattening to tables should happen for the Entity which is the type of the collection, not for the Aspect itself. As an example, for AspectWithErrorCollection.ttl, the resulting table should have columns forerrorNo
,errorText
andstartTimestamp
instead of anitems
column.Describe the solution you'd like
Adjust the SQL Databricks generator to flatten collections of entities into columns. Handle a top-level
items
Property separately if necessary. Make names of additional discriminator columns configurable.The text was updated successfully, but these errors were encountered: