#911 dcat distribution model fix inspect api updates #912

SarahJohnsonONS · 2024-06-06T07:29:54Z

Cubes now being built according to requirements here: #911. Backwards-compatible, so cubes built using version < 0.5.0 can still be inspected.

Unit and behaviour tests are passing. There was a niggly little issue with pyright, which I've fixed temporarily using type: ignore (on L329 of writers/helpers/qbwriter/dsdtordfmodelshelper.py).

github-actions · 2024-06-06T07:36:34Z

ubuntu-latest-python3.11-pandas~2.0 test results

10 files ±0 10 suites ±0 4m 36s ⏱️ +12s
589 tests ±0 589 ✅ ±0 0 💤 ±0 0 ❌ ±0
590 runs ±0 590 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 0249d1e. ± Comparison against base commit 2fcf99c.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-06T07:36:35Z

ubuntu-latest-python3.11-pandas@^1.3 test results

10 files ±0 10 suites ±0 4m 36s ⏱️ +13s
589 tests ±0 589 ✅ ±0 0 💤 ±0 0 ❌ ±0
590 runs ±0 590 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 0249d1e. ± Comparison against base commit 2fcf99c.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-06T07:36:37Z

ubuntu-latest-python3.9- test results

10 files 10 suites 4m 38s ⏱️
589 tests 589 ✅ 0 💤 0 ❌
590 runs 590 ✅ 0 💤 0 ❌

Results for commit 0249d1e.

♻️ This comment has been updated with latest results.

github-actions · 2024-06-06T07:45:43Z

windows-latest-python3.11-pandas~2.0 test results

11 files ±0 12 suites ±0 6m 21s ⏱️ -28s
603 tests ±0 603 ✅ ±0 0 💤 ±0 0 ❌ ±0
617 runs ±0 617 ✅ ±0 0 💤 ±0 0 ❌ ±0

Results for commit 0249d1e. ± Comparison against base commit 2fcf99c.

♻️ This comment has been updated with latest results.

canwaf

Review WIP

pyproject.toml

...i/inspect/multi-attribute-resource-values/multi-attribute-duplicate-labels.csv-metadata.json

canwaf

Add a deperciation warning about old style csvws generated using the 0.1.0-dev1 output/models, and that only the new relational structure will eventually be supported. Further communiucaton at a later date.

canwaf · 2024-06-14T12:19:22Z

src/csvcubed/writers/helpers/qbwriter/dsdtordfmodelshelper.py

+        dcat_dataset_with_metadata = rdf.dcat.Dataset(self._uris.get_dataset_uri())
+        self.cube.metadata.configure_dcat_dataset(dcat_dataset_with_metadata)
+        dcat_dataset_with_metadata.distribution.add(
+            ExistingResource(self._uris.get_distribution_uri()).uri  # type: ignore


Flag for review

Tried to fix this but I have no idea what's going on, so I'll add it as a new ticket.

...cli/inspect/multi-attribute-resource-values/multi-attribute-duplicate-uris.csv-metadata.json

...i/inspect/multi-attribute-resource-values/multi-attribute-duplicate-labels.csv-metadata.json

...cli/inspect/multi-attribute-resource-values/multi-attribute-duplicate-uris.csv-metadata.json

...i/inspect/multi-attribute-resource-values/multi-attribute-duplicate-labels.csv-metadata.json

tests/test-cases/cli/inspect/multi-unit_multi-measure/alcohol-bulletin.csv-metadata.json

...li/inspect/multi-unit_single-measure/new/multi-unit-single-measure-dataset.csv-metadata.json

src/csvcubed/writers/helpers/qbwriter/dsdtordfmodelshelper.py

canwaf · 2024-06-17T14:59:50Z

src/csvcubed/models/rdf/prov.py

+    title: Annotated[
+        str, Triple(DCTERMS.title, PropertyStatus.recommended, map_str_to_en_literal)
+    ]
+    has_primary_source: Annotated[


Why did this show up in here? Should this not be in models?

prov.py was already here (with the Activity class definition). Rob did tell me why some of the models are here instead of in csvcubed-models, but I can't remember exactly what he said - something about it being more flexible?

src/csvcubed/models/inspect/sparqlresults.py

canwaf · 2024-06-17T15:38:11Z

src/csvcubed/models/inspect/sparqlresults.py

@@ -616,3 +629,22 @@ def map_row(row_result: Dict[str, Any]) -> ColumnDefinition:
        )

    return [map_row(row.asdict()) for row in sparql_results]
+
+


Did you use black to lint this?

canwaf · 2024-06-17T15:45:09Z

src/csvcubed/models/cube/qb/catalog.py

        distribution.identifier = self.get_identifier()
        distribution.creator = self.creator_uri
-        distribution.landing_page = set(self.landing_page_uris)
-        distribution.themes = set(self.theme_uris)
-        distribution.keywords = set(self.keywords)



Gonna need you to tell me how we populate the wasGeneratedBy bit

In writers/helpers/qbwriter/dsdtordfmodelshelper.py L280.

src/csvcubed/inspect/sparql_handler/data_cube_repository.py

canwaf · 2024-06-17T15:48:14Z

src/csvcubed/cli/inspectcsvw/metadataprinter.py

@@ -97,13 +97,28 @@ def get_csvw_type_str(csvw_type: CSVWType) -> str:
        else:
            raise InputNotSupportedException()

+    def get_build_minor_version(self) -> int:
+        """Return the minor version of csvcubed used to build the cube."""
+        # TODO Create a class to store csvcubed version information similar to readers/cubeconfig/schema_versions.py


Raise this ticket please.

canwaf

This is a phenominal amount of work, and a great bit of effort. There are no clangers here, and nothing massively needing rework. We can pretty much have all our test cases (behaviour) align to the new object model, and relegate successful inspects on a single test case generated by a previous version of csvcubed.

I will need you to sort out the object model issues I have raised, resolve all the comments appropriately and I can approve.

Excellent work.

…-updates

canwaf

There is one change you need to make, which is commented; however you can merge after you make that change.

src/csvcubed/models/inspect/sparqlresults.py

sonarcloud · 2024-07-17T09:22:12Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

Auto-version-incrementer and others added 14 commits October 13, 2023 11:21

Updating the release version in pyproject.toml

48b73d6

test commit

aba1be7

WIP

973711c

WIP

ad32b40

WIP

bb3e4c3

WIP

a4b567d

WIP

16dc4e1

WIP

c0171f4

Tidy up

db0f896

tidy up

b3a6416

Working

83b1a2b

Added comments

99897bc

fixed pyright errors

dc9ad57

more pyright

dcabee2

canwaf reviewed Jun 14, 2024

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

...i/inspect/multi-attribute-resource-values/multi-attribute-duplicate-labels.csv-metadata.json Outdated Show resolved Hide resolved

Changed #csvqb to #qbDataSet

a88cb22

canwaf reviewed Jun 17, 2024

View reviewed changes

src/csvcubed/models/inspect/sparqlresults.py Outdated Show resolved Hide resolved

canwaf reviewed Jun 17, 2024

View reviewed changes

src/csvcubed/inspect/sparql_handler/data_cube_repository.py Outdated Show resolved Hide resolved

canwaf reviewed Jun 17, 2024

View reviewed changes

canwaf suggested changes Jun 17, 2024

View reviewed changes

SarahJohnsonONS added 2 commits June 20, 2024 13:47

PR comments addressed

712f925

Merge branch 'main' into #911-dcat-distribution-model-fix-inspect-API…

c3093c7

…-updates

SarahJohnsonONS added 5 commits June 20, 2024 14:28

poetry lock

cbc9c90

Merge branch 'main' of https://github.com/GSS-Cogs/csvcubed

1675e6c

Merge branch 'main' into #911-dcat-distribution-model-fix-inspect-API…

80fefdd

…-updates

poetry lock

f8e862f

oops

1451774

canwaf approved these changes Jul 16, 2024

View reviewed changes

src/csvcubed/models/inspect/sparqlresults.py Outdated Show resolved Hide resolved

small change

0249d1e

SarahJohnsonONS merged commit be725e5 into main Jul 17, 2024
16 checks passed

SarahJohnsonONS deleted the #911-dcat-distribution-model-fix-inspect-API-updates branch July 17, 2024 09:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#911 dcat distribution model fix inspect api updates #912

#911 dcat distribution model fix inspect api updates #912

SarahJohnsonONS commented Jun 6, 2024

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

canwaf left a comment

canwaf left a comment

canwaf Jun 14, 2024

SarahJohnsonONS Jun 20, 2024

canwaf Jun 17, 2024

SarahJohnsonONS Jun 18, 2024

canwaf Jun 17, 2024

SarahJohnsonONS Jun 18, 2024

canwaf Jun 17, 2024

SarahJohnsonONS Jun 20, 2024

canwaf Jun 17, 2024

canwaf left a comment

canwaf left a comment

sonarcloud bot commented Jul 17, 2024

		@@ -616,3 +629,22 @@ def map_row(row_result: Dict[str, Any]) -> ColumnDefinition:
		)

		return [map_row(row.asdict()) for row in sparql_results]

#911 dcat distribution model fix inspect api updates #912

#911 dcat distribution model fix inspect api updates #912

Conversation

SarahJohnsonONS commented Jun 6, 2024

github-actions bot commented Jun 6, 2024 • edited Loading

ubuntu-latest-python3.11-pandas~2.0 test results

github-actions bot commented Jun 6, 2024 • edited Loading

ubuntu-latest-python3.11-pandas@^1.3 test results

github-actions bot commented Jun 6, 2024 • edited Loading

ubuntu-latest-python3.9- test results

github-actions bot commented Jun 6, 2024 • edited Loading

windows-latest-python3.11-pandas~2.0 test results

canwaf left a comment

Choose a reason for hiding this comment

canwaf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

canwaf left a comment

Choose a reason for hiding this comment

canwaf left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Jul 17, 2024

Quality Gate passed

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading

github-actions bot commented Jun 6, 2024 •

edited

Loading