Release version 3.0.5 of the event_date_qc library, compliant implementation of all the BDQ Core Time Tests

Release version 3.0.5 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions in the draft BDQ Core standard, from the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes validated implementation of all the core Time tests from BDQ Core as of 2024-11-08, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release passes against all TIME validation test data as of the 2023-11-08 test validation data. This release includes support for the lightweight run of tests from the command line.

Supported BDQ Core tests, with link to draft of quick reference guide.

• AMENDMENT_EVENTDATE_STANDARDIZED
• VALIDATION_DATEIDENTIFIED_INRANGE
• AMENDMENT_EVENT_FROM_EVENTDATE
• VALIDATION_YEAR_NOTEMPTY
• AMENDMENT_MONTH_STANDARDIZED
• VALIDATION_ENDDAYOFYEAR_INRANGE
• VALIDATION_EVENTDATE_CONSISTENT
• MEASURE_EVENTDATE_PRECISIONINSECONDS
• VALIDATION_EVENTDATE_NOTEMPTY
• AMENDMENT_EVENTDATE_FROM_YEARMONTHDAY
• VALIDATION_DAY_STANDARD
• AMENDMENT_DAY_STANDARDIZED
• VALIDATION_YEAR_INRANGE
• VALIDATION_EVENTDATE_INRANGE
• AMENDMENT_EVENTDATE_FROM_YEARSTARTDAYOFYEARENDDAYOFYEAR
• VALIDATION_DATEIDENTIFIED_STANDARD
• AMENDMENT_EVENTDATE_FROM_VERBATIM
• VALIDATION_EVENT_TEMPORAL_NOTEMPTY
• AMENDMENT_DATEIDENTIFIED_STANDARDIZED 
• VALIDATION_MONTH_STANDARD
• VALIDATION_EVENTDATE_STANDARD
• VALIDATION_DAY_INRANGE
• VALIDATION_STARTDAYOFYEAR_INRANGE

Full Changelog: v3.0.2...v3.0.5

Release version 3.0.2 of the event_date_qc library, compliant implementation of all the TG2 Core Time Tests.

Release version 3.0.2 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes validated implementation of all the core Time tests from TDWQ BDQ TG2 as of 2023-07-04, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release passes against all TIME validation test data as of the 2023-07-04 test validation data. This release includes support for the lightweight run of tests from the command line. The v3.0.2 release fixes a bug in the implementation of VALIDATION_EVENTDATE_CONSISTENT in the handling of date ranges including times, and allows build without javadoc generation errors in java versions up to 17.

Supported tests by label and link to rationale management issue:
AMENDMENT_EVENTDATE_STANDARDIZED tdwg/bdq#61
VALIDATION_DATEIDENTIFIED_INRANGE tdwg/bdq#76
AMENDMENT_EVENT_FROM_EVENTDATE tdwg/bdq#52
VALIDATION_YEAR_NOTEMPTY tdwg/bdq#49
AMENDMENT_MONTH_STANDARDIZED tdwg/bdq#128
VALIDATION_ENDDAYOFYEAR_INRANGE tdwg/bdq#131
VALIDATION_EVENTDATE_CONSISTENT tdwg/bdq#67
MEASURE_EVENTDATE_PRECISIONINSECONDS tdwg/bdq#140
VALIDATION_EVENTDATE_NOTEMPTY tdwg/bdq#33
AMENDMENT_EVENTDATE_FROM_YEARMONTHDAY tdwg/bdq#93
VALIDATION_DAY_STANDARD tdwg/bdq#147
AMENDMENT_DAY_STANDARDIZED tdwg/bdq#127
VALIDATION_YEAR_INRANGE tdwg/bdq#84
VALIDATION_EVENTDATE_INRANGE tdwg/bdq#36
AMENDMENT_EVENTDATE_FROM_YEARSTARTDAYOFYEARENDDAYOFYEAR tdwg/bdq#132
VALIDATION_DATEIDENTIFIED_STANDARD tdwg/bdq#69
AMENDMENT_EVENTDATE_FROM_VERBATIM tdwg/bdq#86
VALIDATION_EVENT_TEMPORAL_NOTEMPTY tdwg/bdq#88
AMENDMENT_DATEIDENTIFIED_STANDARDIZED tdwg/bdq#26
VALIDATION_MONTH_STANDARD tdwg/bdq#126
VALIDATION_EVENTDATE_STANDARD tdwg/bdq#66
VALIDATION_DAY_INRANGE tdwg/bdq#125
VALIDATION_STARTDAYOFYEAR_INRANGE tdwg/bdq#130

Release version 3.0.0 of the event_date_qc library, compliant implementation of all the TG2 Core Time Tests.

Release version 3.0.0 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes validated implementation of all the core Time tests from TDWQ BDQ TG2 as of 2023-07-04, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release passes against all TIME validation test data as of the 2023-07-04 test validation data. This release includes support for the lightweight run of tests from the command line.

Supported tests by label and link to rationale management issue:
AMENDMENT_EVENTDATE_STANDARDIZED tdwg/bdq#61
VALIDATION_DATEIDENTIFIED_INRANGE tdwg/bdq#76
AMENDMENT_EVENT_FROM_EVENTDATE tdwg/bdq#52
VALIDATION_YEAR_NOTEMPTY tdwg/bdq#49
AMENDMENT_MONTH_STANDARDIZED tdwg/bdq#128
VALIDATION_ENDDAYOFYEAR_INRANGE tdwg/bdq#131
VALIDATION_EVENTDATE_CONSISTENT tdwg/bdq#67
MEASURE_EVENTDATE_PRECISIONINSECONDS tdwg/bdq#140
VALIDATION_EVENTDATE_NOTEMPTY tdwg/bdq#33
AMENDMENT_EVENTDATE_FROM_YEARMONTHDAY tdwg/bdq#93
VALIDATION_DAY_STANDARD tdwg/bdq#147
AMENDMENT_DAY_STANDARDIZED tdwg/bdq#127
VALIDATION_YEAR_INRANGE tdwg/bdq#84
VALIDATION_EVENTDATE_INRANGE tdwg/bdq#36
AMENDMENT_EVENTDATE_FROM_YEARSTARTDAYOFYEARENDDAYOFYEAR tdwg/bdq#132
VALIDATION_DATEIDENTIFIED_STANDARD tdwg/bdq#69
AMENDMENT_EVENTDATE_FROM_VERBATIM tdwg/bdq#86
VALIDATION_EVENT_TEMPORAL_NOTEMPTY tdwg/bdq#88
AMENDMENT_DATEIDENTIFIED_STANDARDIZED tdwg/bdq#26
VALIDATION_MONTH_STANDARD tdwg/bdq#126
VALIDATION_EVENTDATE_STANDARD tdwg/bdq#66
VALIDATION_DAY_INRANGE tdwg/bdq#125
VALIDATION_STARTDAYOFYEAR_INRANGE tdwg/bdq#130

Release version 2.1.0 of the event_date_qc library, compliant implementation of all the TG2 Core Time Tests.

Release version 2.1.0 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes validated implementation of all the core Time tests from TDWQ BDQ TG2 as of 2022-03-10, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release passes against all TIME validation test data as of v18 except for dataID 987 where the validation data are incorrect and this implementation provides the correct result. This release includes support for the lightweight run of tests from the command line.

What's Changed

• Bump log4j-api from 2.17.0 to 2.17.1 by @dependabot in #13
• Bump log4j-core from 2.17.0 to 2.17.1 by @dependabot in #14

Full Changelog: v2.0.4...v2.1.0

Release version 2.0.4 of the event_date_qc library, implementation of all the TG2 Core Time Tests.

Release version 2.0.4 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes implementation of all the core Time tests from TDWQ BDQ TG2 as of the end of August 2019, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release includes support for the lightweight run of tests from the command line.

Notes on this release: Updated log4j to 2.17.0 to address CVE-2021-44832

Release version 2.0.3 of the event_date_qc library, implementation of all the TG2 Core Time Tests.

Release version 2.0.3 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes implementation of all the core Time tests from TDWQ BDQ TG2 as of the end of August 2019, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release includes support for the lightweight run of tests from the command line.

Notes on this release: Updated log4j to 2.16.0 to address CVE-2021-44228 and CVE-2021-45046 Updated junit to 4.13.1

Release version 2.0.2 of the event_date_qc library, implementation of all the TG2 Core Time Tests.

ISSUE: PURPOSE: Incrementing version to 2.0.2 for release. DESCRIPTION: in pom.
Release version 2.0.0 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release includes implementation of all the core Time tests from TDWQ BDQ TG2 as of the end of August 2019, along with unit tests of both low level methods and higher level framework (matching the test specifications) methods. This release includes support for the lightweight run of tests from the command line.

Notes on this release: Adding unit test coverage. Added test for / without range (plus some other empty values) to unit test for DateUtils.extractInterval() Date range parsing fails for incorrectly formed ranges that include a / but no range. Adding handling for this case to code added to handle incorrect number of digits in ISO dates. Including output of time for million line processing in Runner. ISSUE: tdwg/bdq#76 Adding unit test for validation dateIdentified outofrange. Added unit test with notes concerning where the specification needs clarification (around overlapping ranges). Adding remaining unit tests. Filling in remaining unit tests for DwCEventTG2 methods, covering all TG2 temporal tests. ISSUE: tdwg/bdq#52 Unit test for AMENDMENT_EVENT_FROM_EVENTDATE. ISSUE: tdwg/bdq#86 tdwg/bdq#67 Added unit test for VALIDATION_EVENT_INCONSISTENT, updated handling of empty eventDate to match the specification, added more debug logging. Added unit test for AMENDMENTEVENTDATE_FROMVERBATIM covering multiple cases in current specification, primaraly assessing logic of result type rather than abilty to interpret dates. ISSUE: tdwg/bdq#88 Unit tests for validationEventEmpty. DESCRIPTION: Adding unit test, under interpretation that 'field needed to determine the eventDate' means any field in set, e.g. day alone is compliant. ISSUE: tdwg/bdq#36 Unit test and correction to validationEventdateOutofrange. Adding unit test for method, found error in handling of current date (invoking wrong DateUtils method to get interval for comparison), fixed. Adding more unit tests for DwCEventTG2DQ validations. ISSUE: tdwg/bdq#125 Adding unit test for VALIDATION_DAY_OUTOFRANGE and correcting errors found. Added unit test covering all outcomes in specification, found and fixed errors, particularly in inverted logic acting upon yearParsable. Moving implementation of remaining non-eventDate method from DwCEventTG2DQ. Moved implementation of amendmentDateidentifiedStandardized to static method in DwCOtherDateDQ. Finishing moving test from temporary location in DQTest to TG2DQTest. Moving DwCEventTG2DQ methods to static. changing methods to public static to follow kurator pattern, making follow on changes to Runner and tests. Moving test from temporary location in DQTest to TG2DQTest. ISSUE: tdwg/bdq#125 tdwg/bdq#127 tdwg/bdq#61 Multiple fixes and added unit tests resulting from Arthur Chapman's ongoing review of results of test runs. Set distinct guid for mechanism for DwCEventDateTG2DQ. Java to 1.8 and added commons-lang3 in pom for StringUtils. Moved implementation for dateIdentified into DwCOtherDateDQ. Fix for single digit days and months being recognized as valid ISO date, updates to unit tests in consequence. DateUtils.extractInterval() and extractDate() now returns null when given single digit day or month values. Fix for handling of date ranges with end date before start date. DateUtils.eventDateValid() now returns false on these. added unit tests for validationYearEmpty, amendmentDayStandardized, amendmentEventdateStandardized, validationDateidentifiedNotstandard. ISSUE: tdwg/bdq#52 tdwg/bdq#131 Bug fixes to tests and test harness following Arthur Chapman's iterative reviews of the results of test runs. Fixed a bug in Runner where non-null Event terms were being overwritten with nulls on invocation of amendmentEventFromEventDate. Fixed bugs in validationDateidentifiedNotstandard resulting from copy/paste errors and changed logic. Fixed some other bugs, added a unit test, results now appear sane. Prevent null pointer exception on rare case of null in eventDateInterval. Corrections from exchange of first results with Arthur Chapman, some pre/post amendment validation numbers are odd. isEmpty() not correctly implemented for current expectations of tests. Removing 'NULL' as valid empty value per the current TG2 definition, updating tests in consequence, reducing error reporting. Harden Runner to work against real data. Altering Runner to run tests against tab delimited rather than comma delimited files (tab usual Darwin Core archive content), and allow skipping lines and handling of broken lines when reading data for test. Adding lightweight run of tests from the command line. Adding a Runner class to carry out command line operations. Changing main class in pom from DateUtils to Runner. Adding invocations of tests in DwcEventTG2DQ to Runner, moving main method in DateUtils to a verbatim date extraction method, invoking this from Runner. Reducing log level of some Errors in DateUtils to reduce command line output verbosity. ISSUE: tdwg/bdq#36 Updating VALIDATION_EVENTDATE_OUTOFRANGE to current specification. Implementation using begin and end date parameters, without optional begin, as per current (still under discussion) phrasing of the specification. Using Interval to test range contained within rather than integer year values, matching notes in issue specifying that day counts. Split off no-parameter test (taking the guid) from parameterized test. Fix javadoc tag problems. Fixing problems with @return and @param values resulting from java 8 evaluation of structure of javadoc annotations in remarks. ISSUE: tdwg/bdq#147 tdwg/bdq#125Updating validations for dwc:day based on latest changes in issues. DESCRIPTION: Implementation of validation day notstandard to match amendment_day_standardized testing only dwc:day and checking for integer in range 1-31, and validation_day_inrange to use day month year and return internal prerequisites not met on month or year only when those values are required to evaluate a day value (29-31). Resolving more concerns for date tests and TG2 specifications. ISSUE: tdwg/bdq#84 tdwg/bdq#26 tdwg/bdq#130 tdwg/bdq#61 Updating tests to reflect updates in definitions and latest discussion in tdwg/bdq issues. Removing confirmed unused isMonthInRange() method. Updating VALIDATION_STARTDAYOFYEAR_OUTOFRANGE to match new specification. Switching from #141 to #84 to test year for valid range. Correcting handling of ambiguity in AMENDMENT_EVENTDATE_STANDARDIZED and AMENDMENT_DATEIDENTIFIED_STANDARDIZED to conform with specifications. Implement CORE TIME tests. ISSUE: tdwg/bdq/#140 tdwg/bdq/#69 tdwg/bdq/#76 tdwg/bdq/#147 tdwg/bdq/#131 tdwg/bdq/#88 tdwg/bdq/#67 tdwg/bdq/#33 tdwg/bdq/#66 tdwg/bdq/#36 tdwg/bdq/#126 tdwg/bdq/#130 tdwg/bdq/#131 tdwg/bdq/#49 tdwg/bdq/#141 tdwg/bdq/#84 tdwg/bdq/#26 tdwg/bdq/#127 tdwg/bdq/#52 tdwg/bdq/#86 tdwg/bdq/#93 tdwg/bdq/#132 tdwg/bdq/#61 tdwg/bdq/#128 PURPOSE: Implementations of the currently (2019Aug12) defined core tests related to time. DESCRIPTION: Java class, method signatures, and annotations generated with kurator-ffdq from csv extracted from issues by bdq_issue_to_csv, then implementations added for each of the tests. TODO comments indicate remaining issues raised by implementation. The DwCEventTG2DQ class is expected to get merged into the DwCEventDQ and DwCOtherDateDQ classes, but is committed in parallel here to view all the tests in one place for comparison. ISSUE: tdwg/bdq#151 Adding some example data data. Adding a reference set of various cases of values in dwc date related terms and example interpretation of those values. Adding support for some uncommon month abbreviations.

Release version 2.0.0 of the event_date_qc library, updated FFDQ support.

Release version 2.0.0 of the FilteredPush event_date_qc library for testing data quality of date data. This library provides low level tests of date data and higher level tests for DarwinCore Event terms with those tests described with RDF metadata following the standard biodiversity data quality test descriptions by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

This release incorporates changes to the API to support FFDQ, and updates some tests to current TG2 test definitions. Notes on changes in this release: Creating a clean no-snapshots referenced release. Updating pom to reference 2.0.0 ffdq-api instead of snapshot, incrementing version to 2.0.0, updating README. Correcting Javadoc build error. ISSUE: #8 tdwg/bdq#125 Updating VALIDATION_DAY_OUTOFRANGE in line with proposal for internal prerequisites not met in issue, applying correct guid to issue, updating handling of internal prerequisites not met. ISSUE: #8 tdwg/bdq#147 Updating VALIDATION_DAY_NOTSTANDARD. VALIDATION_DAY_NOTSTANDARD now returns NOT_COMPLIANT for any dwc:day value except an integer in the range 1-31, prerequisites not met no longer thrown internally to this implementation as that is only returned if term is absent. Adding command line options to parse dates from standard input and from a command line argument. Adding to DateUtils main method -i to read a line from standard input and -v date to read a date as a command line argument. Preventing null pointer exception on problem case in date parsing. Reducing logging messages. Reducing logging level for joda exception related to out of range token values in date parsing. Correcting version of ffdq-api to 2.0.0-SNAPSHOT to fit with standard semantic versioning scheme used in kurator. Fixes to example data to allow publishing in IPT. Fixed two duplicate occurrenceId values. Updated date modified for new records. Adding command line use documentation. Improving use on the command line. Adding slightly better error condition handling to DateUtils.main(), adding a log4j2 configuration file to suppress warning message from log4j. Added description of command line options. Allowing run on a list of dates from the jar. Adding main class to the shaded jar manifest in the pom. Adding example data (modified from real data) to provide at least one case for each of the current BDQ TG2 Event related tests. Added a column concatenating all of the resource relationship terms for provision in a remark term when resource relationship terms aren't available (e.g. for mapping into IPT). Updated version to 2.0.0 to reflect the changes to the ffdq API Consistent with the current versions of kurator-ffdq and ffdq-api Merging from the ffdq branch to support 2.0-SNAPSHOT API Version Changes from branch include new set of return types used in the RDF representation of the framework. Updates to the DQ classes and CSV to work with ffdq test runner. Adding csv file generated from tdwg/bdq test issues with label=Time on Feb 8. Updating dq class with changes to amendment values in kurator-ffdq. The underlying object as part of AmendmentValue is now a java Map. Minor fix to missing result values when result state is CHANGED. Reformatting the csv file to match the spreadsheet of standardized tests. Same format however contains only the event_date_qc tests. Added an xlsx version that contains rows taken from spreadsheet. The full row data from the spreadsheet of standardized tests for tests defined in the DwCEventDQ.csv. Still need to add the missing tests. Adding the IDs from the spreadsheet of standardized tests. Minor fix, no change in the extractDateFromVerbatim has empty result. In the report, no change should have no value for the result. Was an empty data resource instead. Minor fix, adding comment to extract date from start/end date test. Using maven to include dependencies in jar for use with test runner . The ffdq test runner will run tests from a class on the classpath. Packaging dependencies in the jar will enable test runner to run tests from the event_date_qc-1.0.4-SNAPSHOT.jar. Minor fix, eventDateConsistentWithVerbatim returns null result state. Cleanup and updated the tests to reflect recent changes in ffdq. Updated the ffdq return types for all tests implemented in DwCEventDQ.

Release version 1.1.0 of the event_date_qc library, adds parameterized tests.

Release version 1.1.0 of the FilteredPush event_date_qc library for testing data quality of date data, in particular, testing DarwinCore Event terms with tests described by the TDWG Biodiversity Data Quality Interest Group Task Group 2 on tests and assertions.

Changes in this release include: Adding some additional test cases to DateUtilsTest. Adding a test data file with modified real data that exersises many of the event date tests. Annotating the lowerBound argument for test method VALIDATION_YEAR. Adds supports for parameterized tests using the new @parameter annotation on method arguments in the new tests. The Parameter annotation in the ffdq-api project will be used by test runner to interpret method arguments that should map to workflow parameters. Currently, the test runner doesn't have support for method parameters that are unannotated. See kurator-org/kurator-web#21Implementing two different interpreations of event date in range, marking one implementation as 3cff4dc4-72e9-4abe-9bf3-8a30f1618432 in what seems to be the desired consensus. Marking the other implementation with a new uuid as a potential alternative (or additional test). See discussion in tdwg/bdq#36

Release version 1.0.4, improved framework support

David Lowery's work in the ffdq branch. The ffdq test runner will run tests from a class on the classpath. Updated the ffdq return types for all tests implemented in DwCEventDQ. Adding the IDs from the spreadsheet of standardized tests as of Jan 3 2018. Reformatting the csv file to match the spreadsheet of standardized tests. Added a populate event date from verbatim plus atomic parts amendment to comply with EVENTDATE_FROM_VERBATIM in TG2 spreadsheet. Added modified and dateIdentified tests from the spreadsheet. Additional tests based on running data from Tracy through DateValidator. Changed eventsAreSameInterval() to ignore times. Added validation to check eventDate and verbatimEventDate for consistency with ffdq annotations. Updated the term parameters to work with any namespace prefix. More verbatimDate parsing including handling of square brackets. Cleanup of redundant unit tests.