[BUG] CTS failures #7148

planetf1 · 2022-11-29T15:12:28Z

Is there an existing issue for this?

I have searched the existing issues

Current Behavior

Seeing CTS failures in both in-mem & graph (but this is on the experimental v4 build)- and I understand @dwolfson is also seeing in XTDB on v3.14

This was seen when testing v4 (java 17/gradle)

In my case I did a test run on Nov 22nd with in-memory using 'main'.
That is currently failing 16 tests of 6105 with 6089 passing

Expected Behavior

CTS to pass

Steps To Reproduce

Run CTS with current main - using inmemory for example
retrieve results with

http --timeout=600 --verify=no GET https://localhost:9443/servers/cts/open-metadata/conformance-suite/users/garygeeke/report > out.json

Print out the failing assertions with

cat out.json | jq -r '.testLabResults.testResultsFromWorkbenches[].failedTestCases[].unsuccessfulAssertions[]'

The result is

repository-relationship-reference-copy-lifecycle-03: DiscoveredNestedDataField reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DiscoveredNestedDataField refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: RelatedKeyword reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: RelatedKeyword refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: LastAttachmentLink reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: LastAttachmentLink refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: AttachedRating reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: AttachedRating refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: DataFieldAnalysis reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DataFieldAnalysis refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: AnnotationExtension reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: AnnotationExtension refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: DiscoveredAnnotation reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DiscoveredAnnotation refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: SchemaAttributeDefinition reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: SchemaAttributeDefinition refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: SchemaTypeDefinition reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: SchemaTypeDefinition refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: TranslationLink reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: TranslationLink refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: AnnotationReviewLink reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: AnnotationReviewLink refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: DataClassDefinition reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DataClassDefinition refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: DiscoveredDataField reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DiscoveredDataField refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: AttachedTag reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: AttachedTag refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: SearchKeywordLink reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: SearchKeywordLink refreshed reference relationship matches original.
repository-relationship-reference-copy-lifecycle-03: DataProfileLogFile reference relationship matches the relationship that was saved.
repository-relationship-reference-copy-lifecycle-11: DataProfileLogFile refreshed reference relationship matches original.

Environment

- Egeria:
- OS:
- Java:
- Browser (for UI issues):
- Additional connectors and integration:

Any Further Information?

No response

The text was updated successfully, but these errors were encountered:

planetf1 · 2022-11-29T15:14:52Z

The above output is from the graph repo

planetf1 · 2022-11-29T15:17:52Z

will repeat with current code from main

planetf1 · 2022-11-29T18:34:44Z

The CTS tests from main are clean.

This means it is one of

intermittent
specific to the v4 changes (gradle etc)
fixed by changes in main yet to be pushed to the v4 branch (~2 weeks)

planetf1 · 2022-11-29T19:10:45Z

repeating test after updating v4 from master

planetf1 · 2022-12-05T17:54:29Z

After incorporating updates from main, the CTS (graph) continues to fail in the same way

planetf1 · 2022-12-05T18:07:39Z

Server log: https://gist.github.com/8e242fa5769a8856976122d5e29b7303
Nothing abnormal noticed

planetf1 · 2022-12-10T12:56:23Z

Having retested with odpi/egeria-charts#225 (egeria-release-3.14), the graph CTS is failing in the same way

This is going to block the release - will discuss/investigate more on Monday.

planetf1 · 2022-12-12T09:28:28Z

Looking through recent commits:

d669a35 made a change to reference relationships
f37c0ad affects the storage of classification/entities

planetf1 · 2022-12-13T16:53:24Z

@mandy-chessell Do you have any thoughts as to whether the above have changes may have resulted in this cts behaviour?

mandy-chessell · 2022-12-14T10:19:44Z

d669a35 made a change to one of the converters in the OMAS layer to populate extra properties in an OMAS interface bean - highly unlikely to impact the CTS which operates at the OMRS level.

f37c0ad made changes to entity/classifications and all of the failures are in the management of relationships.

I had a clean CTS run on my machine before pushing the changes into main. (Built with maven)

The recent changes I made that broke the CTS was to add support for DEREGISTERED_REPOSITORY instance provenance type. The CTS was storing an instance, retrieving it and using equals() to determine that it had not changed. The problem was that the CTS was using a nonsense metadata collection id in the instance. Before the DEREGISTERED_REPOSITORY instance provenance type fix, no check was made on the validity of the metadata collection id. The local metadata collection now validates the metadata collection id to determine if it is an active member of the cohort. If it is not then the instance provenance type is updated to DEREGISTERED_REPOSITORY before the instance is returned. This change causes the equals() method to fail in the CTS.

The fix I added to the CTS was to update the instance provenance type to DEREGISTERED_REPOSITORY in the original instance before the test for equals().

I am thinking that the CTS fixes missed the branch or somehow I missed fixing the relationship tests. But then I do not know how it worked on my machine.

I will run the CTS on main to see if the failure occurs on our latest code. This will isolate whether there is a missing fix and we have a new/different problem, or the CTS fix missed the branch.

The fact that it fails on the latest V4.0 branch suggests a new problem.

mandy-chessell · 2022-12-14T11:38:46Z

The way to identify the test that is failing is to search the CTS code for the part of the assertion message that does not include the typeName.

For example, for failing test message ….

repository-relationship-reference-copy-lifecycle-03: DiscoveredNestedDataField reference relationship matches the relationship that was saved.

Search for reference relationship matches the relationship that was saved

This gives you the assertion message.

private static final String assertion3      = testCaseId + "-03";
private static final String assertionMsg3   = " reference relationship matches the relationship that was saved.";

Find where it is used in the code and this give you the condition and profile

verifyCondition((newRelationship.equals(retrievedReferenceCopy)),
                assertion3,
                testTypeName + assertionMsg3,
                RepositoryConformanceProfileRequirement.REFERENCE_COPY_STORAGE.getProfileId(),
                RepositoryConformanceProfileRequirement.REFERENCE_COPY_STORAGE.getRequirementId());

Using this technique, the other failing test is

verifyCondition((newRelationship.equals(refreshedRelationshipRefCopy)),
                        assertion11,
                        testTypeName + assertionMsg11,
                        RepositoryConformanceProfileRequirement.REFERENCE_COPY_STORAGE.getProfileId(),
                        RepositoryConformanceProfileRequirement.REFERENCE_COPY_STORAGE.getRequirementId());

Both failures are using equals() to test that what was retrieved was what is expected.

planetf1 · 2022-12-14T12:33:07Z

Thanks - I had looked at the cts code,and established it was that comparison - was just wondering what you thought ref recent changes. It's been failing on v4 for a couple of weeks (ON GRAPH ONLY)

I'm concerned about releasing 3.14 until we understand the cts failure

I'm also today looking at #353 - so we can pick up on cts issues quicker & ensure consistency

mandy-chessell · 2022-12-14T15:35:19Z

If it is graph only it is probably a new issue and not related to the DEREGISTERED_REPOSITORY change which was in the OMRS and not the connectors.

planetf1 · 2022-12-14T18:16:45Z

Rerunning CTS on 3.14 (k8s, via egeria-cts chart)
: InMemory - PASS 2:40
: Graph - FAILED -- as in original report

mandy-chessell · 2022-12-15T08:41:28Z

This is the comparison of the two relationships. The retrieved headerVersion is 0 and should be 1 in the relationship, and each entityProxy. In the comparison below, retrievedReferenceCopy is on the left and newRelationship is on the right.

The headerVersion property is used to discover whether the OMRS is talking to a future version of OMRS that uses later structures. The value should be 1. 0 is the default value in the bean and suggests it is not being set.

It hints at a problem in the way instances are stored/retrieved from the graph store. I do not know what has changed to cause it to fail now and why it is only affecting relationships.

mandy-chessell · 2022-12-15T09:06:22Z

Further investigation shows that RepositoryElementHeader where headerVersion is stored, does not have an equals() method. This means that although this is two errors:

Missing equals() method in RepositoryElementHeader
GraphRepository not setting headerVersion correctly

It is not reason that the CTS is failing.

mandy-chessell · 2022-12-15T11:57:41Z

This is the comparison of the relationships for a failing case.

What we see is that the newRelationship incorrectly includes a null qualifiedName property in the entityProxies for each end of the relationship. This is an invalid property for the type.

The error is in the CTS since it is setting up the newRelationship. The question is, why has this changed?

mandy-chessell · 2022-12-16T09:30:44Z

I have located the source of the problem in EntityProxy.java. It is setting up qualifiedName whether it is null or not. Fixing that led to an NPE in Subject Area OMAS that was dereferencing uniqueProperties without checking it was not null.

The CTS is now running cleanly on graph

planetf1 · 2022-12-17T08:36:44Z

CTS tests

Run via egeria-cts k8s chart (as is usual for release)

Graph

Reporting test failures

        "testCaseCount": 6106,
        "testFailedCount": 1884,
        "testPassCount": 4222,

Summary output:

              Metadata sharing MANDATORY_PROFILE             NOT_CONFORMANT [  70597 /   1060 ]
              Reference copies  OPTIONAL_PROFILE             NOT_CONFORMANT [   6938 /   1884 ]
          Metadata maintenance  OPTIONAL_PROFILE             NOT_CONFORMANT [  13302 /    824 ]
                 Dynamic types  OPTIONAL_PROFILE             UNKNOWN_STATUS [      0 /      0 ]
                 Graph queries  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [    528 /      0 ]
             Historical search  OPTIONAL_PROFILE      CONFORMANT_NO_SUPPORT [    530 /      0 ]
                Entity proxies  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2759 /      0 ]
       Soft-delete and restore  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2592 /      0 ]
                Undo an update  OPTIONAL_PROFILE      CONFORMANT_NO_SUPPORT [    406 /      0 ]
           Reidentify instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2650 /      0 ]
               Retype instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  16365 /      0 ]
               Rehome instance  OPTIONAL_PROFILE             UNKNOWN_STATUS [      0 /      0 ]
                 Entity search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  62878 /      0 ]
           Relationship search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   8254 /      0 ]
        Entity advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  44800 /      0 ]
  Relationship advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   9312 /      0 ]

FAIL [241911/3768]

In-memory

Summary:
"testCaseCount": 6104,
"testFailedCount": 1884,
"testPassCount": 4220,

yet:

➜  inmem ~/src/cts/scripts/cts-analyze.py
              Metadata sharing MANDATORY_PROFILE    CONFORMANT_FULL_SUPPORT [  71657 /      0 ]
              Reference copies  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   8528 /      0 ]
          Metadata maintenance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  14063 /      0 ]
                 Dynamic types  OPTIONAL_PROFILE             UNKNOWN_STATUS [      0 /      0 ]
                 Graph queries  OPTIONAL_PROFILE CONFORMANT_PARTIAL_SUPPORT [    186 /      0 ]
             Historical search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [    530 /      0 ]
                Entity proxies  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2759 /      0 ]
       Soft-delete and restore  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2592 /      0 ]
                Undo an update  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   1218 /      0 ]
           Reidentify instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2650 /      0 ]
               Retype instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  16365 /      0 ]
               Rehome instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   1590 /      0 ]
                 Entity search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  60722 /      0 ]
           Relationship search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   8232 /      0 ]
        Entity advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  40464 /      0 ]
  Relationship advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   9264 /      0 ]

PASS [240820/0]

The new, trial CTS test runs

See https://github.com/planetf1/cts.
These do not check the pass/fail count in the summary, instead relying on negative/positive evidence points in the full output

Both FAILED

Identical results from graph in terms of profile ( https://github.com/planetf1/cts/actions/runs/3715784372/jobs/6301358261 )

Inmemory showing a few issues https://github.com/planetf1/cts/actions/runs/3715784372/jobs/6301358199 - also a fail

planetf1 · 2022-12-17T08:44:57Z

Checked with MAIN

The SAME cts graph failures are reported
(inmemory running now)
Yet this is to be expected, since the PR #7247 is yet to be merged

Therefore the test may be invalid. Perhaps the container image (tagged 3.14) is cached ? Yet this wouldn't affect the trial test runs since there is no persistent container registry (k8s cluster built each time) and yet they had the same results

planetf1 · 2022-12-17T08:54:11Z

The inmemory CTS on 3.14 (k8s/usual way) (which is using latest image) reports one exception:

                            "Unexpected Exception Exception : CTS test repository-entity-reference-copy-lifecycle caught exception AssertionFailureExc
eption from method saveEntityReferenceCopy whilst trying to save a reference copy of an entity of type Regulation.  Exception message was : Failed Ass
ertion:  reference entity retrieved with mappingProperties.Regulation.  Method was invoked with parameters: entity : EntityDetail{entityProperties=Ins
tanceProperties{propertyNames=java.util.HashMap$KeyIterator@5598f6b5, propertyCount=8, instanceProperties={summary=PrimitivePropertyValue{primitiveVal
ue=TestsummaryValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a
-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='n
ull', typeName='null'}, domainIdentifier=PrimitivePropertyValue{primitiveValue=42, primitiveDefCategory=PrimitiveDefCategory{code=5, name='int', javaC
lassName='java.lang.Integer', guid='7fc49104-fd3a-46c8-b6bf-f16b6074cd35'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Pr
imitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, qualifiedName=PrimitivePropertyValue{primitiveValue=TestqualifiedNa
meValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c
4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeNa
me='null'}, scope=PrimitivePropertyValue{primitiveValue=TestscopeValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassNam
e='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive'
, typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, jurisdiction=PrimitivePropertyValue{primitiveValue=TestjurisdictionValue, p
rimitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, insta
ncePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}
, description=PrimitivePropertyValue{primitiveValue=TestdescriptionValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassN
ame='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitiv
e', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, title=PrimitivePropertyValue{primitiveValue=TesttitleValue, primitiveDefC
ategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyC
ategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, priority=P
rimitivePropertyValue{primitiveValue=TestpriorityValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.Str
ing', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescriptio
n='A primitive type.'}, typeGUID='null', typeName='null'}}}, properties=InstanceProperties{propertyNames=java.util.HashMap$KeyIterator@68ab9e02, prope
rtyCount=8, instanceProperties={summary=PrimitivePropertyValue{primitiveValue=TestsummaryValue, primitiveDefCategory=PrimitiveDefCategory{code=11, nam
e='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode
=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, domainIdentifier=PrimitivePropertyValue{primitiveVa
lue=42, primitiveDefCategory=PrimitiveDefCategory{code=5, name='int', javaClassName='java.lang.Integer', guid='7fc49104-fd3a-46c8-b6bf-f16b6074cd35'},
 instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='
null'}, qualifiedName=PrimitivePropertyValue{primitiveValue=TestqualifiedNameValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string',
javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName
='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, scope=PrimitivePropertyValue{primitiveValue=TestscopeValue, pri
mitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instanc
ePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'},
jurisdiction=PrimitivePropertyValue{primitiveValue=TestjurisdictionValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassN
ame='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitiv
e', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}, description=PrimitivePropertyValue{primitiveValue=TestdescriptionValue, p
rimitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, insta
ncePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}
, title=PrimitivePropertyValue{primitiveValue=TesttitleValue, primitiveDefCategory=PrimitiveDefCategory{code=11, name='string', javaClassName='java.la
ng.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategory=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDesc
ription='A primitive type.'}, typeGUID='null', typeName='null'}, priority=PrimitivePropertyValue{primitiveValue=TestpriorityValue, primitiveDefCategor
y=PrimitiveDefCategory{code=11, name='string', javaClassName='java.lang.String', guid='b34a64b9-554a-42b1-8f8a-7d5c2339f9c4'}, instancePropertyCategor
y=InstancePropertyCategory{typeCode=1, typeName='Primitive', typeDescription='A primitive type.'}, typeGUID='null', typeName='null'}}}, classification
s=null, headerVersion=1, type=InstanceType{typeDefName='Regulation', typeDefCategory=TypeDefCategory{typeCode=6, typeName='EntityDef', typeDescription
='An object or concept of interest.'}, typeDefGUID='e3c4293d-8846-4500-b0c0-197d73aba8b0', typeDefVersion=1, typeDefDescription='Identifies a regulati
on related to data that must be supported.', typeDefDescriptionGUID='null', typeDefSuperTypes=[TypeDefLink{guid='c403c109-7b6b-48cd-8eee-df445b258b33'
, name='GovernanceDriver', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value
 equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}, TypeDefLink{guid='578a3500-9ad3-45fe-8ada-e4e9572c37c8', name='Governanc
eDefinition', status=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null', replacedByTypeName='null'}, TypeDefLink{guid='a32316b8-dc8c-48c5-b12b-71c1b2a080bf', name='Referenceable', status
=TypeDefStatus{ordinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedBy
TypeGUID='null', replacedByTypeName='null'}, TypeDefLink{guid='4e7761e8-3969-4627-8f40-bfe3cde85a1d', name='OpenMetadataRoot', status=TypeDefStatus{or
dinal=1, name='ActiveTypeDef', description='TypeDef available and in use.  This is the default value equivalent to null'}, replacedByTypeGUID='null',
replacedByTypeName='null'}], validStatusList=[InstanceStatus{ordinal=15, name='Active', description='The instance is approved and in use.'}, InstanceS
tatus{ordinal=99, name='Deleted', description='The instance has been deleted and is no longer available.'}], validInstanceProperties=[qualifiedName, a
dditionalProperties, summary, implications, domainIdentifier, outcomes, scope, domain, description, title, priority, results, jurisdiction]}, instance
ProvenanceType=InstanceProvenanceType{ordinal=4, name='Deregistered Repository', description='The instance comes from a metadata repository that used
to be a member of the one of the local repository's cohorts but it has been deregistered. The metadata collection id remains the same. If the reposito
ry rejoins the cohort then these elements can be refreshed from the rejoining repository.'}, metadataCollectionId='remotec5-7484-4d71-923b-afb48f8e57a
1', metadataCollectionName='remote-metadataCollection-not-TUT_MDR', replicatedBy='null', instanceLicense='null', status=InstanceStatus{ordinal=15, nam
e='Active', description='The instance is approved and in use.'}, createdBy='OMAGServer', updatedBy='null', maintainedBy=null, createTime=Fri Dec 16 19
:44:48 UTC 2022, updateTime=null, version=1, statusOnDelete=null, mappingProperties={integerMappingPropertyKey=12, stringMappingPropertyKey=stringMapp
ingPropertyValue}, instanceURL='null', GUID='remote14-38ea-4540-bd4a-86bae7e97a49', reIdentifiedFromGUID='null'}"

And 3474 Assertion failures, suggestion the overall test count showing failures is correct (whilst the profile results are not)

planetf1 · 2022-12-17T20:51:29Z

I reran the tests (k8s) :

tests run clean on 3.13
inmem passes on 3.14 ( (as it did 3 other times, albeit prior to the latest fix). Suggests a one-off/intermittent failure above)
graph FAILS on 3.14

planetf1 · 2022-12-19T09:47:31Z

Whilst graph consistently fails, I believe CTS failures for inmemory are inconsistent. To avoid more clutter I'll open up a new issue for inmemory

planetf1 · 2022-12-20T06:21:36Z

By reverting the header version fix, CTS graph now passes. PR opened for 3.14

It's possible the error was caused by conflict resolution, but also that it's genuine. @mandy-chessell your call whether we merge that main fix anyway and then run cts if clean in your environment with it? Or we can build a local image to test if needed

mandy-chessell · 2022-12-21T10:07:35Z

Unfortunately, the exception shown in comment #7148 (comment) is an error in the CTS. It is catching an assertion failure rather than an exception from Egeria.

The assertion failure it masks is probably the real problem. I will merge the current PR and retest with the fix to the exception management.

Fix errors in headers and entity proxies detected by the CTS (#7148)

planetf1 · 2022-12-21T14:32:21Z

If you want to try running the CTS pipelines (to avoid impact on local environment)

Will only work on merged updates
A new container image is made by the 'merge' pipeline at https://github.com/odpi/egeria/actions/workflows/merge-maven.yml. This typically takes an hour, but can take longer -- or fail (and must be restarted)
this image is posted to https://quay.io/repository/odpi/egeria?tab=tags - 'latest' is used for main.
Once the image is ready, the experimental cts github pipeline can be launched at https://github.com/planetf1/cts/actions/workflows/cts.yml by clicking on 'launch workflow'. It takes ~ 4.5 hours to cover our 3 repo connectors. (let me know if there are permission issues and you want to do this)
The output file can be downloaded (and we can figure out capture of additional info, or better interpretation of the results)

planetf1 · 2022-12-21T23:27:30Z

see CTS results after main -> https://github.com/planetf1/cts/actions/runs/3751230156 (though note inmem and xtdb are both showing some intermittency)

planetf1 · 2023-01-11T10:03:19Z

CTS graph errors continue to be reported by the pipeline above.

Additionally, I did a standalone test on a 3 x 16GB/8CPU k8s cluster, and had the same results for graph, which continues to fail in 'main' with:

  graph ~/src/cts/scripts/cts-analyze.py
              Metadata sharing MANDATORY_PROFILE             NOT_CONFORMANT [  70833 /   1062 ]
              Reference copies  OPTIONAL_PROFILE             NOT_CONFORMANT [   6950 /   1887 ]
          Metadata maintenance  OPTIONAL_PROFILE             NOT_CONFORMANT [  13329 /    825 ]
                 Dynamic types  OPTIONAL_PROFILE             UNKNOWN_STATUS [      0 /      0 ]
                 Graph queries  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [    528 /      0 ]
             Historical search  OPTIONAL_PROFILE      CONFORMANT_NO_SUPPORT [    531 /      0 ]
                Entity proxies  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2782 /      0 ]
       Soft-delete and restore  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2598 /      0 ]
                Undo an update  OPTIONAL_PROFILE      CONFORMANT_NO_SUPPORT [    406 /      0 ]
           Reidentify instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   2655 /      0 ]
               Retype instance  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  16365 /      0 ]
               Rehome instance  OPTIONAL_PROFILE             UNKNOWN_STATUS [      0 /      0 ]
                 Entity search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  62926 /      0 ]
           Relationship search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   8248 /      0 ]
        Entity advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [  44800 /      0 ]
  Relationship advanced search  OPTIONAL_PROFILE    CONFORMANT_FULL_SUPPORT [   9312 /      0 ]

FAIL [242263/3774]

planetf1 · 2023-03-01T11:57:59Z

CTS is clean for inmemory & has been clean for a while, so the issue is addressed
(xtdb is failing, but this is fixed in v4 of the xtdb connector, which is not yet available due to dependency work still needed)

planetf1 added bug Something isn't working triage New bug/issue which needs checking & assigning labels Nov 29, 2022

planetf1 added the conformance-testing Egeria conformance testing label Nov 29, 2022

planetf1 changed the title ~~[BUG] CTS failures~~ Release 4 only - [BUG] CTS failures Nov 29, 2022

planetf1 self-assigned this Dec 5, 2022

planetf1 removed the triage New bug/issue which needs checking & assigning label Dec 5, 2022

planetf1 changed the title ~~Release 4 only - [BUG] CTS failures~~ [BUG] CTS failures Dec 10, 2022

planetf1 mentioned this issue Dec 10, 2022

[RELEASE] Release 3.14 #7126

Closed

33 tasks

mandy-chessell mentioned this issue Dec 16, 2022

Fix errors in headers and entity proxies detected by the CTS (#7148) #7247

Merged

planetf1 mentioned this issue Dec 19, 2022

CTS failures - inmemory #7254

Closed

mandy-chessell added a commit that referenced this issue Dec 21, 2022

Merge pull request #7247 from mandy-chessell/egeria-code

f8d19ed

Fix errors in headers and entity proxies detected by the CTS (#7148)

mandy-chessell self-assigned this Dec 21, 2022

planetf1 mentioned this issue Jan 24, 2023

[RELEASE] Release 3.15 #7258

Closed

33 tasks

mandy-chessell mentioned this issue Jan 29, 2023

Add new tests to CTS #7328

Merged

planetf1 closed this as completed Mar 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] CTS failures #7148

[BUG] CTS failures #7148

planetf1 commented Nov 29, 2022 •

edited

Loading

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Dec 5, 2022

planetf1 commented Dec 5, 2022

planetf1 commented Dec 10, 2022 •

edited

Loading

planetf1 commented Dec 12, 2022

planetf1 commented Dec 13, 2022

mandy-chessell commented Dec 14, 2022 •

edited

Loading

mandy-chessell commented Dec 14, 2022 •

edited

Loading

planetf1 commented Dec 14, 2022 •

edited

Loading

mandy-chessell commented Dec 14, 2022

planetf1 commented Dec 14, 2022 •

edited

Loading

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 16, 2022

planetf1 commented Dec 17, 2022

planetf1 commented Dec 17, 2022

planetf1 commented Dec 17, 2022 •

edited

Loading

planetf1 commented Dec 17, 2022

planetf1 commented Dec 19, 2022

planetf1 commented Dec 20, 2022

mandy-chessell commented Dec 21, 2022

planetf1 commented Dec 21, 2022 •

edited

Loading

planetf1 commented Dec 21, 2022

planetf1 commented Jan 11, 2023

planetf1 commented Mar 1, 2023

[BUG] CTS failures #7148

[BUG] CTS failures #7148

Comments

planetf1 commented Nov 29, 2022 • edited Loading

Is there an existing issue for this?

Current Behavior

Expected Behavior

Steps To Reproduce

Environment

Any Further Information?

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Nov 29, 2022

planetf1 commented Dec 5, 2022

planetf1 commented Dec 5, 2022

planetf1 commented Dec 10, 2022 • edited Loading

planetf1 commented Dec 12, 2022

planetf1 commented Dec 13, 2022

mandy-chessell commented Dec 14, 2022 • edited Loading

mandy-chessell commented Dec 14, 2022 • edited Loading

planetf1 commented Dec 14, 2022 • edited Loading

mandy-chessell commented Dec 14, 2022

planetf1 commented Dec 14, 2022 • edited Loading

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 15, 2022

mandy-chessell commented Dec 16, 2022

planetf1 commented Dec 17, 2022

planetf1 commented Dec 17, 2022

planetf1 commented Dec 17, 2022 • edited Loading

planetf1 commented Dec 17, 2022

planetf1 commented Dec 19, 2022

planetf1 commented Dec 20, 2022

mandy-chessell commented Dec 21, 2022

planetf1 commented Dec 21, 2022 • edited Loading

planetf1 commented Dec 21, 2022

planetf1 commented Jan 11, 2023

planetf1 commented Mar 1, 2023

planetf1 commented Nov 29, 2022 •

edited

Loading

planetf1 commented Dec 10, 2022 •

edited

Loading

mandy-chessell commented Dec 14, 2022 •

edited

Loading

mandy-chessell commented Dec 14, 2022 •

edited

Loading

planetf1 commented Dec 14, 2022 •

edited

Loading

planetf1 commented Dec 14, 2022 •

edited

Loading

planetf1 commented Dec 17, 2022 •

edited

Loading

planetf1 commented Dec 21, 2022 •

edited

Loading