Add support for inmemory repository as a native provider in cts/pts #138
Assignees
Comments
planetf1
added a commit
to planetf1/egeria-charts
that referenced
this issue
Mar 28, 2022
Signed-off-by: Nigel Jones <[email protected]>
planetf1
added a commit
that referenced
this issue
Mar 28, 2022
#138 add support for in-memory-repository to cts/pts charts
planetf1
added a commit
to planetf1/egeria-charts
that referenced
this issue
Mar 28, 2022
…ucture issue Signed-off-by: Nigel Jones <[email protected]>
planetf1
added a commit
that referenced
this issue
Mar 28, 2022
#138 additional version update to trigger gh pages after infrastructu…
|
Further issue notes with ordering: Will re-order... |
|
Note the above is only setting the collection name, and as a 200 http response is still returned, configuration continues. Also opened up docs issue to suggest updating the documentation around the sequencing of these calls. |
|
After fix: |
planetf1
added a commit
to planetf1/egeria-charts
that referenced
this issue
Mar 29, 2022
Signed-off-by: Nigel Jones <[email protected]>
planetf1
added a commit
that referenced
this issue
Mar 29, 2022
#138 correct ordering of configuration api calls
planetf1
added a commit
to planetf1/egeria-charts
that referenced
this issue
Mar 29, 2022
Signed-off-by: Nigel Jones <[email protected]>
planetf1
added a commit
that referenced
this issue
Mar 29, 2022
#138 increment version for publishing
|
With these changes done, CTS is still not launching: Leave things to start... until..... This last message repeats, and the test never gets underway. This looks like a cohort/server initialization ordering issue. Need to check - is kafka working ok? What do the local and remote registrations on the server look like? |
|
Note full log from the startup logs on the platform is at https://gist.github.com/68291150111fb9aaca366ef220a9afe2 |
|
Checking cohort members, it seems neither member knows about the other, ie remote registrations: Both are registered: |
|
To clarify - on the hostnames - I'd done a |
|
Full cohort config is added to the gists in the log below: https://gist.github.com/8c1a80b2e22daafe83d163d8004befc5 |
|
From the overall server logs
We see both servers try and register This all looks good. Looks like a problem perhaps with consumer group/offset ? or topics/kafka backend... |
|
With strimzi we can get the kafka topics via the CRD ie: |
|
(logging debug process here for benefit of future attempts) Commands can also be issued direct on the strimzi kafka brokers - in this case I have just one. Nice to see the strimzi container has the client tools present: |
|
|
This is pretty much what I'd expect
The offsets are consistent with this having started from a clean slate. So why aren't the cohort members communicating? Why did the registration process not seem to work? Any ideas @mandy-chessell |
|
Checking topic in more detail: |
|
Taking a look at one of these topics (after a restart.. so the ids differ from above commands) This is all as I'd expect.. |
|
After more testing
Yet the same infrastructure (clean each time, fresh strimzi cluster), charts are identical. The only difference is that in the inmemcase I pass whilst in the graph case, the default value is used which is (from values.yaml): The config logs prove this is respected, and the value is only used for this expression in the configuration job: Timing, or config differences Next Steps
|
|
My only thought here is that most repos (Graph, XTDB, etc) will have some initialization time before they're "ready" and joining into the cohort -- I suspect that the in-memory repo's initialization time is negligible (basically instantaneous). Most likely, then, this is something akin to a race condition -- e.g. where the in-memory repository has finished its initialization just as / just before the CTS driver starts its wait loop (?) A simple test fix would be to add a "sleep" in between the steps where the CTS server (instance) is started and the TUT server (instance) is started and see if that resolve it? |
|
I agree, though our cohort design shouldn't allow for a gap so I think we do also need to understand where the issue is in cohort initialization (or indeed is specific to the cts workbench code in the server) I can add a sleep in the run - and / or maybe start tut first - that may give us reliability. If so I'll externalize so it's easy to revert back to the broken behaviour for future debugging |
|
I've left order as-is and added configurable delay. For in-mem it consistently works with 3s, fails with 2s - though this could well be quite specific to my infrastructure. I've set the default as 10s, which should be reasonable. The main focus of this issue is completed now. However the cohort config failure is a concern. |
|
With these changes, CTS inmem completed successfully with our typical profile results. |
1 task
|
CTS now working. Issue opened against egeria to understand any issues with the core. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently if the cts/pts charts are setup to use the 'native' providers and we set
then the cts (for example) will fail to initialize with
This is because the logic in cts/pts for native is
This condition should be extended to allow for the in-mem repository & perform the appropriate configuration
The text was updated successfully, but these errors were encountered: