Enable Oracle API to send logs to OpenSearch #386

RMCampos · 2023-08-02T12:30:36Z

Following same changes to Backend API, update the Oracle API service allowing it to send logs to OpenSearch.

Backend API PR: Feat/655 send backend logs opensearch #202
Backend API Issue: Send API logs to opensearch #203

Updates on October, 4th:
Pending items for this task

Changes pushed into Funbucks. Here: feat: files and changes for another spar service bcgov-nr/nr-funbucks#478
Test lambda parsing locally with apm and event stream processing

The text was updated successfully, but these errors were encountered:

mgaseta · 2023-08-21T13:11:27Z

Stealing this one for me 🥷 🦹

mgaseta · 2023-08-22T14:42:08Z

Moving to blocked for 2 reasons:

We need to talk with OneTeam about how to make one instance of the fluentbit deal with the logs of both the Backend API and the Oracle API.
I can't connect to BC Gov's VPN due an certificate issue, I'm already talking with Encora's IT team to solve this.

For now, all the changes are pushed to the working branch, they can be checked here: 7e5d2b4

SLDonnelly · 2023-09-06T20:39:02Z

@mbystedt @andrwils would either of you be able to help Matheus with the issue described above? Matheus can provide any additional info that you need

mbystedt · 2023-09-06T21:09:11Z

We need to talk with OneTeam about how to make one instance of the fluentbit deal with the logs of both the Backend API and the Oracle API.

I think we're missing some information about the ask here. Feel free to setup a meeting. One instance of Fluentbit can easily handle quite a number of different sources of logs. There's no difference between having a single input source or 10. The only thing that becomes complicated is tags. We came up with a couple strategies for keeping tags straight in Funbucks for the on-premise that we can share.

mgaseta · 2023-09-06T21:52:25Z

Sorry, that was already solved, I forgot to update the ticket... The situation right now is that we can't see the generated logs on opensearch, we just need to get some directions on how to make it work! You can check the changes we made on this PR, which is also linked to this issue. Thanks!

mbystedt · 2023-09-06T22:34:26Z

@mgaseta Well, now I know who's responsible for the 8000+ errors triggering our alerts. This change should be rolled back until the Fluentbit configuration is fixed.

I can take a look at the output from FluentBit. But, it'll be much quicker if you just use the public docker image (ghcr.io/bcdevops/nr-apm-stack-lambda:main) that can be used to test your output. Unless it's exceedingly obvious, that's what I will be doing, anyway. Docs are here: https://bcdevops.github.io/nr-apm-stack/#/testing

I'll update this with the docker command. podman run --rm -p 3000:3000 ghcr.io/bcdevops/nr-apm-stack-lambda:main

Since it's not a Funbucks template, I have no way to generate a local FluentBit setup. You're on your own with the configuration unless you can provide that local equivalent. Your pipeline to getting new logs to production should have a way to generate a local FluentBit setup. Otherwise, you'll be wasting a lot of time copying and pasting and modifying things manually.

Edit: Fixed the image location

RMCampos · 2023-09-07T13:48:05Z

Hi @mbystedt so sorry about all of those errors. We forgot to delete the PR deployment, since that was only a preview. Now is gone.

There should be a Funbucks template, though. I'll push a commit and open a PR on Funbucks repo. Very likely we'll need some help to do those changes, but I think we can use the PR comments section :)

Once we have Funbucks in place, we can work with nr-apm-stack-lambda.

Thanks for the help, as always.

cc: @mgaseta @SLDonnelly

mbystedt · 2023-09-07T15:29:38Z

@RMCampos Excellent. Once there's that template PR in Funbucks, if you give us a sample then we'll be able to have a better idea what is the problem. The errors are about it not having an index. So, it's likely a fingerprint issue.

Let us know if that lambda container works for you. We moved all the integration docs to https://bcdevops.github.io/nr-apm-stack. Welcome any thoughts you have. Feel free to do a PR if you want to. The files in the docs folder of the apm-stack repo.

SPAR Issue bcgov/nr-spar#386 These are some of changes required to get another SPAR service sending logs to OpenSearch. Note that some questions were made as comments, we could use comments and reviews to get that working.

Issue #478 Also related to SPAR issue bcgov/nr-spar#386 This change restore all SPAR Postgres files to its previous state which are the same as the ones in the main branch.

Issue #478 Also related to SPAR issue bcgov/nr-spar#386 These files are related to the second SPAR backend service called Oracle API which is responsible for fetching information from Oracle THE database. Having this service integrated with OpenSearch will give SPAR a way of seeing all requests made, not only the new Postgres API one.

* feat: files and changes for another spar service SPAR Issue bcgov/nr-spar#386 These are some of changes required to get another SPAR service sending logs to OpenSearch. Note that some questions were made as comments, we could use comments and reviews to get that working. * chore: get spar postgres template files back Issue #478 Also related to SPAR issue bcgov/nr-spar#386 This change restore all SPAR Postgres files to its previous state which are the same as the ones in the main branch. * feat: add spar oracle service files Issue #478 Also related to SPAR issue bcgov/nr-spar#386 These files are related to the second SPAR backend service called Oracle API which is responsible for fetching information from Oracle THE database. Having this service integrated with OpenSearch will give SPAR a way of seeing all requests made, not only the new Postgres API one.

issue #386

* feat: creating and adjusting scripts * feat: creating new config files for fluentbit * feat: adjusting scripts * fix: fixing comment * fix: adjusting property * fix: fixing fluentbit deploy path on the workflows * feat: scape double quotes for file logging issue #386 * feat: update log patterns and logging levels issue #386 * ci: add opensearch oracle-api environment flag issue #386 * feat: add oracle-api config files for fluentbit issue #386 --------- Co-authored-by: Ricardo Campos <[email protected]> Co-authored-by: Craig Yu <[email protected]>

* chore: improve docs and oracle development (#534) * docs: update readme file with latest notes No issue-related. This change updates the repo readme file which contains information of the existing services and applications. * docs: improve documentation in general No issue * feat: allows oracle api to run and develop from docker No issue * test: vegetation code test improvements * chore: fix checkstyle * ci: add env var for oracle ca certs file * feat: remove jssecacerts file and add script for fetching it --------- Co-authored-by: Derek Roberts <[email protected]> * fix: wrong keycloak realm url for prod (#580) * fix: wrong keycloak realm url for prod issue #553 * ci: disable triggers temporarily * fix: spring pagination is zero-based (#584) * fix: spring pagination is zero-based issue #582 * fix: get triggers back wrongly merged in a previous pr * fix: default page index value for swagger when finding seedlots by user issue #582 * feat: backend and oracle decode fam cognito token issue #481 * [Snyk] Security upgrade axios from 1.4.0 to 1.6.0 (#592) * fix: frontend/package.json to reduce vulnerabilities The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-AXIOS-6032459 * package-lock.json --------- Co-authored-by: snyk-bot <[email protected]> Co-authored-by: Derek Roberts <[email protected]> * Update babel deps for subdep vulterability (#599) * Bump vitest for subdep get-func-name vulerability (#601) * Use override to bring postcss up to patched ver (#606) * Feat/386 enable oracle api to send logs to opensearch (#436) * feat: creating and adjusting scripts * feat: creating new config files for fluentbit * feat: adjusting scripts * fix: fixing comment * fix: adjusting property * fix: fixing fluentbit deploy path on the workflows * feat: scape double quotes for file logging issue #386 * feat: update log patterns and logging levels issue #386 * ci: add opensearch oracle-api environment flag issue #386 * feat: add oracle-api config files for fluentbit issue #386 --------- Co-authored-by: Ricardo Campos <[email protected]> Co-authored-by: Craig Yu <[email protected]> * feat: add user id provided by the idp and email * chore: remove comment with real name issue #481 * chore: remove comment with real information issue #481 * ci: add build arg for backend issue #481 * ci: add build arg for backend round 2 issue #481 * ci: add build arg for backend issue #481 * feat: add default aws cognito issuer uri issue #481 * ci: add missing env var to oracle api issue #481 * ci: fix env var for postgres and oracle issue #481

* Feat/480 replace keycloak by fam for authentication on fe (#578) * feat: replace keycloak by fam Issue #480 These changes are the first changes to get FAM working with SPAR since FAM is the preferred authentication library for the ministry applications. * feat: remove context not required anymore Issue #480 This change removes the Context Authentication file created when using keycloak, not needed anymore, because now there's an authentication service created that will handle everything. * feat: fam working locally in the front end Issue #480 This change updates files in the front end in order of getting it working with FAM and AWS Cognito authentication. Note that this is a working in progress, which means you may find some console log. * feat: disable authentication in the back end Issue #480 This commit update the back end service to remove all authentication and authorization handling for now, since it will be tackled in a future task issue #481. * fix: wrong docker compose yml identation Issue #480 Simply fix the docker-compose.yml file identation. * feat: disable backend security Issue #480 This change disables the security in the back end REST api and allows actuator endpoints to be accessed. * feat: add context provider for a better usage Issue #480 This change get back all context-related files and components to work closely with what we have for keycloak. * feat: create header between FE and BE Issue #480 A simple header was created to enable users to keep using SPAR client with usual activities until we get #481 finished. * test: fix user authentication helper test Issue #480 Fix tests and javadoc broken checkstyle. * test: split context and provider and fix land test Issue #480 Having context and provider splited into two files make it possible to mock values for testing components. * tests: get all tests passing related to auth Issue #480 Now all tests are working and passing. There's still issues related to favourite activities mock and morage js, though. * feat: remove old use auth hook created Issue #480 Remove use auth and replace it by use context with the authentication context name. * feat: add runtime hint for http servlet request Issue #480 Runtime Hints required for graal vm and cloud native. * chore: add javadoc comments Issue #480 * feat: add runtime hints for cloud native Issue #480 * ci: add new env vars to workflows Issue #480 Yep, just that. * feat: prevent access from missing env var Issue #480 * chore: fix warnings and issues raised by vscode Issue #480 * test: add user authentication test for temporary case * VITE_AWS_DOMAIN * VITE_AWS_DOMAIN try 2 * VITE_AWS_DOMAIN try 3 * chore: remove unused code issue #480 * chore: remove all keycloak related references issue #480 * chore: improve vite init global issue #480 * ci: crated docker compose test file and fix dockerfiles issue #480 * feat: improve login redirects for a better ux issue #480 * feat: add try catch on signout issue #480 * feat: remove await from logout * Test heap size builds * Dial down max-old-space-size * feat: add refresh token function issue #480 * feat: get cypress working with FAM issue #480 * chore: fix authentication issues on oracle-api * feat: improve authentication for business bce-id issue #480 * test: get cypress working with fam issue #480 * test: update cypress test workflow to use oracle dev issue #480 * chore: remove uneeded variable from landing page * fix: refactor routing, fix cypress, housekeeping * fix: remove code smells * fix: change sonar exclusion * feat: sign out feature redirect issue issue #480 * test: fix cypress test user name issue #480 * fix: remove try catch blocks * fix: remove role req --------- Co-authored-by: Ricardo Campos <[email protected]> Co-authored-by: Derek Roberts <[email protected]> * feat: replace jwt decode and roles to use fam authorization on be (#585) * chore: improve docs and oracle development (#534) * docs: update readme file with latest notes No issue-related. This change updates the repo readme file which contains information of the existing services and applications. * docs: improve documentation in general No issue * feat: allows oracle api to run and develop from docker No issue * test: vegetation code test improvements * chore: fix checkstyle * ci: add env var for oracle ca certs file * feat: remove jssecacerts file and add script for fetching it --------- Co-authored-by: Derek Roberts <[email protected]> * fix: wrong keycloak realm url for prod (#580) * fix: wrong keycloak realm url for prod issue #553 * ci: disable triggers temporarily * fix: spring pagination is zero-based (#584) * fix: spring pagination is zero-based issue #582 * fix: get triggers back wrongly merged in a previous pr * fix: default page index value for swagger when finding seedlots by user issue #582 * feat: backend and oracle decode fam cognito token issue #481 * [Snyk] Security upgrade axios from 1.4.0 to 1.6.0 (#592) * fix: frontend/package.json to reduce vulnerabilities The following vulnerabilities are fixed with an upgrade: - https://snyk.io/vuln/SNYK-JS-AXIOS-6032459 * package-lock.json --------- Co-authored-by: snyk-bot <[email protected]> Co-authored-by: Derek Roberts <[email protected]> * Update babel deps for subdep vulterability (#599) * Bump vitest for subdep get-func-name vulerability (#601) * Use override to bring postcss up to patched ver (#606) * Feat/386 enable oracle api to send logs to opensearch (#436) * feat: creating and adjusting scripts * feat: creating new config files for fluentbit * feat: adjusting scripts * fix: fixing comment * fix: adjusting property * fix: fixing fluentbit deploy path on the workflows * feat: scape double quotes for file logging issue #386 * feat: update log patterns and logging levels issue #386 * ci: add opensearch oracle-api environment flag issue #386 * feat: add oracle-api config files for fluentbit issue #386 --------- Co-authored-by: Ricardo Campos <[email protected]> Co-authored-by: Craig Yu <[email protected]> * feat: add user id provided by the idp and email * chore: remove comment with real name issue #481 * chore: remove comment with real information issue #481 * ci: add build arg for backend issue #481 * ci: add build arg for backend round 2 issue #481 * ci: add build arg for backend issue #481 * feat: add default aws cognito issuer uri issue #481 * ci: add missing env var to oracle api issue #481 * ci: fix env var for postgres and oracle issue #481 --------- Co-authored-by: Ricardo Campos <[email protected]> Co-authored-by: Derek Roberts <[email protected]>