Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Send API logs to opensearch #203

Closed
1 of 2 tasks
RMCampos opened this issue Jun 9, 2023 · 6 comments · Fixed by #202
Closed
1 of 2 tasks

Send API logs to opensearch #203

RMCampos opened this issue Jun 9, 2023 · 6 comments · Fixed by #202
Assignees
@RMCampos
Labels
back-end back-end related task devops devops related task feature implement or replace a feature

Comments

@RMCampos
Copy link
Collaborator

RMCampos commented Jun 9, 2023 •
edited
Loading

Describe the task
Enable SPAR backend Services to send application logs to OpenSearch.

This issue is linked to Jira task FSADT2-655

Acceptance Criteria

  • Send Backend-API logs to SPAR OpenSearch Dashboard
  • Send Oracle-API logs to SPAR OpenSearch Dashboard

Additional context
@RMCampos RMCampos self-assigned this Jun 9, 2023
@RMCampos
Copy link
Collaborator Author

RMCampos commented Jun 9, 2023

PR #202 is related to this issue!

RMCampos added a commit that referenced this issue Jun 9, 2023
@RMCampos
commit to link issue #203 to PR
6a2ce0e
@craigyu craigyu linked a pull request Jun 13, 2023 that will close this issue
Feat/655 send backend logs opensearch #202
Merged
9 tasks
@RMCampos
Copy link
Collaborator Author

Blocked due to missing AWS authentication keys and secrets. Also we need to test it locally, with Funbucks, before shipping this up.

@RMCampos
Copy link
Collaborator Author

Unblocked after a meeting with OneTeam on July, 18th. Grace provided feedback on the log parsing and some issues that need to be fixed. Those are:

  • Do not send messages with double quotes, only single quotes.
  • Change event.dataset to application.log.utc
  • Change path files, both DB and Log Files
  • Remove items that are already being sent on the logs
  • Add Add service.name spar_postgres_api to the filters
  • Add Add service.type postgres_api to the filters

@RMCampos RMCampos mentioned this issue Jul 19, 2023
Merged
9 tasks
@RMCampos RMCampos added back-end back-end related task devops devops related task feature implement or replace a feature labels Jul 27, 2023
@RMCampos
Copy link
Collaborator Author

RMCampos commented Aug 1, 2023

All related pain points and troubleshooting notes were added here: https://github.com/bcgov/nr-spar/wiki/Observability-with-OpenSearch-from-OpenShift#troubleshooting.

Finally this task is finished.

Image

@RMCampos RMCampos mentioned this issue Aug 2, 2023
Closed
2 tasks
@RMCampos
Copy link
Collaborator Author

RMCampos commented Aug 2, 2023

Created a separate task for Oracle API: #386

@RMCampos
Copy link
Collaborator Author

RMCampos commented Aug 3, 2023

Some updates were made in the filter and parser files. Related PR on Funbucks: bcgov-nr/nr-funbucks#463

RMCampos added a commit that referenced this issue Aug 4, 2023
@RMCampos
Feat/655 send backend logs opensearch (#202)
cf8f8af 
* feat: add logback configuration start

* fix: cpu and memory values according to main

* feat: add required ecs fields

* feat: add more required fields to match ecs

* feat: right ecs version and spar team email

* commit to link issue #203 to PR

* fix: tests and missing files

* fix: commit missing files

* tests: add testes to docker

* ci: fix ci file

* ci: remove it parameter from docker run

* ci: fix secrets expression to vars

* ci: fix fluent-bit config-map deployment

* tests: run tests on docker with sonarcloud params

* ci: add missing parameter and fix sonar on docker tests

* fix: add user to dockerfile tests and decrease pvc size for logs

* ci: fix user on dockerfile tests

* ci: fix dockerfile tests

* ci: remove user and get back to working version

* fix: fluent-bit image version and secrets

* fix: deployment labels

* ci: fix missing parameter value and deployments name and labels

* feat: add fluentbit configuration and volumes

* ci: fix volume mounts and pvc for logs

* ci: fix wrong deployment file

* ci: fix wrong indentation of backend openshift deployment

* ci: fix volume name

* ci: fix parameters of backend deployment config file

* ci: fix env reference for aws secrets

* ci: add backend and frontend to trigger database deployments

* ci: skipping trigger until we get the workflow working

* ci: fix missing env vars

* ci: for init deployment - for now

* feat: add kinesis vars to missing fluentbit configmap

* ci: add fluentbit to matrix strategies test and prod

* ci: fix fluentbit config map parameters

* ci: fix secrets workflow

* ci: kinesis parameters not required

* ci: change fluentbit to another file

* ci: fix wrong syntax file

* ci: fix imageChangeParams for fluentbit

* ci: fix missing volume for backend

* feat: improve logging system

* chore: remove unused code and fix order on yaml files

* feat: improve logs to file only for production and env names

* ci: fix missing backend env var

* ci: fix wrong parameter in upper case

* feat: split fluentbit deploy into separate job

* ci: fix zone parameter missing

* ci: fix fluentbit volumes

* ci: empty commit to trigger ci

* ci: remove pvc from fluentbit configmap

* ci: get configmap pvc back

* ci: fix config volume names and remove pvc

* ci: fix fluentbit config map output

* ci: attempt to figure it out configmap vars

* ci: change fluentbit aws role name and stream

* ci: change fluentbit db path

* ci: empty commit to trigger ci

* chore: empty lines cleanups

* feat: use application.log.utc for event.dataset

* feat: change exceptions to not print messages with quotes on console

* feat: fluentbit configurations

* feat: aws access parameters

* feat: fluentbit debug image

* ci: remove readiness and liveness probes

* feat: update env var for opensearch envs

* ci: bound volume mount for the logs

* ci: fix volume claim name for backend and fluentbit

* feat: fix profile for be compatible with cloud native

* fix: time format parser pattern

* ci: fluent bit log format set to debug

* ci: increase pvc capacity for the logs

* ci: empty commit to trigger ci

* ci: fix permissions for the container and logback profiles

* ci: fix file permission syntax

* ci: log mode for fluentbit and spring profiles for logback

* feat: add fluent bit hostname envvar and new filter option

* fix: right envvar name for hostname

* chore: restore unrelated file changes

* chore: restore files not related to this change

* tests: update test profile for not generating file logs

* chore: restoring uneeded changes and improving organisation properties

* chore: improving organisation properties
@RMCampos RMCampos mentioned this issue Aug 4, 2023
Merged
9 tasks
DerekRoberts pushed a commit that referenced this issue May 13, 2024
@RMCampos @DerekRoberts
Feat/655 send backend logs opensearch (#202)
b86e448 
* feat: add logback configuration start

* fix: cpu and memory values according to main

* feat: add required ecs fields

* feat: add more required fields to match ecs

* feat: right ecs version and spar team email

* commit to link issue #203 to PR

* fix: tests and missing files

* fix: commit missing files

* tests: add testes to docker

* ci: fix ci file

* ci: remove it parameter from docker run

* ci: fix secrets expression to vars

* ci: fix fluent-bit config-map deployment

* tests: run tests on docker with sonarcloud params

* ci: add missing parameter and fix sonar on docker tests

* fix: add user to dockerfile tests and decrease pvc size for logs

* ci: fix user on dockerfile tests

* ci: fix dockerfile tests

* ci: remove user and get back to working version

* fix: fluent-bit image version and secrets

* fix: deployment labels

* ci: fix missing parameter value and deployments name and labels

* feat: add fluentbit configuration and volumes

* ci: fix volume mounts and pvc for logs

* ci: fix wrong deployment file

* ci: fix wrong indentation of backend openshift deployment

* ci: fix volume name

* ci: fix parameters of backend deployment config file

* ci: fix env reference for aws secrets

* ci: add backend and frontend to trigger database deployments

* ci: skipping trigger until we get the workflow working

* ci: fix missing env vars

* ci: for init deployment - for now

* feat: add kinesis vars to missing fluentbit configmap

* ci: add fluentbit to matrix strategies test and prod

* ci: fix fluentbit config map parameters

* ci: fix secrets workflow

* ci: kinesis parameters not required

* ci: change fluentbit to another file

* ci: fix wrong syntax file

* ci: fix imageChangeParams for fluentbit

* ci: fix missing volume for backend

* feat: improve logging system

* chore: remove unused code and fix order on yaml files

* feat: improve logs to file only for production and env names

* ci: fix missing backend env var

* ci: fix wrong parameter in upper case

* feat: split fluentbit deploy into separate job

* ci: fix zone parameter missing

* ci: fix fluentbit volumes

* ci: empty commit to trigger ci

* ci: remove pvc from fluentbit configmap

* ci: get configmap pvc back

* ci: fix config volume names and remove pvc

* ci: fix fluentbit config map output

* ci: attempt to figure it out configmap vars

* ci: change fluentbit aws role name and stream

* ci: change fluentbit db path

* ci: empty commit to trigger ci

* chore: empty lines cleanups

* feat: use application.log.utc for event.dataset

* feat: change exceptions to not print messages with quotes on console

* feat: fluentbit configurations

* feat: aws access parameters

* feat: fluentbit debug image

* ci: remove readiness and liveness probes

* feat: update env var for opensearch envs

* ci: bound volume mount for the logs

* ci: fix volume claim name for backend and fluentbit

* feat: fix profile for be compatible with cloud native

* fix: time format parser pattern

* ci: fluent bit log format set to debug

* ci: increase pvc capacity for the logs

* ci: empty commit to trigger ci

* ci: fix permissions for the container and logback profiles

* ci: fix file permission syntax

* ci: log mode for fluentbit and spring profiles for logback

* feat: add fluent bit hostname envvar and new filter option

* fix: right envvar name for hostname

* chore: restore unrelated file changes

* chore: restore files not related to this change

* tests: update test profile for not generating file logs

* chore: restoring uneeded changes and improving organisation properties

* chore: improving organisation properties
DerekRoberts pushed a commit that referenced this issue May 14, 2024
@RMCampos @DerekRoberts
Feat/655 send backend logs opensearch (#202)
99581a8 
* feat: add logback configuration start

* fix: cpu and memory values according to main

* feat: add required ecs fields

* feat: add more required fields to match ecs

* feat: right ecs version and spar team email

* commit to link issue #203 to PR

* fix: tests and missing files

* fix: commit missing files

* tests: add testes to docker

* ci: fix ci file

* ci: remove it parameter from docker run

* ci: fix secrets expression to vars

* ci: fix fluent-bit config-map deployment

* tests: run tests on docker with sonarcloud params

* ci: add missing parameter and fix sonar on docker tests

* fix: add user to dockerfile tests and decrease pvc size for logs

* ci: fix user on dockerfile tests

* ci: fix dockerfile tests

* ci: remove user and get back to working version

* fix: fluent-bit image version and secrets

* fix: deployment labels

* ci: fix missing parameter value and deployments name and labels

* feat: add fluentbit configuration and volumes

* ci: fix volume mounts and pvc for logs

* ci: fix wrong deployment file

* ci: fix wrong indentation of backend openshift deployment

* ci: fix volume name

* ci: fix parameters of backend deployment config file

* ci: fix env reference for aws secrets

* ci: add backend and frontend to trigger database deployments

* ci: skipping trigger until we get the workflow working

* ci: fix missing env vars

* ci: for init deployment - for now

* feat: add kinesis vars to missing fluentbit configmap

* ci: add fluentbit to matrix strategies test and prod

* ci: fix fluentbit config map parameters

* ci: fix secrets workflow

* ci: kinesis parameters not required

* ci: change fluentbit to another file

* ci: fix wrong syntax file

* ci: fix imageChangeParams for fluentbit

* ci: fix missing volume for backend

* feat: improve logging system

* chore: remove unused code and fix order on yaml files

* feat: improve logs to file only for production and env names

* ci: fix missing backend env var

* ci: fix wrong parameter in upper case

* feat: split fluentbit deploy into separate job

* ci: fix zone parameter missing

* ci: fix fluentbit volumes

* ci: empty commit to trigger ci

* ci: remove pvc from fluentbit configmap

* ci: get configmap pvc back

* ci: fix config volume names and remove pvc

* ci: fix fluentbit config map output

* ci: attempt to figure it out configmap vars

* ci: change fluentbit aws role name and stream

* ci: change fluentbit db path

* ci: empty commit to trigger ci

* chore: empty lines cleanups

* feat: use application.log.utc for event.dataset

* feat: change exceptions to not print messages with quotes on console

* feat: fluentbit configurations

* feat: aws access parameters

* feat: fluentbit debug image

* ci: remove readiness and liveness probes

* feat: update env var for opensearch envs

* ci: bound volume mount for the logs

* ci: fix volume claim name for backend and fluentbit

* feat: fix profile for be compatible with cloud native

* fix: time format parser pattern

* ci: fluent bit log format set to debug

* ci: increase pvc capacity for the logs

* ci: empty commit to trigger ci

* ci: fix permissions for the container and logback profiles

* ci: fix file permission syntax

* ci: log mode for fluentbit and spring profiles for logback

* feat: add fluent bit hostname envvar and new filter option

* fix: right envvar name for hostname

* chore: restore unrelated file changes

* chore: restore files not related to this change

* tests: update test profile for not generating file logs

* chore: restoring uneeded changes and improving organisation properties

* chore: improving organisation properties
DerekRoberts pushed a commit that referenced this issue May 14, 2024
@RMCampos @DerekRoberts
Feat/655 send backend logs opensearch (#202)
7956b63 
* feat: add logback configuration start

* fix: cpu and memory values according to main

* feat: add required ecs fields

* feat: add more required fields to match ecs

* feat: right ecs version and spar team email

* commit to link issue #203 to PR

* fix: tests and missing files

* fix: commit missing files

* tests: add testes to docker

* ci: fix ci file

* ci: remove it parameter from docker run

* ci: fix secrets expression to vars

* ci: fix fluent-bit config-map deployment

* tests: run tests on docker with sonarcloud params

* ci: add missing parameter and fix sonar on docker tests

* fix: add user to dockerfile tests and decrease pvc size for logs

* ci: fix user on dockerfile tests

* ci: fix dockerfile tests

* ci: remove user and get back to working version

* fix: fluent-bit image version and secrets

* fix: deployment labels

* ci: fix missing parameter value and deployments name and labels

* feat: add fluentbit configuration and volumes

* ci: fix volume mounts and pvc for logs

* ci: fix wrong deployment file

* ci: fix wrong indentation of backend openshift deployment

* ci: fix volume name

* ci: fix parameters of backend deployment config file

* ci: fix env reference for aws secrets

* ci: add backend and frontend to trigger database deployments

* ci: skipping trigger until we get the workflow working

* ci: fix missing env vars

* ci: for init deployment - for now

* feat: add kinesis vars to missing fluentbit configmap

* ci: add fluentbit to matrix strategies test and prod

* ci: fix fluentbit config map parameters

* ci: fix secrets workflow

* ci: kinesis parameters not required

* ci: change fluentbit to another file

* ci: fix wrong syntax file

* ci: fix imageChangeParams for fluentbit

* ci: fix missing volume for backend

* feat: improve logging system

* chore: remove unused code and fix order on yaml files

* feat: improve logs to file only for production and env names

* ci: fix missing backend env var

* ci: fix wrong parameter in upper case

* feat: split fluentbit deploy into separate job

* ci: fix zone parameter missing

* ci: fix fluentbit volumes

* ci: empty commit to trigger ci

* ci: remove pvc from fluentbit configmap

* ci: get configmap pvc back

* ci: fix config volume names and remove pvc

* ci: fix fluentbit config map output

* ci: attempt to figure it out configmap vars

* ci: change fluentbit aws role name and stream

* ci: change fluentbit db path

* ci: empty commit to trigger ci

* chore: empty lines cleanups

* feat: use application.log.utc for event.dataset

* feat: change exceptions to not print messages with quotes on console

* feat: fluentbit configurations

* feat: aws access parameters

* feat: fluentbit debug image

* ci: remove readiness and liveness probes

* feat: update env var for opensearch envs

* ci: bound volume mount for the logs

* ci: fix volume claim name for backend and fluentbit

* feat: fix profile for be compatible with cloud native

* fix: time format parser pattern

* ci: fluent bit log format set to debug

* ci: increase pvc capacity for the logs

* ci: empty commit to trigger ci

* ci: fix permissions for the container and logback profiles

* ci: fix file permission syntax

* ci: log mode for fluentbit and spring profiles for logback

* feat: add fluent bit hostname envvar and new filter option

* fix: right envvar name for hostname

* chore: restore unrelated file changes

* chore: restore files not related to this change

* tests: update test profile for not generating file logs

* chore: restoring uneeded changes and improving organisation properties

* chore: improving organisation properties
DerekRoberts pushed a commit that referenced this issue May 14, 2024
@RMCampos @DerekRoberts
Feat/655 send backend logs opensearch (#202)
53b9ece 
* feat: add logback configuration start

* fix: cpu and memory values according to main

* feat: add required ecs fields

* feat: add more required fields to match ecs

* feat: right ecs version and spar team email

* commit to link issue #203 to PR

* fix: tests and missing files

* fix: commit missing files

* tests: add testes to docker

* ci: fix ci file

* ci: remove it parameter from docker run

* ci: fix secrets expression to vars

* ci: fix fluent-bit config-map deployment

* tests: run tests on docker with sonarcloud params

* ci: add missing parameter and fix sonar on docker tests

* fix: add user to dockerfile tests and decrease pvc size for logs

* ci: fix user on dockerfile tests

* ci: fix dockerfile tests

* ci: remove user and get back to working version

* fix: fluent-bit image version and secrets

* fix: deployment labels

* ci: fix missing parameter value and deployments name and labels

* feat: add fluentbit configuration and volumes

* ci: fix volume mounts and pvc for logs

* ci: fix wrong deployment file

* ci: fix wrong indentation of backend openshift deployment

* ci: fix volume name

* ci: fix parameters of backend deployment config file

* ci: fix env reference for aws secrets

* ci: add backend and frontend to trigger database deployments

* ci: skipping trigger until we get the workflow working

* ci: fix missing env vars

* ci: for init deployment - for now

* feat: add kinesis vars to missing fluentbit configmap

* ci: add fluentbit to matrix strategies test and prod

* ci: fix fluentbit config map parameters

* ci: fix secrets workflow

* ci: kinesis parameters not required

* ci: change fluentbit to another file

* ci: fix wrong syntax file

* ci: fix imageChangeParams for fluentbit

* ci: fix missing volume for backend

* feat: improve logging system

* chore: remove unused code and fix order on yaml files

* feat: improve logs to file only for production and env names

* ci: fix missing backend env var

* ci: fix wrong parameter in upper case

* feat: split fluentbit deploy into separate job

* ci: fix zone parameter missing

* ci: fix fluentbit volumes

* ci: empty commit to trigger ci

* ci: remove pvc from fluentbit configmap

* ci: get configmap pvc back

* ci: fix config volume names and remove pvc

* ci: fix fluentbit config map output

* ci: attempt to figure it out configmap vars

* ci: change fluentbit aws role name and stream

* ci: change fluentbit db path

* ci: empty commit to trigger ci

* chore: empty lines cleanups

* feat: use application.log.utc for event.dataset

* feat: change exceptions to not print messages with quotes on console

* feat: fluentbit configurations

* feat: aws access parameters

* feat: fluentbit debug image

* ci: remove readiness and liveness probes

* feat: update env var for opensearch envs

* ci: bound volume mount for the logs

* ci: fix volume claim name for backend and fluentbit

* feat: fix profile for be compatible with cloud native

* fix: time format parser pattern

* ci: fluent bit log format set to debug

* ci: increase pvc capacity for the logs

* ci: empty commit to trigger ci

* ci: fix permissions for the container and logback profiles

* ci: fix file permission syntax

* ci: log mode for fluentbit and spring profiles for logback

* feat: add fluent bit hostname envvar and new filter option

* fix: right envvar name for hostname

* chore: restore unrelated file changes

* chore: restore files not related to this change

* tests: update test profile for not generating file logs

* chore: restoring uneeded changes and improving organisation properties

* chore: improving organisation properties
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RMCampos RMCampos

Labels
back-end back-end related task devops devops related task feature implement or replace a feature
Projects
SPAR Sprints | Goal: Improving the SPAR user experience; CONSEP seed testing activities usability testing; finish lo-fi wireframes for additional seed testing activities (family lot, reporting)
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Feat/655 send backend logs opensearch bcgov/nr-spar
2 participants
@RMCampos @craigyu