NRM Fider

OpenShift templates for customized image of Fider, used within Natural Resources Ministries and ready for deployment on OpenShift. Fider is an open-source Go application with a PostgreSQL relational database for persistent data.

BC Gov's SMTP configuration does not support SMTPS, but Fider's SMTP Go library enforces smtps, so we have forked the code and removed the SMTPS authn from our deployment. This means that this image can ONLY be deployed within the BC Gov network, as only devices on the BC Gov network can see the standard BC Gov SMTP Server.

Prerequisites

Network Security Policies are in place; Aporeto is no longer supported on OCP, so see https://developer.gov.bc.ca/Openshift-Useful-Pro-Tips

For builds:

Administrator access to an Openshift Project namespace

Once built, this image may be deployed to a separate namespace with the appropriate system:image-puller role.

For deployments:

Administrator access to an Openshift Project namespace
the oc CLI tool, installed on your local workstation
access to this public GitHub Repo

Once deployed, any visitors to the site will require a modern web browser (e.g. Chrome, Edge, FF, Opera etc.).

Files

OpenShift Fider app template for Fider application
OpenShift Database service template for PostgreSQL Database, acting as the datastore for the Fider application

Build

Custom Image

To create an image stream using this forked code (replace <tools-namespace> with your -tools project namespace).

oc -n <tools-namespace> create istag fider-bcgov:latest

oc -n <tools-namespace> process -f ci/openshift/fider-bcgov.bc.yaml | oc -n <tools-namespace> apply -f -

oc -n <tools-namespace> start-build nrm-feedback

Tag with the correct release version, matching the major.minor release tag at the source repo. For example:

oc -n <tools-namespace> tag fider-bcgov:latest fider-bcgov:0.21.1

NOTE: To update our Fider image, we would update the submodule (e.g. 0.21.1) and then tag this build as latest, and plan for a re-deploy using the newer image.

git submodule update --init --recursive

Out-of-the-box Image

If and when we solve the SMTPS issue, we should build directly off the Fider image, referencing https://github.com/getfider/fider in the deployment, and no longer use custom builds.

Deploy

Database Deployment

Deploy the DB using the correct FEEDBACK_NAME parameter (e.g. an acronym that is prefixed to fider):

oc -n <project> new-app --file=./ci/openshift/postgresql.dc.yaml -p FEEDBACK_NAME=<feedback>

All DB deployments are based on the out-of-the-box OpenShift Database Image.

Prepare DB for application install

Although Fider is setup to auto-install upon deployment, the OpenShift DB template disallows the application account to install DB extensions. Therefore, run the the following via oc rsh, with the correct FEEDBACK_NAME and credentials:

oc -n <project> rsh $(oc -n <project> get pods | grep -v -e "-postgresql.*-deploy" | grep Running | awk '{print $1}')

psql ${POSTGRESQL_DATABASE} -c "CREATE EXTENSION pg_trgm;"

exit

NOTE: the ${POSTGRESQL_DATABASE} text is exactly as written, since the app has access to these environment variables (set during the new-app step above).

Application Deployment

Deploy the Application specifying:

the feedback-specific parameter (i.e. <feedback>fider)
your project namespace that contains the image, and
a @gov.bc.ca email account that will be used with the apps.smtp.gov.bc.ca SMTP Email Server:

oc -n <project> new-app --file=./ci/openshift/fider-bcgov.dc.yaml -p FEEDBACK_NAME=<feedback>fider -p IS_NAMESPACE=<tools-namespace> -p EMAIL_SMTP_USERNAME=<Email.Address>@gov.bc.ca

Log into the Fider installation

After sixty seconds, the application will have finished the initial install. Open the app in a browser, to set the admin user. The URL will be of the form https://<xyz>fider.apps.silver.devops.gov.bc.ca/.

Example Deployment

As a concrete example of a feedback initiative with the acronym acme, deployed in the project namespace 599f0a-dev, here are the steps:

Deployment Steps

Database Deployment

oc -n 599f0a-dev new-app --file=./ci/openshift/postgresql.dc.yaml -p FEEDBACK_NAME=acmefider

--> Deploying template "599f0a-dev/nrmfeedback-postgresql-dc" for "./ci/openshift/postgresql.dc.yaml" to project 599f0a-dev

     nrmfeedback-postgresql-dc
     ---------
     A new PostgreSQL application has been created in your project

     * With parameters:
        * Feedback Name=acmefider
        * Memory Limit=512Mi
        * PostgreSQL Connection Password=kTl3UhrB5cFI5hF4 # generated
        * Database Volume Capacity=2Gi

--> Creating resources ...
    secret "acmefider-postgresql" created
    persistentvolumeclaim "acmefider-postgresql" created
    deploymentconfig.apps.openshift.io "acmefider-postgresql" created
    service "acmefider-postgresql" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/acmefider-postgresql'
    Run 'oc status' to view your app.

Prepare DB for application install

After thirty seconds, the database pod should be up.

oc -n 599f0a-dev rsh $(oc -n 599f0a-dev get pods | grep -v -e "acmefider-postgresql.*-deploy" | grep Running | awk '{print $1}')

Run the install commands in this shell:

psql ${POSTGRESQL_DATABASE} -c "CREATE EXTENSION pg_trgm;"

Type exit to exit the remote shell.

Application Deployment

oc -n 599f0a-dev new-app --file=./ci/openshift/fider-bcgov.dc.yaml -p FEEDBACK_NAME=acmefider -p IS_NAMESPACE=599f0a-tools EMAIL_SMTP_USERNAME=[email protected]

--> Deploying template "599f0a-dev/nrmf-feedback-dc" for "./ci/openshift/fider-bcgov.dc.yaml" to project 599f0a-dev

     nrmf-feedback-dc
     ---------
     A new Fider application, customized with noTLS, has been created in your project

     * With parameters:
        * Namespace=599f0a-tools
        * Image Stream=fider-bcgov
        * Version of Fider Product Feedback=0.18.0
        * Feedback Product Name=acmefider
        * Fider Go Environment=production
        * Fider application logging level=INFO
        * Fider Go Environment=TywM6VICeJ2J0jgmyn4gwS0F7km3f5sq # generated
        * SMTP Host=apps.smtp.gov.bc.ca
        * SMTP Port=25
        * SMTP [email protected]
        * SMTP No-Reply [email protected]
        * Google SoMe AppID=
        * Google SoMe Secret=
        * FaceBook SoMe AppID=
        * FaceBook SoMe Secret=
        * GitHub SoMe AppID=
        * GitHub SoMe Secret=
        * CPU_LIMIT=500m
        * MEMORY_LIMIT=1Gi
        * CPU_REQUEST=500m
        * MEMORY_REQUEST=1Gi
        * REPLICA_MIN=2
        * REPLICA_MAX=3

--> Creating resources ...
    secret "acmefider-jwt" created
    deploymentconfig.apps.openshift.io "acmefider-app" created
    service "acmefider" created
    route.route.openshift.io "acmefider" created
--> Success
    Access your application via route 'acmefider.apps.silver.devops.gov.bc.ca'
    Run 'oc status' to view your app.

Log into the Fider app

After sixty seconds, the application will be ready to accept connections at https://acmefider.apps.silver.devops.gov.bc.ca. Initially, this redirect you to https://acmefider.apps.silver.devops.gov.bc.ca/signup, bringing you the screen:

Once you've confirmed the setup detail, you'll be sent a confirmation email, and you'll have to click the embedded link:

Using Environmental variables to deploy

As this is a template deployment, it may be easier to set environment variable for the deployment, so using the same PROJECT of 599f0a-dev and a new FEEDBACK project of eao:

Deployment Steps

Set the environment variables

On a workstation logged into the OpenShift Console:

export TOOLS=599f0a-tools
export PROJECT=599f0a-dev
export FEEDBACK=eao

On Windows the commands to set environment variables would instead be:

set TOOLS=599f0a-tools
set PROJECT=599f0a-dev
set FEEDBACK=eao

If running commands in a Windows environment (and not in the openshift terminal), environment variables are formatted as %ENV_NAME% instead of <ENV_NAME> or ${ENV_NAME}. Commands in these instructions may need to be adapted for Windows.

Database Deployment

oc -n ${PROJECT} new-app --file=./ci/openshift/postgresql.dc.yaml -p FEEDBACK_NAME=${FEEDBACK}fider

--> Deploying template "599f0a-dev/nrmfeedback-postgresql-dc" for "./ci/openshift/postgresql.dc.yaml" to project 599f0a-dev

     nrmfeedback-postgresql-dc
     ---------
     A new PostgreSQL application has been created in your project

     * With parameters:
        * Feedback Name=eaofider
        * Memory Limit=512Mi
        * PostgreSQL Connection Password=yOmOOBMbY8k3aHiy # generated
        * Database Volume Capacity=2Gi

--> Creating resources ...
    secret "eaofider-postgresql" created
    persistentvolumeclaim "eaofider-postgresql" created
    deploymentconfig.apps.openshift.io "eaofider-postgresql" created
    service "eaofider-postgresql" created
--> Success
    Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
     'oc expose svc/eaofider-postgresql'
    Run 'oc status' to view your app.

Prepare DB for application install

After thirty seconds, the database pod should be up.

oc -n ${PROJECT} rsh $(oc -n ${PROJECT} get pods | grep -v -e "${FEEDBACK}fider-postgresql-.-deploy" | grep Running | grep ${FEEDBACK}fider | awk '{print $1}')

On the shell, enter:

psql ${POSTGRESQL_DATABASE}  -c "CREATE EXTENSION pg_trgm;"
exit

Application Deployment

oc -n ${PROJECT} new-app --file=./ci/openshift/fider-bcgov.dc.yaml -p FEEDBACK_NAME=${FEEDBACK}fider -p IS_NAMESPACE=${TOOLS} -p [email protected]

--> Deploying template "599f0a-dev/nrmf-feedback-dc" for "./ci/openshift/fider-bcgov.dc.yaml" to project 599f0a-dev

     nrmf-feedback-dc
     ---------
     A new Fider application, customized with noTLS, has been created in your project

     * With parameters:
        * Namespace=599f0a-tools
        * Image Stream=fider-bcgov
        * Version of Fider Product Feedback=0.19.1
        * Feedback Product Name=gwfider
        * Fider Go Environment=production
        * Fider application logging level=ERROR
        * Fider Go Environment=ivbfbDFjYyK7omNoVllbvJbhyiHwJd1R # generated
        * SMTP Host=apps.smtp.gov.bc.ca
        * SMTP Port=25
        * SMTP [email protected]
        * SMTP No-Reply [email protected]
        * Google SoMe AppID=
        * Google SoMe Secret=
        * FaceBook SoMe AppID=
        * FaceBook SoMe Secret=
        * GitHub SoMe AppID=
        * GitHub SoMe Secret=
        * CPU_LIMIT=500m
        * MEMORY_LIMIT=1Gi
        * CPU_REQUEST=80m
        * MEMORY_REQUEST=196Mi
        * REPLICA_MIN=2
        * REPLICA_MAX=3

--> Creating resources ...
    secret "gwfider-jwt" created
    deploymentconfig.apps.openshift.io "gwfider-app" created
    horizontalpodautoscaler.autoscaling "gwfider" created
    service "gwfider" created
    route.route.openshift.io "gwfider" created
--> Success
    Access your application via route 'gwfider.apps.silver.devops.gov.bc.ca' 
    Run 'oc status' to view your app.

Log into the Fider app

After sixty seconds, you may navigate to the setup page at: https://iitdfider.apps.silver.devops.gov.bc.ca/signup

Once you've confirmed the setup detail, you'll be sent a confirmation email like this, and you'll have to click the link:

When finished, remember to unset the environment variables:

unset TOOLS PROJECT FEEDBACK

Integration with Platform SSO KeyCloak Shared instance

After requesting access via https://github.com/BCDevOps/devops-requests, use the GUI to navigate to:

Administration -> Site Settings

Authentication -> Add New

OAuth2
Client ID     : dds-portfolio
Client Secret : <xxxxx>-<xxxx>-<xxxx>-<xxxx>-<xxxxx>
Authorize URL : https://loginproxy.gov.bc.ca/auth/realms/standard/protocol/openid-connect/auth
Token URL     : https://loginproxy.gov.bc.ca/auth/realms/standard/protocol/openid-connect/token
Scope: openid
Profile API URL: (empty)
JSON PATH
	ID: 	sub
	Name: 	displayname
	Email: email

FAQ

DB login

To login into the database, open the DB pod terminal (via OpenShift Console or oc rsh) and enter:

psql -U ${POSTGRESQL_USER} ${POSTGRESQL_DATABASE}

DB Cleanup

to clean-up database deployments:

To re-deploy just the database, first idle the database service and then delete the deployed objects from the last run, with the correct FEEDBACK_NAME, such as:
```
oc -n <project> idle <feedback>fider-postgresql
oc -n <project> delete secret/<feedback>fider-postgresql svc/<feedback>fider-postgresql pvc/<feedback>fider-postgresql dc/<feedback>fider-postgresql
```
NOTE: This resets the persistent database, so you will lose all your Fider data. If you wish to only redeploy the DB runtime but not reset the data, then omit the pvc/<feedback>fider-postgresql object from the above command.

or if using environment variables:

  oc -n ${PROJECT} idle ${FEEDBACK}fider-postgresql
  oc -n ${PROJECT} delete secret/${FEEDBACK}fider-postgresql svc/${FEEDBACK}fider-postgresql pvc/${FEEDBACK}fider-postgresql dc/${FEEDBACK}fider-postgresql

App Cleanup

to clean-up application deployments:

  oc -n <project> delete dc/<feedback>fider-app svc/<feedback>fider route/<feedback>fider secret/<feedback>fider-jwt hpa/<feedback>fider-app

Or if using environment variables:

  oc -n ${PROJECT} delete dc/${FEEDBACK}fider-app svc/${FEEDBACK}fider route/${FEEDBACK}fider secret/${FEEDBACK}fider-jwt hpa/${FEEDBACK}fider-app

to reset all deployed objects (this will destroy all data and persistent volumes). Only do this on a botched initial install or if you have the DB backed up and ready to restore into the new wiped database.

oc -n <project> delete all,secret,pvc,hpa -l app=<feedback>fider

or if using environment variables:

oc -n ${PROJECT} delete all,secret,pvc,hpa -l app=${FEEDBACK}fider

Git Submodule

Git SubModule was created via:

git submodule add https://github.com/getfider/fider

NOTE: In hindsight, we should've explicitly tracked the master branch

> git submodule add -b master https://github.com/getfider/fider

To update this LimeSurvey git submodule from the upstream repo, from root of repo:

  git submodule update --remote

Upgrade Fider Git submodule

This is an example when updating from v19.0 to v19.1

First check status of the submodle

~/p/nrm-feedback ❯❯❯ git submodule  status  
-74ac88ed1f6dfac1a3adfb93eb9c8c325f0018a8 fider

~/p/nrm-feedback ❯❯❯ more .gitmodules
[submodule "fider"]
        path = fider
        url = https://github.com/getfider/fider
        branch = main        

~/p/nrm-feedback ❯❯❯ git submodule update --remote --merge

NOTE: Git Submodule update instructions above are yet to be 100% confirmed. TODO - confirm instructions work as advertise above

NOTE: Fider often updates the fider\Dockerfile so be sure to integrate the changes to .\Dockerfile.openshift and .\Dockerfile.dev

For the former, be sure to change:

COPY . .

to

COPY fider .

TODO

check for image triggers which force a reploy (image tags.. latest -> v0.19.0)
work around OCP Build issue which shows up as Error shutting down storage: A layer is mounted: layer is in use by a container but this masks the underlying issue - solution is apparently to fix file permissions, or to work around by git config --global --add safe.directory. I've temporarily downgraded Golang to an older version (pre-v18) to avoid this.

Done

documented KeyCloak integration
health checks for application containers
appropriate resource limits (multi-replica pods supported)
created fider-bcgov.bc.json file
integrated with apps.smtp.gov.bc.ca:25 without TLS (e.g. x509 error due to vwall.gov.bc.ca on cert)
added health checks for each of the database container
converted ci/openshift/_.json to _.yaml
tested DB backup/restore and transfer with Backup-Containers
tested git submodule update

SMTPS Issue

When configured with apps.smtp.gov.bc.ca

ERROR [2019-11-30T02:54:11Z] [WEB] [6N0EEiyyCVMpNWxD3w9N7J6XjVnlFT6Y] Error Trace:
- app/handlers/apiv1/invite.go:31
- failed to send email with template invite_email (app/pkg/email/smtp/smtp.go:104)
- x509: certificate is valid for vwall.gov.bc.ca, www.vwall.gov.bc.ca, not apps.smtp.gov.bc.ca

When using straight non-TLS STMP connection

ERROR [2019-11-30T03:17:59Z] [WEB] [7g7GXPSR57065JvQYioR9nvj7Z7Zy0pJ] Error Trace:
- app/handlers/apiv1/invite.go:31
- failed to send email with template invite_email (app/pkg/email/smtp/smtp.go:104)
- dial tcp 74.125.20.108:587: connect: connection timed out

I suspect apps.smtp.gov.bc.ca starts the TLS handshake which tells Fider SMTP utility to expect a properly configured SSL Certificate.

Name		Name	Last commit message	Last commit date
Latest commit History 69 Commits
backup-container		backup-container
ci/openshift		ci/openshift
docs		docs
fider @ c962c41		fider @ c962c41
.gitignore		.gitignore
.gitmodules		.gitmodules
Dockerfile.dev		Dockerfile.dev
Dockerfile.openshift		Dockerfile.openshift
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
sample.env		sample.env

License

bcgov/nrm-feedback

Folders and files

Latest commit

History

Repository files navigation

Table of Contents

NRM Fider

Prerequisites

Files

Build

Custom Image

Out-of-the-box Image

Deploy

Database Deployment

Prepare DB for application install

Application Deployment

Log into the Fider installation

Example Deployment

Database Deployment

Prepare DB for application install

Application Deployment

Log into the Fider app

Using Environmental variables to deploy

Set the environment variables

Database Deployment

Prepare DB for application install

Application Deployment

Log into the Fider app

Integration with Platform SSO KeyCloak Shared instance

FAQ

DB login

DB Cleanup

App Cleanup

Git Submodule

Upgrade Fider Git submodule

TODO

Done

SMTPS Issue

About

Resources

License

Stars

Watchers

Forks

Releases 2

Packages 0

Contributors 2

Languages

Packages