[Bug]: Cypress tests gives false failures in the deployment CI

[vedhav]

The issue

Some cypress tests as part of the deploy.yml workflow fail in the CI while they all succeed locally. Because of this, we are ignoring the results of the test. If this starts to work we can make sure to deploy only after we verify all the modules are loaded in the app.

This workflow ran on d19a0ba

Workflow file

---
name: Deploy Apps 🚀

on:
  push:
    branches:
      - main
      - dev
  workflow_dispatch:
  schedule:
    - cron: "12 3 * * *"

concurrency:
  group: publish-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true

env:
  SHINYAPPSIO_ACCOUNT: genentech
  APP_PREFIX: NEST
  GITHUB_PAT: ${{ secrets.REPO_GITHUB_TOKEN }}

jobs:
  deploy:
    permissions:
      contents: write
    defaults:
      run:
        shell: bash
    name: Publish 🗞
    runs-on: ubuntu-latest
    container:
      image: ghcr.io/insightsengineering/rstudio_4.3.1_bioc_3.17:latest
    if: >
      !contains(github.event.commits[0].message, '[skip deploy]')
    strategy:
      fail-fast: false
      matrix:
        directory:
          [
            "RNA-seq",
            "basic-teal",
            "efficacy",
            "exploratory",
            "longitudinal",
            "early-dev",
            "patient-profile",
            "python",
            "safety",
          ]
        channel: ["stable", "dev"]
    steps:
      - name: Set channel related constants
        run: |
          if [ "${{ matrix.channel }}" = "stable" ]; then
            echo "BRANCH_NAME=main" >> $GITHUB_ENV
          else
            echo "BRANCH_NAME=dev" >> $GITHUB_ENV
          fi

      - name: Checkout repo 🛎
        uses: actions/checkout@v3
        with:
          ref: "${{ env.BRANCH_NAME }}"

      - name: Check if cypress test exists
        id: find-cypress
        run: |
          if [ -d ${{ matrix.directory }}/tests/cypress ]; then
            echo "has-cypress-tests=true" >> $GITHUB_OUTPUT
          else
            echo "Skipping frontend tests because cypress directory does not exist."
            echo "has-cypress-tests=false" >> $GITHUB_OUTPUT
          fi

      - name: Setup system dependencies for cypress and python app
        run: >
          apt-get update && apt-get install --yes
          libgtk2.0-0 libgbm-dev libnotify-dev libgconf-2-4 xvfb python3.10-venv

      - name: Setup Node
        uses: actions/setup-node@v3
        if: steps.find-cypress.outputs.has-cypress-tests == 'true'
        with:
          node-version: 16

      - name: Restore renv from cache
        uses: actions/cache@v3
        env:
          CACHE_KEY: renv-${{ runner.arch }}-${{ runner.os }}-${{ matrix.directory }}-${{ matrix.channel }}
        with:
          path: ${{ matrix.directory }}/renv/library
          key: ${{ env.CACHE_KEY }}-${{ hashFiles(format('{0}/renv.lock', matrix.directory)) }}
          restore-keys: ${{ env.CACHE_KEY }}-

      - name: Update renv.lock file with updated GitHub packages
        shell: Rscript {0}
        run: |
          setwd("${{ matrix.directory }}")
          lockfile <- renv::lockfile_read()
          pkg_name_structure <- ifelse("${{ matrix.channel }}" == "stable", "%s/%s@*release", "%s/%s")
          for (package in lockfile$Packages) {
              if (package$Source == "GitHub") {
                  # TODO: After teal.slice is released we have to remove this change. This workaround is because update to {bslib} breaks current release of teal.slice
                  if (package$Package == "teal.slice" && "${{ matrix.channel }}" == "stable") {
                    renv::record("insightsengineering/teal.slice@103491c67e26a9a85d636bd105a806904a0111ec")
                  } else {
                    renv::record(sprintf(pkg_name_structure, package$RemoteUsername, package$Package))
                  }
              }
          }

      - name: Install R packages using renv and update the renv snapshot
        shell: Rscript {0}
        working-directory: ${{ matrix.directory }}
        run: |
          options(renv.config.cache.symlinks = FALSE)
          lockfile_pkgs <- renv::lockfile_read()$Package
          github_pkgs <- names(lockfile_pkgs)[sapply(lockfile_pkgs, function(x) x$Source == "GitHub")]
          renv::restore(clean = TRUE)
          renv::update(exclude = github_pkgs)
          renv::snapshot()

      - name: Print the new renv.lock file for ${{ matrix.directory }}
        working-directory: ${{ matrix.directory }}
        run: cat renv.lock

      - name: Front end test to check if the app works fine
        if: steps.find-cypress.outputs.has-cypress-tests == 'true'
        continue-on-error: true
        uses: cypress-io/github-action@v6
        with:
          build: npm install cypress --save-dev
          working-directory: ${{ matrix.directory }}/tests
          start: npm run run-app
          wait-on: "http://localhost:3333"
          wait-on-timeout: 500

      - name: Install deploy R package dependencies
        shell: Rscript {0}
        working-directory: ${{ matrix.directory }}
        run: |
          install.packages(c("BiocManager", "rsconnect"))

      - name: Deploy 🖨 ${{ matrix.directory }} 🎨
        shell: Rscript {0}
        working-directory: ${{ matrix.directory }}
        run: |
          rsconnect::setAccountInfo(
            name = "${{ env.SHINYAPPSIO_ACCOUNT }}",
            token = "${{ secrets.SHINYAPPSIO_TOKEN }}",
            secret = "${{ secrets.SHINYAPPSIO_SECRET }}",
            server = "shinyapps.io"
          )
          rsconnect::deployApp(
            appFiles = c("app.R"),
            appName = rsconnect::generateAppName("${{ env.APP_PREFIX }}_${{ matrix.directory }}_${{ matrix.channel }}"),
            appTitle = "${{ env.APP_PREFIX }}_${{ matrix.directory }}_${{ matrix.channel }}",
            account = "${{ env.SHINYAPPSIO_ACCOUNT }}",
            upload = TRUE,
            logLevel = "normal",
            lint = FALSE,
            forceUpdate = TRUE
          )

      - name: Commit updated renv.lock file
        continue-on-error: true
        run: |
          git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git config --global user.name "github-actions"
          git config --global --add safe.directory /__w/teal.gallery/teal.gallery
          git fetch
          git stash
          git checkout ${{ env.BRANCH_NAME }}
          git pull
          git stash apply
          git add ${{ matrix.directory }}/renv.lock
          git add ${{ matrix.directory }}/renv/activate.R
          if [ -n "$(git diff --staged)" ]; then
            git commit -m "[skip deploy] Update renv.lock file for ${{ matrix.directory }} app"
            git push origin ${{ env.BRANCH_NAME }}
          else
            echo "renv.lock was not modified. Nothing to commit.";
          fi

Cypress failure log

Run cypress-io/github-action@v6
  with:
    build: npm install cypress --save-dev
    working-directory: safety/tests
    start: npm run run-app
    wait-on: http://localhost:3333
    wait-on-timeout: 500
    record: false
    publish-summary: true
    component: false
  env:
    SHINYAPPSIO_ACCOUNT: genentech
    APP_PREFIX: NEST
    GITHUB_PAT: ***
    BRANCH_NAME: dev
/usr/bin/docker exec  4ccaa337[2](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:2)be99db061c64860e5e20c9a042a12e52d21570b15e5d27ac7987227 sh -c "cat /etc/*release | grep ^ID"
Received 5299526 of 5299526 (100.0%), [3](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:3)0.6 MBs/sec
Cache Size: ~5 MB (5299526 B)
/usr/bin/tar -xf /__w/_temp/9210877e-020c-[4](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:4)108-96e8-24fb[5](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:5)91c9979/cache.tgz -P -C /__w/teal.gallery/teal.gallery -z
Cache restored successfully
Received 1782510[6](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:6)[7](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:7) of 17[8](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:8)251067 (100.0%), 174.7 MBs/sec
Cache Size: ~170 MB (178251067 B)
/usr/bin/tar -xf /__w/_temp/[9](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:9)a72dd60-0f05-4fcf-870e-5f91963af6c5/cache.tgz -P -C /__w/teal.gallery/teal.gallery -z
Cache restored successfully
/__t/node/16.20.2/x64/bin/npm ci

added 201 packages, and audited 202 packages in 2s

39 packages are looking for funding
  run `npm fund` for details

3 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
/__t/node/16.20.2/x64/bin/npx cypress cache list
┌─────────┬───────────────────┐
│ version │ last used         │
├─────────┼───────────────────┤
│ 13.3.0  │ a few seconds ago │
└─────────┴───────────────────┘
build app command "npm install cypress --save-dev"
current working directory "/__w/teal.gallery/teal.gallery/safety/tests"
/__t/node/16.20.2/x64/bin/npm install cypress --save-dev

changed 1 package, and audited 202 packages in [10](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:10)s

39 packages are looking for funding
  run `npm fund` for details

3 moderate severity vulnerabilities

To address all issues, run:
  npm audit fix

Run `npm audit` for details.
start server command "npm run run-app"
current working directory "/__w/teal.gallery/teal.gallery/safety/tests"
waiting on "http://localhost:3333" with timeout of 500 seconds
/__t/node/16.20.2/x64/bin/npm run run-app

> run-app
> cd .. && Rscript -e "shiny::runApp(port = 3333)"

The following package(s) are missing entries in the cache:
- packrat
- rsconnect
- rstudioapi
These packages will need to be reinstalled.

The following package(s) have broken symlinks into the cache:
- packrat
- rsconnect
- rstudioapi
Use `renv::repair()` to try and reinstall these packages.

Loading required package: shiny

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union


Loading required package: ggmosaic
Loading required package: ggplot2
Loading required package: shinyTree
Loading required package: teal
Loading required package: teal.data
Loading required package: teal.slice
Loading required package: teal.transform
Loading required package: magrittr
Registered S3 method overwritten by 'teal':
  method        from      
  c.teal_slices teal.slice

You are using teal version 0.14.0.9024

Attaching package: ‘teal’

The following objects are masked from ‘package:teal.slice’:

    as.teal_slices, teal_slices

Loading required package: tern
Loading required package: rtables
Loading required package: formatters

Attaching package: ‘rtables’

The following object is masked from ‘package:utils’:

    str

Registered S3 method overwritten by 'tern':
  method   from 
  tidy.glm broom
[INFO] 2023-[11](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:11)-28 14:08:09.6658 pid:3630 token:[] teal.modules.general Initializing tm_front_page
[INFO] 2023-11-28 14:08:09.6694 pid:3630 token:[] teal.modules.general Initializing tm_data_table
[INFO] 2023-11-28 14:08:09.6723 pid:3630 token:[] teal.modules.general Initializing tm_variable_browser
[INFO] 2023-11-28 14:08:09.6805 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary
[INFO] 2023-11-28 14:08:09.6879 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_summary
[INFO] 2023-11-28 14:08:09.6949 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events
[INFO] 2023-11-28 14:08:09.7021 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_by_grade
[INFO] 2023-11-28 14:08:09.7109 pid:3630 token:[] teal.modules.clinical Initializing tm_t_events_patyear
[INFO] 2023-11-28 14:08:09.7505 pid:3630 token:[] teal.modules.clinical Initializing tm_t_smq
[INFO] 2023-11-28 14:08:09.7613 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary_by
[INFO] 2023-11-28 14:08:09.8386 pid:3630 token:[] teal.modules.clinical Initializing tm_t_shift_by_grade
[INFO] 2023-11-28 14:08:10.1735 pid:3630 token:[] teal.modules.clinical Initializing tm_t_abnormality_by_worst_grade
[INFO] 2023-11-28 14:08:10.2414 pid:3630 token:[] teal.modules.clinical Initializing tm_t_summary_by
[INFO] 2023-11-28 14:08:10.2584 pid:3630 token:[] teal.modules.clinical Initializing tm_t_exposure
[INFO] 2023-11-28 14:08:10.2880 pid:3630 token:[] teal.modules.clinical Initializing tm_t_abnormality
[INFO] 2023-11-28 14:08:10.2966 pid:3630 token:[] teal.modules.clinical Initializing tm_t_mult_events
[INFO] 2023-11-28 14:08:10.3034 pid:3630 token:[] teal.modules.clinical Initializing tm_t_shift_by_arm
[INFO] 2023-11-28 14:08:10.4894 pid:3630 token:[] teal.modules.clinical Initializing tm_g_lineplot
Warning in rlang::hash(hashables) :
  'package:sparkline' may not be available when loading

Listening on http://[12](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:12)7.0.0.1:3333
It looks like this is your first time using Cypress: [13](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:13).6.0

[STARTED] Task without title.
[SUCCESS] Task without title.

Opening Cypress...
[3853:1128/[14](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:14)0813.765659:ERROR:object_proxy.cc(590)] Failed to call method: org.freedesktop.portal.Settings.Read: object_path= /org/freedesktop/portal/desktop: unknown error type: 

DevTools listening on ws://127.0.0.1:33071/devtools/browser/0b0e4dd2-2809-413a-be0b-5ebeea4a837f

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        13.6.0                                                                         │
  │ Browser:        Electron 114 (headless)                                                        │
  │ Node Version:   v20.8.1 (/__e/node20/bin/node)                                                 │
  │ Specs:          1 found (app.cy.js)                                                            │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  app.cy.js                                                                       (1 of 1)


  app
    ✓ Starts (459ms)
[INFO] 2023-11-28 14:08:17.9232 pid:3630 token:[9b1ea8f3] teal Initializing reporter_previewer_module
[INFO] 2023-11-28 14:08:28.1798 pid:3630 token:[5ab3181b] teal Initializing reporter_previewer_module
    ✓ Has 12 tabs ([16](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:17)628ms)
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADSL-filter-ADSL_AGE-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEX-filter-ADEX_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADLB-filter-ADLB_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
Warning: The 'plotly_relayout' event tied a source ID of 'teal-main_ui-filter_panel-active-ADEG-filter-ADEG_AVAL-inputs-histogram_plot' is not registered. In order to obtain this event data, please add `event_register(p, 'plotly_relayout')` to the plot (`p`) that you wish to obtain event data from.
[INFO] [20](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:21)[23](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:24)-11-28 14:08:40.9578 pid:3630 token:[7f4a4[27](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:28)5] teal Initializing reporter_previewer_module
    1) Navigates to all tabs without error


  2 passing (34s)
  1 failing

  1) app
       Navigates to all tabs without error:
     AssertionError: Timed out retrying after 4000ms: Expected to find element: `.shiny-busy`, but never found it. Queried from:

              > cy.get(html)
      at Context.eval (webpack:///./cypress/e2e/app.cy.js:30:23)




  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        3                                                                                │
  │ Passing:      2                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  1                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     33 seconds                                                                       │
  │ Spec Ran:     app.cy.js                                                                        │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


  (Screenshots)

  -  /__w/teal.gallery/teal.gallery/safety/tests/cypress/screenshots/app.cy.js/app --     (1[28](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:29)0x720)
      Navigates to all tabs without error (failed).png                                              


====================================================================================================

  (Run Finished)


       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  app.cy.js                                00:[33](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:34)        3        2        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed ([100](https://github.com/insightsengineering/teal.gallery/actions/runs/7019861588/job/19098283022#step:12:101)%)                     00:33        3        2        1        -        -  

Error: Cypress tests: 1 failed

[cicdguy]

Agreed. Ideally we'd want to deploy the apps following successful test runs.
I'll take a look into possible solution to probe for readiness+liveness of the apps prior to the test execution.