Upgrading from v1.x to 2.0.0

Upgrading from v1.x to 2.0.0

The 2.0.0 release represents a major upgrade to the Roadmap codebase. Please follow the instructions below to ensure that your site functions properly after upgrade.

We highly recommend that you backup your existing database and that you perform these tasks in a development environment first so that you can confirm the functionality of your system before running the upgrade in your production environment!

Overview of data model changes

• Added new table stats to facilitate the Usage Statistics page available to Admins
• Added new versionable_id field to the following tables to help ensure accurate transfer of template customizations when a funder template changes: annotations, questions, sections, phases
• Added default: false to boolean fields: guidance_groups.published, guidance_groups.optional_subset, notes.archived, orgs.is_other
• Added limit: 80 to users.email
• Dropped unused tables: file_types, file_uploads, splash_logs
• Removed unused fields: guidances.question_id, orgs.wayfless_entity, orgs.parent_id, orgs.banner_text, orgs.logo_file_name, phases.slug, plans.slug, users.orcid_id, users.shibboleth_id
• Various index changes in support of the above changes

Overview of model validations

• Guidance records must now have at least one associated theme if they are published.
• Guidance and Templates can no longer be published if they belong to the 'Other' Org, orgs.is_other

Database upgrade

The new version of the Roadmap codebase includes an effort to improve data integrity. This was accomplished by adding data validation checks at the database and Model levels. You will need to run through 2 separate scripts to ensure that these new validations do not have issues with your existing data.

Part One

Please run rake upgrade:v2_0_0_part_1

The Part one script runs 3 separate steps to prepare your system for Roadmap v2.0.0:

1. Updates existing records in your database to prepare them for some new database validations that prevent NULL values in boolean fields:

• guidance_groups.optional_subset - any NULL values will be set to 'false'
• guidance_groups.published - any NULL values will be set to 'false'
• notes.archived - any NULL values will be set to 'false'
• orgs.is_other - any NULL values will be set to 'false'

2. The standard Rails database migrations will be run
3. A script that checks your existing data to determine if any records would become invalid due to the new validations added to each of the models

The 3rd script will output a report to your console. Please review the report and resolve any issues that have been reported to prepare your data for the new version.

In the example below, we can see that there are 36 orphaned records in my answers table, one orphaned guidance group, and one template record with a missing family_id and 3 user records that do not have a unique email address:

Checking Annotation records
  0 records with a empty text field
  0 orphaned records due to nil or missing org
  0 orphaned records due to nil or missing question
  0 records with a empty type field
  0 records that are not unique per (type, question_id, org_id)
Checking Answer records
  0 orphaned records due to nil or missing plan
  0 orphaned records due to nil or missing user
  36 orphaned records due to nil or missing question
  0 records that are not unique per (question_id, plan_id)
Checking ExportedPlan records
  0 orphaned records due to nil or missing plan
  0 records with a empty format field
Checking Guidance records
  0 records with a empty text field
  0 orphaned records due to nil or missing guidance_group
  0 records that do not have a valid value for [:published], should be [true, false]
Checking GuidanceGroup records
  0 records with a empty name field
  0 records that are not unique per (name, org_id)
  1 orphaned records due to nil or missing org
  0 records that do not have a valid value for [:optional_subset], should be [true, false]
  0 records that do not have a valid value for [:published], should be [true, false]

...

Checking Template records
  0 records that have an invalid [:links]. See the TemplateLinksValidator for further details
  0 records with a empty title field
  0 orphaned records due to nil or missing org
  0 records with a empty locale field
  0 records with a empty version field
  0 records that are not unique per (version, family_id)
  0 records with a empty visibility field
  1 records with a empty family_id field
  0 records with a empty title field
Checking Theme records
  0 records with a empty title field
Checking TokenPermissionType records
  0 records with a empty token_type field
  0 records that are not unique per (token_type)
Checking User records
  0 records with a empty email field
  3 records that are not unique per (email)

Any issues you find in the report were either caused by bugs in an older version of the codebase or were problems carried over when you upgraded your system from DMPonline_v3.

Correct any data issues

You should now take the time to correct any issues reported in part one. You can either fix them manually in your database or add new rules to the lib/data_cleanup/rules directory. Any rules you add to this directory will be picked up and run during part two.

TODO: Add explanation of issues reported and how to address them via data_cleanup rules

Part Two

The second script will first repair any invalid data based on rules you have added to lib/data_cleanup/rules in the prior section above.

1. Cleans up any invalid records by processing the rules you have defined in the prior section.
2. Runs a script to initialize all of the new versional_id fields that were added to the phases, sections, questions and annotations tables.
3. Normalizes your languages abbreviations
4. Populates the new usage statistics tables

If you have a lot of invalid data cleanup to do, you may want to run that script individually for each of your rules as you create them. To do so, please run rake data_cleanup:clean_invalid_records and then rake data_cleanup:find_known_invalidations to verify that your rules have in fact fixed the problem(s).

*Please note that the task that updates the versional_id fields can take in excess of 15 minutes depending on the size of your database.

Updating your deployment scripts

This release also involved some changes to the basic infrastructure of the system. Please check and update your deployment scripts as needed for your individual installations.

• Tests are now run via RSpec instead of MiniTest. Use rpec spec/ to run the tests
• Modifications were made to the Webpack configuration. To compile you assets for production (or any situation where you do not want Webpack to watch your assets for changes) you should now use npm run bundle -- -p --no-watch
• We have updated the Gemfile to segregate deployment specific gems so that you can exclude them as needed when running bundle install --without [groupA] [groupB]. For example: bundle install --without mysql puma thin:
  • puma
  • thin
  • mysql
  • postgres
• You will need to setup a job (e.g. via CRON) that runs the rake stat:build_monthly task at the beginning of each month

Changes to the developer guide

We have also updated our Wiki to improve our developer guides. Please review the updated documentation.