vulekamali

This is the source code for the South African National and Provincial Budget Portal.

In addition to providing user-friendly overviews of the national and provincial budget data, this django app provides Single Sign-on (SSO) and support for maintaining correct and consistent data for the various tools and services around vulekamali.

This django app used to just coordinate and serve data, hence being called datamanager, but now serves the actual website, so it will be renamed appropriately.

Code Structure vision

Browser support requirements

See browserslist

HTML, CSS, Javascript

We use a yarn (Node.js) and webpack-based build system for CSS and Javascript.

You can find frontend code in the following places:

• HTML (Django template language)
  • Page templates in budgetportal/templates
  • Includes in assets/js/...html
• Javascript (ES6)
  • The package for the main site has package.json in the root, and source files in assets/js
    • This is bundled as frontend-v1.bundle.js
  • A package providing react views for certain pages in packages/webapp
    • This is bundled as webapp.bundle.js
• CSS (SCSS) in assets/scss
• Webflow-built pages
  • webflow.js and other assets are served from the webflow app static directory
  • JS to integrate the webflow frontend with the django project is also built using webpack
    • This is bundled as vulekamali-webflow.bundle.js

Most of the HTML has been migrated from Jekyll as simply and quickly as possible and we're figuring out how best to structure it going forward. Find a balance between fitting in to the existing style and moving towards standard django template best practises for the time being.

When vulekamali was a jekyll site, there was an attempt to have a component- oriented file structure so that the HTML (Liquid template language), SCSS, and Javascript for a component were all in the same folder.

When migrating to Django, we have tried to keep this structure but split out HTML (django template language) into the django template folder, SCSS in assets/scss and Javascript in assets/js but ran into trouble with javascript reading some HTML files. We decided to leave Django template include files in assets/js for the time being and set that up as an additional template directory.

Webflow

We are using for new content and are potentially replacing the current frontend with one built using webflow. Webflow is a web design tool, which provides hosting and integrates with their own content management system. We don't use their hosting or CMS for production pages of vulekamali.gov.za, but rather export the HTML, CSS and Javascript and use that for those content areas on this site. Page data is either provided to the client via JSON embedded in the page, or via AJAX requests to APIs.

By using webflow, we can build semi-working interactive prototypes in a convenient design tool, refine these into mockups, export these and use it as the ready-built frontend. There is no need for a web developer to write markup and style the page manually. This means we can preview changes to the design in webflow, and reflect those changes in this django site with little further frontend development.

The webflow frontend is in the Django app budgetportal.webflow.

Webflow HTML, CSS and Javascript must not be edited manually. Those edits will be overridden on the next import from Webflow.

See the budgetportal.webflow docs for how to work on the webflow-based parts of the site.

Python code

We're moving towards the following code structure:

+-------+
| views |
+-------+
  |   |
  |   V
  | +-----------+
  | | summaries |
  | +-----------+
  |    |   |   |
  |    |   |   |
  V    V   |   V
+--------+ |  +----------+          +------+
| models | |  | datasets |---HTTP-->| CKAN |
+--------+ |  +----------+          +------+
           |
           V
      +--------------+          +--------------+
      | openspending |---HTTP-->| OpenSpending |
      +--------------+          +--------------+

A lot of summary code has been implemented as part of models but should now start moving to summaries, and just call out to models and datasets as needed.

Setting up Development environment

Install frontend dependencies

First time you clone this repo, and any time frontend dependencies might have changed

yarn

Run JS and CSS incremental build for the assets managed by yarn (except webapp package)

This is an incremental build and will rebuild the root bundle until you Ctrl+C out of it.

yarn build:dev

When it's done with the initial build it just sits waiting to build changes looking like this - it won't show anything when building subsequent changes unless there are errors:

Build changes in webapp package.

In another shell, build the webapp package. This is not an incremental build and needs to be rebuilt for each change in this package.

yarn build:webapp

Start development server

Point the hostname minio at the host running the dev server, probably 127.0.0.1, using your hosts file:

127.0.0.1    localhost minio

Start the database:

docker-compose up db

Setup the database - either by running migrations against a new database, or by loading a dump from elsewhere:

If you're setting up a new database:

docker-compose run --rm app python manage.py migrate
docker-compose run --rm app python manage.py loaddata development-first-user

Then run the server

docker-compose up

Now you can login with initial the development superuser:

Username: admin@localhost Password: password

A fixture is needed to set this up instead of createsuperuser because Django Allauth is configured to require verified email addresses.

If you want to reset the database and start fresh:

docker-compose down --volumes

And then:

docker-compose up

Load data

Load an initial set of financial years, spheres and governments. You might need to add more recent ones manually in the admin interface.

You can download data from the production datamanager to use in your test environment as follows:

for year in 2016-17 2017-18 2018-19 2019-20; do curl https://vulekamali.gov.za/${year}/national/departments.csv > departments-national-${year}.csv; done
for year in 2016-17 2017-18 2018-19 2019-20; do curl https://vulekamali.gov.za/${year}/provincial/departments.csv > departments-provincial-${year}.csv; done

You can load this data into your environment with:

docker-compose run --rm app python manage.py loaddata years-spheres-governments \
                                                      video-language \
                                                      events \
                                                      homepage \
                                                      quarters \
                                                      menu
for year in 2016-17 2017-18 2018-19 2019-20; do docker-compose run --rm app python manage.py load_departments ${year} national departments-national-${year}.csv; done
for year in 2016-17 2017-18 2018-19 2019-20; do docker-compose run --rm app python manage.py load_departments ${year} provincial departments-provincial-${year}.csv; done

Finally load the 2019-20 featured national infrastructure projects on the Infrastructure Project Parts admin page via the Import buton. Find the file to import at budgetportal/fixtures/infrastructure-projects-2019-20.csv.

---

Create and run database migrations with

docker-compose run --rm app python manage.py makemigrations

Development best practises

• Always maintain or improve test coverage
• Follow the principles of the test pyramid

Python/Django

• Get better debugging with python manage.py runserver_plus
• Format your code using Black: budgetportal manage.py discours --exclude budgetportal/bulk_upload.py

React stuff (package.json and packages/webapp/package.json)

TBC

Classic django+bower stuff:

• Put javascript into budgetportal/static/javascript/app.js
• Put SCSS stylesheets into budgetportal/static/stylesheets/app.scss
• Install new asset packs with Bower: bower install -Sp package-to-install

Frontend development

Root NPM package

See package.json and webpack.config.js

packages/webapp NPM package

See README.md

Specifically, it is set up so that development can happen both directly against the webapp, e.g.

cd packages/webapp
yarn start

or

cd packages/webapp
yarn start:storybook

Or as part of the django app where it gets embedded

yarn build:webapp

and reloading the django page.

Single Sign-on (SSO)

To use the Single Sign-on functionality in local development, make sure you set the relevant environment variables to match your local setup, e.g.

HTTP_PROTOCOL=http \
DISCOURSE_SSO_SECRET=... \
CKAN_SSO_URL=http://localhost/user/login \
EMAIL_HOST=localhost \
EMAIL_PORT=2525 \
EMAIL_USE_TLS= \
python manage.py runserver

Logout from ckan might not send you to the right URL to logout from DataManager. You can manually go to logout from this app at http://localhost:8000/accounts/logout/.

Additional environment variables sometimes needing customisation for development

Environment variable	Description
DEBUG_CACHE	Enable the django app cache. Normally disabled by DEBUG=True, this enables it for development - see more in settings.py.
ROBOTS_DENY_ALL	Set to true to disallow robots, e.g. to avoid sandbox getting indexed by google
GUNICORN_WORKERS_ARG	e.g. --workers 2 - use this to specify the number of workers gunicorn should run.

Running tests

All tests

docker-compose run --rm test

Specific tests, e.g.

docker-compose run --rm test python manage.py test budgetportal.tests.test_bulk_upload.BulkUploadTestCase

Production deployment

Initial Deployment

You'll need:

• API key for a CKAN user who can modify datasets in the national-treasury organisation

Create and configure the app

dokku apps:create budgetportal
dokku config:set budgetportal DJANGO_DEBUG=false \
                              DISABLE_COLLECTSTATIC=1 \
                              DJANGO_SECRET_KEY=some-secret-key \
                              CKAN_API_KEY=... \
                              DATABASE_URL=postgresql://... \
                              EMAIL_HOST_PASSWORD=... \
                              DISCOURSE_SSO_SECRET=... \
                              RECAPTCHA_PRIVATE_KEY=... \
                              SENTRY_DSN=https://[email protected]/1730285 \
                              AWS_ACCESS_KEY_ID=... \
                              AWS_SECRET_ACCESS_KEY=... \
                              AWS_STORAGE_BUCKET_NAME=... \
                              SOLR_URL=... \
                              COMMENTS_ENABLED=True

git push dokku master
dokku run budgetportal python manage.py migrate
dokku run budgetportal python manage.py createsuperuser
dokku ps:scale budgetportal worker=1

Also use dokku domains to configure the hostnames that your app will serve.

Authentication

Apart from the superuser, additional users authenticate using either username+password or OAuth with social media accounts, e.g. Google and Facebook.

To enable this, we use django-allauth add social media account providers which provide verified email addresses in Django Admin's Social Accounts section.

For Google, set up an OAuth Client ID in Google API Console

Loading departments in bulk

The departments and their metadata are loaded using the Django Manage Command load_departments.

The command can be called to update the production database using

DATABASE_URL=postgresql://...produser...:...prodpasswd...@proddbhost/budgetportal python manage.py load_departments 2018-19 national departments.csv

where

• 2018-19 is the financial year slug
• national is the sphere slug
• departments.csv is a CSV file as follows:

Required columns:

• government - government name
• department_name
• vote_number

Optional columns:

• is_vote_primary - TRUE or FALSE
• intro
• website_url

Markdown syntax must be used for formatting intro. e.g. 2 line breaks will result in new paragraphs. Use headings like ## Vote purpose

e.g. for national

government	department_name	vote_number	is_vote_primary	intro	website_url
South Africa	The Presidency	1	TRUE	## Vote purpose Facilitate a common programme towards the ... ## Mandate To serve the president in the execution of his ...	http://www.thepresidency.gov.za/
South Africa	Centre for Public Service Innovation	10	FALSE	## Vote purpose Facilitate the unearthing, development and practical ... ## Mandate The responsibility for public sector innovation is vested in the Minister of Public Service...	www.cpsi.co.za

e.g. for provincial

government	department_name	vote_number	is_vote_primary	intro	website_url
Eastern Cape	Health	3	TRUE	## Vision A quality health service to the people of the ... ## Mission To provide and ensure accessible comprehensive integrated ... ## Core functions and responsibilities The strategic objectives are in line with the implementation

License

MIT License