Add securedrop keyring package and update Release Key in sys-firewall

[emkll]

Status

Ready for review. Still marked as draft as to drop the final commit on this branch once this is approved and the securedrop-keyring package is uploaded to apt-test for final testing

Description of Changes

Fixes #438

Adds securedrop-keyring package (see freedomofpress/securedrop-builder#171)

This will require all end users to run securedrop-admin --apply prior to the key expiry date, once the following packages are uploaded to the package repos:

• rpm containing these changes
• securedrop-keyring package
• (optional, but ensures new template builds use it) securedrop-workstation-config package

Some challenges with handing sys-firewall state, which will cause a significant (~300 lines, including tests) change to the Preflight Updater:

Some of the challenges I've observed (see commit messages for more details) are as follows, mostly boiling down to the fact that the updater only enforces dom0 state:

1. We cannot include sys-firewall changes as part of dom0 changes unless there's a significant refactoring of the current logic.
2. Using the Updater will allow us to have test coverage which could be easier to manage complex conditional logic and provide more robust logging of failures.
3. We don't want to always want apply sys-firewall changes: every time we apply the state, as running it will requie the mgmt vm to power on, apply state and power off, significantly increasing time on every updater run
4. files in /srv/salt are not accessible by the user (require sudo), so using a hard-coded hash to compare those files will work in both dev and staging/prod scenarios without unnecessary permissions

Testing

Prerequisites

(Steps below not required if securedrop-keyring package has been uploaded to apt-test.freedom.press)

• Build securedrop-keyring 0.1.4 package as per Add securedrop-keyring package securedrop-builder#171
• copy package to dom0:/srv/salt/sd/sd-workstation/

Clean Install

• make clean to remove all existing SDW VMs
• make clone on this branch
• make all completes without errors
• make test complete without errors

Upgrade install

• start with an existing install (0.3.0, but master also works since no changes have been introduced since 0.3.0
• make clone on this branch
• make all completes without errors
• make test complete without errors

Checklist

If you have made code changes

• Linter (make flake8) passes in the development environment (this box may
  be left unchecked, as flake8 also runs in CI)

If you have made changes to the provisioning logic

• All tests (make test) pass in dom0 of a Qubes install

[conorsch]

Overall the strategy is sound. There's a bit of variance in the keyring tests, worth a closer look on why tildes are appended to the keyring files. Once that's resolved, I suggest moving to ready-for-review state.

[conorsch]

Not on my system:

user@host:~$ qubesdb-read /name
sd-whonix
user@host:~$ find /etc/apt/trusted.gpg.d/| grep -i securedrop
/etc/apt/trusted.gpg.d/securedrop-keyring.gpg

After patching the test, I was able to get it to pass.

[conorsch]

Different number of hyphens? That's a head-scratcher. I suspect that number is determined by the length of filepath:

>>> hyphens = "----------------------------------------------"
>>> len(hyphens)
46
>>> keyring_path = "/etc/apt/trusted.gpg.d/securedrop-keyring.gpg"
>>> len(keyring_path)
45
>>>

Indeed, matches test output:

FAIL: test_debian_keyring_config (test_vms_platform.SD_VM_Platform_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 248, in test_debian_keyring_config
    self._ensure_keyring_package_exists_and_has_correct_key(vm, is_whonix=True)
  File "/home/user/securedrop-workstation/tests/test_vms_platform.py", line 138, in _ensure_keyring_package_exists_and_has_correct_key
    self.assertEqual(fpf_gpg_pub_key_info, results.split('\n'))
AssertionError: Lists differ: ['/et[86 chars]------', 'pub   rsa4096 2016-10-20 [SC] [expir[219 chars]ss>'] != ['/et[86 chars]-----', 'pub   rsa4096 2016-10-20 [SC] [expire[218 chars]ss>']

First differing element 1:
'----------------------------------------------'
'---------------------------------------------'

  ['/etc/apt/trusted.gpg.d/securedrop-keyring.gpg',
-  '----------------------------------------------',
?   -

+  '---------------------------------------------',
   'pub   rsa4096 2016-10-20 [SC] [expires: 2021-06-30]',
   '      22245C81E3BAEB4138B36061310F561200F4AD77',
   'uid           [ unknown] SecureDrop Release Signing Key',
   'uid           [ unknown] SecureDrop Release Signing Key '
   '<[email protected]>']

----------------------------------------------------------------------
Ran 70 tests in 256.289s

[emkll]

There was also some variance to what appears to be a default .gnupg.conf in the Whonix template

[eloquence]

Given the limited pilot size and the fact that this enables automatic keyring updates in future, during sprint planning today, we agreed that it's reasonable to rely on a one-time securedrop-admin --apply run to ensure the keyring is present where required. This and associated PRs can be de-scoped accordingly, and I'll take point on pinging our admins/users to run the command & verify that it was successful.

The clearer we can make the failure case for the user, the better, so we don't have to go through pages of scrollback to find out what went wrong.

[emkll]

Updated this pr and associated packaging PR based on the feedback above, this is now ready for re-review. The next steps should be as follows:

• Use this PR to test securedrop-keyring package (Add securedrop-keyring package securedrop-builder#171)
• Merge Add securedrop-keyring package securedrop-builder#171
• Upload securedrop-keyring to apt-test (since we will not be building nightlies of that package in CI): Adds securedrop-keyring package to apt-test securedrop-apt-test#45
• Drop 3beffa2
• Final review on this PR, once Adds securedrop-keyring package to apt-test securedrop-apt-test#45 is merged

[zenmonkeykstop]

Environment:

• Environment type: Qubes dev
• Client version: n/a
• Workstation version: 438-keyring-package
• Server version: n/a

Test purpose:

PR review

Test steps:

Followed PR fresh install test plan as far as make all

Test results:

make all fails with:

Set up logging VMs early
sd-log-buster-template:
  ----------
            ID: topd-always-passes
      Function: test.succeed_without_changes
          Name: foo
        Result: True
       Comment: Success!
       Started: 10:14:12.879001
      Duration: 0.482 ms
       Changes:   
  ----------
            ID: dsa-4371-update
      Function: cmd.script
        Result: True
       Comment: Nothing to do, apt already fixed.
       Started: 10:14:12.884514
      Duration: 209.577 ms
       Changes:   
  ----------
            ID: update
      Function: pkg.uptodate
        Result: True
       Comment: Upgrade ran successfully
       Started: 10:14:20.806458
      Duration: 32419.894 ms
       Changes:   
                ----------
                securedrop-log:
                    ----------
                    new:
                        0.1.1-dev-20200604-060619+buster
                    old:
                        0.1.1-dev-20200602-060809+buster
  ----------
            ID: notify-updates
      Function: cmd.run
          Name: /usr/lib/qubes/upgrades-status-notify
        Result: True
       Comment: Command "/usr/lib/qubes/upgrades-status-notify" run
       Started: 10:14:53.236175
      Duration: 6055.619 ms
       Changes:   
                ----------
                pid:
                    2210
                retcode:
                    0
                stderr:
                stdout:
  ----------
            ID: install-python-apt-for-repo-config
      Function: pkg.installed
        Result: True
       Comment: All specified packages are already installed
       Started: 10:15:05.051223
      Duration: 12.668 ms
       Changes:   
  ----------
            ID: configure-apt-test-apt-repo
      Function: pkgrepo.managed
          Name: deb [arch=amd64] https://apt-test.freedom.press buster main
        Result: True
       Comment: Package repo 'deb [arch=amd64] https://apt-test.freedom.press buster main' already configured
       Started: 10:15:05.067941
      Duration: 76.912 ms
       Changes:   
  ----------
            ID: install-securedrop-keyring-package
      Function: file.managed
          Name: /opt/securedrop-keyring.deb
        Result: False
       Comment: Source file salt://sd/sd-workstation/securedrop-keyring_0.1.4+buster_all.deb not found in saltenv 'base'
       Started: 10:15:05.181078
      Duration: 4.238 ms
       Changes:   
  ----------
            ID: install-securedrop-keyring-package
      Function: cmd.run
          Name: apt install -y /opt/securedrop-keyring.deb
        Result: False
       Comment: Command "apt install -y /opt/securedrop-keyring.deb" run
       Started: 10:15:05.185490
      Duration: 54.931 ms
       Changes:   
                ----------
                pid:
                    2614
                retcode:
                    100
                stderr:
                    
                    WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
                    
                    E: Unsupported file /opt/securedrop-keyring.deb given on commandline
                stdout:
                    Reading package lists...
  ----------
            ID: install-securedrop-log-package
      Function: pkg.installed
        Result: False
       Comment: One or more requisite failed: fpf-apt-test-repo.install-securedrop-keyring-package
       Started: 10:15:05.241131
      Duration: 0.008 ms
       Changes:   

The ~ version of the keyring is apparently a backup, so wouldn't expect to see it in a fresh install

[conorsch]

@zenmonkeykstop The apt error E: Unsupported file /opt/securedrop-keyring.deb given on commandline is rather cryptic, but indicates that filepath can't be found—see the preceding error on the copy task:

Comment: Source file salt://sd/sd-workstation/securedrop-keyring_0.1.4+buster_all.deb not found in saltenv 'base

So double-check that you've got that deb file in sd-workstation/ locally, both in sd-dev and dom0, then try again!

[zenmonkeykstop]

Sorry, thought I'd added another comment - I'd gotten past that after looking at the salt config (also added the necessary setup instructions in the test plan for other potential reviewers). Seeing the same test failure as you due to the ~ in the keyring name not being present, but otherwise all good.

[emkll]

Thanks @zenmonkeykstop for the review and for updating the test plan. I think i have figured out the source of the confusion (which was entirely on my end):

The issue was with the cleanup of whonix-gw-15: because the keyring is in /etc, the deb package views it as a conffile. As a result, it would not re-install the keyring in place after an apt remove operation. The ~ ended up being a red herring, the issue was with the install process in my local environment after running make clean.

I have updated tests, as well as the clean logic for whonix-gw. I have confirmed this is working locally, the next steps should still be in #563 (review) . This is now ready for re-review

[zenmonkeykstop]

make test is passing for me now - this is in a good state to proceed with the related PRs and then test with the securedrop-keyring deb on apt-test.

[emkll]

Dropped the last commit, ready for final review

[zenmonkeykstop]

Stepped through fresh install scenario again, one new failure on make test:

 ======================================================================
FAIL: test_log_dirs_properly_named (test_log_vm.SD_Log_Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kog/securedrop-workstation/tests/test_log_vm.py", line 50, in test_log_dirs_properly_named
    self.assertFalse("host" in log_dirs)
AssertionError: True is not false

Confirmed that there is a hosts directory in /home/user/QubesIncomingLogs containing syslog.log.

[conorsch]

Am able to reproduce the same isolated test failure that @zenmonkeykstop reports. Haven't investigated yet, but let's take another look.

[zenmonkeykstop]

Looks like the host thing is a timing issue. rsyslog is already running on sd-whonix before /rw/config/rc.local runs to fix up the config with the correct hostname, so it syncs up logs as host to begin with. Then after rsyslog restart, it syncs as sd-whonix.

Verified by:

• disabling rsyslog in whonix-gw-15 and rebooting
• updating rc.local on sd-whonix to enable rsyslogd after the config change and rebooting
• observing new logs synced to sd-log - only sd-whonix logs are synced.

[zenmonkeykstop]

Given that the logging test failure (as documented in comments) is unrelated, this PR LGTM!