Finalize Debian packaging strategy and developer guidelines #136

eloquence · 2018-08-30T21:17:28Z

We've reached a preliminary agreement to package SecureDrop Workstation code, such as the SDK and the client, as Debian packages. To do so, we need clear developer packaging guidelines that should be committed as documentation to this repository. Preparatory to creating these guidelines, some initial research and experimentation is required.

Review turnaround cycles for recent security vulnerabilities in dependencies used by SecureDrop (e.g., Flask) to determine if using official Debian packages for Python dependencies is an option
- This review concluded that we will need to create our own wheels at least for some packages and distribute them inside .debs, while ideally installing them into a virtualenv using, e.g., https://github.com/spotify/dh-virtualenv
Create developer guidelines with example package
Final test run and sign-off by @redshiftzero

2018-09-07: Updated and re-scoped per discussion.

2018-10-04: Updated per outcomes of sprint planning

redshiftzero · 2018-08-30T21:24:12Z

Thanks for filing this @eloquence, note that it makes sense to put all our packaging code in a new repo (but we can keep this tracking issue here for visibility)

kushaldas · 2018-09-07T21:48:34Z

https://packages.debian.org/search?keywords=python-flask says Sid has 1.0.2-1: all and Stretch on 0.12.1-1: all. I could not find any bug tracker entry for the same https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=python-flask.

redshiftzero · 2018-09-07T22:13:35Z

Let's look at sqlalchemy, alembic, werkzeug, cryptography, pyqt, python-gnupg. If all the Debian packages of these Python dependencies aren't updated promptly after security releases or do not have critical functionality that we need, then we need to re-evaluate the packaging strategy. If it's just one or two packages that are not promptly updated for security fixes, perhaps we can take action just to determine how the process is for getting those packages updated (e.g. we can file a ticket for the python-flask package noting the recent CVE).

kushaldas · 2018-09-07T22:29:30Z

python-werkzeug

https://packages.debian.org/search?keywords=Werkzeug&searchon=names&suite=stretch&section=all

0.11.15+dfsg1-1

python-sqlalchemy

https://packages.debian.org/search?keywords=sqlalchemy&searchon=names&suite=stretch&section=all

1.0.15+ds1-1

python-alembic

https://packages.debian.org/search?suite=stretch&section=all&arch=any&searchon=names&keywords=alembic

0.8.8-2

http://alembic.zzzcomputing.com/en/latest/changelog.html has a lot of bug fixes.

python-cryptography

https://packages.debian.org/search?suite=stretch&section=all&arch=any&searchon=names&keywords=cryptography

1.7.1-3

We have a CVE https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10903 but, the debian version is too
old

python-gnupg (There is no python3 package)

https://packages.debian.org/search?suite=stretch&section=all&arch=any&searchon=names&keywords=python-gnupg

0.3.9-1: all

pyqt5-dev

https://packages.debian.org/search?suite=stretch&section=all&arch=any&searchon=names&keywords=pyqt

5.7+dfsg-5

There is a CVE in Qt.

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15011

Flask

https://packages.debian.org/search?keywords=python-flask

0.12.1-1

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1000656

gnupg

https://packages.debian.org/search?suite=stretch&section=all&arch=any&searchon=names&keywords=gnupg

2.1.18-8~deb9u2

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-12020

redshiftzero · 2018-09-07T22:44:54Z

Given the situation with CVEs in multiple important packages, and several Python dependencies we need not being packaged for Debian, we should re-evaluate the packaging strategy. The possible options are:

Package all of these Python dependencies ourselves as separate Debian packages. I think this is an unreasonable amount of work.
Use wheels for our dependencies (cross-linking with the related [RFE] Isolate yourself from the system Python securedrop#3407), in a similar manner to what we've been doing with securedrop-app-code.
Use a different base OS (cc Use Debian-based templates for all VMs #111) that is better with keeping Python packages updated. I think we should stick with our decision to use Debian as we benefit from significant team knowledge of Debian, the packaging of many other packages that we want in the workstation (e.g. MAT), and the EOL issue (I know there is an ongoing discussion about the ability for us to update base templates in Workstation update process #24, but I'm mentioning here for completeness).

I think we should re-evaluate option 2 - what do you think? Do you see significant downsides to that approach?

kushaldas · 2018-09-07T22:51:44Z

I think we should re-evaluate option 2 - what do you think? Do you see significant downsides to that approach?

None of the Operating Systems provide a formal way to install/update/maintain packaging of weel based installation. The whole reason we have python wheel so that people don't have to compile for their platforms in the virtualenvs and use system packages where people want to update/maintain in a safer way.

Qubes still uses option 3 has their default option :)

redshiftzero · 2018-09-07T23:03:15Z

Regarding option 3, I'm looking around and we're hitting similar problems (though overall the packages are more recent versions): some packages are not being packaged for Fedora (e.g. python-gnupg packaged for Fedora is not the fork of python-gnupg we are using in SecureDrop), and it looks like Flask is old as in Debian (0.12 series). I think we need to have control over updating these dependencies no?

conorsch · 2018-09-07T23:12:41Z

I think we should re-evaluate option 2

Agreed. While the goal of using strictly repo-hosted packages to install all application dependencies is a noble one, what we really care about is the ability to ship updates in a timely manner, to ensure a minimal attack surface over the life of a workstation installation.

Why should we not use wheel? It seems like it would provide us flexibility over both Debian and Fedora repos in terms of updating dependencies when we deem it necessary.

None of the Operating Systems provide a formal way to install/update/maintain packaging of weel based installation. The whole reason we have python wheel so that people don't have to compile for their platforms in the virtualenvs and use system packages where people want to update/maintain in a safer way.

That doesn't sound untenable to me. Maintaining wheel packages is a bit of a pain on the packagers, but I'm willing to absorb that cost if it results in a stronger workstation config over time.

eloquence · 2018-09-07T23:52:29Z

To recap our ad hoc discussion today:

We're already using wheels inside Debian packages for SecureDrop server code, and are comfortable doing so here (including the maintenance burden this entails), with one caveat: As outlined in [RFE] Isolate yourself from the system Python securedrop#3407 using dh-virtualenv or a similar virtualenv solution offers important additional protection against conflicts with system packages, and we should ensure that we do so from the start in Qubes.
With that agreed upon approach in mind, Kushal will attempt to package securedrop-sdk and create packaging guidelines as outlined in this ticket, as part of the current sprint. The SDK itself still requires modifications to work with the current proxy code, so the .deb will be of limited usefulness at this stage, and that's fine.

redshiftzero · 2018-09-10T18:59:30Z

Hey @kushaldas - you mentioned in chat today that you'd been doing packaging using dh-virtualenv 🎉 - can you update this ticket with the documentation and (a pointer to) the scripts you used to build those debs (we can make another repo for this)? Several people on the team should run through these docs so we all are comfortable packaging

kushaldas · 2018-09-10T20:07:13Z

@redshiftzero I just started working on the packaging and learning part itself, I will update this ticket during the current sprint with full documentation + examples.

kushaldas · 2018-09-24T18:54:22Z

I have pushed the initial documentation at https://github.com/freedomofpress/securedrop-debian-packaging-guide Time to start building 🏭

redshiftzero · 2018-09-24T19:11:26Z

Hey @kushaldas et al., ReadTheDocs is now set to build the SecureDrop packager documentation off master: https://securedrop-debian-packaging-guide.readthedocs.io/en/latest/

(btw: see PR adding link to README.md here: freedomofpress/securedrop-debian-packaging-guide#1)

eloquence · 2018-09-24T22:13:03Z

Per our discussion today at sprint planning, securedrop-sdk will not itself need to be packaged, but will instead be installed as a dependency via dh-virtualenv (by the client package). Re-scoped the ticket accordingly.

redshiftzero · 2018-09-24T23:28:32Z

One outstanding question based on the current documentation is how to ensure integrity of dependencies from PyPI - i.e. we want to at minimum be using the hashes in Pipfile.lock during the build process. We could ensure that the hashes of the downloaded tarballs match what is in Pipfile.lock (via a custom script if necessary) - this is something we would verify prior to signing new debian packages (cc @emkll)

kushaldas · 2018-09-25T19:13:32Z

We should be able to use computepipfilehash tool to generate requirements.txt with proper hashes. I will update the documentation for the same.

kushaldas · 2018-09-25T23:12:04Z

The guide is now updated to use the tool . We are now using hashes in every part of the pip usage. @redshiftzero @emkll ^^

kushaldas · 2018-09-25T23:14:47Z

If there is any mismatch in the hashes, the debian build command will fail.

THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    Werkzeug==0.14.1 from file:///home/user/packaging/securedrop-proxy-0.1.0/localwheels/Werkzeug-0.14.1-py2.py3-none-any.whl (from -r ./requirements.txt (line 1)):
        Expected sha256 e224b7be15904cc4fbfdcd0d13dff5c7500d026e634af48c85766d5362fa7b37
             Got        e224b7be15904cc4fbfdcd0d13dff5c7500d026e634af48c85766d5362fa7b36

eloquence · 2018-09-27T16:29:57Z

Recapping where we are with this:

The packaging guide repo is at https://github.com/freedomofpress/securedrop-debian-packaging-guide and https://securedrop-debian-packaging-guide.readthedocs.io/ is generated from it
We need to have others (probably @joshuathayer and @conorsch per discussion yesterday) attempt to do a dry run using this guide before we can call this done, probably packaging the current client code (Create Debian package securedrop-client#22).

Question:

Is the example code cited in the guide published somewhere? If not I would suggest committing it to an examples directory or similar.

eloquence · 2018-10-04T21:18:23Z

Per discussion at sprint planning today:

@kushaldas will address the outstanding issue in the guide (we need to account for the fact that wheel builds are not idempotent; see Rebuild only the wheels which are already not present locally securedrop-debian-packaging-guide#3)
@kushaldas will attempt to publish the current securedrop-client and securedrop-proxy packages
@redshiftzero will attempt a final dry run of creating a package following this guide, and close this issue if there are no remaining concerns

redshiftzero · 2018-10-05T23:35:38Z

@kushaldas - looks like the upstream issue to address freedomofpress/securedrop-debian-packaging-guide#3 on our side is waiting for feedback

One other question: can you explain why this freedomofpress/securedrop-debian-packaging-guide@8c3772e change was made? i.e. why the sd-buildwheels and sd-downloadsources commands were added (and can you provide a pointer to the source for those)?

kushaldas · 2018-10-06T02:24:46Z

@redshiftzero I will work with upstream on that wheel issue.

One other question: can you explain why this freedomofpress/securedrop-debian-packaging-guide@8c3772e change was made? i.e. why the sd-buildwheels and sd-downloadsources commands were added (and can you provide a pointer to the source for those)?

I added those as helper command in computepipfilehash package. We can have the full commands also in the guide (instead of those two).

The sources: sd-downloadsources and sd-buildwheels.

redshiftzero · 2018-10-09T21:40:16Z

ok great, thanks for the pointer. Let's include the full command in the guide just for clarity (and we'll eventually be automating this entire deb building process anyway as we do on the server side)

redshiftzero · 2018-10-09T23:14:17Z

@redshiftzero will attempt a final dry run of creating a package following this guide, and close this issue if there are no remaining concerns

I did a dry run building the securedrop-proxy package, which all went well! minor docs changes over in freedomofpress/securedrop-debian-packaging-guide#4

@kushaldas will address the outstanding issue in the guide (we need to account for the fact that wheel builds are not idempotent; see freedomofpress/securedrop-debian-packaging-guide#3)
@kushaldas will attempt to publish the current securedrop-client and securedrop-proxy packages

(but not closing yet as these are outstanding)

eloquence · 2018-10-25T15:45:46Z

To quote @redshiftzero:

if the C libs aren't reproducibly built (why the binary wheels aren't reproducible for certain libraries) then we are unfortunately stuck for now

So, closing this issue, we can revisit the reproducibility issue down the road if we find a way around it.

eloquence added the epic label Aug 30, 2018

eloquence assigned kushaldas Aug 30, 2018

eloquence removed the epic label Sep 5, 2018

redshiftzero mentioned this issue Sep 7, 2018

[securedrop-proxy] Add proper setup.py based on setuptools freedomofpress/securedrop-proxy#4

Closed

eloquence added this to the 0.1alpha milestone Sep 8, 2018

eloquence mentioned this issue Sep 11, 2018

Create Debian package freedomofpress/securedrop-client#22

Closed

eloquence mentioned this issue Sep 21, 2018

[securedrop-proxy] Create Debian package freedomofpress/securedrop-proxy#6

Closed

eloquence mentioned this issue Sep 24, 2018

Create an API proxy as a qrexec service #107

Closed

3 tasks

eloquence closed this as completed Oct 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Finalize Debian packaging strategy and developer guidelines #136

Finalize Debian packaging strategy and developer guidelines #136

eloquence commented Aug 30, 2018 •

edited

Loading

redshiftzero commented Aug 30, 2018

kushaldas commented Sep 7, 2018

redshiftzero commented Sep 7, 2018

kushaldas commented Sep 7, 2018 •

edited

Loading

redshiftzero commented Sep 7, 2018

kushaldas commented Sep 7, 2018

redshiftzero commented Sep 7, 2018

conorsch commented Sep 7, 2018

eloquence commented Sep 7, 2018 •

edited

Loading

redshiftzero commented Sep 10, 2018

kushaldas commented Sep 10, 2018

kushaldas commented Sep 24, 2018

redshiftzero commented Sep 24, 2018

eloquence commented Sep 24, 2018

redshiftzero commented Sep 24, 2018

kushaldas commented Sep 25, 2018 •

edited

Loading

kushaldas commented Sep 25, 2018

kushaldas commented Sep 25, 2018

eloquence commented Sep 27, 2018

eloquence commented Oct 4, 2018

redshiftzero commented Oct 5, 2018

kushaldas commented Oct 6, 2018

redshiftzero commented Oct 9, 2018

redshiftzero commented Oct 9, 2018

eloquence commented Oct 25, 2018

Finalize Debian packaging strategy and developer guidelines #136

Finalize Debian packaging strategy and developer guidelines #136

Comments

eloquence commented Aug 30, 2018 • edited Loading

redshiftzero commented Aug 30, 2018

kushaldas commented Sep 7, 2018

redshiftzero commented Sep 7, 2018

kushaldas commented Sep 7, 2018 • edited Loading

python-werkzeug

python-sqlalchemy

python-alembic

python-cryptography

python-gnupg (There is no python3 package)

pyqt5-dev

Flask

gnupg

redshiftzero commented Sep 7, 2018

kushaldas commented Sep 7, 2018

redshiftzero commented Sep 7, 2018

conorsch commented Sep 7, 2018

eloquence commented Sep 7, 2018 • edited Loading

redshiftzero commented Sep 10, 2018

kushaldas commented Sep 10, 2018

kushaldas commented Sep 24, 2018

redshiftzero commented Sep 24, 2018

eloquence commented Sep 24, 2018

redshiftzero commented Sep 24, 2018

kushaldas commented Sep 25, 2018 • edited Loading

kushaldas commented Sep 25, 2018

kushaldas commented Sep 25, 2018

eloquence commented Sep 27, 2018

eloquence commented Oct 4, 2018

redshiftzero commented Oct 5, 2018

kushaldas commented Oct 6, 2018

redshiftzero commented Oct 9, 2018

redshiftzero commented Oct 9, 2018

eloquence commented Oct 25, 2018

eloquence commented Aug 30, 2018 •

edited

Loading

kushaldas commented Sep 7, 2018 •

edited

Loading

eloquence commented Sep 7, 2018 •

edited

Loading

kushaldas commented Sep 25, 2018 •

edited

Loading