Dependency on rust removes support for a number of platforms #5771

mgorny · 2021-02-08T08:58:33Z

I would like to report that the newly added dependency on Rust has made it impossible to package cryptography for a number of supported Gentoo architectures (and these are architectures where people actually use Python packages that depend on cryptography).

Please see Platform Support. Besides degrading a few of our targets to 'Tier 3', Rust completely does not support at least alpha, hppa, ia64, m68k, s390 (not -x). The cost of adjusting cryptography's C code to these platforms (even if we assumed they wouldn't work out of the box as they do so far) is much less than the cost of porting the whole Rust. You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

Right now it's a really hard struggle to even get Rust packaged for non-Tier 1 architectures (i.e. where upstream does not provide bootstrap binaries).

LindezaGrey · 2021-02-08T09:16:33Z

destroyed my CI/CD Pipeline this morning 😆

clanzett · 2021-02-08T09:36:51Z

It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.

tiran · 2021-02-08T09:48:40Z

It is also impossible to build on ANY alpine linux version/architecture.

Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).

Please remove this dependency as soon as possible.

Not going to happen. Rust dependency will stay to replace C code with an actual safe language.

You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.

clanzett · 2021-02-08T09:53:09Z

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

mgorny · 2021-02-08T09:56:38Z

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

tiran · 2021-02-08T10:06:29Z

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

silverfisk · 2021-02-08T10:13:18Z

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

clanzett · 2021-02-08T10:13:35Z

Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

clanzett · 2021-02-08T10:13:59Z

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

Smirl · 2021-02-08T10:23:30Z

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:

BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed.
We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform.
cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0.
cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.

This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.

Update
I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.

tiran · 2021-02-08T10:28:11Z

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.

mgorny · 2021-02-08T11:00:56Z

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

tiran · 2021-02-08T11:13:50Z

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

public · 2021-02-08T11:17:22Z

@tiran

I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed.
      =============================DEBUG ASSISTANCE=============================

Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

mgorny · 2021-02-08T11:17:54Z

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

That's a very roundabout way of showing the middle finger to users of 'other hardware platforms' that are not supported by Rust (or LLVM) at all.

fourstepper · 2021-02-08T11:35:27Z

CI testing went down the drain with this in GitLab CI with docker:stable-dind, as it's based off of Alpine. I can imagine this happening all over the place.

tiran · 2021-02-08T11:37:44Z

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.

The documentation for Alpine builds has been improved #5763 already. I have a pending PR for Fedora, CentOS, and Debian at #5764 . If you have any suggestions for better build instructions, please help by opening a PR.

thesamesam · 2021-02-08T11:39:31Z

For myself, I'd like to note that a heads-up in the CHANGELOG / Release Notes (I may have missed it) would've helped:

when considering the move ("we're considering doing this, please give feedback")
when it was decided it should actually happen ("release X will optionally contain Rust, release Y will have mandatory Rust")

I don't see any mention before the release in the CHANGELOG or in the emails to python-announce-list.

That's the one thing which distro maintainers or packagers really should be reading on every new release. Including it as a warning in previous releases would've helped us out here to both object/raise concerns and begin preparations.

cdesousa · 2021-02-08T11:53:28Z

To be fair the developers did ask for feedback (in a not very inviting way):

We are interested in feedback in things we can do to make this a smooth process, particularly from folks who are re-distributing cryptography. Please note: feedback of the form “don’t use Rust” is not productive and will be disregarded

I highly suspect there will be many more complaints throughout the day about stuff breaking. I had stuff breaking on amazon linux and was confused as to why pip install ansible was asking for rust, seems that one is tagged to latest cryptography in requirements.txt (which is of course not directly the fault of the devs here).

MrMino · 2021-02-08T11:55:35Z

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about first impressions 😄. That's a good reminder to update the alpine images though.

Hands up: who here watches every update of every transitive dependency in each app/environment they have to maintain?

BTW. This actually gives me ideas about dependency change notification systems for developers. Notifying users about incompatible dependency changes has been an issue in this industry for what, last 25 years? How much has changed wrt this during that time?

clanzett · 2021-02-08T12:02:40Z

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.
I suppose this is perhaps a bit naive. I don't blame you, I'm not sure what else you could've done to inform your users either. That said, you need to realize that your project has just introduced itself to a few thousands developers by killing their CICD runs. Talk about reminders to update to the newest alpine 😄.

I totally agree. I really think we did not even scratch the tip of the iceberg. There might be some "interesting" problems ahead.

And don't get me wrong @tiran, I also agree with you regarding your security concerns, but this switch could have really done some serious damage. We will see...

xgqt · 2021-02-08T12:20:36Z

Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

other hardware platforms

lol

ambv · 2021-02-08T12:39:23Z

It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.

Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.

You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.

In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.

xgqt · 2021-02-08T12:45:28Z

Let's depend on Rust because everything does!
Let's depend on glibc because everything does!
Let's depend on GNU coreutils because everything does!
Let's depend on systemd because everything does!
Let's depend on Linux because everything does!

Afternote: should be "HARD depend" - without the ability to toggle building with the given component.

manuelmarquezs · 2021-02-08T13:24:14Z

We have been able to fix our alpine Pipelines by following the suggested changes in the documentation but they are now extremely slow. We have gone from 30s to 4min. The build of the cryptography package that used to take a few seconds now takes minutes.

We have updated alpine, installed all the dependencies listed in the docs: sudo apk add gcc musl-dev python3-dev libffi-dev openssl-dev cargo and we are using Rust>1.45.0.

Any idea on why the compilation with Rust is that slow??
How could we speed it up??

Building wheels for collected packages: cryptography
  Building wheel for cryptography (PEP 517): started
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): still running...
  Building wheel for cryptography (PEP 517): finished with status 'done'

wvxvw · 2021-02-08T13:26:43Z

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

cryptography is a transitive dependency in our code: through jira, which requires oauthlib. Of course, in part, the blame is on pip, which makes it very hard to signal that the new version of the package is incompatible / wouldn't try to find the actually compatible version. But you, as a maintainer, also did a very bad job at signaling this to pip.

If you really wanted this breaking change, the only real way to do this as of right now (wrt' tools like pip) is to create a package with a different name. Changing versions won't really help due to transitive dependencies other package maintainers may have no control over.

I'd be totally fine with there being a cryptography2, with newest bells and whistles. But, I have zero incentives for updating the dependencies for our automation tools. It's already too damn hard to find a combination that works. The "benefits" of Rust being a "more secure language" are completely worthless in my case.

felixfontein · 2021-02-08T13:31:47Z

@wvxvw creating a new package would actually make the situation a lot worse. Except if the Python module name is also renamed, which would require everyone to adjust there code. Which would also be worse IMO.

tiran · 2021-02-08T13:35:17Z

It's really not just "some other platforms". Ubuntu 18 is also affected... We are using system distribution of Python to run some automation on Ubuntu images built for AWS and Azure.

Ubuntu Bionic LTS has recent Rust and Cargo version. pip is the problem here. Ubuntu 18.04 comes with an old, unsupported pip version. I recommend that you install a newer pip version in a virtual env. It might also be worth the trouble to file an update request with your vendor.

# echo $(source /etc/os-release; echo $VERSION)
18.04.5 LTS (Bionic Beaver)
# apt update
# apt install -y rustc python3-pip
# rustc -V
rustc 1.47.0
# pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

MarkSG93 · 2021-02-09T12:00:54Z

It sounds like y'all are getting an unexpected lesson about the flip-side of open source software. I would like to push back on some of the entitlement I'm sensing reading in this thread.

Before I begin, I'd like to remind you that security is a numbers game. If the cryptography maintainers can help 90% of their users by switching to a modern memory-safe language, then they'd be irresponsible holding back just because among the remaining 10% there exist fringe platforms which can't even run a Rust compiler.

You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

It sounds like you're saying that your time as a Gentoo and NetBSD maintainer is worth more than the time of cryptography maintainers. They are volunteers as well, y' know. You expect those volunteers to keep their security-focused project dependent on inherently insecure technology because it would make your own job easier. Your goals and requirements might not be matching the goals and plans of the maintainers of cryptography. It might be unfortunate for you but it really is as simple as that.

In fact, your only real recourse long-term is exactly to band up volunteers to make Rust work on your platform. Or pay some hired guns to do the initial work for you. After all, you will be seeing more and more projects depending on this programming language as time goes on. Yelling at creators of those projects for choosing a language that is inconvenient for you is both ineffective and laughable.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

It's a security-focused package. If you are not already paying attention to its development then you're putting your users at risk.

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

I invite you to do that. Please put your money and time where your mouth is. Report back in a year's time how it went.

Nobody is saying that it's a bad idea. Don't push a minor patch version for a very clear breaking change and none of these conversations would even be happening. Stop making assumptions about peoples systems and software and use correct versioning. If this was cryptography 4.0.0 then nobody would care because all their builds and docker images still work, but because it's 3.x a lot of us are up a creek without a paddle.

noonien · 2021-02-09T12:04:37Z

ITT: People that use (probably not formally verified) CRYPTOGRAHPY packages, distributed as BINARY BLOBS, that cannot be verified to be compiled from a certain commit, versions of which get AUTOMATICALLY UPDATED in their CIs, making even source builds untrusted.

You guys are surprised your builds break? And then blame the author?

I suggest you ask for a refund.

tamasgal · 2021-02-09T12:05:58Z

Holy crap, now I need to install Rust on like 20 base Docker images which we use for around 300 software projects in our collaboration. Seriously?

kaniini · 2021-02-09T12:07:21Z

Thank you very much! That's a valuable piece of information. I filed https://bugs.python.org/issue43179 for __s390__.

To be clear, hppa and m68k are still quite alive and well in Linux, please don't use that as justification to remove support for those archs. I like running Python on my Amiga.

tiran · 2021-02-09T12:07:24Z

Holy crap, now I need to install Rust on like 20 base Docker images which we use for around 300 software projects in our collaboration. Seriously?

No, you don't. There are multiple documented workarounds like updating pip to >19.1 or setting an env var. It's all explained in the error output.

ignaloidas · 2021-02-09T12:14:46Z

@tiran I've just run the queries on the BigTable of PyPi download data. It shows what pypistats.org doesn't, and that is that 87% of cryptography downloads over the last 7 days were source downloads. This goes against your claims of the majority of users using binary wheels. You don't know who your users are.

Edit:

Query for reference:

SELECT file.type, COUNT(*) AS num_downloads
FROM `the-psf.pypi.file_downloads`
WHERE file.project = 'cryptography'
 -- Only query the last 7 days of history
 AND DATE(timestamp)
   BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 7 DAY)
   AND CURRENT_DATE()
GROUP BY file.type;

msva · 2021-02-09T12:19:59Z

I've deleted all my comments in this issue to stop affecting anyone who can be affected in negative manner by them.

But I still want to describe my point of view: despite of authors are have all the rights about their code, I personally think that becoming an industry standard adds something like "moral restrictions" and responsibility, even over "all of those who uses it wrong way".

tiran · 2021-02-09T12:30:59Z

@tiran I've just run the queries on the BigTable of PyPi download data. It shows what pypistats.org doesn't, and that is that 87% of cryptography downloads over the last 7 days were source downloads. This goes against your claims of the majority of users using binary wheels. You don't know who your users are.

the-psf.pypi.file_downloads is a download statistics, not a users statistics.

You might be right that majority of our users don't use binary wheels. You might draw wrong conclusion from the statistic, because the data might not reflect a large chunk of our users. Amongst others packages or container image layers are cached at multiple levels. Linux distros and Python distributions like Anaconda have their own distribution channels. It's impossible to draw accurate user statistics from the download stats.

MrMino · 2021-02-09T12:32:39Z

@kaniini I gather that you are interested in a "manymusl" format. I created a thread for it on Python Discourse, I invite you to share your thoughts: https://discuss.python.org/t/wheels-for-musl-alpine/7084/15

The largest obstacle so far is that there'd have to be a champion for the standard. While I'd love to try to contribute in this way, I'm pretty sure it's above my head for now.

ignaloidas · 2021-02-09T12:46:55Z

You might be right that majority of our users don't use binary wheels. You might draw wrong conclusion from the statistic, because the data might not reflect a large chunk of our users. Amongst others packages or container image layers are cached at multiple levels. Linux distros and Python distributions like Anaconda have their own distribution channels. It's impossible to draw accurate user statistics from the download stats.

Yet you used statistics from exact same source to claim that the majority of your users use Linux. It is impossible to draw any reliable user statistics for a project with such wide usage, yet you try to justify your choices with claims about your users that you made up.

MrMino · 2021-02-09T12:47:38Z

You might draw wrong conclusion from the statistic

@tiran @ignaloidas

I don't think its even the right statistic. Ask JFrog how many of their clients have Artifactory, and how many instances those clients have. If you count in those (how would you even count them?), then maybe you can get any sort of real world value for downloads.

Counting it in terms of users is IMO the wrong way to go about it. If I have a small business that does ~1000 CI runs a day, and I lead the business alone, do you count me in the same way as the guy next door who accidentally downloaded paramiko?

These stats say nothing about the scale of the problem, and arguing over them is a waste of time.

naparuba · 2021-02-09T12:52:13Z

But I still want to describe my point of view: despite of authors are have all the rights about their code, I personally think that becoming an industry standard adds something like "moral restrictions" and responsibility, even over "all of those who uses it wrong way".

That's where I am not agree and think the OSS will burnout most mainteners: it's up to tje industry to manage all this additonal load (because it is, and lot of overload for such a project), not to upstream.

If it's managed by a big corpo, it's ok: they have all the power to manage such a load (they already doing it for internal use)
But I think force all upstream projects to manage it is a very wrong thing. All the overload must be managed by where the money/power/time is: the industry itself.

Absolucy · 2021-02-09T12:54:52Z

Probably the most expedient method of supporting minority platforms in Rust would be to add a backend to LLVM that emits plain C code, which can then be fed in to GCC or whatever. Or maybe write an LLVM IR to C decompiler, which would not even require co-operation with upstream. Sounds like a fun project I would like to work on, if I had any spare time...

@ljmccarthy That backend did used to exist. It was removed in LLVM 3.1, which was released in 2012. I think there's some unofficial C backend for modern LLVMs, but there's a decent chance it's not quite production-ready.

To get back on topic -- what do people think about an LTS branch for cryptography 3.3.x, which we would support until some point shortly after the Rust GCC frontend is merged and proven? This would be a reasonable solution that provides an answer for those needing a more portable package now while the portability issues get resolved over time.

@kaniini Problem is, that probably won't happen for years, and does anyone really want to take on the burden of maintaining a
legacy branch intended for devices that probably won't even make up 0.1% of it's actual users?

However, if you're willing to do so, that's great!

rfink · 2021-02-09T13:52:27Z

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

Yep, forced to do the same (AWS Glue broke when importing this for requests lib)

This is to avoid a known issue with a recent version of cryptography [1] [1] pyca/cryptography#5771 Change-Id: I034bcd24031b5881ae49b8bc03bed6654cd1d335

* Update bifrost from branch 'master' to 5360d0e869daaa9bdb6ab0eeedfd1600453bfd4d - Install at least pip version 19.1.1 This is to avoid a known issue with a recent version of cryptography [1] [1] pyca/cryptography#5771 Change-Id: I034bcd24031b5881ae49b8bc03bed6654cd1d335

ddevault · 2021-02-09T14:05:49Z

Switching to Rust and locking out dozens of platforms is worse for security, because now those users have no choice but to rely on outdated security-critical software OR to port Rust to their platform, which is a gargantuan effort (hi, someone who has tried - and failed - to port Rust to a new platform here). A lot of those people can't afford to "upgrade" to newer hardware - much of which brings its own security concerns, like UEFI.

C is not evil. C is not a bad programming language. C is a universally supported language. Cargo culting Rust is harmful to the ecosystem.

pfalcon · 2021-02-09T14:08:31Z

There were questions on how the announcement/notifications could be handled better. Well, it's fair to say that this change affects the Python ecosystem as a whole, and so release notification should have been posted to the https://mail.python.org/mailman3/lists/python-announce-list.python.org/ mailing list. And indeed it was! But I'd say it was pretty meek: https://mail.python.org/archives/list/[email protected]/thread/J6VZ22LY5IS3WG5SMICUSWQQOK5SAFTQ/

Given the effect it caused, it should have a subject:

PyCA cryptography 3.3.2 and 3.4 released - SYSTEM BREAKAGE ahead

and should have started with:

###################### WARNING! #########################
Installing PyCA cryptography from source (as happens with
older pip releases) now requires Rust to be installed. If
you use old pip (if you didn't check, you likely do) and
don't have Rust installed, all packages depending on
PyCA cryptography will be broken!
###################### WARNING! #########################

And I'm only half-joking here, your recent release announcement before/with such a big change really contain too few exclamation marks and all-caps.

And I second what other people said - dear cryptography developers, you are doing the great work of maintaining security package as volunteers! But not only that, you also BREAK SYSTEMS of other people! Please take not less pride in that than in your development work. And if you don't want to take pride in that (why? it's fun), then maybe indeed spend a bit less time hasting to write more Rust, and a bit more time writing "scary" (read: disclosing the truth to users) (pre)release announcements.

Thanks!

alex · 2021-02-09T14:15:45Z

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

ddevault · 2021-02-09T14:17:01Z

C is in fact very bad and I have a mountain of research to prove it: https://www.usenix.org/conference/enigma2021/presentation/gaynor

https://en.wikipedia.org/wiki/Confirmation_bias

alex · 2021-02-09T14:17:34Z

You read my slides very quickly! I can tell you took them seriously and didn't just drop that link immediately.

ddevault · 2021-02-09T14:18:22Z

You're right - I don't take your research seriously!

alex · 2021-02-09T14:20:06Z

I recommend not using this software package then, regardless of what languages its written in. (Of course, I suspect that's already the case -- you're not a user, you just showed up to complain because this is your hobby horse.)

ddevault · 2021-02-09T14:20:51Z

I am a cryptography user, as a matter of fact, and a packager for affected systems. But thanks for the ad-hominem speculation, appreciate it.

alex · 2021-02-09T14:22:42Z

My apologies. I regret engaging with you at all.

First: this is going to be the second to last last comment on this thread. I think it's run its course. Folks should please feel empowered to file new issues, but I'd request that you only file issues for either a) new issues, b) concrete improvements not already discussed here.

Second: Thank you to all the folks who have engaged constructively here. Our release is better for your contributions.

Third: The last comment on this thread will be a summary of everything discussed here, I'll leave that after I finish writing it.

alex · 2021-02-09T14:42:53Z

I now want to summarize the state of various discussions here:

Outdated Rust installations

We now take advantage of an option in setuptools-rust to provide a clear error message when a user's Rust installation is too old.

In the future, we'll be somewhat conservative in increasing our MSRV.

Update: We have lowered the MSRV to 1.41 in #5823 and this change was released in 3.4.5.

No Rust installation at all

We've now documented very aggressively how to obtain a Rust installation, and to update pip (which will solve the problem for folks on manylinux-compatible distributions).

No future steps are expected.

Alpine

Wheels

It looks like there's folks actively interested in specifying many-musl. Excellent! Once that's defined and supported in pip, we'll ship wheels for that platform.

Update September 20, 2021: The musllinux specification and implementation is complete and cryptography was the first project to upload x86_64 and aarch64 wheels. You need pip >= 21.2.4 for musllinux support.

Bootstrapping Rust on supported platforms

We'll reach out off-list to see how we can support this effort.

LTS

We'll engage on the thread that @tiran filed.

SemVer

We'll consider it, however, I am skeptical for the simple reason that this change would not have required a major version bump (no public APIs were changed). Therefore it's clear that what folks wanted was a major version bump regardless of semver.

Comms

In the future, when we know about a change like this well in advance, we'll include it in our CHANGELOG as soon as we know about it.

If a platform for proactively reaching out to PyPI users exists, we'll use it.

Alpha, S390, HPPA, HPUX, AIX, etc.

Best of luck.

I'd encourage you to contribute platform support to LLVM or contribute to gccrs.

My work to promote language-level memory safety will continue unabated.

Miscellany

I'm sure I missed something (though I did just re-read this thread in its entirety).

tzheleznyak mentioned this issue Feb 8, 2021

Fix broken pipeline on cryptography dependency cyberark/cyberark-conjur-cli#183

Merged

6 tasks

fourstepper mentioned this issue Feb 8, 2021

Dependencies to build on Alpine Linux? #5776

Closed

scjody mentioned this issue Feb 9, 2021

Quieten browser init in test setup plotly/dash#1551

Merged

1 task

everflux pushed a commit to trion-development/docker-jenkins-docker-client that referenced this issue Feb 9, 2021

Update to support pyca/cryptography#5771

481bf8f

openstack-mirroring pushed a commit to openstack/bifrost that referenced this issue Feb 9, 2021

Install at least pip version 19.1.1

5360d0e

This is to avoid a known issue with a recent version of cryptography [1] [1] pyca/cryptography#5771 Change-Id: I034bcd24031b5881ae49b8bc03bed6654cd1d335

pyca locked as too heated and limited conversation to collaborators Feb 9, 2021

alex closed this as completed Feb 10, 2021

Dependency on rust removes support for a number of platforms #5771

Dependency on rust removes support for a number of platforms #5771

Comments

mgorny commented Feb 8, 2021 • edited Loading

LindezaGrey commented Feb 8, 2021

clanzett commented Feb 8, 2021

tiran commented Feb 8, 2021

clanzett commented Feb 8, 2021

mgorny commented Feb 8, 2021

tiran commented Feb 8, 2021

silverfisk commented Feb 8, 2021

clanzett commented Feb 8, 2021 • edited Loading

clanzett commented Feb 8, 2021 • edited Loading

Smirl commented Feb 8, 2021 • edited Loading

tiran commented Feb 8, 2021

mgorny commented Feb 8, 2021

tiran commented Feb 8, 2021

public commented Feb 8, 2021 • edited Loading

mgorny commented Feb 8, 2021

fourstepper commented Feb 8, 2021

tiran commented Feb 8, 2021

thesamesam commented Feb 8, 2021 • edited Loading

cdesousa commented Feb 8, 2021 • edited Loading

MrMino commented Feb 8, 2021 • edited Loading

clanzett commented Feb 8, 2021

xgqt commented Feb 8, 2021 • edited Loading

ambv commented Feb 8, 2021

xgqt commented Feb 8, 2021 • edited Loading

manuelmarquezs commented Feb 8, 2021 • edited Loading

wvxvw commented Feb 8, 2021 • edited Loading

felixfontein commented Feb 8, 2021

tiran commented Feb 8, 2021

MarkSG93 commented Feb 9, 2021

noonien commented Feb 9, 2021

tamasgal commented Feb 9, 2021

kaniini commented Feb 9, 2021

tiran commented Feb 9, 2021

ignaloidas commented Feb 9, 2021 • edited Loading

msva commented Feb 9, 2021 • edited Loading

tiran commented Feb 9, 2021

MrMino commented Feb 9, 2021

ignaloidas commented Feb 9, 2021

MrMino commented Feb 9, 2021

naparuba commented Feb 9, 2021

Absolucy commented Feb 9, 2021 • edited Loading

rfink commented Feb 9, 2021

ddevault commented Feb 9, 2021 • edited Loading

pfalcon commented Feb 9, 2021

alex commented Feb 9, 2021

ddevault commented Feb 9, 2021

alex commented Feb 9, 2021

ddevault commented Feb 9, 2021

alex commented Feb 9, 2021

ddevault commented Feb 9, 2021 • edited Loading

alex commented Feb 9, 2021 • edited Loading

alex commented Feb 9, 2021 • edited by reaperhulk Loading

Outdated Rust installations

No Rust installation at all

Alpine

Wheels

Bootstrapping Rust on supported platforms

LTS

SemVer

Comms

Alpha, S390, HPPA, HPUX, AIX, etc.

Miscellany

mgorny commented Feb 8, 2021 •

edited

Loading

clanzett commented Feb 8, 2021 •

edited

Loading

clanzett commented Feb 8, 2021 •

edited

Loading

Smirl commented Feb 8, 2021 •

edited

Loading

public commented Feb 8, 2021 •

edited

Loading

thesamesam commented Feb 8, 2021 •

edited

Loading

cdesousa commented Feb 8, 2021 •

edited

Loading

MrMino commented Feb 8, 2021 •

edited

Loading

xgqt commented Feb 8, 2021 •

edited

Loading

xgqt commented Feb 8, 2021 •

edited

Loading

manuelmarquezs commented Feb 8, 2021 •

edited

Loading

wvxvw commented Feb 8, 2021 •

edited

Loading

ignaloidas commented Feb 9, 2021 •

edited

Loading

msva commented Feb 9, 2021 •

edited

Loading

Absolucy commented Feb 9, 2021 •

edited

Loading

ddevault commented Feb 9, 2021 •

edited

Loading

ddevault commented Feb 9, 2021 •

edited

Loading

alex commented Feb 9, 2021 •

edited

Loading

alex commented Feb 9, 2021 •

edited by reaperhulk

Loading