Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[RFC 0031] Support ppc64(le) architecture #31

Closed
wants to merge 1 commit into from
Closed

[RFC 0031] Support ppc64(le) architecture #31

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 86 additions & 0 deletions rfcs/0031-ppc64-platforms.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
---
feature: ppc64-platforms
start-date: 2018-08-23
author: CrystalGamma
co-authors: (find a buddy later to help out with the RFC)
related-issues: https://github.com/NixOS/nixpkgs/pull/45340 https://github.com/NixOS/nixpkgs/pull/45472 https://github.com/NixOS/nixpkgs/pull/45474 https://github.com/NixOS/nixpkgs/pull/45475
---

# Summary
[summary]: #summary

Bring support for ppc64 platforms, initially especially ppc64le, to nixpkgs/NixOS.

# Motivation
[motivation]: #motivation

NixOS/nixpkgs has long prided itself to be a general-purpose OS/package repository, allowing its packages to be used across a broad range of systems, and providing more packages than ever.

With alternative architectures such as OpenPOWER and RISC-V gaining traction, now is the time to add support for those architectures.
Specifically motivating this request is the OpenPOWER, or powerpc64le, architecture—and its embodiment in the POWER series of microprocessors from IBM.
Fully open source commercially available systems have recently become widely available for this platform and there are active Nix users who desire support.

# Detailed design
[design]: #detailed-design

The purpose of this RFC is to create consensus for general acceptance of ppc64(le) related pull requests (as long as they are technically reasonable).
A general commitment for contributions to not break ppc64(le) is not necessary as long as the architecture stays in only niche use.
This especially means that building a Continuous Integration infrastructure for ppc64le is not required in the near future.

Bringing support for ppc64(le) platforms will be done as follows:

The first step for implementing support is extending lib/ to recognize powerpc64(le) as a CPU architecture and allowing bootstrap files for the architecture to be (cross-)built and a stdenv based on those bootstrap files to be built.
This has already been merged in [nixpkgs#45340](https://github.com/NixOS/nixpkgs/pull/45340).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The more architectures we support, the harder it's going to be to upgrade gcc and other heavily-patched packages. Maintainers don't have access to the more esoteric platforms.

How do you propose that patches be handled if for example gcc has to be upgraded quickly for security-related reasons?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe lib/systems/platforms.nix should be amended with an extra "officiallySupported" flag.


The author of this RFC is already running a NixOS-based system on his POWER9-based computer, so for a basic system without graphical interface, only a handful of packages are known to require changes to build a working system:

* mesa: currently assumes that any non-ARM system is x86. Requires restructuring the driver selection, but to the author's knowledge (since no actual graphical software was tested yet) it should not require any changes to the actual build description. ([nixpkgs#45474](https://github.com/NixOS/nixpkgs/pull/45474))
* strace: also assumes that any non-ARM system is x86 and so tries to use the m32 personality. Requires checking the available machine personalities for all non-x86 architectures. ([nixpkgs#45472](https://github.com/NixOS/nixpkgs/pull/45472), already merged)
* TeX Live: builds LuaJIT as part of mfluajit, which doesn't work because LuaJIT doesn't support 64-bit Power architectures yet; seems to be possible to disable on that platform without too much impact. ([nixpkgs#45475](https://github.com/NixOS/nixpkgs/pull/45475))
* Bootloader tooling:
* the internal option *system.boot.loader.kernelFile* is currently set to the kernelTarget attribute of the host platform definition.
PowerPC kernels however use `zImage` as build target, yet produce a `vmlinux` file.
A one-line change would allow the option to default to the host platform definition's kernelFile attribute, if set.
* A typical bootloader for OpenPOWER systems is Petitboot. It can read a variety of bootloader configuration files, including those of GRUB 2 and SYSLINUX.
Using GRUB, even with *boot.loader.grub.device* = "nodev", as it is, implies building GRUB for the host architecture, which fails for ppc64le (currently because it requires soft-float).

However, the generic-extlinux-compatible bootloader module can be modified to provide options to store the configuration files in a path that Petitboot will consider and use absolute paths starting from the filesystem root, so that Petitboot can load the kernel/initrd images.

Beyond a headless system, the biggest obstacle for ppc64(le) support seems to be Rust, because a lot of desktop software has it as a (direct or transitive) build dependency.
In particular, a lot of GNOME software has a dependency on librsvg (which has parts written in Rust) via their documentation tools.
The author of this RFC plans to create a way to build bootstrapping binaries for Rust on ppc64le (either via cross-building from x86 or from source via mrustc).

Some packages, like OpenBLAS, will need target descriptions for these platforms, but many don't require changes to the actual build process.

Support for Big Endian ppc64 might require changes in a few packages, though the majority of changes are assumed to be upstreamable (see [Unresolved Questions][unresolved]).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about keeping track of this progress in the nixos.wiki ?

While these information give an idea of how much work is left to do, the RFC should also cover other aspects of adding new platforms like the added cost of nix evaluation and maintenance overhead that it creates inside of nixpkgs. I would like to see things addressed like:

  • what are the rules of package maintainers in interaction with the non-official platforms?
  • where is the list of official and unofficial platforms listed?
  • what are the rules for adding and dropping a platform?
  • for non-official platforms, are they keeping their own build farm? if yes, how can this be listed?
  • where should the stdenv bootstrap files be stored?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that I would like to see explored, is if it's possible to store the stdenv out of the main nixpkgs tree, along with the binary cache and all the platform-specific information. Using the @nix-community organisation would be OK for that type of activity if you want.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing that I would like to see explored, is if it's possible to store the stdenv out of the main nixpkgs tree

Would this make Continuous Integration less easy?

One thing I'd definitely want to have as a maintainer is swift feedback on whether changes I make break things for some architecture.

Copy link
Member

@zimbatm zimbatm Mar 4, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how would you be notified that your change broke an architecture that is not checked by ofborg and the main hydra?

let's say I am the maintainer of POWER, I can:

  • add power to the main nixpkgs architectures
  • create my own nixpkgs-power repo. that repo contains the stdenv, overrides and power-specific packages
  • create my own hydra instance that runs whenever nixpkgs master or nixpkgs-power changes => and publishes my binary cache

potentially as the POWER maintainer I could also report breakage issues with links from my hydra instance


# Drawbacks
[drawbacks]: #drawbacks

Supporting another architecture might introduce additional maintenance overhead for nixpkgs.
Generally this is restricted to packages that ship (JIT or AOT) compilers or do some feature selection based on architectures.

If a new Rust bootstrap is created for ppc64le, as long as it isn't adopted for the other platforms, it will be a second rustc bootstrap that may cause maintenance effort.

If a build infrastructure of ppc64le systems should be established, it will cause maintenance effort as well as costs for either procurement of hardware, or use of public cloud services (e. g. IntegriCloud), unless an organisation is willing to sponsor build machines.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you try and talk to IntegriCloud if they are willing to sponsor hardware for open source projects? I can arrange for the NixOS Foundation to talk to them if that helps.

One thing that you have to gage is how much people are interested in using this architecture, and how much effort you are willing to put into that project. This is really on your side I'm afraid, but it would be good to know. Obviously if the port is successful it might attract more users in the future and this hard to foresee. Do you already have a sense of today's user base that you would have?


# Alternatives
[alternatives]: #alternatives

For ppc64(le) in general:

* keep the already-merged stdenv/bootstrapping code, maintain package changes as an overlay
* revert stdenv/bootstrapping support, maintain ppc64le as a nixpkgs fork
* not have ppc64le support in any NixOS-based distribution (barely an alternative)

# Unresolved questions
[unresolved]: #unresolved-questions

Should the nixpkgs/NixOS project host bootstrap files for ppc64le?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The informal NixOS hardware team can provide some file hosting if necessary but it can't look official.


Should Big Endian be in-scope for this RFC?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not, one thing at a time.


# Future work
[future]: #future-work

If Big Endian support on ppc64 is decided to be out of scope for this RFC, it might be of interest for a future project.