[FEATURE REQUEST] Minion support for Linux ppc64 architecture

[lukasraska]

Is your feature request related to a problem? Please describe.
Although Salt Minion is very versatile in terms of environment it can run on (it can probably run on anything that can run Python) and it's currently possible to run minions on ppc64 architecture (either on ppc64le or ppc64 - big endian), there are still few things that are missing.

Minion RPM can already be rebuilt for ppc64le RHEL via rpmbuild (ppc64 big endian arch is a little bit complicated - it can be rebuilt, but cross-compilation with gcc on RHEL has few issues this arch + it's very problematic to virtualize due to endianity, but rpmbuild on existing BE system runs fine),

Describe the solution you'd like
Basic functionality works, all standard core grains are compiled without issues, maybe put it into CI pipeline as well? I expect the ppc64le architecture would be the one to focus on (and keep the BE arch only as a note it can run)

Describe alternatives you've considered
Define custom modules / grains that override core functionality and keep it to myself.

Additional context
Versions report (LE)

Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Aug 18 2020, 12:45:08)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rhel 8.3 Ootpa
        locale: UTF-8
       machine: ppc64le
       release: 4.18.0-240.8.1.el8_3.ppc64le
        system: Linux
       version: Red Hat Enterprise Linux 8.3 Ootpa

Versions report (BE)

Salt Version:
          Salt: 3003
 
Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Aug 13 2020, 03:39:19)
  python-gnupg: Not Installed
        PyYAML: 3.11
         PyZMQ: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: Not Installed
 
System Versions:
          dist: rhel 7.9 Maipo
        locale: UTF-8
       machine: ppc64
       release: 3.10.0-1160.31.1.el7.ppc64
        system: Linux
       version: Red Hat Enterprise Linux Server 7.9 Maipo

Disclaimer: I haven't actually tested if ZeroMQ transport works (we're using TCP transport), so because of the complicated build on the BE architecture, ZMQ is not even installed. But TCP transport works fine.

Please Note
If this feature request would be considered a substantial change or addition, this should go through a SEP process here https://github.com/saltstack/salt-enhancement-proposals, instead of a feature request.

[OrangeDog]

it's very problematic to virtualize

Just install qemu-system-ppc64le, no?

[lukasraska]

it's very problematic to virtualize

Just install qemu-system-ppc64le, no?

That's the little little-endian, but just the "ppc64" is big-endian. So you either need to have big-endian system, or use virtualization platform that can emulate big-endian on little-endian (AFAIK QEMU can't do that currently for PowerPC arch.). So that's why it's a bit tricky

[OrangeDog]

Did you try? This says it just works.
Is it possible to run VM with ppc64le architecture on a host machine with x86_64 architecture?

[dmurphy18]

FYI - The AIX native minion runs on ppc64 as a tiamat based salt-minion, it builds a rpm (AIX) and then works that into an AIX bff (rte) fileset. The build process for GitLab CI is still being worked on, but there is a bash build script that can be worked on. Note: still waiting to move this into the open-source area (constrained by resources at the moment) on GitLab, but will be there eventually.

Also IBM has a version of Salt (3001.1) built by them already available as an rpm, see https://www.ibm.com/support/pages/aix-toolbox-linux-applications-downloads-alpha#S

[dmurphy18]

@lukasraska FYI here is the grains.items for AIX Salt 3003.1 which is going to QA today, if this helps.
Let me know if there are additional grains which you feel should be included, and I shall try to accommodate in future releases.

root@p656-pvm1:/home/cecuser/temp/salt_3003.1# salt-call --local grains.items                  
local:
    ----------
    cpu_flags:
        PV_8_Compat
    cpu_model:
        POWER 8
    cpuarch:
        PowerPC_POWER8
    cwd:
        /home/cecuser/temp/salt_3003.1
    dns:
        ----------
        domain:
        ip4_nameservers:
            - 129.40.242.1
            - 129.40.242.2
        ip6_nameservers:
        nameservers:
            - 129.40.242.1
            - 129.40.242.2
        options:
        search:
            - p656.cecc.ihost.com
        sortlist:
    domain:
        p656.cecc.ihost.com
    fqdn:
        p656-pvm1.p656.cecc.ihost.com
    fqdn_ip4:
        - 129.40.126.113
    fqdn_ip6:
    fqdns:
        - p656-pvm1.p656.cecc.ihost.com
    gid:
        0
    gpus:
    groupname:
        system
    host:
        p656-pvm1
    hwaddr_interfaces:
        ----------
    id:
        p656-pvm1.p656.cecc.ihost.com
    ip4_interfaces:
        ----------
        en0:
            - 129.40.126.113
        en1:
            - 172.24.126.113
        lo0:
            - 127.0.0.1
    ip6_interfaces:
        ----------
        en0:
        en1:
        lo0:
            - ::1
    ip_interfaces:
        ----------
        en0:
            - 129.40.126.113
        en1:
            - 172.24.126.113
        lo0:
            - 127.0.0.1
            - ::1
    ipv4:
        - 127.0.0.1
        - 129.40.126.113
        - 172.24.126.113
    ipv6:
        - ::1
    kernel:
        AIX
    kernelrelease:
        1
    kernelversion:
        7
    locale_info:
        ----------
        defaultencoding:
            ISO8859-1
        defaultlanguage:
            en_US
        detectedencoding:
            ISO8859-1
        timezone:
            EDT
    localhost:
        p656-pvm1
    lvm:
        ----------
        rootvg:
            - hd5
            - hd6
            - hd8
            - hd4
            - hd2
            - hd9var
            - hd3
            - hd1
            - hd10opt
            - hd11admin
            - lg_dumplv
            - livedump
    machine_id:
        67b91966-346b-46e7-b401-ef5e22c7ae43
    manufacturer:
        IBM
    master:
        salt
    mem_total:
        4096
    nodename:
        p656-pvm1
    num_cpus:
        2
    num_gpus:
        0
    os:
        AIX
    os_family:
        AIX
    osarch:
        PowerPC_POWER8
    osfinger:
        AIX-7
    osfullname:
        AIX
    osmajorrelease:
        7
    osrelease:
        7.1.0.0
    osrelease_info:
        - 7
        - 1
        - 0
        - 0
    osrelease_techlevel:
        7100-05
    path:
        /usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin
    pid:
        13369546
    productname:
        8408-E8E
    ps:
        /usr/bin/ps auxww
    pythonexecutable:
        /opt/saltstack/salt/run/run
    pythonpath:
        - /opt/saltstack/salt/run/base_library.zip
        - /opt/saltstack/salt/run/lib-dynload
        - /opt/saltstack/salt/run
    pythonversion:
        - 3
        - 7
        - 10
        - final
        - 0
    saltpath:
        /opt/saltstack/salt/run/salt
    saltversion:
        3003.1
    saltversioninfo:
        - 3003
        - 1
    serialnumber:
        21C490V
    server_id:
        985842948
    shell:
        /usr/bin/ksh
    swap_total:
        1048576
    systemfirmware:
        IBM,FW860.81 (SV860_215)
    systempath:
        - /usr/bin
        - /etc
        - /usr/sbin
        - /usr/ucb
        - /usr/bin/X11
        - /sbin
        - /usr/java8_64/jre/bin
        - /usr/java8_64/bin
    uid:
        0
    username:
        root
    virtual:
        physical
    zfs_feature_flags:
        False
    zfs_support:
        False
    zmqversion:
        4.3.3

[lukasraska]

Did you try? This says it just works.
Is it possible to run VM with ppc64le architecture on a host machine with x86_64 architecture?

@OrangeDog There are two variants under ppc64 architecture - ppc64 (which is the original PowerPC 64-bit, big endian) and ppc64le (also sometimes called ppc64el - the little endian variant). From QEMU docs you cannot emulate big endian ppc64 on little endian host system. According to wiki - The Intel x86 and AMD64 / x86-64 series of processors use the little-endian format. (https://en.wikipedia.org/wiki/Endianness#Current_architectures ), so while you're easily able to run ppc64le architecture (it "just works"), the ppc64 itself is the problematic one.

So far I don't have any BE machine that I could actually use to try to virtualize ppc64 OS, just one or two actual servers (where I recompiled the necessary packages to run Salt). But as far as I understood the BE-variant of Linux on Power is mostly legacy - Debian is not supporting it officially, RedHat only provides ppc64le arch. for RHEL 8 (ppc64 is only for RHEL 7), so it's questionable whether to focus on it (at least from packaging support)

@dmurphy18 Thanks, the AIX minion works fine (the currently supported version) and the grains look fine too. The mentioned issue here is mostly for Linux on Power - all it took to get it working was to invoke rpmbuild on the existing source RPMs (for RHEL on Power), so it's much easier than building packages for AIX I believe. If it's possible, we can test the 3003.1 QA AIX package as well.

[sagetherage]

The discussion of this was brought up some time ago and there was not enough demand for it. In discussing this with some of the Core Team this is still the case and putting this into the Approved milestone as it can be picked up by the open source community to develop.

[dmurphy18]

@lukasraska Going to close this feature request since the Salt Project just moved AIX and hence ppc64 support to a community repository https://github.com/vmware/salt-native-minion-for-aix.

If you examine the script used to build, you will see that rpm's are created using rpmbuild, which are then processed to provide the normal AIX fileset, bff, etc. package. It is a trivial task to only build the rpm, currently only for the salt-minion, but it would be simple to build other Salt components.

I am still the maintainer for the GitHub repository, but VMware is inviting co-maintainers, if you are interested.
Please duplicate the issue if you desire on the Git Hub repository https://github.com/vmware/salt-native-minion-for-aix

[lukasraska]

@dmurphy18 well this has nothing to do with AIX, but with Linux on Power architecture, which has nothing to do with the Salt Native minions (regular minions work just fine on ppc64le if rebuilt from the current SRPM), but it's your call, i'm fine with closing this issue

[dmurphy18]

@lukasraska Understand your feature request has nothing to do with AIX, but I was pointing out the steps in the script which basically build an rpm (on an AIX machine) and then once the rpm is built, the script then goes on to generate the AIX package.

The same steps could be used to build Salt on a Power Linux machine, perhaps not having to lock down the dependencies from the AIX Toolbox, since they are available from Power Linux.
Actually looked at doing this back in 2017/2018 but it was decided that there was not enough market there at the time to progress with it. (RedHat, Ubuntu), and work environment priorities have since changed with SaltStack brought by VMware.

Big fan of the ppc hardware, and Linux at it should run great on it.