Adding osx-arm support #224

mkitti · 2022-09-03T00:08:21Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

…nda-forge-pinning 2022.08.23.13.46.03

conda-forge-linter · 2022-09-03T00:08:26Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

mkitti · 2022-09-03T00:08:38Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2022.09.02.10.01.46

mkitti · 2022-09-03T00:51:38Z

@conda-forge-admin, please rerender

github-actions · 2022-09-03T00:53:00Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/julia-feedstock/actions/runs/2982651707.

mkitti · 2022-09-03T01:13:39Z

@conda-forge-admin, please rerender

github-actions · 2022-09-03T01:15:17Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/julia-feedstock/actions/runs/2982709690.

ngam · 2022-09-29T23:41:07Z

@mkitti do you want to give this a shot as a native build? Not emulated

ngam · 2022-09-29T23:56:49Z

Here are minimal steps to get you started:

clone this repo (main)
add the following to conda-forge.yml

build_platform:
  osx_arm64: osx_64
test: native_and_emulated

rerender (conda smithy rerender)
make sure to have conda-build (or boa for mambabuild)
in root, conda-build recipe --variant-config-file .ci_support/osx_arm64_.yaml

For me, like before, this procedure gets stuck at

    JULIA usr/lib/julia/corecompiler.ji

... but we should be able to build this!

In file included from /Users/ngam/.Mambaforge-MacOSX-arm64/envs/py310/conda-bld/julia_1664495424314/work/src/ircode.c:11:
./serialize.h:112:13: warning: unused function 'write_uint32' [-Wunused-function]
static void write_uint32(ios_t *s, uint32_t i) JL_NOTSAFEPOINT
            ^
./serialize.h:117:17: warning: unused function 'read_uint32' [-Wunused-function]
static uint32_t read_uint32(ios_t *s) JL_NOTSAFEPOINT
                ^
2 warnings generated.
    LINK usr/lib/libjulia-internal.1.8.dylib
ld: warning: -pie being ignored. It is only used when linking a main executable
[cctools-port]: generating fake signature for '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/py310/conda-bld/julia_1664495424314/work/usr/lib/libjulia-internal.1.8.dylib'
    LINK usr/lib/libjulia-internal.1.dylib
    LINK usr/lib/libjulia-internal.dylib
1 warning generated.
    LINK usr/lib/libjulia-codegen.1.8.dylib
ld: warning: -pie being ignored. It is only used when linking a main executable
[cctools-port]: generating fake signature for '/Users/ngam/.Mambaforge-MacOSX-arm64/envs/py310/conda-bld/julia_1664495424314/work/usr/lib/libjulia-codegen.1.8.dylib'
    LINK usr/lib/libjulia-codegen.1.dylib
    LINK usr/lib/libjulia-codegen.dylib
make[2]: Leaving directory '$SRC_DIR/src'
make[2]: Entering directory '$SRC_DIR'
Warning: git information unavailable; versioning information limited
    JULIA usr/lib/julia/corecompiler.ji

MilesCranmer · 2023-01-17T01:28:30Z

Hey @ngam @mkitti, how much work do you think it will be to get this working? Most of the new Apple products are exclusively ARM-based, so I am thinking it will be a good idea to have conda capability set up relatively soon.

mkitti · 2023-01-17T03:48:41Z

I do have the hardware. I suppose I should try to build this and see if I can figure out where it gets stuck.

MilesCranmer · 2023-04-15T15:49:07Z

Hey @mkitti, what is the status of this? Anything I can help with?
Cheers,
Miles

MilesCranmer · 2023-05-23T22:29:43Z

Took a stab at trying to build this, but am getting this weird issue (maybe just a local issue):

Linking ncurses-6.3-headf329_1
critical libmamba filesystem error: cannot copy: Input/output error [/workspace/miniforge3/pkgs/ncurses-6.3-headf329_1/share/terminfo/e/eterm-color] [/workspace/miniforge3/share/terminfo/e/eterm-color]
Traceback (most recent call last):
  File "/workspace/./build-locally.py", line 105, in <module>
    main()
  File "/workspace/./build-locally.py", line 95, in main
    run_osx_build(ns)
  File "/workspace/./build-locally.py", line 34, in run_osx_build
    subprocess.check_call([script])
  File "/opt/conda/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['.scripts/run_osx_build.sh']' returned non-zero exit status 1.

ngam · 2023-05-23T23:43:29Z

@conda-forge-admin, please rerender

mkitti · 2023-05-30T01:47:46Z

Is there a blocker to building openlibm for macos-arm in conda-forge?

mkitti · 2023-05-30T01:49:35Z

Are we cross compiling?

ngam · 2023-05-30T01:56:41Z

Is there a blocker to building openlibm for macos-arm in conda-forge?

It's already available: https://anaconda.org/conda-forge/openlibm/files

recipe/build.sh

Co-authored-by: Mark Kittisopikul <[email protected]>

mkitti · 2023-05-30T02:08:21Z

It's already available: https://anaconda.org/conda-forge/openlibm/files

So the problem is that Julia cannot find openlibm.dylib?

ngam · 2023-05-30T02:09:20Z

It's already available: https://anaconda.org/conda-forge/openlibm/files

So the problem is that Julia cannot find openlibm.dylib?

It may be that it is finding the "wrong" one because we are cross-compiling

ngam · 2023-05-30T02:09:52Z

I am also trying to build locally natively, so we will see how that goes...

mkitti · 2023-05-30T02:10:36Z

It may be that it is finding the "wrong" one because we are cross-compiling

I'm not sure if Julia will cross compile. We're going to get stuck at building the system image.

ngam · 2023-05-30T02:11:23Z

Do you know what would stop the building at Generating REPL precompile statements... 38/41? What's it doing in these steps anyway?

mkitti · 2023-05-30T02:12:35Z

JuliaLang/julia#30338

@staticfloat, are we any closer to cross compiling?

mkitti · 2023-05-30T02:15:29Z

Do you know what would stop the building at Generating REPL precompile statements... 38/41? What's it doing in these steps anyway?

It's running the REPL and tracing what method specializations were compiled while doing so. This generates a series of calls to Base.precompile so that these compiled methods get cached within the system image, sys.{dll,so,dylib}. This makes the REPL fast to start without compilation latency.

ngam · 2023-05-30T02:18:32Z

Well... I guess it is a moot effort for the time being. Let me know if you make any progress on this front.

staticfloat · 2023-05-30T04:13:06Z

Julia can cross-compile as long as you can execute the generated executable; so e.g. if you have rosetta available, I think you can cross-compile Julia x86_64 on aarch64. I'm not sure what kind of cross-compilation is happening here though.

mkitti · 2023-05-30T04:35:18Z

I think we're cross compiling from a x86_64 Azure host running macOS for an aarch64 macOS target.

MilesCranmer · 2023-06-01T19:44:58Z

@conda-forge/help-osx-arm64 is there any way we can use a self-hosted macOS agent for building the arm64 binaries? Or is there a way we can upload binaries manually?

Unless I misunderstand @staticfloat it sounds like Julia may not support osx-64 => osx-arm64 cross-compilation right now. But osx-arm64 => osx-arm64 builds seem to be working.

MilesCranmer · 2023-06-01T19:55:36Z

@ngam @mkitti I also get stuck at 38/41 of the precompilation. This tip might provide some debugging info: JuliaLang/julia#37777 (comment)

Edit: it gets stuck here:

       Profile.@profile while Profile.len_data() < 1000; rand(10,10) * rand(10,10); end

This seems like a bad idea for a precompilation statement... It's an infinite while loop. I guess the profiler isn't getting any samples out of this.

Edit: I will add the following patch file to this branch:

- debug_output = devnull # or stdout
+ debug_output = stdout

mkitti#3

ngam · 2023-06-01T20:21:32Z

@MilesCranmer could you point your PR to this repo?

MilesCranmer · 2023-06-01T20:27:57Z

Sure. Also I added a patch for the infinite precompilation (I'm going to patch this in Julia too, it seems like it is maybe not a great idea to have a unbounded while loop during precompilation...)

Edit: looks like they actually patched it already.

MilesCranmer · 2023-06-01T20:30:49Z

#252

ngam · 2023-06-02T12:34:33Z

is there any way we can use a self-hosted macOS agent for building the arm64 binaries? Or is there a way we can upload binaries manually?

@MilesCranmer we do this for quite a few difficult packages (e.g., PyTorch, TensorFlow, and at least at some point QT) that are usually impossible to build within 6 hours of free public CI. There is a procedure outlined somewhere (under a CFEP, conda-forge enhancement proposal). Once we test the package in #252, I will tag the appropriate people to make it happen and formalize it.

Unless I misunderstand @staticfloat it sounds like Julia may not support osx-64 => osx-arm64 cross-compilation right now.

Thanks @staticfloat and @mkitti; is there an issue upstream to follow progress on this front? Do we think it's happening soon?

staticfloat · 2023-06-02T15:47:10Z

Julia does not support cross compilation of Julia code, the only reason it works the other way is because Apple built an emulator, Rosetta 2.

mkitti · 2023-06-03T02:41:00Z

I think the closest tracking issue is JuliaLang/julia#30338

ngam · 2023-06-05T04:07:15Z

I see, yeah, this doesn’t look like a feature that will be resolved anytime soon (understandably so)

regro-cf-autotick-bot and others added 3 commits August 23, 2022 19:52

migration: OSXArm

5f38bbe

MNT: Re-rendered with conda-build 3.22.0, conda-smithy 3.21.1, and co…

ea8b31d

…nda-forge-pinning 2022.08.23.13.46.03

Move curl to the build system

4d24a10

mkitti requested review from SylvainCorlay, abhi18av, acaprez, bgruening, dfornika, mariusvniekerk and ngam as code owners September 3, 2022 00:08

conda-forge-webservices[bot] and others added 2 commits September 3, 2022 00:10

MNT: Re-rendered with conda-build 3.22.0, conda-smithy 3.21.1, and co…

c6c522f

…nda-forge-pinning 2022.09.02.10.01.46

Patch jldownload to use build env curl

72d3b30

Make jldownload less verbose

7f3119b

Make jldownload less verbose for real

055c2d3

jmuchovej mentioned this pull request Sep 9, 2022

Add osx-arm64 support? (Julia 1.8.0 now supports M1) #226

Closed

1 task

Merge branch 'main' into mkitti/bot-pr_arm_osx_h9dcb98

d3c9551

ngam changed the title ~~Mkitti/bot pr arm osx h9dcb98~~ Adding osx-arm support May 23, 2023

ngam added 2 commits May 29, 2023 21:11

add openlibm to build

76187fc

try vendoring openlibm for osx-arm

a0733ae

mkitti commented May 30, 2023

View reviewed changes

recipe/build.sh Outdated Show resolved Hide resolved

mkitti commented May 30, 2023

View reviewed changes

recipe/build.sh Outdated Show resolved Hide resolved

ngam and others added 2 commits May 29, 2023 22:04

correct logic

0cdb481

Co-authored-by: Mark Kittisopikul <[email protected]>

correct logic

b5226e4

Co-authored-by: Mark Kittisopikul <[email protected]>

MilesCranmer mentioned this pull request Jun 1, 2023

Adding osx-arm support v3 #252

Open

MilesCranmer mentioned this pull request Jul 23, 2023

Switch to juliaup #256

Open

Adding osx-arm support #224

Are you sure you want to change the base?

Adding osx-arm support #224

Conversation

mkitti commented Sep 3, 2022 • edited by ngam Loading

conda-forge-linter commented Sep 3, 2022

mkitti commented Sep 3, 2022

mkitti commented Sep 3, 2022

github-actions bot commented Sep 3, 2022

mkitti commented Sep 3, 2022

github-actions bot commented Sep 3, 2022

ngam commented Sep 29, 2022

ngam commented Sep 29, 2022 • edited Loading

MilesCranmer commented Jan 17, 2023

mkitti commented Jan 17, 2023

MilesCranmer commented Apr 15, 2023

MilesCranmer commented May 23, 2023

ngam commented May 23, 2023 • edited Loading

mkitti commented May 30, 2023

mkitti commented May 30, 2023

ngam commented May 30, 2023

mkitti commented May 30, 2023

ngam commented May 30, 2023

ngam commented May 30, 2023

mkitti commented May 30, 2023

ngam commented May 30, 2023

mkitti commented May 30, 2023

mkitti commented May 30, 2023

ngam commented May 30, 2023

staticfloat commented May 30, 2023

mkitti commented May 30, 2023

MilesCranmer commented Jun 1, 2023

MilesCranmer commented Jun 1, 2023 • edited Loading

ngam commented Jun 1, 2023

MilesCranmer commented Jun 1, 2023 • edited Loading

MilesCranmer commented Jun 1, 2023

ngam commented Jun 2, 2023 • edited Loading

staticfloat commented Jun 2, 2023

mkitti commented Jun 3, 2023

ngam commented Jun 5, 2023 • edited Loading

mkitti commented Sep 3, 2022 •

edited by ngam

Loading

ngam commented Sep 29, 2022 •

edited

Loading

ngam commented May 23, 2023 •

edited

Loading

MilesCranmer commented Jun 1, 2023 •

edited

Loading

MilesCranmer commented Jun 1, 2023 •

edited

Loading

ngam commented Jun 2, 2023 •

edited

Loading

ngam commented Jun 5, 2023 •

edited

Loading