Add support for linux/arm64 and darwin/arm64

[Volatus]

Signed-off-by: Ismayil Mirzali [email protected]

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

closes #101

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[xylar]

@Volatus, I tried to do architecture migration in #102. It may be more successful to comment there, though you are welcome to see if you can make more headway here. You need to add several things to the build sections of the builds as in:
https://github.com/conda-forge/airflow-feedstock/pull/102/files#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9aR25-R27
For now, you will also need to skip airflow-with-ldap for the new architectures because dependencies have not yet been built:
https://github.com/conda-forge/airflow-feedstock/pull/102/files#diff-f3725a55bf339595bf865fec73bda8ac99f283b0810c205442021f29c06eea9aR387-R388

[Volatus]

@xylar thanks for the pointers, I'll see what I can do

[xylar]

@conda-forge-admin, please rerender

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

• Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [393]

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[Volatus]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/airflow-feedstock/actions/runs/4404255160.

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/airflow-feedstock/actions/runs/4404263799.

[Volatus]

@xylar It seems like some issue with yarn but when I tried the yarn install on the Airflow repo everything went fine. Do you think it's possible that the wrapper install script ends up shadowing the real errors underneath? Is there a way I can run the packaging locally to test? (I'm on darwin/arm64)

[xylar]

@Volatus, I also don't understand the yarn issue. It seems to have emerged very recently and since it's appearing in #104, I don't think it's related to anything you changed here.

[xylar]

I asked about it here:
https://matrix.to/#/!SOyumkgPRWoXfQYIFH:matrix.org/$1678705789214366sdSaV:matrix.org?via=matrix.org&via=gitter.im&via=gnome.org
but no one has responded so far.

[Volatus]

@Volatus, I also don't understand the yarn issue. It seems to have emerged very recently and since it's appearing in #104, I don't think it's related to anything you changed here.

Right, and I thought Airflow broke something recently but it did seem like yarn worked fine for me locally on their repo. How would I try running a local build of this feedstock for my host and arch to test some things?

[xylar]

Hmm, I don't think Airflow changed but it does seem like some of the yarn dependencies aren't working anymore. We probably need to figure out how to patch airflow/www/yarn.lock to fix it but I don't have time or expertise to debug this.

Right, and I thought Airflow broke something recently but it did seem like yarn worked fine for me locally on their repo. How would I try running a local build of this feedstock for my host and arch to test some things?

If you can install docker, you can use:
https://github.com/conda-forge/airflow-feedstock/blob/main/build-locally.py
to build locally. I can't use docker (it interferes with my ability to connect to my institution's wifi) so I use:

conda mambabuild -m .ci_support/<config>.yaml recipe

That's not foolproof (it can pick up system libraries occasionally) but works pretty well.

[xylar]

You will need to install mamba and boa for this. boa provides mambabuild. I believe there are currently some compatibility issues with the latest conda, mamba and boa so it might be safest to go with a version of conda from earlier this year so that you can have a working installation of conda and boa. (Very inconvenient, so I hope it gets worked out soon!)

[xylar]

@Volatus, it looks like there is a release candidate for 2.5.2:
apache/airflow#30028
That suggests to me that there might be a new release in a week or so. I think we should just wait for that and try again. Feel free to debug in the meantime but I won't have time to spend on this until after that release.

[xylar]

@Volatus, if you rebase, you should be able to test again. The fix was to constrain yarn <2.

[Volatus]

@Volatus, if you rebase, you should be able to test again. The fix was to constrain yarn <2.

@xylar Will rebase off of main now.

[Volatus]

@xylar So here's the takeaways:

1. Not sure why the MacOS ARM builds failed, might be something with conda-build.
2. linux/aarch64 failed on 3.9 and 3.10, but succeeds for 3.8, which makes perfect sense, due to this change introduced in 3.9. It would seem that passlib, a dependency of sqlalchemy ends up creating a bcrypt hash, which Ubuntu's (the current build environment) glibc cannot handle.

[xylar]

@Volatus, okay, we could wait on OSX-ARM for now (and make a separate PR where we ask for help from the experts).

Regarding aarch64, I don't think there's anything we can do about updating glibc (I've had a discussion with core folks about that on some other feedstock). Anything we can do to patch or bypass this?

[Volatus]

Regarding aarch64, I don't think there's anything we can do about updating glibc (I've had a discussion with core folks about that on some other feedstock). Anything we can do to patch or bypass this?

I was wondering how I can change the Docker image for the aarch64 runs. There is an anvil for aarch64 and it would massively speed up our CI builds. Might even fix the issue with crypt, worth a try I think.

[xylar]

It probably makes sense to skip this test when we're cross-compiling:

airflow-feedstock/recipe/meta.yaml

Line 114 in a1cff27

- airflow --help

[xylar]

I don't think we have the ability to change the docker images. That has to be done consistently across feedstocks by the conda-forge/core folks.

[xylar]

@Volatus, I solved the OSX ARM issue by moving two of the package from host (target architecture) to build (native architecture). That got merged in #106 and should be available soon (if not now).

I'm working on the Linux architectures in #105 but certainly am open to suggestions you might have. Feel free to keep working here if you need a testing grounds.