Maintain Xamarin iOS support by keeping the library "AOT-proof"

[antonfirsov]

Description

Xamarin iOS users need to be able to "seed" open-generic library API-s with the help of AotCompilerTools, otherwise it will lead to runtime exceptions on that platform.

Related: #767, #785, #776, #800

Problems

Currently we are struggling to maintain this feature because of the following reasons:

• There is no one in the core team having experience with Xamarin iOS or any other AOT platform.
• We don't have any regression tests for this, we never know when a change breaks the "AOT-proofness" of an existing API
• We have introduced a massive change with Introduce a non-generic Image base class #904, which will most likely break AOT & Xamarin iOS for many existing API-s in RC1. I have defined a point in epic Epic: introduce pixel-agnostic API across the whole library #907 to investigate this, however in practice it turned out that I hardly have the resources to do so during this summer. (Neither I think @JimBobSquarePants can make it.) Therefore it will most likely not happen for RC1.

Conclusion

With our current prioritites and resources, the only way to maintain this feature is community support. If you can provide any help (add missing AOT seeds, or fix regressions), feel free to open a Pull Request!

Long term solution

If someone could define a mono AOT test configuration for appveyor, that would be ace.

/CC @vicfergar @davilovick @gameleon-dev @Lapinou42 @dmanning23 @swoolcock

EDIT
PS: Could someone do a favor and chack the latest MyGet dev builds from on Xamarin iOS? (JPEG, PNG, Resize, Quantization etc.)

[dmanning23]

Cool, super appreciate the work yall have been putting into ImageSharp. I have some downtime this week and can take a look at the iOS build.

[JimBobSquarePants]

@dmanning23 That would be absolutely amazing if you can, thank you very much!

[dmanning23]

Opened a pull-request with a few changes to fix ImageSharp on iOS. I've verified with these changes that it works for jpg, gif, and image resizing.
I also rolled up some changes I've had in mind for the #800 ticket. I've verified that the AoTCompilerTools.Seed methods never actually have to be called, that code just needs to be present for the AoT compiler to pick it up.

Since these AoT/JIT errors only pop up at runtime, and only on physical iOS devices (doesn't occur on simulator!), it'll be extremely difficult to come up with a CI solution to detect them before they pop up. I've started asking around if any other Xamarin developers know how to detect & fix these issues preemptively.

[leonluc-dev]

@dmanning23
Thanks for looking into this!

I've verified with these changes that it works for jpg, gif, and image resizing.

Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.

[antonfirsov]

Would these changes also fix issues with PNG images? PNG was the format giving the most iOS based JIT/AOT issues in beta-0006.

I don't think so.

Suggested follow-up step:

• Run Image.Load(...) for all the test images within our test suite, inside a Xamarin iOS app (running on a device ...).
• Collect the failing ones into a checklist of missing seeds, so we can address the issue in a systematic way.

[dmanning23]

Ok, I can take a look at .png images also.
Are you having problems loading or saving to .png?

[leonluc-dev]

Mostly loading from PNG bytes. I haven't been able to properly test encoding to PNG yet in an iOS environment.

In beta0006 I can consistently reproduce the following error by loading png image bytes using Image.Load(pngbytes):

Unhandled managed exception: Attempting to JIT compile method 'SixLabors.ImageSharp.Formats.Png.PngScanlineProcessor:ProcessRgbaScanline<SixLabors.ImageSharp.PixelFormats.Rgba32> (SixLabors.ImageSharp.Configuration,SixLabors.ImageSharp.Formats.Png.PngHeader&,System.ReadOnlySpan`1<byte>,System.Span`1<SixLabors.ImageSharp.PixelFormats.Rgba32>,int,int)' while running in aot-only mode. See https://docs.microsoft.com/xamarin/ios/internals/limitations for more information.
 (System.ExecutionEngineException)
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ProcessDefilteredScanline[TPixel] (System.ReadOnlySpan`1[T] defilteredScanline, SixLabors.ImageSharp.ImageFrame`1[TPixel] pixels, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00135] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.DecodePixelData[TPixel] (System.IO.Stream compressedStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabor
s.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x000e5] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.ReadScanlines[TPixel] (System.IO.Stream dataStream, SixLabors.ImageSharp.ImageFrame`1[TPixel] image, SixLabors.ImageSharp.Formats.Png.PngMetaData pngMetaData) [0x00018] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoderCore.Decode[TPixel] (System.IO.Stream stream) [0x00171] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Formats.Png.PngDecoder.Decode[TPixel] (SixLabors.ImageSharp.Configuration configuration, System.IO.Stream stream) [0x00008] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Decode[TPixel] (System.IO.Stream stream, SixLabors.ImageSharp.Configuration config) [0x00015] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image+<>c__DisplayClass55_0`1[TPixel].<Load>b__0 (System.IO.Stream s) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLa
bors.ImageSharp.Image.WithSeekableStream[T] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, System.Func`2[T,TResult] action) [0x0002b] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream, SixLabors.ImageSharp.Formats.IImageFormat& format) [0x00022] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.IO.Stream stream) [0x00000] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load[TPixel] (SixLabors.ImageSharp.Configuration config, System.Byte[] data) [0x00007] in <7430a37e6bca4100b1763856ff8f7867>:0 
  at SixLabors.ImageSharp.Image.Load (System.Byte[] data) [0x00005] in <7430a37e6bca4100b1763856ff8f7867>:0 

A similar error occurs when loading a png image from other sources such as a stream.

[antonfirsov]

Someone has to call all the PNG scanline decoder methods from AotCompilerTools.

I expect similar issues with gif, and probably even certain jpegs.

The following steps do not require any knowledge related of our codebase, and it would be really helpful if someone could do them:

1. Upload our whole test image suite to an iOS device.
2. Run this code from inside a Xamarin.iOS app. (If running into something like OutOfMemoryException: just delete the giant jpegs.)
3. Send back Report.txt

This will help getting a nice overview about how bad the situation is. I want to avoid doing the discussion overhead image-by-image.

[dmanning23]

Awesome, I will have some extra time next week and will create a quick iOS app with those images & code.

[dmanning23]

Ok wrote a quick&dirty iOS app that loads all the images in the test suite, and didn't run into any AoT/JIT errors. Whatever error it was hitting must have been fixed in #949

@gameleon-dev If you want to send me the specific .png I can double check that it will work going forward.

Here is the sample app with output:
https://github.com/dmanning23/ImageSharpTest.iOS

[JimBobSquarePants]

@dmanning23 Legend! 👑

[leonluc-dev]

@dmanning23 Thank you so much!

If it's any help testing, the images I used that generated the error were iOS screenshots. (Taken with the default iOS screenshot feature)

I've attached one of them to this comment.

Attachment: IMG_7829

[dmanning23]

You should be good to go, I was able to load the attached image:

2019-08-07 17:24:44.794 ImageSharpTest.iOS[2745:131288] Testing Content/62653769-9175ed80-b95e-11e9-9a8d-348b071cb714.png
2019-08-07 17:24:45.439 ImageSharpTest.iOS[2745:131288]    ... SUCCESS!

[omatrot]

@antonfirsov Following #1183 I have the same problem with RC1.

[JimBobSquarePants]

@unamed000 Can you please try building from source from this branch js/aot-experiments

I'm hoping the reference to Configuration.Default is enough to trigger the linker to preserve the relevant code since each codec is explicitly referenced in the construction of the default instance. If not' I'll explicitly reference the individual formats and codecs.

[unamed000]

Hi @JimBobSquarePants

It's not working, still the same stacktrace and error with the same line

[JimBobSquarePants]

It's not working, still the same stacktrace and error with the same line

I'm at a loss then....

Am I right in saying you can tell the linker to ignore the SixLabors binaries?

[unamed000]

There is a an attribute that we can tell the linker to ignore a class as far as i know which is:
[Preserve (AllMembers = true)]

But i'm not really so sure about this though.

[JimBobSquarePants]

You're ahead of me then. I've only briefly looked at the documentation. I don't do any application development.

[unamed000]

After trying to disable line by line. I figured out which line is causing this:
It was actually this in ReadFrameColors method. The Rgba32.FromRgb24 method

But i've no idea how to solve this.
I can workaround like this on my project:
private static void SetupAot() { var gifDecoderCore = new GifDecoderCore(null, null); gifDecoderCore.Decode<Rgba32>(null, CancellationToken.None); }

But then, the app will crash immediately without throwing any stacktrace if i send a high dimension gif file. And i guess this is memory issue problem? Is there anyway that i can workaround this?

Edit:
I've another bug the same with Rgba32 for PNG file (PngDecoderCore). Wonder why it's a special error for Rgba32 ....

[antonfirsov]

@unamed000 can you try what happens if you alter the implementation of Rgba32.FromRgb24 ?

Variant 1

[MethodImpl(InliningOptions.ShortMethod)]
public void FromRgb24(Rgb24 source)
{
    this.R = source.R;
    this.G = source.G;
    this.B = source.B;
    this.A = byte.MaxValue;
}

Variant 2 (No inlining)

public void FromRgb24(Rgb24 source)
{
    this.R = source.R;
    this.G = source.G;
    this.B = source.B;
    this.A = byte.MaxValue;
}

EDIT: With and without the SetupAot workaround.

[JimBobSquarePants]

I might have to dig out the Raspberry Pi again if we change that.

Perhaps adding implementation calls for all of the IPixel methods to AotCompilerTools would be better. They're missing currently.

[unamed000]

@antonfirsov
I tried that already, for removing the InliningOptions and not.
The result is InliningOptions does not change anything. It's just my work around that will affect the result.

Also, any idea why a high dimension gif file is breaking on iOS device? Could it be because it's using too much memory?

[antonfirsov]

@JimBobSquarePants the difference between the "small" and "large" gifs might be actually a difference in transFlag here:

ImageSharp/src/ImageSharp/Formats/Gif/GifDecoderCore.cs

Lines 485 to 508 in 521fae3

	if (!transFlag)
	{
	// #403 The left + width value can be larger than the image width
	for (int x = descriptorLeft; x < descriptorRight && x < imageWidth; x++)
	{
	int index = Unsafe.Add(ref indicesRowRef, x - descriptorLeft);
	ref TPixel pixel = ref Unsafe.Add(ref rowRef, x);
	Rgb24 rgb = colorTable[index];
	pixel.FromRgb24(rgb);
	}
	}
	else
	{
	for (int x = descriptorLeft; x < descriptorRight && x < imageWidth; x++)
	{
	int index = Unsafe.Add(ref indicesRowRef, x - descriptorLeft);
	if (transIndex != index)
	{
	ref TPixel pixel = ref Unsafe.Add(ref rowRef, x);
	Rgb24 rgb = colorTable[index];
	pixel.FromRgb24(rgb);
	}
	}
	}

Does this make sense?
If this is true, the AOT compiler might probably fail to properly compile one of the paths.

It's probably worth to try seeding FromRgb24 and similar calls in Seed<TPixel>:

TPixel pixel = default;
pixel.FromRgb24(default);

@unamed000 can you try if adding this changes anything? (Probably in combination with other hacks.)

[antonfirsov]

Also, any idea why a high dimension gif file is breaking on iOS device? Could it be because it's using too much memory?

@unamed000 in your previous comment you stated:

I try to resize the second image into 50x50 but it's still the same issue, same like it was not the dimensions after all?

Can you try if it works with the resized version with any of the hacks?

[unamed000]

Hi @antonfirsov , i'll try your suggestion tomorrow

Can you try if it works with the resized version with any of the hacks?

I actually tried that before, here is the summary for what i did

	Small Image	Big Image
With my hack (SetupAot)	Able to Load	Crash without yielding any stacktrace
Without my hack	JIT Compiler stacktrace (ReadFromColors method)	JIT Compiler stack trace(ReadFromColors method)

So i think with my current hack, i'm able to get rid of the AOT issue, though the crash without stacktrace could be something else, it could be either:

• Something loop forever inside (this is the case i usually see when Xamarin does not yield any stacktrace)
• Too high memory usage.

[antonfirsov]

Too high memory usage.

What if you upscale a different gif? (One that did noes not produce this issue originally)
If it also hangs, can you incrementally find a size where it starts to hang? (or maybe slow down very noticeably)

[unamed000]

It's probably worth to try seeding FromRgb24 and similar calls in Seed:

Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24

the difference between the "small" and "large" gifs might be actually a difference in transFlag here:

The transFlag are the same for both images since the width and height ratio is the same.

What if you upscale a different gif? (One that did noes not produce this issue originally)

I've resize the gif to like your suggestion and here is the result:
250x250 gif, it's working fine

400x400 gif, it's crashing without stacktrace

Though these gif are very light, it crashes when only 400x400 pixels, so i don't think that could be high memory usage?

EDIT:
Sorry for wrong debugging information, actually it was Image.Save method that throw nothing with high dimension gif, it was crashing inside the Encode method of GifEncoderCore. And it was crashing at QuantizeFrame method.

And after further debugging, it was this method which crash silently

ErrorDither.ApplyQuantizationDither

[antonfirsov]

So far it looks like the crash is a completely separate problem. Does it happen on emulator as well?

Can you help us by filing a new GitHub issue for it (with templates filled) so we can track it?

[JimBobSquarePants]

Tried that as well, somehow not working, it's only work if i seed directly for Rgba32.FromRgb24

Just to clarify @unamed000 you are saying the following did not work when added to AotCompilerTools?

TPixel pixel = default;

Rgb24 rgb24 = default;
pixel.FromRgb24(rgb24);

@antonfirsov if that is the case then we should migrate to using T4 for generating the tools. Messier to extend but if it's the only thing that works we have little choice.

I would like to try this though before we go that route.

private static void AotPixelInterface<TPixel>()
    where TPixel : unmanaged, IPixel<TPixel>
{
    TPixel pixel = default;
    Rgba32 rgba32 = default;
    pixel.ToRgba32(ref rgba32);
    pixel.FromRgba32(rgba32);
    pixel.FromScaledVector4(pixel.ToScaledVector4());
    pixel.FromVector4(pixel.ToVector4());

    pixel.FromArgb32(default);
    pixel.FromBgr24(default);
    pixel.FromBgra32(default);
    pixel.FromBgra5551(default);
    pixel.FromL16(default);
    pixel.FromL8(default);
    pixel.FromLa16(default);
    pixel.FromLa32(default);
    pixel.FromRgb24(default);
    pixel.FromRgb48(default);
    pixel.FromRgba64(default);
    pixel.FromRgb24(default);
}

I've added them to https://github.com/SixLabors/ImageSharp/tree/cc84f6f66a73febd3c72ec93a932ac6868662acb for testing.

[unamed000]

@JimBobSquarePants
yes, i tried that before and i didn't workout.
But could you give me a little of explanation why we need to seed these? Even though i copied the entire code into my project seem like i need to seed them as well (I don't have much knowledge about these "seeding" things)

[JimBobSquarePants]

@unamed000 I just updated my branch as I realized I made a mistake in the loader code and was calling the Rgba32 version only. I would pull the latest from that branch and build/reference from that source code.

The reason we seed the code is to attempt to ensure the AOT compiler doesn't try to delete code branches it incorrectly thinks are not being used. By explicitly calling troublesome methods we try to get the compiler to recognize their use.

You shouldn't have to copy the code to your system I wouldn't think.

[unamed000]

@JimBobSquarePants would you indicate which branch you did the update?

[JimBobSquarePants]

https://github.com/SixLabors/ImageSharp/tree/js/aot-experiments

[JimBobSquarePants]

@unamed000 How did you get on building from the source in the branch?

[JimBobSquarePants]

We need someone to raise an issue upstream here for us to track. We can't fix issues with the AOT compiler.

[UltraNamahage]

I had a similar problem with Unity Android IL2CPP, but I found a solution.
Xamarin iOS and Unity Android IL2CPP both work with Mono, so I think you can solve it the same way.

According to the Unity manual, you need to write a specific code to make a particular generic method work.

This should probably cache all interface methods that have generic methods.
For example, this part.

I wanted to shrink a large PNG image, so I created the following code.

/// <summary>
/// Because it calls an internal class, it needs to be placed inside the ImageSharp project.
/// </summary>
static class UnityAoT<TPixel> where TPixel : unmanaged, IPixel<TPixel>
{
    /// <summary>
    /// call from <see cref="SixLabors.ImageSharp.Advanced.AotCompilerTools.Seed{TPixel}"/>
    /// </summary>
    internal static void UnitySeed()
    {
        AotCompileImageProcessor();
        AotCompileResampler();
        AotCompileEncoder();
    }
    private static void AotCompileEncoder()
    {
        default(PngEncoderCore).Encode<TPixel>(default, default, default);
    }
    private static void AotCompileImageProcessor()
    {
        AotCompilerPixelSpecifics<ResizeProcessor>();
    }
    private static void AotCompilerPixelSpecifics<TProc>()
        where TProc : class, ICloningImageProcessor
    {
        default(TProc).CreatePixelSpecificCloningProcessor<TPixel>(default, default, default);
        default(TProc).CreatePixelSpecificProcessor<TPixel>(default, default, default);
    }
    private static void AotCompileResampler()
    {
        AotCompileResamplers<BicubicResampler>();
    }
    private static void AotCompileResamplers<TResampler>()
        where TResampler : struct, IResampler
    {
        // TODO : Methodize it.
        default(ResizeProcessor<TPixel>).ApplyTransform<TResampler>(default);
        default(TResampler).ApplyTransform(default(ResizeProcessor<TPixel>));
    }
}

This is working fine, but doesn't allow me to do anything other than shrink.
I think we need to cache more methods to do all the processing.

[JimBobSquarePants]

@UltraNamahage Thanks for the additional info.

We have a class AotCompilerTools which we have used to for AOT issues previously. Perhaps you can update that?

ImageSharp/src/ImageSharp/Advanced/AotCompilerTools.cs

Line 26 in 36018bd

internal static class AotCompilerTools

[UltraNamahage]

@JimBobSquarePants Please wait a week.