[camera] Support JPEG fallback for Android on ImageStream

[OgulcanCelik]

Description

Currently, ImageStream provides YUV_420_888 on Android and FLEX_RGBA_8888 iOS. However, YUV is a bit problematic, since it's really heavy in terms of performance cost when converting YUV to RGB plane & even default libraries like Image does not support YUV image processing.

Android's ImageReader does not support FLEX_RGBA_8888 but it does support JPEG as ImageFormat. JPEG is already usable by default widgets like Image.memory() so it opens more possibilities to ImageStream on Android, compared to YUV.

So this helps with real-time processing use cases, Dart Image library already supports JPEG decoding and its much faster than trying to convert YUV planes to RGB.

This PR adds a new optional parameter for CameraController to specify which ImageFormat wanted from ImageStream. Since it is optional and the default one is still YUV_420_888, its not a breaking change.

Related Issues

flutter/flutter#26348
flutter/flutter#18968
flutter/flutter#34743

and lots of questions on StackOverflow regarding how to use CameraImage on Android & how to convert YUV images to RGB

Checklist

Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

• I read the Contributor Guide and followed the process outlined there for submitting PRs.
• My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
• All existing and new tests are passing.
• I updated/added relevant documentation (doc comments with ///).
• The analyzer (flutter analyze) does not report any problems on my PR.
• I read and followed the Flutter Style Guide.
• The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
• I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
• I updated CHANGELOG.md to add a description of the change.
• I signed the CLA.
• I am willing to follow-up on review comments in a timely manner.

Breaking Change

Does your PR require plugin users to manually update their apps to accommodate your change?

• Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
• No, this is not a breaking change.

[OgulcanCelik]

@bparrishMines hello, sorry for mention but I believe this is somewhat an urgent issue, it's critical for lots of image process-related apps do you have time for a quick review? Also if you are not accepting feature requests via PRs, may I publish a forked version of camera as a new package, including this PR feature? -Since it will be based on your camera plugin codebase-

[bparrishMines]

Hi @OgulcanCelik

Thanks for the contribution! Unfortunately, the ecosystem team is currently focused on other priorities at the moment and we don't anyone that can focus on new camera features right now. We'll typically follow this prioritization policy. In the meantime, you can always add this branch to your the pubspec.yaml of a Flutter app:

dependencies:
  camera:
    git:
      url: [email protected]:OgulcanCelik/plugins.git
      path: packages/camera

I would only recommend publishing your own plugin if you intend to continue to maintain it. Ultimately, it's up to you though.

[mario-jerkovic]

@OgulcanCelik did you decide if you want to publish this fork as a separate plugin :) as this PR is awesome and it would be a waste not to release it 😃

[OgulcanCelik]

@OgulcanCelik did you decide if you want to publish this fork as a separate plugin :) as this PR is awesome and it would be a waste not to release it

Hey, I don't have time to publish this as a separate plugin because that would mean I have to also maintain it :) Eventually, dev team going to implement something related to this issue I hope, but until then you can use my fork for the camera I think.

Nevertheless, its shame that dev team cant find neither time nor a developer to look for their official plugin development, this isn't a big complicated issue to fix.

[davidmartos96]

@OgulcanCelik Nice PR! I am no Flutter reviewer, but I think it would be nice if instead of using the magic numbers 35, 42 and 256 throughout the code, use the static constants provided by the ImageFormat class. https://developer.android.com/reference/android/graphics/ImageFormat#YUV_420_888

35 == ImageFormat.YUV_420_888

[alexcohn]

@OgulcanCelik which device did you use to develop and test this change? While Android camera does mandate support for JPEG, there is no such requirement to support JPEG streaming.

[joaoavf]

I wasn't able to get a jpeg stream in two different devices: Xiaomi Mi Lite 2 and Motorola G7 Play.

The stream keeps returning YUV_420_888 (35) despite the CameraControler being set to jpeg (256).

Has anyone been able to implement this properly?

[alexcohn]

As I wrote above, it is impossible to get a JPEG stream from the camera (except that there may exist devices that support this). What's possible, is to convert the preview frames to JPEG in Java and pass them to Flutter. The problem is that I cannot imagine a use case when this kind of image stream may be helpful. Can you?

[joaoavf]

My specific use case is reading a code (similar to a barcode), but as the code is somewhat hard to read (light orange on a white background), I have to make quite a few retries until getting it right, and being able to make a lot of retries in a short amount of time greatly improves the user experience.

The current bottleneck I have is converting from YUV to an Image object, which is somewhat expensive in the code I am currently using.

[OgulcanCelik]

@OgulcanCelik which device did you use to develop and test this change? While Android camera does mandate support for JPEG, there is no such requirement to support JPEG streaming.

I wasn't able to get a jpeg stream in two different devices: Xiaomi Mi Lite 2 and Motorola G7 Play.

The stream keeps returning YUV_420_888 (35) despite the CameraControler being set to jpeg (256).

Has anyone been able to implement this properly?

After the official response from the dev team, I left Flutter and went with React Native instead. Especially the official camera plugin missing so many important endpoints.

When made this PR, I tried it with Xiaomi Redmi 8, OnePlus 5T and Xiaomi Mi A2 Lite didn't notice any problem, so I'm not sure what's the issue, maybe this is the reason, but still:

@OgulcanCelik which device did you use to develop and test this change? While Android camera does mandate support for JPEG, there is no such requirement to support JPEG streaming.

https://developer.android.com/reference/android/graphics/ImageFormat#JPEG

Interesting, official documentation says This format is always supported as an output format for the android.hardware.camera2 API, and as a picture format for the older Camera API though, so this means all devices should feed JPEG right?

[alexcohn]

@OgulcanCelik so why do you prefer JPEG to RGB (imagine for a moment that your device can deliver both)?

[OgulcanCelik]

Actually I prefer RGB image feed with separate channels so I can manipulate the image easily but RGB didn't work properly so I went with jpeg, it worked on all 3 phones I tried. Also decoding jpeg isn't a hard task like converting YUV to RGB.

[joaoavf]

Oğulcan, it is interesting to know that it is working on the Xiaomi Mi A2 Lite as it is the same model I am trying to use here.

I wonder what I might have set up wrong. The only change I have made was to add the androidFormatCode: 256 parameter on CameraController. I am consuming from a stream created with startImageStream. Is there anything I've missed here? (It is still returning YUV instead of jpeg).

[alexcohn]

@joaoavf From my measurements, the Image class is very slow. It's not surprising, because it's pure dart and manipulates byte lists. It can decode Jpeg, but the overhead is more than building Image directly from yuv420 in dart. At any rate, I have a much faster way to convert CameraImage to RGB (it does not build an Image, but rather a bmp). I see that in your experiment you actually use luminance. This data you can get directly from the CameraImage (it's planes.get(0)).

[mvanbeusekom]

this feature is implemented in PRs #3364 and #3359, therefore I will be closing this PR