Feature request: overlay images with transparency (was: Resize, combine multiple images)

[kembs]

Hello,

I was wondering if I could get help if this is possible with this library. I've been looking at the documentation and the code but can't find any info for combining multiple images on top of each other.

I assume I'd to do something like resize them beforehand and save them to buffers, then actually joining them?

Thanks

[lovell]

Hi Kyle, would you be more interested in:

1. blending two images of the same dimensions, perhaps taking advantage of an alpha transparency channel in the top image, or
2. stitching two images edge-to-edge?

The underlying libvips library can handle both; neither are currently exposed via this module.

[lovell]

@kembs ?

[lovell]

Closing due to inactivity but please do feel free to reopen if this is still something you're looking for.

[oaleynik]

@lovell I would be interested in blending two images of the different dimensions, perhaps taking advantage of an alpha transparency channel in the top image. I saw that vips has im_insert operation which could place one image over another at specified position. However, number of bands should be equal for both images or one of them should have one band.

This feature will let me fully switch to sharp and remove graphicsmagick fallback for this operation

[lovell]

Hi @oaleynik, libvips provides an [ifthenelse](http://www.vips.ecs.soton.ac.uk/supported/current/doc/html/libvips/libvips-Hi conversion.html#vips-ifthenelse) operator that could be used for this - the transparency layer could be extracted and used as the cond image.

The API might look something like:

var image1 = sharp('image1.jpg');
var image2 = sharp('image2.png');
image1
  .overlayWith(image2, top, left)
  .toBuffer(function(err, data) {
    ...
  });

In this example data would be in JPEG format as you're operating on image1. The top and left parameters could be made optional, where not providing them results in the images being centred.

Does this make sense?

(I'm planning to migrate to the new vips8 C++ API in 7.42.0+ later this year. This feature will be much easier to implement after doing so.)

[oaleynik]

@lovell looks awesome! Exactly what I need and I think it will be useful for others.

[nebkam]

If I understand correctly, this would make watermarking images a breeze, and fits my use case perfectly
👍

[aseemk]

+1 😄

[gasi]

FYI, I started exploring this and will try to report back on my progress later this week.

[lovell]

Great news, thanks @gasi!

[pschneider]

@gasi Looking forward to it. Thanks!

[gasi]

@lovell: I have a proof-of concept and while it’s rough, I’d appreciate your feedback: https://github.com/gasi/sharp/blob/89e6dfbd3ce0fe776b85a768d0e9c809693a0af6/test/unit/composite.c

As stated in the PR (gasi#2), I have a feeling there must be a more concise and potentially more performant approach using vips_ifthenelse, but I need to move forward to hit a deadline 😄

[gasi]

Got it working using vips_ifthenelse but I am not sure it helps readability. However, I assume it’s faster (not tested yet), so probably the better long-term approach: gasi@e18b5d3

[gasi]

I started exploring how to integrate having two images being composited into the C++ extension. My guideline is your code snippet from above:

var image1 = sharp('image1.jpg');
var image2 = sharp('image2.png');
image1
  .overlayWith(image2, top, left)
  .toBuffer(function(err, data) {
    ...
  });

@lovell: Would you like this to be part of the ResizeWorker or should this be a separate worker?

[lovell]

Thanks for your great start on this feature Daniel.

I'm also unsure if the use of vips_ifthenelse will be faster, but if it produces good-enough results with less code to maintain, that sounds like a suitable approach. When we move to the new vips8 C++ interface (see #152) all the object (de)referencing goes away which will significantly reduce complexity and LOC.

It's worth noting all these techniques assume linear RGB. JPEGs without a profile are usually in a non-linear sRGB space so gamma correction may be required. Don't worry about this for now - a possible future enhancement.

The ResizeWorker is increasingly misnamed; I consider it more of a general PipelineWorker these days. Encapsulating the overlay/composite logic in another method or class would, of course, be a good idea.

We'll need to vips_embed the image to be overlaid within a fully-transparent image of the output image size before vips_ifthenelse. vips_black can be used to help generate the transparency. We can stipulate/assume the image to be overlaid is the same size or smaller to keep this step as simple as possible.

I suspect the most common use of this feature will be for watermarks. This would suggest a good place to apply this operation is just before the final conversion to sRGB.

As for the JS API, we could start out more simply by supporting an experimental overlayWith(filename, top, left) rather than passing in a full-blown sharp instance to avoid dealing with things like waiting for multiple input Streams to be ready etc. right now.

[gasi]

First of all, thanks for all the helpful comments, @lovell. I have an end-to-end proof of concept of overlayWith inside sharp now. It currently does the minimum work we require for our use case (composite two PNGs of the same size without specifying top or left). My primary goal would be to take the existing approach and bring it up to sharp’s coding guidelines, so it could land in one of the next releases as experimental API. Depending on time and resources, I might add some of the other feature extensions outlined in this issue.

I'm also unsure if the use of vips_ifthenelse will be faster, but if it produces good-enough results with less code to maintain, that sounds like a suitable approach.

👍

When we move to the new vips8 C++ interface (see #152) all the object (de)referencing goes away which will significantly reduce complexity and LOC.

Excited for this! Indeed, I can’t wait to get rid of all these t[…] intermediate result references.

The ResizeWorker is increasingly misnamed; I consider it more of a general PipelineWorker these days. Encapsulating the overlay/composite logic in another method or class would, of course, be a good idea.

Agreed, PipelineWorker seems like a more appropriate name. As of now, I include my external composite function (should probably be Composite to match the rest) from composite.c. Let me know if that works or if there are benefits to having this in a .cc file like common.cc / sharp::IsJpeg, etc.

We'll need to vips_embed the image to be overlaid within a fully-transparent image of the output image size before vips_ifthenelse. vips_black can be used to help generate the transparency. We can stipulate/assume the image to be overlaid is the same size or smaller to keep this step as simple as possible.

As stated above, we currently don’t have the use case of embedding smaller images but if it’s easy enough, one of us could add it later on.

I suspect the most common use of this feature will be for watermarks. This would suggest a good place to apply this operation is just before the final conversion to sRGB.

Thanks for the tip! This is what I have done in my proof of concept.

As for the JS API, we could start out more simply by supporting an experimental overlayWith(filename, top, left) rather than passing in a full-blown sharp instance to avoid dealing with things like waiting for multiple input Streams to be ready etc. right now.

👍 I simply implemented overlayWith(filename) for now.

I am calling it a night 🌃 now but since you are in London with the day ahead of you, I thought I’d share my latest status. Please add your comments / thoughts to my pull request and I will try to address them tomorrow: gasi#2 😄

P.S. I have been playing around with ImageMagick’s compare and was wondering whether you’ve considered automating sharp’s unit tests by checking in the expected output images and algorithmically comparing them against the test output using compare?

[gasi]

It's worth noting all these techniques assume linear RGB. JPEGs without a profile are usually in a non-linear sRGB space so gamma correction may be required. Don't worry about this for now - a possible future enhancement.

As an amateur photographer, I have always heard about gamma but didn’t understand its meaning until I watched this video recommended by our intern Tony: https://www.youtube.com/watch?v=LKnqECcg6Gw&feature=youtu.be

[lovell]

Thank you @gasi, I've commented inline on your latest additions.

I've created the knife branch for v0.11.0 work-in-progress, of which this will be part, so please aim any Pull Requests in its general direction.

Containing this feature in a separate .c file makes a lot of sense right now as I will need to refactor/split ResizeWorker ahead of #152. This means your code will move into something like an Operations class as part of that.

Task #122 covers adding the use of perceptual (i.e. frequency-based) image hashing to test assertions - feel free to add ideas and comments there.

[gasi]

Thanks, @lovell. I rebased my work against knife, tightened it up, and submitted a PR against the knife branch: #207

I added some comments explaining my work and/or asking some questions. I’d appreciate if you could take a look and I will make sure to address your feedback 👍

This means your code will move into something like an Operations class as part of that.

👍

Task #122 covers adding the use of perceptual (i.e. frequency-based) image hashing to test assertions - feel free to add ideas and comments there.

Good idea. For now, I went with a simple approach using gm.compare. Please let me know what you think.

[gasi]

I uploaded more changes to address your feedback, @lovell: #207

[lovell]

Update: building on @gasi's great work on this feature, commit 1091be3 on the knife branch adds support for compositing any alpha-channel containing image (including 2 channel greyscale+alpha) over any image, even if that underlying image does not itself contain an alpha channel.

It currently still requires that the two images have the same dimensions.

For watermarking, I think all that remains is the ability to apply a gravity to the overlay image so its dimensions can differ from the underlying image.

[lovell]

Need to test PNGs with bit depths other than 8 - see #237

[lovell]

This feature is now in territory. #239 created to track support of differing dimensions.

[oaleynik]

@lovell wonderful! Can't wait for it!

[lovell]

Now in master awaiting release.

[MPSinclair]

What about combining more than 2 images (all of the same dimensions)?

[lovell]

@MPSinclair You'll need to make two passes to do this. Raw pixel input/output is supported so this will remain fast, albeit at the cost of having the whole uncompressed image in memory. Happy to provide more details in a new question/issue if required.