Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make thumbnails for uploading pictures #2206

Merged
merged 28 commits into from
Aug 5, 2019

Conversation

compulim
Copy link
Contributor

@compulim compulim commented Jul 19, 2019

Fixes #2138.

Changelog Entry

Description

Problem

Yesterday

When we attach a picture, Direct Line will echo back the activity with attachment of URL pointing to Direct Line temporary storage. Web Chat will replace the image with the URL echoed by Direct Line.

Today/tomorrow

When we attach a picture, Direct Line will not echo back the attachment URL, but a placeholder image. Since Web Chat replace the URL with a placeholder image, the UX is not great.

Resolution

We are making thumbnail of every attaching picture, save it in channelData.attachmentThumbnails. As a bonus, since the thumbnail live inside the activity, it will survive F5.

By default, we are making thumbnail of 720x360 as JPEG of quality 0.6. It yields thumbnail of size around 20 KB with our testing images.

We detect whether the upload is a picture or not using filename extension (.gif/.jpeg/.jpg/.png). We are not using fingerprint of the file (e.g. GIF87a/GIF89a). We do not support PDF files.

Since thumbnail has a toll on the CPU, we are using few modern browser APIs to offload the work:

The minimum requirement is Chrome 69 (for OffscreenCanvas). Although Firefox 46 support OffscreenCanvas thru an experimental flag, it do not support OffscreenCanvas.getContext('2d'). So this feature is disabled for Edge (UWP)/Firefox/Safari automatically thru capability check.

Specific Changes

Since thumbnail is a UI feature and requires specific user agent (browser) to work, we are adding it to component only. Devs using core will not have thumbnail creation.

  • Modify UploadButton.js
    • connectUploadButton.sendFiles function updated to make thumbnail when picture is uploaded
  • Added downscaleImageToDataURL.js and its singleton worker for offloading work
    • Send the ArrayBuffer to worker
    • Worker decode the picture, downscale it, encode it, and send it back as Base64 string
  • Added styleOptions for toggling this feature, max width/height, image type, and image quality
  • Added tests
    • WebDriver currently do not support uploading multiple files at the same time

  • Testing Added

@coveralls
Copy link

coveralls commented Jul 19, 2019

Coverage Status

Coverage increased (+0.6%) to 65.571% when pulling 77c2191 on compulim:feat-thumbnail-upload into 9320172 on microsoft:master.

Copy link
Contributor

@corinagum corinagum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm concerned about this only being available on Chrome. this seems like an important feature that will only be partially supported on the browsers we support. Thoughts?

CHANGELOG.md Outdated Show resolved Hide resolved
__tests__/setup/pageObjects/sendFile.js Outdated Show resolved Hide resolved
@compulim
Copy link
Contributor Author

Talked to Chris and he actually suggest this (prefer Web Worker even it only works in Chrome for now).

I am unsure if he is aware that Firefox doesn't work yet.

corinagum
corinagum previously approved these changes Jul 26, 2019
@compulim compulim requested a review from corinagum August 1, 2019 23:38
@compulim
Copy link
Contributor Author

compulim commented Aug 1, 2019

Now support both Web Worker and non-Web Worker mode. Edge, Firefox, and IE11 will run in non-Web Worker mode.

Firefox has experimental (behind-the-flag) support for OffscreenCanvas, but it lack of encoding (convertToBlob). Instead of doing partial work in Web Worker, we will use non-Web Worker for Firefox.

@compulim compulim merged commit 8faf2bc into microsoft:master Aug 5, 2019
@compulim compulim deleted the feat-thumbnail-upload branch August 5, 2019 20:53
@Sakshi0921
Copy link

@corinagum Do we need to update/change webchat or anything else from our side to see this fix in our bots?
We are still facing this problem in our projects.

@corinagum
Copy link
Contributor

This bug wil be fixed in our next release in October. If you need these changes immediately, please see our documentation on pointing to our MyGet feed for latest bits.

@compulim compulim mentioned this pull request Aug 22, 2019
@compulim compulim mentioned this pull request Oct 4, 2019
13 tasks
compulim added a commit to compulim/BotFramework-WebChat that referenced this pull request Oct 5, 2019
* Add downscale on upload

* Clean up

* Add entry

* Separate worker into another file

* Check for downscale support

* Add comment

* Add style options for thumbnail

* Typo

* Add test

* Rename styleOptions for upload thumbnail

* Fix ESLint

* Add upload text file

* Cleanup

* Update entry

* Update entry

* Fix broken propTypes

* More robust capability check

* Fix tests

* Fix tests

* Add snapshot

* Apply suggestions from code review

Co-Authored-By: Corina <[email protected]>

* Incorporate PR comments

* Thumbnail support for Edge, Firefox, and IE11

* Fix ESLint

* Fix promise chaining

* Fix test
@compulim compulim mentioned this pull request Oct 25, 2019
55 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Make thumbnail of upload image into channelData
4 participants