Feature: Make thumbnail of upload image into channelData #2138

shunyh · 2019-07-02T07:11:10Z

[Edit by corinagum]: this issue has a workaround. Please see @EricDahlvang's comment at the bottom of this thread:

@handsomeyao77 Please see the workaround described here: microsoft/botbuilder-dotnet#2143 (comment)

Original comment:

Hi team,
I get an frequency error, the pictures can't be rendered in chat window, it's broken.

I view this picture src attribute in html:
src="https://bcdirectlineauseast.blob.core.windows.net/at27055/LSfhGXesbMy-original"

I think this is invalidate, because the right src in html is:
src="https://directline.botframework.com/attachments/F7RM2rUmRwUI1uS3WxU8LF-0/0000113/0/win10-gbi-19.jpg?t=ew0KICAiYWxnIjogIlJTMjU2IiwNCiAgImtpZCI6ICJBT08tZXhGd2puR3lDTEJhOTgwVkxOME1tUTgiLA0KICAieDV0IjogIkFPTy1leEZ3am5HeUNMQmE5ODBWTE4wTW1ROCIsDQogICJ0eXAiOiAiSldUIg0KfQ.ew0KICAiYm90IjogImNoYXRib3Qtc2VhLWludCIsDQogICJzaXRlIjogImd0amo3anhFVWFjIiwNCiAgImNvbnYiOiAiRjdSTTJyVW1Sd1VJMXVTM1d4VThMRi0wLTAwMDAxMTMiLA0KICAibmJmIjogMTU2MjA0OTY3OSwNCiAgImV4cCI6IDE1NjIwNTE0NzksDQogICJpc3MiOiAiaHR0cHM6Ly9kaXJlY3RsaW5lLmJvdGZyYW1ld29yay5jb20vIiwNCiAgImF1ZCI6ICJodHRwczovL2RpcmVjdGxpbmUuYm90ZnJhbWV3b3JrLmNvbS8iDQp9.LKtl-Od1isNFau_40g1SMD4YAF6_qrI7sxC9qRXqWru5UJBzCSHfLWMKlzZbNk3yUMvagSbk2ThXkLxihuWeDIR3wMPVllsz7ubEiaIRqRuRhYtb3I8HpPiERmij1G4SSEar4ZOw1B7B5ugMK3bEf_0o1Ir4B6GNtOU0v6yMzbl52FZUzHjMVeHxDeLUdLF1m79YKWH5zaa74pX_yrXz0BQnahFlXRemAyL8IdKOuOHU7e9j4Zzag6Ca7Ump3xbBa9BCvAzrp_i28zlE7oM8qCzaShLX1iVNWLMcS0-YYGODedlILEOdnn3TCWdJGkPtFidNsQNI0q81fjBwsixOWg"

The webchat version is V3.

We have multiple chat dialog in the same time, if we switch between these dialog, the image message in chat window often broken.

To Reproduce

Steps to reproduce the behavior:

Create more than 2 chat dialog
Chatting with any one(Dialog1), send text or image to him/her
Then switch to another one(Dialog2), send text or image to him/her
Now switch back to 2# (Dialog1), the image which your sent is broken in chat window.

Please kindly help me on this issue, thanks very much!

tdurnford · 2019-07-02T07:46:43Z

Seems related to #2084

tdurnford · 2019-07-02T18:04:24Z

Unfortunately v3 of Web Chat is in sunset mode right now and we're not performing maintenance that doesn't fall under security issues for v3.

Could you try out your bot on a basic v4 web page made by following our sample 01.a? This is just to test if your errors are persisting to v4 of Web Chat. If so, we'll be able to investigate the problem and file this as a bug. Otherwise, my immediate fix recommendation would be to use v4 through CDN or MyGet until we get v4 available via embed, which will hopefully be in the upcoming weeks.

shunyh · 2019-07-03T08:08:36Z

@tdurnford , thanks your quickly response.

I try to upgrade webchat version to V4, it seems this error still exists:

Besides, the image can't be show as itself size, it looks like limited in 480px*240 px.

tdurnford · 2019-07-03T17:57:06Z

@handsomeyao77 I'm having a difficult time reproducing this issue. Can you share the files you are uploading? It looks like you are sending a text file and a png. Also, double check the file size. Direct Line limits attachments to 4MB.

tdurnford · 2019-07-04T00:08:03Z

@handsomeyao77 I was able to reproduce the error. The issue only occurs if you pass the Direct Line secret to Direct Line instead of exchanging it for a token first. Try using the code snippet below to test your bot.

<!DOCTYPE html>
<html lang="en-US">
<head>
  <title>WebChat</title>
  <script src="https://cdn.botframework.com/botframework-webchat/latest/webchat.js"></script>
  <style>
    html, body { height: 100% }
    body { 
      margin: 0;
      }
    #webchat {
      height: 100%;
    }
  </style>
</head>
<body>
  <div id="webchat" role="main"></div>
  <script>
    (async function() {
      const webChatSecret = '<DIRECT_LINE_SECRET>';
      const res = await fetch('https://directline.botframework.com/v3/directline/tokens/generate', { 
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${webChatSecret}`,
          'Content-Type': 'application/json'
        },
        body: JSON.stringify({ user: { id: 'dl_user_123' }})
      });
      const { token } = await res.json();
      const directLine = window.WebChat.createDirectLine({ token });
      window.WebChat.renderWebChat({
        directLine,
      }, document.getElementById('webchat'));
    })().catch(err => console.log(err));
  
  </script>
</body>

Thank you @stevkan for recommending this fix.

shunyh · 2019-07-04T07:42:12Z

@tdurnford , thanks your solution.
I used token instead secret to init DirectLine, just like the sample your provided

Unfortunately, it occurs again, the image src attribute in html is: https://bcdirectlineeasia2.blob.core.windows.net/at27104/8RLaaFI7rYl-original

shunyh · 2019-07-05T10:35:50Z

@tdurnford , this bug is really urgent for me, could you give me more helps?

EricDahlvang · 2019-07-05T18:37:12Z

@handsomeyao77 Please see the workaround described here: https://github.com/microsoft/botbuilder-dotnet/issues/2143#issuecomment-508830039

shunyh · 2019-07-08T03:30:11Z

@EricDahlvang , thank you!

I read microsoft/botbuilder-donet#2143 related info, this is different from our scenario.

Some times we upload image successfully:

But if I refresh our bot page or switch to other same host page , the image gets broken, the request URL is invalid.

We save the images in our storage too, but we can't render images in chat window, all the messages in chat window are from bot-framework service.
We have to provide other page to show the images to user. It's not convenient to us to process huge business cases in short time.

EricDahlvang · 2019-07-08T17:22:51Z

Hi @handsomeyao77

The best solution to workaround what you are experiencing is to create a thumbnail of the uploaded file and add it to channelData, then set the contentUrl to that channelData in DIRECT_LINE/INCOMING_ACTIVITY as demonstrated in the linked example.

shunyh · 2019-07-10T03:02:05Z

Hi@EricDahlvang , thanks your response.

I am trying the workaround your mentioned: create a downscale image then add to channelData.

Now, I am using this sample React-web-Chat-Redux to create a listener to listen upload file event.

But I found that this sample has two ReactWebChats component.

One in App.js, the other in webchat.js.

Why do we have to init two WebChats? We only one WebChat in our page.

tdurnford · 2019-07-10T04:58:21Z

There is really just one instance of the ReactWebChat component in the project. The component created in the WebChat.js file acts a container for the ReactWebChat component imported from the BotFramework-WebChat dependency. All of the logic in this file is responsible for creating the DirectLine Connection and the Custom Store Middleware. In theory, all of this code could be moved to the App.js file, but that wouldn't be the best design for larger scale apps.

The App.js file imports the component created in WebChat.js - the one that is acting as a container - and adds it to the App. The ReactWebChat component in this file is different from the ReactWebChat component in the WebChat.js file and should probably be renamed to just WebChat or WebChatContainer to avoid confusion. If you would like, you change the component name in the App.js file in your project to help clarify the difference.

App.js

import { connect } from 'react-redux';
import React from 'react';

import WebChatContainer from './WebChat';

class App extends React.Component {
  render() {
    const { props: {
      backgroundColor,
      dispatch
    } } = this;

    return (
      <div id="app" style={{ backgroundColor }}>
        <WebChatContainer appDispatch={ dispatch } />
      </div>
    )
  }
}

export default connect(
  ({ backgroundColor }) => ({ backgroundColor })
)(App)

Hope this helps!

shunyh · 2019-07-10T08:27:12Z

Hi @EricDahlvang, here are some summaries what i met share to you.

#1. Create a listener.

#2. Listener implement .

export default function(dispatch) {
    return () => next => action => {
    if (action.type === 'DIRECT_LINE/POST_ACTIVITY') {
         if (action.payload.activity && action.payload.activity.attachments) {
           const downscaledImages = Promise.all(
           action.payload.activity.attachments.map(
           ({ contentType, contentUrl }) => /^image\//.test(contentType) ? downscaleImage(contentUrl, 
              contentType, 480) : null
         )
       );
        console.log(downscaledImages);
        (action.payload.activity.channelData || (action.payload.activity.channelData = {})).thumbnails = 
         downscaledImages;
     }
   } else if (action.type === 'DIRECT_LINE/INCOMING_ACTIVITY') {
     const activity = action.payload.activity;

     if (activity && activity.attachments && activity.channelData && activity.channelData.thumbnails) {
       action.payload.activity.attachments = activity.attachments.map((attachment, index) => ({
         ...attachment,
         contentUrl: activity.channelData.thumbnails[index] || attachment.contentUrl
       }));
       console.log(activity.attachments);
     }
   }
   return next(action);
};

}

#3. Upload an image, render in chat window successfully, the contentUrl looks like bot-framework-service, not a base64 data.

#4. But if i refresh the page, the contentUrl is invalid, not shown as a base64Data, so image broken.

Is there something wrong?
I think this workaround will convert image to base64 instead use bot-framework Url as img src. Is it right?

shunyh · 2019-07-10T08:33:00Z

Append some related functions:

async function downscaleImage(imageURL, contentType, maxSize) {
  const image = document.createElement('img');
  const imageLoadPromise = eventToPromise(image, 'load');

  image.src = imageURL;
  alert(image.src);
  await imageLoadPromise;

  const canvas = document.createElement('canvas');

  canvas.height = image.height > image.width ? maxSize : ~~(image.height / image.width * maxSize);
  canvas.width = image.width > image.height ? maxSize : ~~(image.width / image.height * maxSize);
  alert(canvas.height);
  alert(canvas.width);
  canvas.getContext('2d').drawImage(image, 0, 0, canvas.width, canvas.height);

  const dataURL = canvas.toDataURL(contentType);

  return dataURL;
}
function eventToPromise(target, name) {
  return new Promise((resolve, reject) => {
  const handler = event => {
     target.removeEventListener(name, handler);
     resolve(event);
  };
}
  target.addEventListener(name, handler);
  });

}

shunyh · 2019-07-10T08:49:53Z

@tdurnford Thanks your explanation, it really helps me a lot.

compulim · 2019-07-10T21:37:14Z

@cwhitten We talked about making thumbnail in Web Chat via <canvas>.

Could you give me some suggestions in this topic? E.g. do we need to use Web Worker if available?

cwhitten · 2019-07-29T21:09:51Z

We're going to defer the logic into a WebWorker on browsers that support it - if the client is on a browser that does not fully support the isolation to run inside a WebWorker we will run the same logic on the main thread. cc @compulim

shunyh · 2019-07-30T07:32:03Z

Hi team,

Thanks for your workaround: #2138 (comment), we fixed the image broken after refresh issue.

But, we encountered the other image related error, image src is a default placeholder:

just like this : #2152.

The new error is different from the old one, should I create a new issue on GitHub?

aamalhot · 2019-08-02T09:48:16Z

Hi Team,
We are facing the same default placeholder issue. Do we have any ETA for the fix? Also, please share any known issues/ limitations once the fix is deployed.

Thanks!

compulim · 2019-08-02T09:54:15Z

The PR #2206 for this issue is pending for review.

It will generate thumbnails and save it as part of the activity (i.e. channelData). The downscaling is verified to work in IE11 and all modern browsers including Safari.

shunyh added bug Indicates an unexpected problem or an unintended behavior. Pending labels Jul 2, 2019

corinagum removed the Pending label Jul 3, 2019

corinagum assigned tdurnford Jul 3, 2019

corinagum added the needs-repro Waiting for repro or investigation label Jul 3, 2019

corinagum removed the needs-repro Waiting for repro or investigation label Jul 5, 2019

corinagum pinned this issue Jul 5, 2019

corinagum added 4.6 front-burner labels Jul 5, 2019

corinagum unassigned tdurnford Jul 5, 2019

cwhitten mentioned this issue Jul 8, 2019

Image Upload returns default placeholder from directline #2152

Closed

corinagum added the p1 Painful if we don't fix, won't block releasing label Jul 9, 2019

corinagum assigned compulim Jul 10, 2019

compulim changed the title ~~Picture broken in chat window~~ Feature: Make thumbnail of upload image into channelData Jul 10, 2019

compulim mentioned this issue Jul 19, 2019

Make thumbnails for uploading pictures #2206

Merged

1 task

Kaiqb added the Customer label Jul 23, 2019

corinagum removed the Triage-E label Jul 30, 2019

compulim closed this as completed in #2206 Aug 5, 2019

corinagum unpinned this issue Aug 6, 2019

daveta added Bot Services Required for internal Azure reporting. Do not delete. Do not change color. customer-reported Required for internal Azure reporting. Do not delete. customer-replied-to Required for internal reporting. Do not delete. labels Aug 23, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Make thumbnail of upload image into channelData #2138

Feature: Make thumbnail of upload image into channelData #2138

shunyh commented Jul 2, 2019 •

edited by corinagum

Loading

tdurnford commented Jul 2, 2019

tdurnford commented Jul 2, 2019

shunyh commented Jul 3, 2019

tdurnford commented Jul 3, 2019

tdurnford commented Jul 4, 2019

shunyh commented Jul 4, 2019

shunyh commented Jul 5, 2019

EricDahlvang commented Jul 5, 2019

shunyh commented Jul 8, 2019

EricDahlvang commented Jul 8, 2019

shunyh commented Jul 10, 2019

tdurnford commented Jul 10, 2019

shunyh commented Jul 10, 2019 •

edited

Loading

shunyh commented Jul 10, 2019

shunyh commented Jul 10, 2019

compulim commented Jul 10, 2019

cwhitten commented Jul 29, 2019

shunyh commented Jul 30, 2019 •

edited

Loading

aamalhot commented Aug 2, 2019

compulim commented Aug 2, 2019

Feature: Make thumbnail of upload image into channelData #2138

Feature: Make thumbnail of upload image into channelData #2138

Comments

shunyh commented Jul 2, 2019 • edited by corinagum Loading

Original comment:

To Reproduce

tdurnford commented Jul 2, 2019

tdurnford commented Jul 2, 2019

shunyh commented Jul 3, 2019

tdurnford commented Jul 3, 2019

tdurnford commented Jul 4, 2019

shunyh commented Jul 4, 2019

shunyh commented Jul 5, 2019

EricDahlvang commented Jul 5, 2019

shunyh commented Jul 8, 2019

EricDahlvang commented Jul 8, 2019

shunyh commented Jul 10, 2019

tdurnford commented Jul 10, 2019

shunyh commented Jul 10, 2019 • edited Loading

shunyh commented Jul 10, 2019

shunyh commented Jul 10, 2019

compulim commented Jul 10, 2019

cwhitten commented Jul 29, 2019

shunyh commented Jul 30, 2019 • edited Loading

aamalhot commented Aug 2, 2019

compulim commented Aug 2, 2019

shunyh commented Jul 2, 2019 •

edited by corinagum

Loading

shunyh commented Jul 10, 2019 •

edited

Loading

shunyh commented Jul 30, 2019 •

edited

Loading