Cloud Export integration, saving & display #1192

jywarren · 2020-01-10T22:20:43Z

Fixes #1190 and addresses #1191

This will require some manual testing!

codeclimate · 2020-01-10T22:21:24Z

Code Climate has analyzed commit 9a26a4d and detected 0 issues on this pull request.

View more on Code Climate.

codecov · 2020-01-10T22:35:07Z

Codecov Report

Merging #1192 into main will not change coverage by %.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1192   +/-   ##
=======================================
  Coverage   73.28%   73.28%           
=======================================
  Files          40       40           
  Lines        1400     1400           
=======================================
  Hits         1026     1026           
  Misses        374      374

Impacted Files	Coverage Δ
app/controllers/export_controller.rb	`90.90% <100.00%> (ø)`

jywarren · 2020-01-10T23:29:03Z

I was able to get this running locally but was blocked by a CORS limitation:

Access to XMLHttpRequest at 'http://export.mapknitter.org/export' from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

I'm surprised -- this is working elsewhere. Checking with @icarito on this...

jywarren · 2020-01-10T23:29:45Z

Hm, yeah - it should work: https://github.com/publiclab/mapknitter-exporter-sinatra/pull/44/files#diff-cc95738088603531796e0d0f246a5d77R20

we allow *

jywarren · 2020-01-10T23:30:13Z

Maybe we need to allow different ports?

jywarren · 2020-01-10T23:30:26Z

aha https://stackoverflow.com/questions/46942956/cors-not-allowed-when-origin-includes-port-number

jywarren · 2020-01-10T23:33:37Z

Also having trouble getting it to build at https://jenkins.laboratoriopublico.org/job/Mapknitter-Unstable/

icarito · 2020-01-11T00:17:55Z

Checking the issue in building looks the same as Mapknitter-Stable

icarito · 2020-01-11T00:24:11Z

I was able to reproduce in the command line, looking for a fix

icarito · 2020-01-11T00:37:14Z

I figured it out, it was a permissions issue! Apparently somewhere in the process we create files by root? I've chowned them and it should work now - will keep an eye to see at what part of the process it happens again.

icarito · 2020-01-11T01:22:49Z

Okay so this branch is running in unstable staging branch. Stable is running also.
It calls my attention the presence of a core file. This usually indicates a crash somewhere, to pay attention.

icarito · 2020-01-11T01:24:20Z

I guess there are no debugging symbols in our passenger build but GDB reports this:

Core was generated by `Passenger AppPreloader: /app (forking...)                                     '.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007fc4e85b3fff in ?? ()
(gdb) bt
#0  0x00007fc4e85b3fff in ?? ()
#1  0x0000000000000000 in ?? ()
(gdb)

jywarren · 2020-01-11T01:41:47Z

I got a sentry notice of a migration that didn't run, fwded it. Not sure if related!

…

On Fri, Jan 10, 2020, 8:24 PM Sebastian Silva ***@***.***> wrote: I guess there are no debugging symbols in our passenger build but GDB reports this: Core was generated by `Passenger AppPreloader: /app (forking...) '. Program terminated with signal SIGABRT, Aborted. #0 0x00007fc4e85b3fff in ?? () (gdb) bt #0 0x00007fc4e85b3fff in ?? () #1 0x0000000000000000 in ?? () (gdb) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1192?email_source=notifications&email_token=AAAF6J5LA35BR5RGEGFOWJTQ5ENUJA5CNFSM4KFOBK22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIVVH4I#issuecomment-573264881>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF6J2E2SZJOQHGDEZICTDQ5ENUJANCNFSM4KFOBK2Q> .

sentry-io · 2020-01-11T01:43:24Z

Sentry issue: MAPKNITTER-M

jywarren · 2020-01-11T06:05:42Z

Hm. I'm still getting this locally:

jquery.self-e200ee796ef24add7054280d843a80de75392557bb4248241e870fac0914c0c1.js?body=1:9838 POST http://export.mapknitter.org/export 500 (Internal Server Error)
send @ jquery.self-e200ee796ef24add7054280d843a80de75392557bb4248241e870fac0914c0c1.js?body=1:9838
ajax @ jquery.self-e200ee796ef24add7054280d843a80de75392557bb4248241e870fac0914c0c1.js?body=1:9435
_defaultFetchStatusUrl @ leaflet.distortableimage.self-3f907497d15bb5abe7bee4c432b0cee39ec3d28c608315257e7af8999b49adf1.js?body=1:2807
(anonymous) @ leaflet.distortableimage.self-3f907497d15bb5abe7bee4c432b0cee39ec3d28c608315257e7af8999b49adf1.js?body=1:2830
startExport @ leaflet.distortableimage.self-3f907497d15bb5abe7bee4c432b0cee39ec3d28c608315257e7af8999b49adf1.js?body=1:2761
addHooks @ leaflet.distortableimage.self-3f907497d15bb5abe7bee4c432b0cee39ec3d28c608315257e7af8999b49adf1.js?body=1:1790
enable @ leaflet.toolbar.self-226640c481c6ad02405acc7f132301ba4ced168b1f7e9db10f3c86cf9babf70f.js?body=1:1
enable @ leaflet.toolbar.self-226640c481c6ad02405acc7f132301ba4ced168b1f7e9db10f3c86cf9babf70f.js?body=1:1
handler @ LeafletEnvironmentalLayers.self-bfc185ab0b58c254f69ba81d8a71d041a396d8e4ba15a90522dc0e1046225c5b.js?body=1:14320

edit:1 Access to XMLHttpRequest at 'http://export.mapknitter.org/export' from origin 'http://localhost' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Ah, i bet the 500 error is causing the CORS error, i think that happened before. Let me check, i bet the format is not quite right.

jywarren · 2020-01-11T06:06:16Z

ah yes, it's because my images are local, so the exporter can't read them. I guess i have to test on unstable.

jywarren · 2020-01-11T06:09:48Z

app/assets/javascripts/mapknitter/Map.js

+            // exportUrl: 'http://34.74.118.242/api/v2/export/', // to swap to JS exporter
+            // exportStartUrl: 'http://34.74.118.242/api/v2/export/', // to swap to JS exporter
+            // fetchStatusUrl: fetchStatusUrl,
+            handleStatusResponse: handleStatusResponse


it's not clear to me that this is getting used; need to console.log a bit.

jywarren · 2020-01-11T06:10:01Z

app/assets/javascripts/mapknitter/Map.js

+
+            // save the location of the status URL
+            $.ajax({	
+              url: "/export",	


for example this doesn't seem to be running...

jywarren · 2020-01-11T17:09:33Z

Syncing this with publiclab/Leaflet.DistortableImage#519 too cc @sashadev-sky -- thanks!

jywarren · 2020-01-13T23:25:08Z

If folks could try this out it'd be great cc @VladimirMikulic @cesswairimu

shift-select the images and press the export button in upper left of the map. Also, the Exports => Exports tab should show exports you've started from that page...

they won't properly run from local maps, so heads up.

Almost there! I'll try to pull in the latest LDI version too!

VladimirMikulic · 2020-01-13T23:43:53Z

@jywarren I've started a few exports but they are not shown in the "Exports" tab. No console errors.

jywarren · 2020-01-14T02:19:10Z

Ah, in this initial implementation they're only showing on page refresh. But a later refactor won't be hard!

…

On Mon, Jan 13, 2020, 6:43 PM Vladimir Mikulic ***@***.***> wrote: @jywarren <https://github.com/jywarren> I've started a few exports but they are not shown in the "Exports" tab. No console errors. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1192?email_source=notifications&email_token=AAAF6J2ICXJSW4QPHXI7WHTQ5T4DVA5CNFSM4KFOBK22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI2XDOY#issuecomment-573927867>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAF6J3UY5ISBU3E4E4BF4DQ5T4DVANCNFSM4KFOBK2Q> .

jywarren · 2020-01-14T04:52:43Z

I think we've got this, pretty much. I need help resolving the promise to cancel the export, though -- @sashadev-sky @VladimirMikulic do you have any tips for me here?

I'll try pushing this to unstable as soon as I can for testing!

cesswairimu · 2020-01-14T12:45:46Z

Awesome 🎉 🎉

jywarren · 2020-01-14T15:32:02Z

app/assets/javascripts/mapknitter/Map.js

+      // repeatedly fetch the status.json
+      var updateInterval = setInterval(function intervalUpdater() {
+// need to bust cache with a rotating timestamp...
+        $.ajax('http://export.mapknitter.org/' + status_url + '?' + Date.now(), { // bust cache with timestamp


TODO: after we get the first status.json response, update the URL to go directly to Google Cloud, so we pay less

VladimirMikulic · 2020-01-14T15:55:10Z

I think we've got this, pretty much. I need help resolving the promise to cancel the export, though -- @sashadev-sky @VladimirMikulic do you have any tips for me here?

You could wrap handleStatusResponse in a promise and then resolve out of it once the export is done/cancelled.

jywarren · 2020-03-24T15:46:34Z

😕 -- oddly there is no tags.js that I can see... https://github.com/publiclab/mapknitter/tree/main/app/assets/javascripts/

But, are we able to compare to the manifest I referenced above where I did have it working locally? Or is that how you're comparing it?

Thanks!

icarito · 2020-03-24T18:50:57Z

Yes the comparison was from the manifests, where I found tags.js. If it's not familiar then it's likely it's an artifact of the build.

icarito · 2020-03-24T19:18:03Z

I found some missing commands when comparing the Makefile that we use in Plots2 vs this one, this looks to be the issue.

icarito · 2020-03-24T20:21:52Z

I feel I'm getting closer. I can reproduce the issue I think, that is, I seem to be unable to build assets in unstable, I get the following output instead of the expected:

jenkins@tycho:/srv/mapknitter_unstable/mapknitter$ docker-compose exec web rake assets:precompile
yarn install v1.22.4
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
Done in 0.34s.

icarito · 2020-03-24T21:35:16Z

Allright! I slowly reviewed all differences with plots2 and found .yarnrc which helps put assets in the right location for rails assets pipeline to find. So now it does build production assets!
Please review!

jywarren · 2020-03-24T23:02:15Z

Wow, it seems to be working!!!

I think we can merge this now. There is some follow-up fixing we ought to do but it's good enough to publish... let me review the checklist first!

jywarren · 2020-03-24T23:05:29Z

Yes, the next step is to merge this, the test on stable. 🙌

I'm waiting for my export to complete but otherwise all good!

OK the export worked well. But the alert didn't show. Will check this then merge.

jywarren · 2020-03-24T23:06:39Z

Ah, i had removed the completion alert. Adding it back in then this is good to merge -- @icarito if you see it pass with my new commit, feel free to!

jywarren · 2020-03-24T23:13:26Z

app/assets/javascripts/mapknitter/Map.js

+          data = JSON.parse(data);
+          console.log(data, data.status, data.jpg);
+          if (data && data.status == "complete") {
+            alert("Export completed at " + data.cm_per_pixel + " cm/px. JPG available at https://mapknitter-exports-warps.storage.googleapis.com/" + data.jpg.split('public/warps')[1] + " -- Please refresh page to view completed exports.");


Here I set the alert to show the JPG and direct towards a reload. then cleared the interval. Wooo!

icarito

Looks ok on unstable so far!

icarito · 2020-03-25T06:09:54Z

Ah, i had removed the completion alert. Adding it back in then this is good to merge -- @icarito if you see it pass with my new commit, feel free to!

Sure, except I don't seem to have the privileges to do so...

jywarren · 2020-03-25T20:10:08Z

Hi @icarito i'm not sure the latest code is on unstable... it's not giving me an alert. Can you check to be sure the latest commits are up there? Thanks!

I'm testing at: https://unstable.mapknitter.org/maps/testmap/edit

icarito · 2020-03-25T20:33:31Z

Hi Jeff, Indeed you seem to be pushing to the branch on your fork, instead of /unstable/on the main repo. It is not automatic like PRs are! On my dev machine I pulled your branch into the `unstable` branch and then pushed it to github, triggering redeployment (in progress now). One quick way to push to unstable is `git push origin HEAD:unstable` /in addition to/pushing to your branch. Glad to be of service! El mié, 25 de mar de 2020 a las 13:10, Jeffrey Warren <[email protected]> escribió:

…

Hi @icarito <https://github.com/icarito> i'm not sure the latest code is on unstable... it's not giving me an alert. Can you check to be sure the latest commits are up there? Thanks! I'm testing at: <https://unstable.mapknitter.org/maps/testmap/edit> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1192 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABQYS6E2ARDQTO4AZUBP5DRJJQK5ANCNFSM4KFOBK2Q>.

jywarren · 2020-03-26T18:31:02Z

Ah, sorry!!! it's been a while since i did it!

jywarren · 2020-03-26T18:52:39Z

Yay!!!

Merging!

jywarren mentioned this pull request Jan 10, 2020

Cloud exporter publication roadmap #1191

Open

25 tasks

jywarren changed the title ~~Export display~~ Cloud Export integration, saving & display Jan 10, 2020

jywarren commented Jan 11, 2020

View reviewed changes

This was referenced Jan 11, 2020

Feature/cancel export rebased & bug fixes to customizable export UI publiclab/Leaflet.DistortableImage#519

Merged

FEATURE: Make collection export option dynamic publiclab/Leaflet.DistortableImage#523

Merged

jywarren commented Jan 14, 2020

View reviewed changes

Add asset compilation to Makefile

e1a620d

icarito added 4 commits March 24, 2020 15:35

Try exec instead of run

1a710d8

Move asset building

f83471c

Wrap yarn call in Makefile

af6abe7

Mimic plots2 deployment config

15a1e40

icarito added 2 commits March 24, 2020 17:24

Find assets!

005534a

Restore Makefile

c4ff692

Update Map.js

96300a4

jywarren commented Mar 24, 2020

View reviewed changes

icarito approved these changes Mar 25, 2020

View reviewed changes

jywarren mentioned this pull request Mar 25, 2020

MapKnitterExporter architecture discussion #298

Open

jywarren added 2 commits March 25, 2020 16:00

Update _cloud_exports.html.erb

3926966

Update _cloud_exports.html.erb

42102a3

Update Map.js

4c4117e

jywarren merged commit 36f6d77 into publiclab:main Mar 26, 2020

jywarren mentioned this pull request Mar 31, 2020

Load testing! publiclab/image-sequencer-app#19

Open

5 tasks

jywarren mentioned this pull request Mar 23, 2022

Bump image-sequencer from 1.4.1 to 3.7.2/3 and switch uglifier to terser #1616

Merged

Cloud Export integration, saving & display #1192

Cloud Export integration, saving & display #1192

Conversation

jywarren commented Jan 10, 2020 • edited Loading

codeclimate bot commented Jan 10, 2020

codecov bot commented Jan 10, 2020 • edited Loading

Codecov Report

jywarren commented Jan 10, 2020

jywarren commented Jan 10, 2020

jywarren commented Jan 10, 2020

jywarren commented Jan 10, 2020

jywarren commented Jan 10, 2020

icarito commented Jan 11, 2020

icarito commented Jan 11, 2020

icarito commented Jan 11, 2020

icarito commented Jan 11, 2020

icarito commented Jan 11, 2020

jywarren commented Jan 11, 2020 via email

sentry-io bot commented Jan 11, 2020

jywarren commented Jan 11, 2020

jywarren commented Jan 11, 2020

jywarren Jan 11, 2020

Choose a reason for hiding this comment

jywarren Jan 11, 2020

Choose a reason for hiding this comment

jywarren commented Jan 11, 2020

jywarren commented Jan 13, 2020

VladimirMikulic commented Jan 13, 2020

jywarren commented Jan 14, 2020 via email

jywarren commented Jan 14, 2020

cesswairimu commented Jan 14, 2020

jywarren Jan 14, 2020

Choose a reason for hiding this comment

VladimirMikulic commented Jan 14, 2020

jywarren commented Mar 24, 2020

icarito commented Mar 24, 2020

icarito commented Mar 24, 2020

icarito commented Mar 24, 2020

icarito commented Mar 24, 2020

jywarren commented Mar 24, 2020

jywarren commented Mar 24, 2020

jywarren commented Mar 24, 2020

jywarren Mar 24, 2020

Choose a reason for hiding this comment

icarito left a comment

Choose a reason for hiding this comment

icarito commented Mar 25, 2020

jywarren commented Mar 25, 2020

icarito commented Mar 25, 2020 via email

jywarren commented Mar 26, 2020

jywarren commented Mar 26, 2020

jywarren commented Jan 10, 2020 •

edited

Loading

codecov bot commented Jan 10, 2020 •

edited

Loading