rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

nreese · 2021-02-24T14:18:55Z

This PR moves advanced setting ml:fileDataVisualizerMaxFileSize to the file_upload plugin and renames the setting to fileUp\load:maxFileSize. There is a migration script to migrate ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize. The setting is now displayed in the general section.

This PR updates geojson upload from a hard coded 50MB to using fileUpload:maxFileSize. This means that geojson upload size will increase from 50MB to 100MB and upto 1GB. Because of this increased file size, geojson importing had to be re-worked so that the entire file is never held in memory at a single time. Chunks of the file are read and indexed in Elasticsearch. The biggest change for users is that they will no longer get to preview the entire file. Instead, they get to preview only the first 10000 features or 3MB, whichever is less.

elasticmachine · 2021-02-24T14:18:57Z

Pinging @elastic/kibana-gis (Team:Geo)

elasticmachine · 2021-02-24T14:20:05Z

Pinging @elastic/ml-ui (:ml)

nreese · 2021-02-24T14:22:41Z

@elasticmachine merge upstream

nreese · 2021-02-24T15:08:15Z

#92504 is resolved by changes made to this PR.

jgowdyelastic

LGTM

thomasneirynck

Thx for adding the icon and tooltip for the preview-layer in the LayerTOC

created #93261 and #93254 for separate follow-up

x-pack/plugins/file_upload/public/components/import_complete_view.tsx

x-pack/plugins/file_upload/public/importer/geojson_importer/geojson_importer.ts

kindsun · 2021-03-02T19:55:51Z

@nreese Overall working really smoothly. Great to see GeoJSON Upload handling large files! A few questions/comments:

Does this PR need a more recent merged version of master or is the number doubling still an issue? Note the docCount vs. the found number. Again, might just need an updated master.

Looks like we're handling files with mixed types differently than before (albeit one of them always failed before). Looks like it defaults incorrectly to geo point now? I've attached the file used below for testing. In a perfect world we should get a point, a zig-zag line and a square. Or if the square really is invalid, then just the first two.

Old left, new right:

aGeoJsonFile.zip

We were previously lowercasing the index & index pattern name to save users the hassle of having to do it when ingesting a file with mixed casing in the name. Might be worth doing a toLowerCase() on this name again:

nreese · 2021-03-02T20:06:03Z

Does this PR need a more recent merged version of master or is the number doubling still an issue? Note the docCount vs. the found number. Again, might just need an update master.

Needs a more recent version of master. That has been fixed but is not in this branch yet.

Looks like we're handling files with mixed types differently then before (albeit one of them always failed before). Looks like it defaults incorrectly to geo point now? I've attached the file used below for testing. In a perfect world we should get a point, a zig-zag line and a square. Or if the square really is invalid, then just the first two.

Have you tried indexing the file as geo_shape?

kindsun · 2021-03-02T20:13:05Z

Have you tried indexing the file as geo_shape?

Sorry, I should've been more clear. I'm just referring to the default type-detection settings which determine if the file is geo point or geo shape upfront. The picture above shows how each are ingested by default without the user overriding any settings.

nreese · 2021-03-02T21:00:34Z

Sorry, I should've been more clear. I'm just referring to the default type-detection settings which determine if the file is geo point or geo shape upfront. The picture above shows how each are ingested by default without the user overriding any settings.

ok, I have updated the logic for auto selecting geo field type. If a file contains both points and shapes then geo_shape is selected. If a file contains just points then geo_point is selected. With this logic in place, the file loads the same as in previous versions, without the user having to configure geo field type.

kindsun · 2021-03-02T22:54:40Z

I'm unable to upload the attached file. The progress indicator stays at 0 until it eventually times out.

Following the timeout, it progresses to the results screen showing a bad gateway error but also allowing the user to still Add layer.

Then it ends up in a strange state where there are no source details but there is a layer.

ne_10m_urban_areas_landscan.zip

FWIW, I tested the file in mapshaper.org and it does appear to be valid but with a lot of intersections. Regardless though, we might want to handle it with something other than a 502 if possible and ensure the user can't add it as a layer.

nreese · 2021-03-02T23:43:37Z

Discussed offline with @aaronjcaldwell about ne_10m_urban_areas_landscan.zip. This file is causing loaders.js problems. Loaders can not cleanly iterator over file without problems. Fixing loaders.js should be done outside of this PR.

There was a problem identified where when in an error state, the wizard incorrectly allowed users to advance to adding the layer. This issue has been resolved.

kindsun

Thanks for addressing feedback. This is a long overdue improvement to Maps' file upload capabilities. Also great to have alignment with ML so we can hopefully all benefit from each other's work from here forward. Very excited to see where we can take it from here! 🍾

LGTM w/ green CI!

code review
tested locally in chrome

…d:maxFileSize and increase max geojson upload size to 1GB (elastic#92620) * rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize * add saved object migration * file preview * importing status * remove console statement * import complete view * fix geojson_importer test * tslint * i18n fixes * cleanup * update documenation for advanced setting rename * advanced settings usage_collection * remove ml:fileDataVisualizerMaxFileSize from schemas and types * add copy buttons for import response and fix geojson upload functional tests * tslint * remove clipboard-read check * return early if env does not support reading from clipboard * fix reporting tests * review feedback * update GeoJsonFileSource to support showing results trimmed icon and tooltip * add fileUpload to useMlKibana context and replace dependencyCache with useMlKibana * tslint * review feedback * lower case file name * default to selecting geo_shape when file contains both points and shapes * fix wizard onError callback to not advance to next step Co-authored-by: Kibana Machine <[email protected]>

…d:maxFileSize and increase max geojson upload size to 1GB (#92620) (#93358) * rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize * add saved object migration * file preview * importing status * remove console statement * import complete view * fix geojson_importer test * tslint * i18n fixes * cleanup * update documenation for advanced setting rename * advanced settings usage_collection * remove ml:fileDataVisualizerMaxFileSize from schemas and types * add copy buttons for import response and fix geojson upload functional tests * tslint * remove clipboard-read check * return early if env does not support reading from clipboard * fix reporting tests * review feedback * update GeoJsonFileSource to support showing results trimmed icon and tooltip * add fileUpload to useMlKibana context and replace dependencyCache with useMlKibana * tslint * review feedback * lower case file name * default to selecting geo_shape when file contains both points and shapes * fix wizard onError callback to not advance to next step Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

* master: (45 commits) Add outcome of node scripts/build_api_docs (elastic#93399) [Lens] fix long field name on field stats panel doesn't wrap (elastic#93279) [Bug] Fix filter creation for numeric scripted fields in Discover (elastic#93224) [uptime] Fix anomaly alert edit (elastic#93025) Consolidate @babel/* packages and use latest compatible version (elastic#93264) [Search Embeddable] Add highlighting when searching (elastic#93178) [APM] Add missing bottom border to header (elastic#93179) [CI] No longer collect APM span stack traces (elastic#93263) [XY Chart] Fix "No data to display" error when using IP range aggregation to split series (elastic#93024) update generated public api docs API DOCS Step 3/3 (elastic#92929) chore(NA): look for bazel packages on npm_module folder during distributable build (elastic#93262) rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB (elastic#92620) [kbn/optimizer] allow customizing the limits path from the script (elastic#93153) [Alerting][Docs] Adding template for documenting alert and action types (elastic#92830) [jenkins] convert baseline capture job to use tasks (elastic#93288) removing the linked issue in comments from PR (elastic#93303) chore(NA): do not include fs within a storybook build (elastic#93294) [Maps] Update Map extent queries to use bounding box logic for both point and shape queries (elastic#93156) Add searchDuration to EQL and Threshold rules (elastic#93149) ...

kibanamachine · 2021-03-04T21:03:26Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fileUpload`	197	199	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fileUpload`	826.0KB	825.0KB	-1.0KB
`maps`	2.7MB	2.7MB	+724.0B
`ml`	6.4MB	6.4MB	-452.0B
`triggersActionsUi`	1.6MB	1.5MB	-23.9KB
total			-24.6KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fileUpload`	11.5KB	13.7KB	+2.2KB
`triggersActionsUi`	104.0KB	104.1KB	+82.0B
total			+2.3KB

Unknown metric groups

async chunk count

id	before	after	diff
`triggersActionsUi`	41	42	+1

History

💚 Build #110581 succeeded 5968394
💚 Build #110526 succeeded eedb74e
💚 Build #110507 succeeded e80285a
💚 Build #110395 succeeded 7f4ce1c
💔 Build #110366 failed a1cbbfb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

nreese added 9 commits February 22, 2021 11:37

rename ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize

a102bab

add saved object migration

bf4bf6a

file preview

1db54f7

importing status

eb63959

remove console statement

1665c45

import complete view

aca5463

fix geojson_importer test

1889840

tslint

21c38f2

i18n fixes

e7f6866

nreese added release_note:enhancement [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation v8.0.0 v7.13.0 labels Feb 24, 2021

nreese requested review from kindsun and jgowdyelastic February 24, 2021 14:18

nreese requested review from a team as code owners February 24, 2021 14:18

nreese added :ml Feature:File Upload labels Feb 24, 2021

nreese changed the title ~~rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUppload:maxFileSize and increase max geojson upload size to 1GB~~ rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB Feb 24, 2021

cleanup

07ff940

kibanamachine and others added 2 commits February 24, 2021 09:22

Merge branch 'master' into file_upload_max_size

1818acf

update documenation for advanced setting rename

ed7a0fa

nreese mentioned this pull request Feb 24, 2021

[Maps][File upload] GeoJSON file fails in preview but works on index #92504

Closed

advanced settings usage_collection

295ef6c

nreese requested a review from a team as a code owner February 24, 2021 16:36

tslint

7f4ce1c

jgowdyelastic approved these changes Mar 2, 2021

View reviewed changes

thomasneirynck mentioned this pull request Mar 2, 2021

[Maps][File upload] Remove usage of max file size advanced setting 1GB limit in geo file upload #93261

Closed

thomasneirynck approved these changes Mar 2, 2021

View reviewed changes

kindsun reviewed Mar 2, 2021

View reviewed changes

x-pack/plugins/file_upload/public/components/import_complete_view.tsx Outdated Show resolved Hide resolved

x-pack/plugins/file_upload/public/importer/geojson_importer/geojson_importer.ts Outdated Show resolved Hide resolved

merge with master

ef47a78

review feedback

812b511

nreese requested a review from kindsun March 2, 2021 20:14

nreese added 2 commits March 2, 2021 13:15

lower case file name

e80285a

default to selecting geo_shape when file contains both points and shapes

eedb74e

fix wizard onError callback to not advance to next step

5968394

kindsun approved these changes Mar 2, 2021

View reviewed changes

nreese merged commit bd9b349 into elastic:master Mar 3, 2021

nreese mentioned this pull request Mar 3, 2021

[7.x] rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB (#92620) #93358

Merged

This was referenced Mar 4, 2021

[Maps] chunk geojson upload to keep import requests under 1MB #93678

Merged

[Maps] Error seen in maps while uploading a geoJSON - Uncaught (in promise) TypeError: Invalid attempt to spread non-iterable instance #60710

Closed

thomasneirynck mentioned this pull request Mar 9, 2021

[Maps] Layer in preview mode shows different results after uploading geojson file when feature-count exceeds ES-search limit #40091

Closed

jportner mentioned this pull request Mar 15, 2021

Fix invalid Reporting integration test data #94616

Merged

nreese mentioned this pull request Apr 1, 2021

[Maps] GeoJSON upload can't index polygons #61794

Closed

nreese mentioned this pull request Apr 12, 2021

[Maps] Increase geojson upload file size #49240

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

nreese commented Feb 24, 2021 •

edited

Loading

elasticmachine commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

nreese commented Feb 24, 2021

nreese commented Feb 24, 2021

jgowdyelastic left a comment

thomasneirynck left a comment

kindsun commented Mar 2, 2021 •

edited

Loading

nreese commented Mar 2, 2021

kindsun commented Mar 2, 2021 •

edited

Loading

nreese commented Mar 2, 2021

kindsun commented Mar 2, 2021 •

edited

Loading

nreese commented Mar 2, 2021

kindsun left a comment

kibanamachine commented Mar 4, 2021

async chunk count

rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

rename advanced setting ml:fileDataVisualizerMaxFileSize to fileUpload:maxFileSize and increase max geojson upload size to 1GB #92620

Conversation

nreese commented Feb 24, 2021 • edited Loading

elasticmachine commented Feb 24, 2021

elasticmachine commented Feb 24, 2021

nreese commented Feb 24, 2021

nreese commented Feb 24, 2021

jgowdyelastic left a comment

Choose a reason for hiding this comment

thomasneirynck left a comment

Choose a reason for hiding this comment

kindsun commented Mar 2, 2021 • edited Loading

nreese commented Mar 2, 2021

kindsun commented Mar 2, 2021 • edited Loading

nreese commented Mar 2, 2021

kindsun commented Mar 2, 2021 • edited Loading

nreese commented Mar 2, 2021

kindsun left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 4, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

async chunk count

History

nreese commented Feb 24, 2021 •

edited

Loading

kindsun commented Mar 2, 2021 •

edited

Loading

kindsun commented Mar 2, 2021 •

edited

Loading

kindsun commented Mar 2, 2021 •

edited

Loading