Custom beat: Cannot export and re-import dashboard. Missing steps?

[joshuar]

Hey all,

I'm developing a custom beat against the 6.4 branch. I'm following the beats developer guide. My problem is with the export/import of dashboards.

So I have a clean Elasticsearch/Kibana running from the official Docker images. I'll use my beat to index data into ES and then build out some visualisations and a dashboard in KB. Then I go to export the dashboard to include it in my beat with:

go run vendor/github.com/elastic/beats/dev-tools/cmd/dashboards/export_dashboards.go -dashboard 79a7d0e0-a755-11e8-8251-ff8bb68f52ef -output _meta/kibana/6/dashboard/mydashboard.json 

I believe _meta/kibana/6/dashboard/ is the correct location for these dashboards.

I then wipe my Elasticsearch clean (i.e, curl -XDELETE localhost:9200/_all), logout and log back into Kibana so it recreates its index with its OOTB defaults.

I then run make update in my beat which seems to create a _meta/kibana.generated/6 directory which contains both the dashboard and index pattern.

I'll then re-run the setup of my beat and specify this directory with ./mybeat -d "*" -e setup -E "setup.dashboards.directory=_meta/kibana.generated", which seems to load the index pattern and dashboards/visualisations correctly.

Then run my beat to re-generate my data and then try to view my imported dashboard. But then I get:

The visualisations themselves work individually, but the dashboard seems to be broken. I have to recreate the dashboard from scratch. The time is borked and no visualisations are displayed. I have to delete this dashboard object and re-create with the visualisations.

Is this a know problem? Or am I missing some step here?

[ruflin]

Could you check the content of your dashboard file inside kibana.generated to see how it looks like? Does it have encoded json inside or not? Can you compare it to what you get after you exported it?

[joshuar]

It does contain JSON but looks like a completely different file. If I diff the export in _meta/kibana/6/dashboard vs the generated one in _meta/kibana.generated/6/dashboard it appears the entire file is different.

Interestingly, if I export the dashboard with:

go run vendor/github.com/elastic/beats/dev-tools/cmd/dashboards/export_dashboards.go -dashboard 79a7d0e0-a755-11e8-8251-ff8bb68f52ef -output _meta/kibana/6/dashboard/mydashboard.json 

Then wipe my .kibana index and try to re-import by running the beat with the setup command pointed at the dashboards I just exported:

/mybeat -d "*" -e setup -E "setup.dashboards.directory=_meta/kibana"

I get the following error:

Loaded index template
Loading dashboards (Kibana must be running and reachable)
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory _meta/kibana: Failed to import dashboard: Failed to load directory _meta/kibana/6/dashboard:
  error loading _meta/kibana/6/dashboard/node_stats_overall.json: failed to parse [visualization.kibanaSavedObjectMeta.searchSourceJSON]. Response: {"objects":[{"id":"55f0a7a0-a68b-11e8-80c7-5b7f0ebd2d9a","type":"visualization","error":{"message":"failed to parse [visualization.kibanaSavedObjectMeta.searchSourceJSON]"}},{"id":"2a2f4570-b00b-11e8-bd2f-2dc03ea3fd59","type":"visualization","error":... (truncated)
  error loading _meta/kibana/6/dashboard/thread_pools_overall.json: failed to parse [visualization.kibanaSavedObjectMeta.searchSourceJSON]. Response: {"objects":[{"id":"58eac910-ab52-11e8-922c-c7db1716ef83","type":"visualization","error":{"message":"failed to parse [visualization.kibanaSavedObjectMeta.searchSourceJSON]"}},{"id":"6d69e860-ab54-11e8-922c-c7db1716ef83","type":"visualization","error":... (truncated)

[joshuar]

Okay I did some more digging, and it looks like the export_dashboards.go tool does not properly export the panelsJSON field for the dashboard. If I run for example:

go run vendor/github.com/elastic/beats/dev-tools/cmd/dashboards/export_dashboards.go -dashboard 79a7d0e0-a755-11e8-8251-ff8bb68f52ef -output _meta/kibana/6/dashboard/mydashboard.json 

mydashboard.json contains all the necessary saved searches and visualisations for the dashboard, but the dashboard object itself has "null" as the value for the panelsJSON field. If I run the export API request that the tool is using directly against Kibana, i.e.:

curl -XGET 'localhost:5601/api/kibana/dashboards/export?dashboard=79a7d0e0-a755-11e8-8251-ff8bb68f52ef'

It's the same JSON response, but the panelsJSON field is correctly exported with it's value.

[joshuar]

Alright, it looks like this line is the culprit.

Catching the error thrown by json.Unmarshal produces:

error: json: cannot unmarshal array into Go value of type common.MapStr

Indeed the panelsJSON value is an array in my case. Is this a known issue?

[ruflin]

@joshuar Thanks for digging into the code here directly. Any chance you could send me an email with the dashboard file that is exported when you run ./metricbeat export dashboard -id 79a7d0e0-a755-11e8-8251-ff8bb68f52ef > dashboard.json? This works on Metricbeat 6.4 as an example. I'm wondering if there is something new in the dashboard structure from Kibana that we don't know about yet.

[joshuar]

Okay scratch that last comment (deleted it) as that was with a broken dashboard. So I just recreated a new dashboard in Kibana, saved it, then ran the export command above. Here is the JSON export:

dashboard.zip

[ruflin]

@ycombinator Could you have a look at this one?

[ycombinator]

Yep, I got it.

[ycombinator]

@joshuar I believe #8954 should've fixed this issue. Any chance you could work off master and try to reproduce the issue? If master is not an option, the fix should be getting backported to 6.x (for release in 6.6) shortly: #9102

[ChrsMark]

Hello there, facing a similar problem.

Having exported a Kibana dashboard with:

go run vendor/github.com/elastic/beats/dev-tools/cmd/dashboards/export_dashboards.go -dashboard 79a7d0e0-a755-11e8-8251-ff8bb68f52ef -output /kibana/6/dashboard/mydashboard.json 

I'm not able to import with:

 ./custombeat -v -e -d "*" setup -E setup.dashboards.enabled=true

The error I get is:

failed to parse [visualization.uiStateJSON]. Response: {"objects":[{"id":"12fe0dc0-0374-11e9-980d-c5ba02cd266a","type":"visualization","error":{"message":"failed to parse [visualization.uiStateJSON]"}},{"id":"70f41ec0-0373-11e9-980d-c5ba02cd266a","type":"visualization","error":{"message":"failed to parse... (truncated)

So comparing my json file with the one @joshuar uploaded I find that uiStateJSON, visState, searchSourceJSON are decoded instead of being strings. So to tackle this, I comment out the decoding options on

beats/libbeat/dashboards/decode.go

Line 30 in 1ed8e32

"attributes.uiStateJSON",

, export again and everything works fine.

So I'm wondering if this is something expected, or there is an incompatibility, bug, or problematic behavior.

cc: @ycombinator

[ycombinator]

Hey @ChrsMark, what version/branch are you developing your custom beat from?

[ChrsMark]

Hello @ycombinator, here is the info:

chrismark@chrismark-dev:~/go-home/src/github.com/nfvsap/natsbeat$ (cd ../../elastic/beats/ && git log -1 --decorate)
commit 6c88537d3a1e925abfd57a043563d90b3acb27b0 (HEAD -> master, origin/master, origin/HEAD)
Author: Andrew Kroh <-->
Date:   Tue Dec 18 01:39:33 2018 -0500

    Update auditbeat.reference.yml (#9606)

[jsoriano]

I think #9097 solved this issue with the added --decode flag.

[jlind23]

Backlog grooming: Closing it for now until further activity, can still be reopened if needed.