Support restart upgrades to OpenSearch Dashboards 1.0 from Kibana v7.10.x

[dblock]

Upgrading your cluster from ElasticSearch to OpenSearch and OpenSearch Dashboards is just like upgrading your cluster between versions of ElasticSearch and Kibana. Specifically, OpenSearch supports rolling upgrades and restart upgrades from ElasticSearch 6.8.0 through ElasticSearch 7.10.2 and to OpenSearch 1.0. OpenSearch Dashboards supports restart upgrades from Kibana 6.8.0 through Kibana 7.10.2 and to OpenSearch Dashboards 1.0.

[dblock]

@seanneumann which versions do we want to be drop-in replacement for?

[dblock]

#325

[seanneumann]

Primary focus is to support Kibana 7.10.2 to OpenSearch Dashboard 1.0.0.

[kavilla]

We were using #231 but will use this issue instead.

On 14th Jan 2021, Elastic announced license change for Elastic and Kibana going forward no longer making them open-sourced LINK. Amazon forked ElasticSearch and Kibana and launched fork publicly LINK.

As we prepare to make a General Availability release for the OSS distribution, Kibana 7.10.2 will need to migrate to OpenSearch Dashboards 1.0.0.

While renaming we broke stuff that would not allow OpenSearch-Dashboards to be a simple drop-in replacement for Kibana.

Issues (not finalized):

• Index renamed from .kibana to .opensearch_dashboards
  • Will impact auto-migration because it will look for the default index which is now currently .opensearch_dashboards and will not see any index of that name so it will not migrate but just create the new index.
• Object keys that have be renamed that will impact the ability to automatically migrate saved objects for .kibana to .opensearch_dashboards because it has no value to map to:
  • kibanaSavedObjectMeta → opensearchDashboardsSavedObjectMeta
  • esTypes → opensearchTypes
  • timelion.* → timeline.*
• Object values that will have to be renamed that will impact the ability to automatically migrate saved objects that might impact the functionality:
  • indexRefName:"opensearchDashboardsSavedObjectMeta.searchSourceJSON.index"}
  • URLs stored in the saved objects that depend on URL that contains the route: kibana
  • Indices for example if the expression had .es() we changed to .opensearch() this will break any saved visualizations since it will try to query for .es(*) which does not exist.
  • DQL (previously KQL) can have queries saved for example: _index: ".kibana" would need to be _index: ".opensearch_dashboards"
• In general, any URLs that contains the route kibana
  • localhost:5603/qmg/app/management/kibana/* → localhost:5603/qmg/app/management/opensearch-dashboards/*
  • localhost:5603/qmg/app/kibana_overview#/ → localhost:5603/qmg/app/opensearch_dashboards_overview#/
• Map settings were changed, however this might not have any impact for people.
  • includeElasticMapsService → includeOpenSearchMapsService
  • proxyElasticMapsServiceInMaps → proxyOpenSearchMapsServiceInMaps
• Depending on what content is being set payload size will increase with the field name change. For example:
  • kibanaSavedObjectMeta is 21 bytes and opensearchDashboardsSavedObjectMeta is 35 bytes.
• Plugin impact
  • Plugins that depended on the default index being .kibana were impacted
• yaml file configurations
• vis_type_vega config.kibana → config.opensearchDashboards
• Settings are not being migrated and when the settings are loaded it is loaded by version, so even though the settings go carried over it used the settings 1.0.0-rc instead of the saved 7.x version.
• We had to remove telemetry within the application because it was coupled with pre-fork resources 142d0e0 and 142d0e0, there is telemetry items like dql-telemetry that we turned into kql-telemetry. Although, we made this change I am not able to create any problems and it seems to be purely for internal usage on the core application. Will leave dql-telemetry untouched.
  • There is an error about conflict on the search telemetry but that is from pre-fork and has no impact to the application and was not due to rebranding.

[kavilla]

There are a couple solutions that we are thinking about the most important tenant being "drop-in", the consideration on keeping .kibana index the same seems like the definite outcome as of right now. Just how we deal with the changed fields that we updated that will definitely require code change.

[kavilla]

Kibana did not support rolling upgrades, OpenSearch Dashboards will not for this release either.

[kavilla]

To summarize, keep settings breaking (kibana.yml --> opensearch_dashboards.yml). We could provide a tool for updating or provide documentation.

For breaking changes will restore the default index and restore data fields (not aliasing, aliasing has pitfalls). This shouldn't really trigger a migration and will hit the tenant of allowing customers to rollback from OpenSearch Dashboards to Kibana. However, this isn't really fully forward compatible with future expectations (whatever that will be perhaps 0 reference to kibana), so there will require more work on that end. But once we go from Kibana to OpenSearch Dashboards 1.0 then OpenSearch Dashboards 1.0 to vNext, we are not blocked by the tenant of allowing rolling back form OpenSearch Dashboards to Kibana. This was discussed to be the safest option and relatively reduce time to meet announced timelines. Another CON will be customers could use dev tools and see these values, but this is not a violation of license since it is functionality issue (or MAGIC WORDS). If it is a magic word then it is allowed even if it might show up on the screen.

However, we need to see if we can replace URL with 302 or restore back to kibana for example: localhost:5603/qmg/app/management/kibana/* → localhost:5603/qmg/app/management/opensearch-dashboards/*.

[dblock]

I opened a feature request #365 to support rolling upgrades in the future.

[tmarkley]

To summarize, keep settings breaking (kibana.yml --> opensearch_dashboards.yml). We could provide a tool for updating or provide documentation.

Where are we providing documentation for this?

[dblock]

I opened opensearch-project/documentation-website#71

[kavilla]

Will close this issue for tracking purposes, but will open if any issues related to GA prop up.

[shawnz]

Will it be supported to upgrade from Kibana 7.10.x to ODS 1.1? If not that would be somewhat difficult for us since 1.0 doesn't have an RPM distribution.