Adds new blog post announcing opensearch hadoop #1650
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| --- | ||
| layout: post | ||
| title: "Announcing the general availability of the OpenSearch Hadoop client" | ||
| authors: | ||
| - hvamsi | ||
| - nknize | ||
| date: 2023-05-06 12:15:00 -0700 | ||
| categories: | ||
| - releases | ||
| meta_keywords: opensearch hadoop, apache spark, apache hive, apache hadoop, openseearch, mapreduce, hdfs | ||
| meta_description: OpenSearch Hadoop is now generally available with support for multiple versions of OpenSearch to run on Spark and Hive. | ||
| twittercard: | ||
| description: OpenSearch Hadoop is now generally available with support for multiple versions of OpenSearch to run on Spark and Hive. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| excerpt: We are excited to announce the release of the new OpenSearch Hadoop connector. This tool enables efficient interaction between your Hadoop-based Big Data operations and OpenSearch clusters, supporting all versions of OpenSearch. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| --- | ||
|
|
||
| We are excited to announce the release of the new OpenSearch-Hadoop connector. This tool enables efficient interaction between your Hadoop-based Big Data operations and OpenSearch clusters, supporting all versions of OpenSearch. | ||
|
|
||
| ## OpenSearch Hadoop connector features: | ||
|
There was a problem hiding this comment. Can we add an opening paragraph here? For example, "The OpenSearch-Hadoop connector includes the following features:" (and remove the colon from the heading)
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - **Versatility**: Compatible with Scala up to version 2.13.x and Spark up to version 3.2.x, the connector facilitates data processing and analysis operations across different environments. | ||
| - **Memory and Input/Output (I/O) Efficient**: The connector is designed with a focus on performance. It uses pull-based parsing and supports bulk updates to and direct conversion of native types, resulting in efficient memory and network I/O usage. | ||
| - **Adaptive I/O**: The connector can detect transport errors and retry automatically. In case of node failures, it can reroute requests to available nodes. If OpenSearch is overloaded, the connector can detect data rejection and resend it. | ||
| - **Data Co-location Integration**: The connector integrates with Hadoop to expose network access information, enabling co-located OpenSearch and Hadoop clusters to be aware of each other, thus reducing network I/O. | ||
| - **Secure Access**: Supports identity and access management (IAM) for AWS-managed OpenSearch, ensuring secure access to your AWS resources. | ||
|
|
||
| ## Compatibility with OpenSearch | ||
|
|
||
| The following matrix shows the compatibility of [`opensearch-hadoop`](https://central.sonatype.com/artifact/org.opensearch.client/opensearch-hadoop) with versions of [`OpenSearch`](https://opensearch.org/downloads.html#opensearch). | ||
|
|
||
| | Client version | OpenSearch version | Elasticsearch version | | ||
| | -------------- | ------------------ | --------------------- | | ||
| | 1.0.1 | 1.0.0-2.8.0 | 7.10 | | ||
|
|
||
| ## Compatibility with Spark and Scala | ||
|
|
||
| | Client version | Spark version | Scala version(s) | | ||
| | -------------- | ------------- | ---------------- | | ||
| | 1.0.1 | 2.2.3 | 2.10 | | ||
| | 1.0.1 | 2.4.4 | 2.11/2.12 | | ||
| | 1.0.1 | 3.2.4 | 2.12/2.13 | | ||
|
|
||
| ## Compatibility with AWS Glue | ||
|
|
||
| | Client version | Spark version | Glue version(s) | | ||
| | -------------- | ------------- | --------------- | | ||
| | 1.0.1 | 2.4.4 | 2 | | ||
| | 1.0.1 | 3.2.4 | 3/4 | | ||
|
|
||
| This connector is a crucial tool for anyone looking to leverage the full power of OpenSearch alongside their Hadoop ecosystem. Download and start using it today. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the meta with the following:
Meta_keywords: OpenSearch Hadoop, Apache Hadoop, OpenSearch Hadoop client
Meta_description: The OpenSearch Hadoop connector is now generally available with support for multiple versions of OpenSearch running on Spark and Hive.