-
Notifications
You must be signed in to change notification settings - Fork 478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adds new blog post announcing opensearch hadoop #1650
Open
harshavamsi
wants to merge
16
commits into
opensearch-project:main
Choose a base branch
from
harshavamsi:opensearch_hadoop_blog
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
66f5e9f
Adds new blog post announcing opensearch hadoop
harshavamsi b28eb91
Fix vale grammar errors
harshavamsi bb8a3f7
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 8d90bbc
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 615e2d3
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi b738d94
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi b9d86ea
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 14d65f7
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi ecbce98
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 71dd439
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 15cf7b5
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 8d2f092
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 8b1b822
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 6c2f17d
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 1d24543
Update _posts/2023-06-05-opensearch-hadoop-launch.markdown
harshavamsi 34d6892
Add download links
harshavamsi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
--- | ||
layout: post | ||
title: "Announcing the general availability of the OpenSearch Hadoop client" | ||
authors: | ||
- hvamsi | ||
- nknize | ||
date: 2023-05-06 12:15:00 -0700 | ||
categories: | ||
- releases | ||
meta_keywords: opensearch hadoop, apache spark, apache hive, apache hadoop, openseearch, mapreduce, hdfs | ||
meta_description: OpenSearch Hadoop is now generally available with support for multiple versions of OpenSearch to run on Spark and Hive. | ||
twittercard: | ||
description: OpenSearch Hadoop is now generally available with support for multiple versions of OpenSearch to run on Spark and Hive. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
excerpt: We are excited to announce the release of the new OpenSearch-Hadoop connector. This tool enables efficient interaction between your Hadoop-based Big Data operations and OpenSearch clusters, supporting all versions of OpenSearch. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
--- | ||
|
||
We are excited to announce the release of the new OpenSearch-Hadoop connector. This tool enables efficient interaction between your Hadoop-based Big Data operations and OpenSearch clusters, supporting all versions of OpenSearch. | ||
|
||
## OpenSearch-Hadoop connector features: | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Versatility**: Compatible with Scala up to version 2.13.x and Spark up to version 3.2.x, the connector facilitates data processing and analysis operations across different environments. | ||
- **Memory and I/O Efficient**: The connector is designed with a focus on performance. It uses pull-based parsing, supports bulk updates, and direct conversion to/of native types, resulting in efficient memory and network I/O usage. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Adaptive I/O**: The connector can detect transport errors and retry automatically. In case of node failures, it can re-route requests to available nodes. If OpenSearch is overloaded, the connector can detect data rejection and resend it. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Facilitates Data Co-Location**: The connector integrates with Hadoop to expose network access information, enabling co-located OpenSearch and Hadoop clusters to be aware of each other, thus reducing network I/O. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- **Secure Access**: Supports IAM for AWS-managed OpenSearch, ensuring secure access to your AWS resources. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
## Compatibility with OpenSearch | ||
|
||
The below matrix shows the compatibility of the [`opensearch-hadoop`](https://central.sonatype.com/artifact/org.opensearch.client/opensearch-hadoop) with versions of [`OpenSearch`](https://opensearch.org/downloads.html#opensearch). | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
| Client version | OpenSearch version | Elasticsearch version | | ||
| -------------- | ------------------ | --------------------- | | ||
| 1.0.1 | 1.0.0-2.8.0 | 7.10 | | ||
|
||
## Compatibility with spark and scala | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
| Client version | Spark version | Scala version(s) | | ||
| -------------- | ------------- | ---------------- | | ||
| 1.0.1 | 2.2.3 | 2.10 | | ||
| 1.0.1 | 2.4.4 | 2.11/2.12 | | ||
| 1.0.1 | 3.2.4 | 2.12/2.13 | | ||
|
||
## Compatibility with AWS Glue | ||
|
||
| Client version | Spark version | Glue version(s) | | ||
| -------------- | ------------- | --------------- | | ||
| 1.0.1 | 2.4.4 | 2 | | ||
| 1.0.1 | 3.2.4 | 3/4 | | ||
|
||
This connector is a crucial tool for anyone looking to leverage the full power of OpenSearch alongside their Hadoop ecosystem. Download and start using it today. | ||
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
harshavamsi marked this conversation as resolved.
Show resolved
Hide resolved
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update the meta with the following:
Meta_keywords: OpenSearch Hadoop, Apache Hadoop, OpenSearch Hadoop client
Meta_description: The OpenSearch Hadoop connector is now generally available with support for multiple versions of OpenSearch running on Spark and Hive.