Elasticsearch csv Ingest Processor

This processor can parse CSV data and stores it as individual fields. This filter can also parse data with any separator, not just commas.

Deprecated

Official CSV ingest processor have released on 7.6.0. This plugin deprecated. Please use official CSV ingest processor.

Installation

ES version	Command
7.5.2	`./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.2/ingest-csv-7.5.2.0.zip`
7.5.1	`./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.1/ingest-csv-7.5.1.0.zip`
7.5.0	`./bin/elasticsearch-plugin install https://github.com/johtani/elasticsearch-ingest-csv/releases/download/7.5.0/ingest-csv-7.5.0.0.zip`
7.4.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.2.0/ingest-csv-7.4.2.0.zip`
7.4.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.1.0/ingest-csv-7.4.1.0.zip`
7.4.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.4.0.0/ingest-csv-7.4.0.0.zip`
7.3.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.2.0/ingest-csv-7.3.2.0.zip`
7.3.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.1.0/ingest-csv-7.3.1.0.zip`
7.3.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.3.0.0/ingest-csv-7.3.0.0.zip`
7.2.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.2.0.0/ingest-csv-7.2.0.0.zip`
7.0.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/7.0.0.0/ingest-csv-7.0.0.0.zip`

Older versions (6.8.5 - 5.5.0)

ES version	Command
6.8.5	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.8.5.0/ingest-csv-6.8.5.0.zip`
6.8.4	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.8.4.0/ingest-csv-6.8.4.0.zip`
6.7.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.7.1.0/ingest-csv-6.7.1.0.zip`
6.7.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.7.0.0/ingest-csv-6.7.0.0.zip`
6.6.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.2.0/ingest-csv-6.6.2.0.zip`
6.6.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.1.0/ingest-csv-6.6.1.0.zip`
6.6.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.6.0.0/ingest-csv-6.6.0.0.zip`
6.5.4	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.4.0/ingest-csv-6.5.4.0.zip`
6.5.3	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.3.0/ingest-csv-6.5.3.0.zip`
6.5.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.2.0/ingest-csv-6.5.2.0.zip`
6.5.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.1.0/ingest-csv-6.5.1.0.zip`
6.5.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.5.0.0/ingest-csv-6.5.0.0.zip`
6.4.3	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.3.0/ingest-csv-6.4.3.0.zip`
6.4.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.2.0/ingest-csv-6.4.2.0.zip`
6.4.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.1.0/ingest-csv-6.4.1.0.zip`
6.4.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.4.0.0/ingest-csv-6.4.0.0.zip`
6.3.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.2.0/ingest-csv-6.3.2.0.zip`
6.3.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.1.0/ingest-csv-6.3.1.0.zip`
6.3.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.3.0.0/ingest-csv-6.3.0.0.zip`
6.2.4	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.4.0/ingest-csv-6.2.4.0.zip`
6.2.4	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.4.0/ingest-csv-6.2.4.0.zip`
6.2.3	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.3.0/ingest-csv-6.2.3.0.zip`
6.2.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.2.0/ingest-csv-6.2.2.0.zip`
6.2.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.1.0/ingest-csv-6.2.1.0.zip`
6.2.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.2.0.0/ingest-csv-6.2.0.0.zip`
6.1.3	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.3.0/ingest-csv-6.1.3.0.zip`
6.1.2	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.2.0/ingest-csv-6.1.2.0.zip`
6.1.1	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.1.1.0/ingest-csv-6.1.1.0.zip`
6.0.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/6.0.0.0/ingest-csv-6.0.0.0.zip`
5.6.3	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/5.6.3.0/ingest-csv-5.6.3.0.zip`
5.5.0	`./bin/elasticsearch-plugin install https://oss.sonatype.org/service/local/repositories/releases/content/info/johtani/elasticsearch/plugin/ingest/ingest-csv/5.5.0.0/ingest-csv-5.5.0.0.zip`

Usage

PUT _ingest/pipeline/csv-pipeline
{
  "description": "A pipeline to do whatever",
  "processors": [
    {
      "csv" : {
        "field" : "my_field",
        "columns" : ["a", "b"]
      }
    }
  ]
}

PUT /my-index/_doc/1?pipeline=csv-pipeline
{
  "my_field" : "a_value,b_value"
}

GET /my-index/_doc/1

then, the doc has 3 fields like this.
{
  "my_field" : "a_value,b_value",
  "a" : "a_value",
  "b" : "b_value"
}

Configuration

Parameter	Use	Required
field	Field name of where to read the content from	Yes
columns	Define a list of column names.	Yes
quote_char	Define the character used to quote CSV fields. If this is not specified the default is a double quote ".	No
separator	Define the column separator value. If this is not specified, the default is a comma ,.	No
max_chars_per_column	Define the maximum amount of chars a column can have in it, before the plugin gives up. This is set to limit the amount of memory consumed. Valid values are 2 - 64000. Default is 4096	No

Setup

Install Java 12 + compiler

wget https://download.java.net/java/GA/jdk12/GPL/openjdk-12_linux-x64_bin.tar.gz tar -xzf openjdk-12_linux-x64_bin.tar.gz export JAVA_HOME=$(pwd)/jdk-12 export PATH=$(pwd)/jdk-12/bin:$PATH

In order to install this plugin, you need to create a zip distribution first by running

./gradlew clean check

This will produce a zip file in build/distributions.

After building the zip file, you can install it like this

bin/plugin install file:///path/to/ingest-csv/build/distribution/ingest-csv-0.0.1-SNAPSHOT.zip

Bugs & TODO

Need more test, like using separator and quote_char
and todos...

Thanks & More information about making Ingest Processor Plugin

Thanks for making great cookiecutter template @spinscale ! If you are interested in making Ingest Processor Plugin, you should use https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor

Name		Name	Last commit message	Last commit date
Latest commit History 70 Commits
.github/workflows		.github/workflows
gradle/wrapper		gradle/wrapper
licenses		licenses
src		src
.gitignore		.gitignore
LICENSE.txt		LICENSE.txt
NOTICE.txt		NOTICE.txt
README.md		README.md
build.gradle		build.gradle
gradle.properties		gradle.properties
gradlew		gradlew
gradlew.bat		gradlew.bat
settings.gradle		settings.gradle
uploadArchives.gradle		uploadArchives.gradle

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Elasticsearch csv Ingest Processor

Deprecated

Installation

Usage

Configuration

Setup

Install Java 12 + compiler

Bugs & TODO

Thanks & More information about making Ingest Processor Plugin

About

Releases 3

Packages

Contributors 2

Languages

License

johtani/elasticsearch-ingest-csv

Folders and files

Latest commit

History

Repository files navigation

Elasticsearch csv Ingest Processor

Deprecated

Installation

Usage

Configuration

Setup

Install Java 12 + compiler

Bugs & TODO

Thanks & More information about making Ingest Processor Plugin

About

Resources

License

Stars

Watchers

Forks

Releases 3

Packages 0

Contributors 2

Languages

Packages