Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDBC input plugins fetch data from mysql multiple times #342

Open
kamit78 opened this issue Jul 4, 2019 · 1 comment
Open

JDBC input plugins fetch data from mysql multiple times #342

kamit78 opened this issue Jul 4, 2019 · 1 comment

Comments

@kamit78
Copy link

kamit78 commented Jul 4, 2019

Steps to reproduce-

  1. I have created multiple config files to fetch data from each table in mysql and put in /etc/logstash/conf.d folder
  2. Start logstash service as sudo systemctl start logstash
  3. Run following command to execute files
    /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/spt_audit_event.conf
  4. Data successfully fetched

Actual Results

  1. Number of documents in elasticsearch keep on increasing and reached to 8581812 but there are only 600 rows in mysql table

Is it bug in plugin or I'm doing something wrong ?

Following is configuration file
input{
jdbc {
jdbc_driver_library => "/usr/share/jdbc_driver/mysql-connector-java-5.1.47.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://:3306/"
jdbc_user => ""
jdbc_password => ""
statement => "select * from spt_identity"
}
}

output {
elasticsearch {
"hosts" => "localhost:9200"
"index" => ""
}
stdout {}
}

@aleksamagicka
Copy link

Look up this and this. That should allow you to gather only new ones, according to the timestamps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants