The first data is lost

[StuCurry]

I have two data in my mongodb

_id ObjectId                                   time String    ip String

1 5c2492cb768d4d1d5c440e59 "16:52" No field
2 5c2492d9768d4d1d5c440e5a No field "100.99.9.1"

Only the second data can be read through the plugin

[2019-01-09T10:38:01,459][INFO ][logstash.inputs.mongodb ] Registering MongoDB input
{
"logdate" => "2018-12-27T08:52:41+00:00",
"log_entry" => "{"_id"=>BSON::ObjectId('5c2492d9768d4d1d5c440e5a'), "ip"=>"100.99.9.1"}",
"@Version" => "1",
"@timestamp" => 2019-01-09T02:38:03.720Z,
"host" => "greenvm-l14185v1",
"ip" => "100.99.9.1",
"mongo_id" => "5c2492d9768d4d1d5c440e5a",
"uid" => "5c2492d9768d4d1d5c440e5a"
}

My logstash.conf

input {
mongodb {
uri => 'mongodb://ip:27017/log'
placeholder_db_dir => '/tmp/logstash-mongodb/'
placeholder_db_name =>'logstash_sqlite.db'
collection => 'audit_log'
}
}

[TheVastyDeep]

That is correct, the code sorts the since column in ascending order and uses the first value as its placeholder. Then when it does a fetch it fetches everything greater than the placeholder, so the first value is not fetched.

See also this issue, which links to a PR which changes collection.find({:_id => {:$gt => last_id_object}}) to use $gte. That could result in duplicates, but would be in the spirit of logstash's "at least once" delivery model.

[binsonHao]

I have the same problem