Feature/java implementation #93

ph · 2016-06-09T14:23:56Z

Not ready for production, do not use.

ph · 2016-06-17T14:31:04Z

@talevy there is a faillings test in when the when the client authenticate with a secondary ca, I am looking at fixing this but You can do a first run :)

ph · 2016-06-20T15:34:15Z

@talevy this PR need to run with logstash master.

talevy · 2016-06-21T06:07:48Z

Gemfile

@@ -2,3 +2,4 @@ source 'https://rubygems.org'

 # Specify your gem's dependencies in logstash-mass_effect.gemspec
 gemspec
+gem "logstash-core", :path => "~/es/logstash"


leaving a note here to remember we need this to go before merging

ph · 2016-06-23T20:01:02Z

@talevy Can you take a second look?
@spinscale I have upgrade to netty 4.1.1, I had a few issues with the upgrade that the BeatsParser test catched and I have fixed all of them. I am still doing some test locally for freeing up the bytebuffer to make sure It doesnt have a leak.

spinscale · 2016-06-24T08:08:13Z

build.gradle

+    compile 'io.netty:netty-tcnative-boringssl-static:1.1.33.Fork17'
+    compile 'org.apache.logging.log4j:log4j-api:2.6.1'
+    compile 'org.apache.logging.log4j:log4j-core:2.6.1'
+    compile 'org.lucee:javassist:3.9.0.GA'


thats not the origininal javassist package, try http://mvnrepository.com/artifact/org.javassist/javassist

ph · 2016-06-30T19:16:17Z

I did another pass @spinscale !

spinscale · 2016-07-01T07:10:55Z

src/main/java/org/logstash/beats/BeatsParser.java

+        READ_JSON(-1),
+        READ_DATA_FIELDS(-1);
+
+        private int value;


name it length?

spinscale · 2016-07-01T07:17:19Z

left some minor comments, but I like this. hope to get this into more testing soon! :)

spinscale · 2016-07-04T07:04:27Z

quick update: Be careful with jackson and dependencies.. I just checked out this issue in the rss plugin and I see a reference to afterburner already logstash-plugins/logstash-input-rss#21

jordansissel · 2016-07-06T17:07:27Z

Failing to use this plugin:

% bin/logstash -e 'input { beats { } }'
fetched an invalid config {:config=>"input { beats { } }output { stdout { codec => rubydebug } }", :reason=>"Couldn't find any input plugin named 'beats'. Are you sure this is correct? Trying to load the beats input plugin resulted in this error: no such file to load -- vendor/jar-dependencies/logstash-input-beats-3.1.0.beta1", :level=>:error}

Noting the 'no such file to load', I checked a few things and everything seems OK to me:

% grep logstash-input-beats Gemfile*
Gemfile:gem "logstash-input-beats", "3.1.0.beta1", :path => "vendor/local_gems/18ae8921/logstash-input-beats-3.1.0.beta1-java"
Gemfile.jruby-1.9.lock:  remote: vendor/local_gems/18ae8921/logstash-input-beats-3.1.0.beta1-java
Gemfile.jruby-1.9.lock:    logstash-input-beats (3.1.0.beta1-java)
Gemfile.jruby-1.9.lock:  logstash-input-beats (= 3.1.0.beta1)!

^^^ Correct version installed, it seems.

And the file it should have tried to load does exist:

vendor/local_gems/18ae8921/logstash-input-beats-3.1.0.beta1-java/vendor/jar-dependencies/logstash-input-beats-3.1.0.beta1.jar

Weird?

jordansissel · 2016-07-06T18:07:53Z

lib/logstash/inputs/beats.rb

 require "logstash/util"
-require "thread_safe"
+require "vendor/jar-dependencies/logstash-input-beats-3.1.0.beta1.jar"


This causes Logstash to fail to load the plugin because the file is not found because it's the wrong path.

The .gemspec puts vendor/jar-dependencies into ruby's library path, so you can just do require "logstash-input-beats-3.1.0.beta1.jar" here. If I make this change, I can load this plugin into logstash.

jordansissel · 2016-07-12T00:02:15Z

If you throw garbage at a plain text beats input, you get all kinds of fun exceptions:

To reproduce:

% bin/logstash --log.level debug -e 'input { beats { port => 12345 } }'
...
% dd if=/dev/urandom bs=4k count=1 | nc localhost 12345

Sample exception logged by Logstash:

Jul 11, 2016 3:05:14 PM io.netty.channel.AbstractChannelHandlerContext invokeExceptionCaught
WARNING: An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: Illegal initial capacity: -1982301064
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:36)
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:339)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: Illegal initial capacity: -1982301064
        at java.util.HashMap.<init>(HashMap.java:448)
        at java.util.HashMap.<init>(HashMap.java:467)
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:125)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
        ... 9 more

jordansissel · 2016-07-12T00:15:40Z

Sending a MAX_INT as the json payload length causes fun errors>

Reproduction:

>> o = { "hello" => "world" }; TCPSocket.new("localhost", 12345).tap { |x| x.write(["1W", 1].pack("A*N")); x.write(["2J", 1, -1, o.to_json].pack("A*NNA*")); }.close

Error: io.netty.handler.codec.DecoderException: java.lang.NegativeArraySizeException

Full output

15:19:36.794 [nioEventLoopGroup-3-2] ERROR org.logstash.beats.BeatsHandler - Exception
io.netty.handler.codec.DecoderException: java.lang.NegativeArraySizeException
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:36) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:339) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:339) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:393) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145) [netty-all-4.1.1.Final.jar:4.1.1.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
Caused by: java.lang.NegativeArraySizeException
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:193) ~[logstash-input-beats-3.1.0.beta1.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[netty-all-4.1.1.Final.jar:4.1.1.Final]
        ... 9 more

jordansissel · 2016-07-12T00:52:11Z

Hilarious things happen when Logstash runs out of file descriptors. I triggered this scenario by simply opening lots of connections.

>> 10000.times.collect { TCPSocket.new("localhost", 12345) }.map(&:close)

jordansissel · 2016-07-12T00:59:48Z

.gitignore

+build/
+.idea
+.gradle
+null


jordansissel · 2016-07-12T01:12:32Z

Very poorly-scienced performance comparison on my 1-core VM

% (cd logstash-5.0.0-alpha4-with-new-beats; bin/logstash-plugin list --verbose | grep beats; bin/logstash -e 'input { beats { port => 12345 } } output { stdout { codec => dots } }'  | pv -War > /dev/null)
logstash-input-beats (3.1.0.beta1)
...
[13.5KiB/s] [9.37KiB/s]

Peak was around 13.5Keps (this PR)

% (cd logstash-5.0.0-alpha4; bin/logstash-plugin list --verbose | grep beats; bin/logstash -e 'input { beats { port => 12345 } } output { stdout { codec => dots } }'  | pv -War > /dev/null)
logstash-input-beats (3.0.3)
...
[11.2KiB/s] [4.55KiB/s]

Release (5.0.0-alpha4) beats input peaks at 11.2Keps.

filebeat.yml

% egrep -v '^\s*#|^\s*$' filebeat.yml
filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
      input_type: stdin
output:
  logstash:
    hosts: ["localhost:12345"]
shipper:
logging:
  files:
    rotateeverybytes: 10485760 # = 10MB

jordansissel · 2016-07-12T01:26:34Z

Same test as above with a 2-core VM (same physical machine)

this pr: ~21Keps
version 3.0.3: ~18Keps

I need to test against older versions also.

jordansissel · 2016-07-12T01:49:55Z

I also get this sometimes when shutting down:

java.util.concurrent.RejectedExecutionException: event executor terminated

ph · 2016-07-18T19:21:08Z

answering @jordansissel's comment here.

java.util.concurrent.RejectedExecutionException: event executor terminated I think this is due to the idle check execution pool, I was able to get this error in tests.

Concerning the test what did you use to generate the events, Filebeat from stdin?

jordansissel · 2016-07-18T19:35:44Z

@ph yeah I used filebeat from stdin

  (seq 21000000; sleep 10) | ./filebeat

ph · 2016-07-18T19:41:14Z

@jordansissel My test were done with logstash using all available core, the null output and generator beats which just send a infinite number of events. The stats were gathered at the sending end using libbeat http server. The metric was the number of events ack per seconds from a 30s window. IIRC.

ph · 2016-07-18T19:42:06Z

I am investigating the leak that @talevy has found when running the code locally, with the LeakDetector set to Advanced.

ph · 2016-07-18T19:44:38Z

@jordansissel still using stdout represent more a real world case. :)

ph · 2016-07-18T20:08:16Z

@jordansissel Concerning the Number of FD and when we send garbage, would you be OK to do that in another PR, this one is starting to be pretty big :)

ph · 2016-07-18T20:16:59Z

@jordansissel Can you use pipelining on the beat side under the logstash output configuration?
All the test we did was using 5 for the value, I don't think this option is on by default.

output.logstash:
  compression_level: 3
  hosts:
    - localhost:5044
  pipelining: 5

talevy · 2016-07-19T17:55:40Z

LGTM.

I think this should be merged into master as is and we can open up new issues around @jordansissel to keep track of those separately: #93 (comment)

suyograo · 2016-07-19T18:00:51Z

+1, this PR is getting huge as is.

@ph needs a rebase though

Motivation: The core of Logstash is using more and more java instead of ruby for performance reason and scalability. This PR introduce a lot of internal changes in the plugins and move from ruby to a pure java implementation using the Netty 4 Framework. Also the connection model of this plugin has changed from a blocking to a non blocking implementation. The changes should be backward compatible with the previous settings and will work for both LSF and Filebeat client.

ph · 2016-07-19T18:36:58Z

I've created issues from @jordansissel comments:

#101, #100, #99, #98

elasticsearch-bot · 2016-07-19T18:43:54Z

Pier-Hugues Pellerin merged this into the following branches!

Branch	Commits
master	`01e70cf`

ph mentioned this pull request Jun 9, 2016

Rewrite the beats plugin in pure java #92

Closed

14 tasks

ph force-pushed the feature/java-implementation branch from 73f8986 to f3725d7 Compare June 17, 2016 14:25

ph mentioned this pull request Jun 17, 2016

Handling jar dependency resolution with jruby projects elastic/logstash#5514

Open

ph assigned ph and talevy Jun 17, 2016

talevy reviewed Jun 21, 2016
View reviewed changes

suyograo added the enhancement label Jun 23, 2016

spinscale reviewed Jun 24, 2016
View reviewed changes

spinscale reviewed Jul 1, 2016
View reviewed changes

src/main/java/org/logstash/beats/BeatsParser.java

READ_JSON(-1),

READ_DATA_FIELDS(-1);

private int value;

Copy link

spinscale Jul 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name it length?

jordansissel reviewed Jul 6, 2016
View reviewed changes

jordansissel reviewed Jul 12, 2016
View reviewed changes

.gitignore

build/

.idea

.gradle

null

Copy link

Contributor

jordansissel Jul 12, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null? hah.

ph force-pushed the feature/java-implementation branch from 1fff26f to d1252fb Compare July 19, 2016 18:25

ph mentioned this pull request Jul 19, 2016

Running out of file descriptors leads to undesired effects #98

Open

elasticsearch-bot closed this in 01e70cf Jul 19, 2016

ph changed the title ~~[WIP] Feature/java implementation~~ Feature/java implementation Jul 19, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/java implementation #93

Feature/java implementation #93

ph commented Jun 9, 2016

ph commented Jun 17, 2016

ph commented Jun 20, 2016

talevy Jun 21, 2016

ph commented Jun 23, 2016

spinscale Jun 24, 2016

ph commented Jun 30, 2016

spinscale Jul 1, 2016

spinscale commented Jul 1, 2016

spinscale commented Jul 4, 2016

jordansissel commented Jul 6, 2016

jordansissel Jul 6, 2016

jordansissel commented Jul 12, 2016 •

edited

Loading

jordansissel commented Jul 12, 2016 •

edited

Loading

jordansissel commented Jul 12, 2016

jordansissel Jul 12, 2016

jordansissel commented Jul 12, 2016

jordansissel commented Jul 12, 2016

jordansissel commented Jul 12, 2016

ph commented Jul 18, 2016

jordansissel commented Jul 18, 2016 •

edited

Loading

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016 •

edited

Loading

talevy commented Jul 19, 2016

suyograo commented Jul 19, 2016

ph commented Jul 19, 2016 •

edited

Loading

elasticsearch-bot commented Jul 19, 2016

Feature/java implementation #93

Feature/java implementation #93

Conversation

ph commented Jun 9, 2016

ph commented Jun 17, 2016

ph commented Jun 20, 2016

talevy Jun 21, 2016

Choose a reason for hiding this comment

ph commented Jun 23, 2016

spinscale Jun 24, 2016

Choose a reason for hiding this comment

ph commented Jun 30, 2016

spinscale Jul 1, 2016

Choose a reason for hiding this comment

spinscale commented Jul 1, 2016

spinscale commented Jul 4, 2016

jordansissel commented Jul 6, 2016

jordansissel Jul 6, 2016

Choose a reason for hiding this comment

jordansissel commented Jul 12, 2016 • edited Loading

jordansissel commented Jul 12, 2016 • edited Loading

jordansissel commented Jul 12, 2016

jordansissel Jul 12, 2016

Choose a reason for hiding this comment

jordansissel commented Jul 12, 2016

jordansissel commented Jul 12, 2016

jordansissel commented Jul 12, 2016

ph commented Jul 18, 2016

jordansissel commented Jul 18, 2016 • edited Loading

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016

ph commented Jul 18, 2016 • edited Loading

talevy commented Jul 19, 2016

suyograo commented Jul 19, 2016

ph commented Jul 19, 2016 • edited Loading

elasticsearch-bot commented Jul 19, 2016

jordansissel commented Jul 12, 2016 •

edited

Loading

jordansissel commented Jul 12, 2016 •

edited

Loading

jordansissel commented Jul 18, 2016 •

edited

Loading

ph commented Jul 18, 2016 •

edited

Loading

ph commented Jul 19, 2016 •

edited

Loading