GELFDispatcher - Could not handle GELF message :: Writing GZipped Gelf to TCP socket fails

[farzadpanahi]

I am working on a fork from Gelf4Nlog https://github.com/farzadpanahi/Gelf4NLog to add TCP support. UDP is working fine and basically what I have done is to make the code write to a TCP socket rather than UDP. But it keeps failing with this error on server side:

2013-04-01 13:33:26,460 WARN : org.graylog2.inputs.gelf.GELFDispatcher - Could not handle GELF message.

java.lang.IllegalStateException: Failed to decompress the GELF message payload
at org.graylog2.gelf.GELFMessage.getJSON(GELFMessage.java:150)
at org.graylog2.gelf.GELFProcessor.messageReceived(GELFProcessor.java:62)
at org.graylog2.inputs.gelf.GELFDispatcher.messageReceived(GELFDispatcher.java:77)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:458)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:439)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:84)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:471)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:332)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:35)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
at java.util.zip.GZIPInputStream.readUByte(GZIPInputStream.java:264)
at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:171)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:78)
at java.util.zip.GZIPInputStream.(GZIPInputStream.java:90)
at org.graylog2.plugin.Tools.decompressGzip(Tools.java:159)
at org.graylog2.gelf.GELFMessage.getJSON(GELFMessage.java:139)
... 15 more

It looks like that the server fails to read the payload. Is there anything special that needs to be taken into consideration when writing to graylog-server tcp socket?

When the exact same generated bytes are written to a UDP socket, the graylog server reads the bytes and saves the log successfully.

I would appreciate any help/hint.

Code snippet responsible for writing to TCP socket:

public void Send(byte[] bytes, int length, IPEndPoint ipEndPoint)
{
using (var tcpClient = new TcpClient(ipEndPoint.Address.ToString(), ipEndPoint.Port))
{
var stream = tcpClient.GetStream();
stream.Write(bytes, 0, length);
stream.Close();
}
}

Code snippet responsible for writing to UDP socket:

public void Send(byte[] datagram, int bytes, IPEndPoint ipEndPoint)
{
using (var udpClient = new UdpClient())
{
udpClient.Send(datagram, bytes, ipEndPoint);
}
}

ps: I have also submitted this issue to graylog mailing list here: https://groups.google.com/d/msg/graylog2/2nV-_bLS2E0/WeJ-k9KygwQJ

[brianc]

Yeah same thing happening here.

[brianc]

@farzadpanahi I got this figured out. When you JSON.stringify your gelf packet you need to add a null character to the end of the string so the TCP parser can determine packet boundaries in a stream of unknown-sized packets opposed to UDP's fixed size but unknown delivery order.

var msg = JSON.stringify(yourGelfJsonObject) + '\0';

[farzadpanahi]

Tnx Brian :> I will look into this. It should solve my problem : >

[brianc]

From what I can figure out you cannot use compression either. So you just write JSON.stringified gelf messages + null characters into the tcp socket.

[lennartkoopmann]

Yes, @brianc is right! The NUL byte is used as a frame separator.

Compression should work over TCP too, though.

[farzadpanahi]

@lennartkoopmann, I would suggest you guys add the fact that "The NUL byte is used as a frame separator" to the GELF wiki page.

[michielproce]

When using gelfj (https://github.com/t0xa/gelfj/), the GELF payload is compressed with a GZIPOutputStream. This creates a 10-byte header, along with some \0 bytes. It seems these bytes are delimited by the TCP pipeline in gelf-server, as we only get the first three bytes (in our case the fourth byte was a \0 byte).

How can we fix this? Can we escape the \0 bytes somehow?

The message is gzipped in GelfMessage:gzipMessage() (line 132)
https://github.com/t0xa/gelfj/blob/master/src/main/java/org/graylog2/GelfMessage.java