4 zeros inserted after some metrics

[matejzero]

Hello,

today I have noticed that I'm not receiving some aggregated and renamed metrics. I took a look at the thing I found out that for some reason, carbon-c-relay adds 4 zeros at the end of some metrics

Config example:

rewrite ^servers\.server_example_org\.iscsisvrtop\.gauge-(10.20.30.40)-(.+)
  into servers.server_example_org.iscsisvrtop.organization_1.\2
  ;
aggregate
        ^servers\.server[0-9]_example_org\.iscsisvrtop\.gauge-(.+)-(.+)
    every 10 seconds
    expire after 15 seconds
    timestamp at end of bucket
    compute sum write to
        servers.server_example_org.iscsisvrtop.gauge-\1-\2
    ;
match
  ^collectd.localhost
  ^servers\. server[0-9]_example_org\.iscsisvrtop
  send to blackhole
  stop
  ;

If I look at the output of carbon-c-relay with ngrep, I get metrics like:

1476188810.servers.server_example_org.iscsisvrtop.organization_1.writes 561.0 0000
1476188810.servers.server_example_org.iscsisvrtop.organization_1.reads 281.0 0000 

The leading 0000 are breaking the metrics format and graphite doesn't want to save them.

I tried different versions of carbon-c-relay and it works OK on v2.0, but breaks on v2.1 and v2.2.

Is there something wrong with my config or is it a bug?

[grobian]

feels like a bug to me

[grobian]

The only (big) change between v2.0 and v2.1 is optimiser changes, how many rules do you have in your config?

[matejzero]

I have cca 90 rewrite rules and 10 aggregation rules.

[matejzero]

I did some testing.

This thing starts happening when I have 39 rewrite rules or more. If I disable one, so I only have 38 rewrite rules, it starts working.

It is rewrite rule specific, because if I disable aggregation rule and still keep 39 rewrite rules, it still doesn't work.

I dropped the output of carbon-c-relay into wireshark and this is what I see:

39 rules:

Text: servers.server_example_org.iscsisvrtop.organization_1.iops 100.0 0000 1476443643\n

38 rules:

Text: servers. server_example_org.iscsisvrtop.organization_1.iops 100.000000 1476443685\n

It looks like when there are more then 38 rules, carbon-c-relay for some reason drops one zero out of metric value.

[grobian]

do you have means to edit the sources and compile? In that case it would be interesting if you could disable the optimisation, that is remove the calls to router_optimise() in relay.c

[matejzero]

Will try to do that later today and report back

[grobian]

thanks a lot!

[matejzero]

After removing router_optimise(), it works as it should

[grobian]

Ok, that's good (and bad for me) news. I know where to look now.

[matejzero]

I'm glad I could be of help. I wish I could do more, but C ain't my thing:)

[grobian]

You could help me by testing the last commit to see if it works for you. I'm affraid we can't really group rewrites, since they potentially rewrite to something that would be matched by something else in the same group.

[grobian]

I think this will help, please reopen if it doesn't.

[matejzero]

I can confirm it now works.

Thanks for the fix.