Split regex_t to one per thread

[marks-sortable]

When using carbon-c-relay to perform metric aggregation, strace found a lot of calls to futex. Using gdb, I discovered that we were frequently blocking on a mutex in the regex matching operations. By compiling a separate regex_t for each dispatcher, we can avoid blocking when matching metric names.

[grobian]

Thanks for the analysis and patch!

Quite stupid if you ask me, since the manpage reads The compiled form is not altered during execution of regexec(), so a single compiled RE can be used simultaneously by multiple threads. But that is on Darwin, Linux manpages don't mention any such thing, except that posix declares it thread-safe/mt-safe. I guess they implemented that with a mutex, yay.

I'll have a look, I might modify your work here and there, for I don't like the globals, so I don't think we should introduce new ones.

[marks-sortable]

You're welcome!

Here's somebody else also getting caught off-guard by the internal locks back in 2010: https://sourceware.org/bugzilla/show_bug.cgi?id=11159

[tehlers320]

FYI, i pulled this commit and 274 into v3.1 branch locally and created 3.1.1 pushing it out for testing. So far no problems and no locks. But like i said before our lock up issue seems mitigated by adding excessive threads to our instances (8).

However i do see a big bump in CPU use which i thought i'd share. CPU use increases are fine for me.
%user is shown

edit: actually this is not reliable data thinking it over more. We had a mirror setting go in with this change. This probably needs a lab environment for testing unless the below is not expected to jump the CPU.

"new-cluster" is an any_of match.

aggregate ^something
  some aggregate
  send to 
     old-cluster
     new-cluster
     stop
  ;

... 7 more aggregates

[grobian]

I'm working on this PR at the moment. I'm unhappy about some assumptions done in the code, but I guess at some point I need to chew on those and just merge it.