User management of some sort?

[Hoodoo]

I wonder if user management of any kind is going to be introduced at some stage?

Kibana 2 seems to be almost there, with an option to only see some indices and kibana-auth branch, what's with Kibana 3?

Is there probably a way to somehow stick a Sinatra app to tokenize and filter queries and censor results before giving it back to Kibana without breaking its workflow?

[otisg]

I think this may be somewhat related to #384

[rashidkpc]

Tokenizing and filtering the ES query would be extremely complex, you're much better off implementing this on a different layer.

The best way to accomplish this is a proxy, there are examples for both nginx and apache in the sample/ directory. Combined with Elasticsearch's filtered aliases you'd have a complete solution for segmenting users. This also benefits from being able to use the entire stable of mod_auth modules, which comprise a far richer set of authentication methods than I could ever implement.

[Hoodoo]

I'd just want to thank you for both pointing in right direction and an example config.

A very quick way to set things up using logstash is:

• make logstash put events to different indices based on facility and a source file
• create aliases once (see elasticsearch documentation)
• use the nginx config in this package to set up a couple of virtual hosts

Your admins now see syslog events and developers see errorlogs of their applications and superusers see both, and it is super simple and awesome.

If you like it, create new aliases every day or week with cron. Close to zero skills needed, the combination of logstash, lumberjack and kibana is pretty simple to install, scalable and very configurable.

Once again, thank you very much.

[kimchy]

@Hoodoo thanks for taking the time to explain this in detail, much appreciated!

[Hoodoo]

I have quite an amount of logs to chew through, around 20 gigs a day, maybe more. I've been trying different setups and went with lumberjack -> logstash -> graylog first, but I didn't like the configurability of graylog and overall complexity of the setup (mongo, elastic, graylog server, graylog web interface, ruby 1.9, passenger, nginx, lumberjack). Moreover, I had a server outage yesterday and wanted to maybe use my new toy to research it a bit, but I failed.

So today I've made a lumberjack -> logstash -> kibana setup, it has way fewer components - lumberjack, elastic, logstash, kibana and nginx, it should be easier to maintain and scale (add another elastic or logstash node, deploy new lumberjack config and restart lumberjack and here you go).

The test with actual data is due this week, and when I'm done and when a couple of weeks worth logs get there I'll write as much as I'm allowed about the setup.

[nidkil]

@Hoodoo can you please point me towards the examples you used to get this to work? Or maybe you are able to share your solution? Thx.

[crazyigor]

@Hoodoo i'm interested in your solution too. Could you please send us a howto how you managed your project? Or at least some examples? :-)

[Hoodoo]

Ooops. Didn't mean to be a prick, but here's what I have in quite an isolated environment:

server {
  listen *:80 ;

  server_name kibana.myhost.org;
  access_log /var/log/nginx/kibana.myhost.org.access.log;

# These are public 

  location / {
    root /local/elk-binaries/kibana;
    index index.html index.htm;
  }

  location ~ ^/_aliases$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_aliases$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/_nodes$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_search$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }
  location ~ ^/.*/_mapping {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
  }

  # Password protected end points
  location ~ ^/kibana-int/dashboard/.*$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
    limit_except GET {
      proxy_pass http://127.0.0.1:9200;
      auth_basic "Restricted";
      auth_basic_user_file /local/elk-configs/kibana/kibana.htpasswd;
    }
  }
  location ~ ^/kibana-int/temp.*$ {
    proxy_pass http://127.0.0.1:9200;
    proxy_read_timeout 90;
    limit_except GET {
      proxy_pass http://127.0.0.1:9200;
      auth_basic "Restricted";
      auth_basic_user_file /local/elk-configs/kibana/kibana.htpasswd;
    }
  }
}

Please note that I use this setup in an isolated environment (i.e. there's no access there from outside) and the aggregator does not store any sensible data whatsoever, so I can get away with such a half-assed 'restriction'.

I believe I got this one from examples which were shipped with Kibana BTW.