Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ES doesn't start up on OpenVZ #2768

Closed
jaxxstorm opened this issue Mar 13, 2013 · 14 comments
Closed

ES doesn't start up on OpenVZ #2768

jaxxstorm opened this issue Mar 13, 2013 · 14 comments

Comments

@jaxxstorm
Copy link

After much investigation, I think this may be a bug. I'll let you guys be the judge.

[2013-03-12 19:56:02,696][INFO ][node ] [Holocaust] {0.20.5}[63428]: initializing ...
[2013-03-12 19:56:02,696][DEBUG][node ] [Holocaust] using home [/opt/logging/elasticsearch-0.20.5], config [/opt/logging/elasticsearch-0.20.5/config], data [[/opt/logging/elasticsearch-0.20.5/data]], logs [/opt/logging/elasticsearch-0.20.5/logs], work [/opt/logging/elasticsearch-0.20.5/work], plugins [/opt/logging/elasticsearch-0.20.5/plugins]
[2013-03-12 19:56:02,700][INFO ][plugins ] [Holocaust] loaded [], sites []
[2013-03-12 19:56:02,707][DEBUG][common.compress.lzf ] using [UnsafeChunkDecoder] decoder
[2013-03-12 19:56:02,857][DEBUG][env ] [Holocaust] using node location [[/opt/logging/elasticsearch-0.20.5/data/elasticsearch/nodes/0]], local_node_id [0]
[2013-03-12 19:56:03,464][DEBUG][threadpool ] [Holocaust] creating thread_pool [generic], type [cached], keep_alive [30s]
[2013-03-12 19:56:03,468][DEBUG][threadpool ] [Holocaust] creating thread_pool [index], type [cached], keep_alive [5m]
[2013-03-12 19:56:03,468][DEBUG][threadpool ] [Holocaust] creating thread_pool [bulk], type [cached], keep_alive [5m]
[2013-03-12 19:56:03,468][DEBUG][threadpool ] [Holocaust] creating thread_pool [get], type [cached], keep_alive [5m]
[2013-03-12 19:56:03,469][DEBUG][threadpool ] [Holocaust] creating thread_pool [search], type [cached], keep_alive [5m]
[2013-03-12 19:56:03,469][DEBUG][threadpool ] [Holocaust] creating thread_pool [percolate], type [cached], keep_alive [5m]
[2013-03-12 19:56:03,469][DEBUG][threadpool ] [Holocaust] creating thread_pool [management], type [scaling], min [1], size [5], keep_alive [5m]
[2013-03-12 19:56:03,471][DEBUG][threadpool ] [Holocaust] creating thread_pool [flush], type [scaling], min [1], size [10], keep_alive [5m]
[2013-03-12 19:56:03,471][DEBUG][threadpool ] [Holocaust] creating thread_pool [merge], type [scaling], min [1], size [20], keep_alive [5m]
[2013-03-12 19:56:03,471][DEBUG][threadpool ] [Holocaust] creating thread_pool [refresh], type [scaling], min [1], size [10], keep_alive [5m]
[2013-03-12 19:56:03,471][DEBUG][threadpool ] [Holocaust] creating thread_pool [cache], type [scaling], min [1], size [4], keep_alive [5m]
[2013-03-12 19:56:03,471][DEBUG][threadpool ] [Holocaust] creating thread_pool [snapshot], type [scaling], min [1], size [5], keep_alive [5m]
[2013-03-12 19:56:03,483][DEBUG][transport.netty ] [Holocaust] using worker_count[48], port[9300], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[2/6/1], receive_predictor[512kb->512kb]
[2013-03-12 19:56:03,489][DEBUG][discovery.zen.ping.unicast] [Holocaust] using initial hosts [], with concurrent_connects [10]
[2013-03-12 19:56:03,490][DEBUG][discovery.zen ] [Holocaust] using ping.timeout [3s], master_election.filter_client [true], master_election.filter_data [false]
[2013-03-12 19:56:03,490][DEBUG][discovery.zen.elect ] [Holocaust] using minimum_master_nodes [-1]
[2013-03-12 19:56:03,491][DEBUG][discovery.zen.fd ] [Holocaust] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-03-12 19:56:03,493][DEBUG][discovery.zen.fd ] [Holocaust] [node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-03-12 19:56:03,508][DEBUG][monitor.jvm ] [Holocaust] enabled [true], last_gc_enabled [false], interval [1s], gc_threshold [{default=GcThreshold{name='default', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}, ParNew=GcThreshold{name='ParNew', warnThreshold=1000, infoThreshold=700, debugThreshold=400}, ConcurrentMarkSweep=GcThreshold{name='ConcurrentMarkSweep', warnThreshold=10000, infoThreshold=5000, debugThreshold=2000}}]
[2013-03-12 19:56:04,028][DEBUG][monitor.os ] [Holocaust] Using probe [org.elasticsearch.monitor.os.SigarOsProbe@549ad840] with refresh_interval [1s]
[2013-03-12 19:56:04,030][DEBUG][monitor.process ] [Holocaust] Using probe [org.elasticsearch.monitor.process.SigarProcessProbe@dec3c6d] with refresh_interval [1s]
[2013-03-12 19:56:04,031][DEBUG][monitor.jvm ] [Holocaust] Using refresh_interval [1s]
[2013-03-12 19:56:04,032][DEBUG][monitor.network ] [Holocaust] Using probe [org.elasticsearch.monitor.network.SigarNetworkProbe@6d7ffbf] with refresh_interval [5s]
[2013-03-12 19:56:04,035][DEBUG][monitor.network ] [Holocaust] net_info
host
venet0 display_name [venet0]
address [/10.137.22.179] [/127.0.0.1]
mtu [1500] multicast [false] ptp [true] loopback [false] up [true] virtual [false]
sub interfaces:
venet0:0 display_name [venet0:0]
address [/10.137.22.179]
mtu [1500] multicast [false] ptp [true] loopback [false] up [true] virtual [true]
lo display_name [lo]
address [/0:0:0:0:0:0:0:1%1] [/127.0.0.1]
mtu [16436] multicast [false] ptp [false] loopback [true] up [true] virtual [false]

[2013-03-12 19:56:04,159][DEBUG][indices.store ] [Holocaust] using indices.store.throttle.type [none], with index.store.throttle.max_bytes_per_sec [0b]
[2013-03-12 19:56:04,163][DEBUG][cache.memory ] [Holocaust] using bytebuffer cache with small_buffer_size [1kb], large_buffer_size [1mb], small_cache_size [10mb], large_cache_size [500mb], direct [true]
[2013-03-12 19:56:04,168][DEBUG][script ] [Holocaust] using script cache with max_size [500], expire [null]
[2013-03-12 19:56:04,185][DEBUG][cluster.routing.allocation.decider] [Holocaust] using node_concurrent_recoveries [2], node_initial_primaries_recoveries [4]
[2013-03-12 19:56:04,186][DEBUG][cluster.routing.allocation.decider] [Holocaust] using [cluster.routing.allocation.allow_rebalance] with [indices_all_active]
[2013-03-12 19:56:04,186][DEBUG][cluster.routing.allocation.decider] [Holocaust] using [cluster_concurrent_rebalance] with [2]
[2013-03-12 19:56:04,188][DEBUG][gateway.local ] [Holocaust] using initial_shards [quorum], list_timeout [30s]
[2013-03-12 19:56:04,304][DEBUG][http.netty ] [Holocaust] using max_chunk_size[8kb], max_header_size[8kb], max_initial_line_length[4kb], max_content_length[100mb], receive_predictor[512kb->512kb]
[2013-03-12 19:56:04,313][DEBUG][indices.recovery ] [Holocaust] using max_size_per_sec[0b], concurrent_streams [3], file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2013-03-12 19:56:04,318][DEBUG][indices.memory ] [Holocaust] using index_buffer_size [101.1mb], with min_shard_index_buffer_size [4mb], max_shard_index_buffer_size [512mb], shard_inactive_time [30m]
[2013-03-12 19:56:04,319][DEBUG][indices.cache.filter ] [Holocaust] using [node] weighted filter cache with size [20%], actual_size [202.2mb], expire [null], clean_interval [1m]
[2013-03-12 19:56:04,342][DEBUG][gateway.local.state.meta ] [Holocaust] using gateway.local.auto_import_dangled [YES], with gateway.local.dangling_timeout [2h]
[2013-03-12 19:56:04,342][DEBUG][gateway.local.state.meta ] [Holocaust] took 0s to load state
[2013-03-12 19:56:04,342][DEBUG][gateway.local.state.shards] [Holocaust] took 0s to load started shards state
[2013-03-12 19:56:04,354][ERROR][bootstrap ] {0.20.5}: Initialization Failed ...

  1. NullPointerException[null]

This happens on multiple OpenVZ nodes with interface venet0:0

I have tried setting the host.network IP with no luck.
@spinscale
Copy link
Contributor

Can you set loglevel in logging.yml to DEBUG and check the logfile (not the console output) for a stack trace in order to get more information?

@jaxxstorm
Copy link
Author

That's what is attached above?

Here it is in Gist format:

https://gist.github.com/jaxxstorm/c97e0e80f0b3d77bff4a

@spinscale
Copy link
Contributor

The above is missing the important piece (something has to be written out after the last line), which would help us debugging your problem. Your loglevel is already set correctly, but it seems you pasted this from your console and not from the elasticsearch log file - which is by default in logs/ directory of your elasticsearch installation.

@jaxxstorm
Copy link
Author

Ah, apologies. Here you go:

https://gist.github.com/jaxxstorm/e19be9dcc222e7c1cf6e

@spinscale
Copy link
Contributor

What operating system is this running on? There seem to be problems with the (operating system specific) sigar library. There was a similar problem on the mailinglist today, which was solved by removing sigar (however this is only a workaround, as you are losing many of the monitoring capabilities of elasticsearch).

See https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/8-qlr-7esas

@jaxxstorm
Copy link
Author

This is running on OpenVZ on CentOS 6.3. What monitoring capabilities will I lose?

@spinscale
Copy link
Contributor

Mainly these http://www.elasticsearch.org/guide/reference/api/admin-cluster-nodes-stats.html

Can you read #1975 and hyperic/sigar#15 and check if the same applies to you as well?

There seem to be general problems/needed workarounds with OpenVZ - I am tempted to close this bug :)

@jaxxstorm
Copy link
Author

#1975 looks similar, however I can't see where to specify the user ES runs as.

@jaxxstorm
Copy link
Author

Also, confirmed that removing the sigar libs gets it running, although is possible I'd like to keep them. What perms issues can I investigate?

@spinscale
Copy link
Contributor

If you are not using the debian init script, just start as root with bin/elasticsearch -f and see it if works.
In case you are doing this already I can't help you any further, as I don't have any experience with OpenVZ.

If it works as root, please close this issue - I think it is rather some complex sigar/openvz issue than an elasticsearch problem.

@kimchy
Copy link
Member

kimchy commented Mar 17, 2013

Btw, we did push this: #2776, maybe this will help to still run it with sigar?

@jaxxstorm
Copy link
Author

I shall give it a try, thankyou.

On Sun, Mar 17, 2013 at 10:41 PM, Shay Banon [email protected]:

Btw, we did push this: #2776#2776,
maybe this will help to still run it with sigar?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2768#issuecomment-15032698
.

@jaxxstorm
Copy link
Author

Just wanted to say that push #2776 fixed this completely, runs fine and also I get all my stats back! Thanks!

@spinscale
Copy link
Contributor

Not all your stats, but most of them (no filesystem information). Closing this now.
Thanks for providing all the information!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@kimchy @spinscale @jaxxstorm