SkyWalking not work after two date #4502
Comments
|
until i delete all skywalking es index |
|
Seem like someone delete some of data before. And the OAP can't find the metadata. |
i don't think so! beacuse it reappear always after one day. |
|
I don't know why. Usually deleting index and creating indexes at middle night cause more payload, especially for some some ES cluster. The next 7.0.0 will reduce 50% of the index number, should ease this for you. But basically, that is caused by ES cluster health issue. |
ok！thanks！ |
|
It seems to be a uri encode bug, which has been fixed in elasticsearch-rest-high-level-client-7.2.0. |
Please answer these questions before submitting your issue.
Question
SkyWalking not work after two date ，i can got lots of error information like follow in SkyWalking-oap docker container：
2020-03-12 15:55:30,184 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO -338999 [pool-11-thread-1] ERROR [] - String index out of range: -1
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931) ~[?:1.8.0_212]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.encodePart(RequestConverters.java:1131) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]t org.elasticsearch.client.RequestConverters$EndpointBuilder.addPathPart(RequestConverters.java:1095) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.endpoint(RequestConverters.java:714) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.getStyleRequest(RequestConverters.java:265) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.get(RequestConverters.java:261) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1429) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.get(ElasticSearch7Client.java:187) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.cache.NetworkAddressInventoryCacheEsDAO.getAddressId(NetworkAddressInventoryCacheEsDAO.java:52) ~[storage-elasticsearch-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.cache.NetworkAddressInventoryCache.getAddressId(NetworkAddressInventoryCache.java:67) ~[server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.service.NetworkAddressInventoryRegister.getOrCreate(NetworkAddressInventoryRegister.java:70) ~[server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.standardization.ReferenceIdExchanger.exchange(ReferenceIdExchanger.java:102) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.preBuild(SegmentParseV2.java:166) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2.parse(SegmentParseV2.java:104) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.receiver.trace.provider.parser.SegmentParseV2$Producer.call(SegmentParseV2.java:288) ~[skywalking-trace-receiver-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.library.buffer.DataStreamReader.reCall(DataStreamReader.java:163) ~[library-buffer-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.library.buffer.DataStreamReader.read(DataStreamReader.java:150) ~[library-buffer-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.6.0.jar:6.6.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
2020-0312 15:55:30,432 - org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker -339247 [DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread] ERROR [] - String index out of range: -1
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.String.substring(String.java:1931) ~[?:1.8.0_212]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.encodePart(RequestConverters.java:1131) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters$EndpointBuilder.addPathPart(RequestConverters.java:1095) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.endpoint(RequestConverters.java:714) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.getStyleRequest(RequestConverters.java:265) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RequestConverters.get(RequestConverters.java:261) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1429) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1403) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1373) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:699) ~[elasticsearch-rest-high-level-client-7.0.0.jar:7.0.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client.get(ElasticSearch7Client.java:187) ~[storage-elasticsearch7-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.RegisterEsDAO.get(RegisterEsDAO.java:41) ~[storage-elasticsearch-plugin-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.lambda$onWork$0(RegisterPersistentWorker.java:97) ~[server-core-6.6.0.jar:6.6.0]
at java.util.HashMap$Values.forEach(HashMap.java:981) [?:1.8.0_212]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.onWork(RegisterPersistentWorker.java:95) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker.access$100(RegisterPersistentWorker.java:40) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.oap.server.core.register.worker.RegisterPersistentWorker$PersistentConsumer.consume(RegisterPersistentWorker.java:153) [server-core-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.consume(MultipleChannelsConsumer.java:81) [apm-datacarrier-6.6.0.jar:6.6.0]
at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:52) [apm-datacarrier-6.6.0.jar:6.6.0]
2020-03-12 15:55:40,278 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -349093 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10074 millis, size: 262
2020-03-12 15:55:46,265 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -355080 [I/O dispatcher 1] WARN [] - Bulk [16] executed with failures
2020-03-12 15:55:53,181 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -361996 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9696 millis, size: 230
2020-03-12 15:55:56,266 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -365081 [I/O dispatcher 1] WARN [] - Bulk [17] executed with failures
2020-03-12 15:56:05,254 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -374069 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 8928 millis, size: 101
2020-03-12 15:56:16,266 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -385081 [I/O dispatcher 1] WARN [] - Bulk [18] executed with failures
020-03-12 15:56:18,248 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -387063 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9818 millis, size: 150
2020-03-12 15:56:26,268 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -395083 [I/O dispatcher 1] WARN [] - Bulk [19] executed with failures
2020-03-12 15:56:26,971 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -395786 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 5570 millis, size: 83
2020-03-12 15:56:40,413 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -409228 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10081 millis, size: 215
2020-03-12 15:56:46,269 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -415084 [I/O dispatcher 1] WARN [] - Bulk [20] executed with failures
2020-03-12 15:56:53,656 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -422471 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10048 millis, size: 15
2020-03-12 15:56:56,269 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -425084 [I/O dispatcher 2] WARN [] - Bulk [21] executed with failures
2020-03-12 15:57:06,118 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -434933 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9342 millis, size: 101
2020-03-12 15:57:16,270 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -445085 [I/O dispatcher 1] WARN [] - Bulk [22] executed with failures
2020-03-12 15:57:19,289 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -448104 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 9975 millis, size: 150
2020-03-12 15:57:26,271 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -455086 [I/O dispatcher 1] WARN [] - Bulk [23] executed with failures
2020-03-12 15:57:27,027 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -455842 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 4611 millis, size: 83
2020-03-12 15:57:40,419 - org.apache.skywalking.oap.server.storage.plugin.elasticsearch7.client.ElasticSearch7Client -469234 [pool-12-thread-1] INFO [] - Synchronous bulk took time: 10090 millis, size: 215
2020-03-12 15:57:46,272 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -475087 [I/O dispatcher 1] WARN [] - Bulk [24] executed with failures，
==================
and the SkyWalking-ui docker container will gotsome log as follow：
2020-03-10 08:25:40.017 WARN 1 --- [io-8080-exec-10] o.s.c.n.z.filters.post.SendErrorFilter : Error during filtering
com.netflix.zuul.exception.ZuulException: Forwarding error
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.handleException(RibbonRoutingFilter.java:189)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:164)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:111)
at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112)
at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:193)
at com.netflix.zuul.FilterProcessor.runFilters(FilterProcessor.java:157)
at com.netflix.zuul.FilterProcessor.route(FilterProcessor.java:118)
at com.netflix.zuul.ZuulRunner.route(ZuulRunner.java:96)
at com.netflix.zuul.http.ZuulServlet.route(ZuulServlet.java:116)
at com.netflix.zuul.http.ZuulServlet.service(ZuulServlet.java:81)
at org.springframework.web.servlet.mvc.ServletWrappingController.handleRequestInternal(ServletWrappingController.java:157)
at org.springframework.cloud.netflix.zuul.web.ZuulController.handleRequest(ZuulController.java:44)
at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:50)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1459)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.netflix.client.ClientException: null
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:118)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:152)
at org.springframework.cloud.netflix.zuul.filters.route.support.AbstractRibbonCommand.run(AbstractRibbonCommand.java:49)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10247)
at rx.Observable.subscribe(Observable.java:10214)
at rx.internal.operators.BlockingOperatorToFuture.toFuture(BlockingOperatorToFuture.java:51)
at rx.observables.BlockingObservable.toFuture(BlockingObservable.java:411)
at com.netflix.hystrix.HystrixCommand.queue(HystrixCommand.java:378)
at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:344)
at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:158)
... 67 common frames omitted
Caused by: java.lang.RuntimeException: org.apache.http.conn.HttpHostConnectException: Connect to oap:12800 [oap/172.23.0.2] failed: Connection refused (Connection refused)
at rx.exceptions.Exceptions.propagate(Exceptions.java:58)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:464)
at rx.observables.BlockingObservable.single(BlockingObservable.java:341)
at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112)
... 129 common frames omitted
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to oap:12800 [oap/172.23.0.2] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:82)
at org.springframework.cloud.netflix.ribbon.apache.RibbonLoadBalancingHttpClient.execute(RibbonLoadBalancingHttpClient.java:42)
at com.netflix.client.AbstractLoadBalancerAwareClient$1.call(AbstractLoadBalancerAwareClient.java:104)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:303)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$3$1.call(LoadBalancerCommand.java:287)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:231)
at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.drain(OnSubscribeConcatMap.java:286)
at rx.internal.operators.OnSubscribeConcatMap$ConcatMapSubscriber.onNext(OnSubscribeConcatMap.java:144)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:185)
at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
at rx.Observable.unsafeSubscribe(Observable.java:10151)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
at rx.Subscriber.setProducer(Subscriber.java:209)
at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.subscribe(Observable.java:10247)
at rx.Observable.subscribe(Observable.java:10214)
at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:444)
... 131 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 174 common frames omitted
Bug
Which version of SkyWalking, OS and JRE?
apache/skywalking-oap-server:6.6.0-es7
docker.elastic.co/elasticsearch/elasticsearch:7.6.0
apache/skywalking-ui:6.6.0
Which company or project?
What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.
Requirement or improvement
The text was updated successfully, but these errors were encountered: