Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not run "docker ps -a" in ResourceReaper to avoid TooLongFrameException from Netty #295

Merged
merged 1 commit into from
Mar 1, 2017

Conversation

bsideup
Copy link
Member

@bsideup bsideup commented Feb 20, 2017

dockerClient.listContainersCmd().withShowAll(true).exec() fails with TooLongFrameException from Netty ( see docker-java/docker-java#603 ) if host has a lot of different containers, including stopped ones.

We inspect later, so listContainersCmd is not needed and must be removed to avoid the performance penalty.

@bsideup bsideup requested a review from rnorth February 20, 2017 12:56
@bsideup bsideup added this to the 1.1.10 milestone Feb 20, 2017
@rnorth
Copy link
Member

rnorth commented Mar 1, 2017

Ah, this is an interesting problem. I'm happy to merge - it looks like this check is fairly redundant anyway, and the more specific inspect command later catches the NotFoundException if thrown.

@rnorth rnorth merged commit 8c14b99 into master Mar 1, 2017
@bsideup bsideup deleted the avoid_list_all_containers branch March 1, 2017 20:13
rnorth added a commit that referenced this pull request Mar 12, 2017
## [1.2.0] - 2017-03-12
### Fixed
- Fix various escaping issues that may arise when paths contain spaces (#263, #279)
- General documentation fixes/improvements (#300, #303, #304)
- Improve reliability of `ResourceReaper` when there are a large number of containers returned by `docker ps -a` (#295)

### Changed
- Support Docker for Windows via TCP socket connection (#291, #297, #309). _Note that Docker Compose is not yet supported under Docker for Windows (see #306)
- Expose `docker-java`'s `CreateContainerCmd` API for low-level container tweaking (#301)
- Shade `org.newsclub` and Guava dependencies (#299, #292)
- Add `org.testcontainers` label to all containers created by Testcontainers (#294)
@kiview
Copy link
Member

kiview commented Apr 13, 2017

I've just encountered the following error on the Gitlab CI server of a customer:

15:20:04.378 [dockerjava-netty-1-14] ERROR org.testcontainers.shaded.com.github.dockerjava.core.async.ResultCallbackTemplate - Error during callback
org.testcontainers.shaded.io.netty.handler.codec.TooLongFrameException: object length exceeds 1048576: 1049398 bytes discarded
    at org.testcontainers.shaded.io.netty.handler.codec.json.JsonObjectDecoder.decode(JsonObjectDecoder.java:99)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:72)
    at org.testcontainers.shaded.com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
    at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:233)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
    at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at org.testcontainers.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:961)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:140)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:386)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302)
    at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
    at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.604 sec <<< FAILURE! - in com.setlog.BomImportIntegrationTest
com.setlog.BomImportIntegrationTest  Time elapsed: 12.603 sec  <<< ERROR!
java.lang.ExceptionInInitializerError: null
    at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:739)
    at org.testcontainers.containers.GenericContainer.setDockerImageName(GenericContainer.java:727)
    at org.testcontainers.containers.GenericContainer.<init>(GenericContainer.java:150)
    at org.testcontainers.containers.JdbcDatabaseContainer.<init>(JdbcDatabaseContainer.java:35)
    at org.testcontainers.containers.MySQLContainer.<init>(MySQLContainer.java:14)
    at com.setlog.BomImportIntegrationTest.<clinit>(BomImportIntegrationTest.java:44)
Caused by: org.testcontainers.shaded.io.netty.handler.codec.TooLongFrameException: object length exceeds 1048576: 1049398 bytes discarded
    at org.testcontainers.shaded.io.netty.handler.codec.json.JsonObjectDecoder.decode(JsonObjectDecoder.java:99)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:72)
    at org.testcontainers.shaded.com.github.dockerjava.netty.handler.HttpResponseHandler.channelRead0(HttpResponseHandler.java:33)
    at org.testcontainers.shaded.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:233)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:435)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
    at org.testcontainers.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
    at org.testcontainers.shaded.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:350)
    at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:372)
    at org.testcontainers.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:358)
    at org.testcontainers.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
    at org.testcontainers.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:961)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollDomainSocketChannel$EpollDomainUnsafe.epollInReady(EpollDomainSocketChannel.java:140)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:386)
    at org.testcontainers.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302)
    at org.testcontainers.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877)
    at org.testcontainers.shaded.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
    at java.lang.Thread.run(Thread.java:745)

Does this stem from the same problem? Using testcontainers 1.2.1 here.

@rnorth
Copy link
Member

rnorth commented Apr 14, 2017

Hi @kiview
Given where it's occurring, I think the most likely cause is when populating the list of available images (RemoteDockerImage:58. This would be a separate, but related issue.

Is it likely that there are a large number of images on the CI machine?

As for resolving it, we deliberately fetch all available images and cache it so that we don't have to repeatedly check that the image is available every time we start a container. I will see what practical performance gain this has nowadays, though.

Richard

@kiview
Copy link
Member

kiview commented Apr 14, 2017

@rnorth You're right, there might be a lot of images on the CI server. I'll check later to give you a concrete number and I'll create an dedicated issue once I know more.

@rnorth
Copy link
Member

rnorth commented Apr 14, 2017

I ran a quick test to check, and I think the impact of adding a smarter filter to the image list query will have a negligible impact on performance. We should be able to improve this!

@asafm
Copy link
Contributor

asafm commented Apr 15, 2017 via email

@kiview
Copy link
Member

kiview commented Apr 15, 2017

I'm afraid that would have to be changed upstream in docker-java (which sounds like a good idea for them as well?).

@kiview
Copy link
Member

kiview commented Apr 21, 2017

@rnorth My client has fixed this problem by deleting the unused images, so yes, it was a problem with the list of available images. Sadly I couldn't get image count before they deleted them, so I don't know a hard number for when the error ocured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants