Pubsub java client has pub/ack latency greater than a second

[jrheizelman]

In comparative tests, the gcloud java client's publish latency is over 10x worse than a grpc client in java and the gcloud Python client. Even in relatively low-throughput tests, the client consistently takes 1-2 seconds for a publish call, when other tests are getting results on the order of tens of milliseconds. at the same message size/QPS.

[mziccard]

Even in relatively low-throughput tests, the client consistently takes 1-2 seconds for a publish call

This is very strange. Can you share the code you are using for testing? I just run a quick test and I am getting publish times in the range 50-60milliseconds, which is in line with other user reports.

[jrheizelman]

Sure, we've been using it in a load testing framework here:

https://github.com/GoogleCloudPlatform/pubsub/blob/master/load-test-framework/src/main/java/com/google/pubsub/clients/gcloud/CPSPublisherTask.java

The call is:
Stopwatch stopwatch = Stopwatch.createStarted();
pubSub.publish(topic, messages);
stopwatch.stop();

We've been getting those numbers pretty consistently, even when there's only 1 thread sending the publish requests in at a time. In the same framework, a grpc client (if you want to see it, it's present in the grpc branch of the same repo) is getting 100-150ms pub/ack latency on the same test, same topic as the gcloud one.

[mziccard]

@jrheizelman can you clarify what you mean with "pub/ack latency"? Are you measuring the total time it takes for a message to be published and then received?

[jrheizelman]

No, the time it takes for the publish call to complete (i.e. be acked by the server). In this case, the time elapsed on the stopwatch in the call in my above post, so the amount of time the publish call blocks for.

Similarly, when we tried using the async publish call to see where the slowdown was taking place, the future was returned almost immediately, as you'd expect, but then the get() call took a second+

[mziccard]

@jrheizelman can you share with us the command you are using to start the test? (so we can compare data on a common config)

[jrheizelman]

Closing this issue because from communication with the pubsub teams, this is a known issue that's being worked on.