Pubsub Could Use Autocreate on Topics and Subscriptions

[waprin]

I think this will be a recurring pattern:

topic = pubsub.Topic('book-process-queue')
if not topic.exists():
    topic.create()

subscription = pubsub.Subscription('book-sub', topic)
if not subscription.exists():
    subscription.create()

So it might be a good idea to simplify it to:

topic = pubsub.Topic('book-process-queue', auto_create=True)

gcloud-node does something like this already.

[dhermes]

@tmatsuo WDYT?

[tmatsuo]

Yeah, that seems convenient, although there are some caveats.

• A topic created by auto_create=True will be a dev-null topic which means that messages will be just dropped.
• Implementation should be done in a way that the additional API call is made only when necessary (Don't always check the existence).

[dhermes]

@tmatsuo

• What is a dev-null topic?
• How can one avoid checking existence?

[theacodes]

We had this same conversation at gcloud-node here.

Basically what it boils down to is:

• a dev-null topic is a topic without subscriptions, so any messages sent will be dropped. I think it's out of the scope of the client libraries to avoid this (as it's not an error state), as it should be clearly expressed in docs and samples.
• You can avoid checking by lazily creating. So, create when you get a 404 and retry the original request. That's what we did in node here.

[dhermes]

How is that lazy? That's a GET followed by a POST if unsuccessful.

[theacodes]

The alternative would be to check at construction or before a request. In
most cases the resource does exist, so its a superfluous request.

On Tue, Jun 9, 2015, 4:43 PM Danny Hermes [email protected] wrote:

How is that lazy? That's a GET followed by a POST if unsuccessful.

—
Reply to this email directly or view it on GitHub
#905 (comment)
.

[jgeewax]

I agree, it's "lazy" in the sense that "we'll go based on the assumption that it exists" and a 404 means something different to that kind of topic. 404 to an autocreate topic means "go create the topic, then call the same thing again".

[dhermes]

@jonparrott Maybe I skimmed the node code too quickly / don't understand.

You're saying behavior like this

# foo does NOT exist
topic = Topic('foo', auto_create=True)  # No HTTP request is sent
message = ...
topic.publish(message)  # HTTP request results in 404, so first create and then publish

and

# foo DOES exist
topic = Topic('foo', auto_create=True)  # No HTTP request is sent
message = ...
topic.publish(message)  # HTTP request results in 200, confirmed the item exists

---

Here is some issues I notice with the node approach:

• In the failure case, publish results in 3 requests (404, 200 and 200). We could just as well do a create requests (PUT /projects/{project}/topics/{topic_name}) and just take a 409 or a 200 as a good sign and raise in any other case. This would just be 2 requests to publish (409/200 and 200).
• Tracking whether we are sure or not the item has been created should be done. In other words, a 404 for another reason should not trigger the create request (should not even do it a 2nd time).

[dhermes]

I'm going to send a fix and we can discuss a bit there?

[theacodes]

In the failure case, publish results in 3 requests

Correct, but the failure case only happens once.

We could just as well do a create requests.

You could, but you would need to do this every time the item is constructed.

The trade off is that assuming it exists and handling the failure cases results in 2 extra requests once, whereas looking before you leap means an extra request every single time.

In other words, a 404 for another reason should not trigger the create request

The node one only tries to create once. If the creation method fails, it returns the error to the original callback.

[dhermes]

I definitely overlooked "whereas looking before you leap means an extra request every single time."

Thanks

[dhermes]

Closing. See #949 for context.