pubsub: API refactor

[stephenplusplus]

9/30/14:

PR: #245

9/29/14:

http://stephenplusplus.github.io/gcloud-node/#/docs/master/pubsub

9/22/14:

Outdated:

Initializing

var gcloud = require('gcloud')({ projectId: 'test-project' });
var pubsub = gcloud.pubsub();
// to override global defaults:
// var pubsub = gcloud.pubsub({ keyFilename: '/', etc: '...' });

Topics

// Create a topic
pubsub.createTopic('a-new-topic-name', function(err, topic) {});

// Get topics
pubsub.getTopics(function(err, topics) {});
pubsub.getTopics({ maxResults: 3 }, function(err, topics) {});

// Reference a topic
var topic = pubsub.topic('an-existing-topic-name');

// Publish message
topic.publish('Hi', function(err) {});

// Publish raw message
topic.publishRaw({
  data: 'Hi',
  label: [
    { key: 'priority', numValue: 0 },
    { key: 'foo', stringValue: 'bar' }
  ]
}, function(err) {});

Subscriptions

// Create a subscription
topic.subscribe('a-new-subscription-name', {
  autoAck: true
}, function(err, subscription));

// Get subscriptions
topic.getSubscriptions(function(err, subscriptions) {});
topic.getSubscriptions({ maxResults: 5 }, function(err, subscriptions) {});

// Reference a subscription
var subscription = topic.subscription('an-existing-subscription-name');

// Handle a message event
subscription.on('message', function(message) {});

// Close a subscription
subscription.close();

Example Usage

Create a topic and publish a message

pubsub.createTopic('newTopic', function(err, topic) {
  if (err) {/*...*/}
  topic.publish('Hey!', function(err) {
    if (err) {/*...*/}
  });
});

Publish a message to a topic

pubsub.topic('myTopic')
  .publish('Hey!', function(err) {
    if (err) {/*...*/}
  });

Create a subscription on a topic to listen for messages

pubsub.topic('myTopic')
  .subscribe('newSubscription', function(err, subscription) {
    if (err) {/*...*/}
    subscription.on('message', function() {
      console.log(msg);
    });
  });

Get an existing subscription on a topic to listen for messages, and auto ack them.

pubsub.topic('myTopic')
  .subscription('mySubscription', { autoAck: true })
  .on('message', function() {
    console.log(msg);
  });

Old recommended api, for historic purposes:

// New PubSub creation
var PubSub = require('lib/pubsub');
var pubsub = new PubSub({ projectId: 'test-project' });

// Create Topic
var TOPIC_NAME = 'puppies';
var PuppiesTopic = new pubsub.Topic(TOPIC_NAME);

// Publish Message (simple)
PuppiesTopic.publish('Hi', function(err) {});

// Publish Message (specific [raw format])
PuppiesTopic.publishRaw({
  data: 'Hi',
  label: [
    { key: 'priority', numValue: 0 },
    { key: 'foo', stringValue: 'bar' }
  ]
}, function(err) {});

// Subscribe to a Topic (specific [subscription name specified])
var puppyLogger = PuppiesTopic.subscribe({
  name: 'puppyLogger',
  autoAck: true,
  ackDeadlineSeconds: 60
});

puppyLogger.on('message', function(message) {
  console.log('New message from a puppy!', message);
});

// Close subscription after 5 minutes
setTimeout(puppyLogger.close, 300000);

// Get Subscriptions for a topic (simple)
PuppiesTopic.getSubscriptions(function(err, subscriptions) {});

// Get Subscriptions for a topic (specific)
PuppiesTopic.getSubscriptions({ maxResults: 5 }, function(err, subscriptions) {});

// Get Individual Subscription
PuppiesTopic.getSubscription('puppyLogger', function(err, subscription) {});

// Get Topics (simple)
pubsub.getTopics(function(err, topics) {});

// Get Topics (specific)
pubsub.getTopics({ maxResults: 5 }, function(err, topics) {});

// Get Inidividual Topic
pubsub.getTopic(TOPIC_NAME, function(err, topic) {});

[rakyll]

Looks generally good.

Two issues:

• Is there a way we provide creation of topics and subscriptions? https://developers.google.com/pubsub/v1beta1/topics/create and https://developers.google.com/pubsub/v1beta1/subscriptions/create
• publishMessage may be misleading. Message is the raw message format's entity name. My initial propose is to provide topic.publish(<String>) and topic.publishRaw(<Message>rawMessage), but they proposed me to use publishMessage(rawMessage). What about publish and publishRaw instead. I think they are more self-explanatory.

[stephenplusplus]

Is there a way we provide creation of topics...

I was thinking that would happen with the initial new pubsub.Topic('Topic Name') - if a topic with that name already exists, it would still return a "new Topic" object, but would refer to the already existing topic.

...and subscriptions?

I forgot to annotate what I did with that. Both of these bits would create a subscription:

// Subscribe to a Topic (simple [auto subscription name generated])
var puppyLogger = PuppiesTopic.subscribe({
  autoAck: true,
  ackDeadlineSeconds: 60
});

// Subscribe to a Topic (specific [subscription name specified])
var puppyLogger = PuppiesTopic.subscribe({
  name: 'puppyLogger',
  autoAck: true,
  ackDeadlineSeconds: 60
});

The first one would generate a subscription name for you, but if you wanted to provide one, simply provide a "name" property in the configuration object. I'm not sure if that's a great idea, definitely need more thoughts on that. My thought was, I may not care about naming it, as I never intend to refer to it again from another connection. If I do need to, I can provide a name. Is that a common enough scenario, or should we require user specification of a name?

publishMessage may be misleading.

publish and publishRaw sgtm. They definitely can't be mistaken for anything else that way.

[rakyll]

If I do need to, I can provide a name. Is that a common enough scenario, or should we require user specification of a name?

I filed an issue against API that name should be generated if not provided, and team said my assumption about subscriptions being transient is wrong. In the typical scenario, a user will create a set of topics and subscriptions via a script before deploying their app. They will embed the topic and subscription names to their app config and will never run CRUD operations on topics and subscriptions from the application context.

[rakyll]

b/16462864

[stephenplusplus]

No problemo. Updated the first post with these changes.

[rakyll]

Looks great!

[rakyll]

@stephenplusplus, I think we can start working on this issue. Would you like to focus on it?

[stephenplusplus]

Yep! I actually have a good start, just fine tuning and writing tests
currently.

On Monday, August 11, 2014, Burcu Dogan [email protected] wrote:

@stephenplusplus https://github.com/stephenplusplus, I think we can
start working on this issue. Would you like to focus on it?

—
Reply to this email directly or view it on GitHub
#98 (comment)
.

[rakyll]

Great!

[stephenplusplus]

#107 (comment)

[rakyll]

Opening this issue again, we still need to improve the API.

[tmatsuo]

Have 2 questions.

// Subscribe to a Topic (specific [subscription name specified])
var puppyLogger = PuppiesTopic.subscribe({
name: 'puppyLogger',
autoAck: true,
ackDeadlineSeconds: 60
});

In my understanding, the above code will create a new subscription if not exist. If the subscription with that name exists, what will happen? What if the existing subscription has a push endpoint (push subscription)?

Why the connection needs projectId? Potentially Pub/Sub can be used for building a kind of system dashboard across multiple projects. In that case, you need to create as much connections as project numbers with the current lib.

[tmatsuo]

Generally I like the natural taste of the suggested API. Do you have any plan for providing a similar shortcut for push subscriptions?

[rakyll]

I think you meant push notifications. There is currently no plan for an out of the box solution, because it involves adding a handler to user's existing web server code. We can provide a middleware for that, but currently not in our scope.

The rest of the questions above is what we should address once we get back working on the API improvements.

[stephenplusplus]

I've updated the first post with a new proposal and simple example usages. I liked the other approach as well (at the bottom of the first post), but there was too much magic involved that could be confusing.

Please let me know your thoughts!

[stephenplusplus]

// @ryanseys @silvolu

[tmatsuo]

Does pubsub.getTopic() invoke the underlining API call?

[stephenplusplus]

It does. To eliminate two api calls when doing one operation like subscribing to a topic, we would have to keep a subscribe function outside the scope of a topic, similar to what we have now on master:

pubsub.subscribe({
  topic: 'name',
  name: 'sub-name'
}, function(err) {});

My proposals have mainly been for creating a clear hierarchy: Topic > Subscription. However, situations like above make that tricky to pull off efficiently.

[tmatsuo]

So do you think that hierarchy matters over performance?

[stephenplusplus]

So do you think that hierarchy usability matters over performance?

Sometimes. In this case, I'm not sure, but am leaning towards sticking to a similar API to what we have on master, just with a fresh coat of paint. What are your thoughts? (and anyone else here?)

[tmatsuo]

How about to just allow creating Topic/Subscription object without any API calls?