Pub/Sub: IAM Permissions Required to Subscribe Unclear

[MarkHerhold]

If I create an IAM service account with the View and Subscribe roles, I am unable to subscribe to an existing topic in pub/sub. It seems I must use an admin role. Some clarification on the roles required to do various things would be nice. I'm not sure if the issue is being cause by the node client or if I am actually missing roles that are needed.

Environment details

• OS: Mac OSX
• Node.js version: v6.3.0
• npm version: 3.10.3
• @google-cloud/pubsub version: 0.1.1

Steps to reproduce

const pubsub = require('@google-cloud/pubsub');

const JOB_QUEUE = 'new_job';

const pubsubClient = pubsub({
    projectId: 'myproject-goes-here',
    keyFilename: '/Users/Mark/creds.json'
});

const newJobTopic = pubsubClient.topic(JOB_QUEUE);

newJobTopic.subscribe(JOB_QUEUE, {
    reuseExisting: true
}, function(err, subscription) {
    console.error(err); // Error: User not authorized to perform this action.
});

If I create a service account that has an admin pub/sub role, I can connect to the service as expected.

[stephenplusplus]

@tmatsuo any insight?

[MarkHerhold]

Hi @stephenplusplus @tmatsuo

Sorry to bother you guys again, but I would really like to know the answer to this. If I need to have admin privileges to publish to a topic then I need to reconsider my application design or find a new pub/sub provider. Thanks!

[jgeewax]

If I need to have admin privileges to publish to a topic then I need to reconsider my application design

Did you mean subscribe to a topic?

If I need to have admin privileges to subscribe to a topic then I need to reconsider my application design

You shouldn't need admin rights to publish to a topic... If you do, that's just broken. Can you confirm if that's happening? You should be required to have write privileges on the topic in order to publish, but I don't think that's the issue you're seeing.

I suspect that the issue here is that "subscribing" here (with auto-create) is actually "creating a subscription", which the "Pub/Sub Subscriber" role should allow you to do, but doesn't appear to do... (https://cloud.google.com/pubsub/access_control#tbl_roles)

Can you test adding the pubsub.subscriptions.create permission on the project for your service account? If that works, then we know the fix, and I can go to the Pub/Sub team and we can figure out what the right way to solve this is...

If that does work, I'd guess that the right fix is to allow "subscribers" to create subscriptions, as well as update and delete subscriptions they've created.

[MarkHerhold]

@jgeewax My apologies for the typo. I did mean to say subscribe, as that is what is in my code sample. I can try your suggestions and get back to you later today. Thanks!

[stephenplusplus]

@MarkHerhold how did it go?

[MarkHerhold]

Sorry for the delay. I chose to pursue a solution that doesn't involve the use of GCP's Pub/Sub. I will still try to answer to your question though.

Can you test adding the pubsub.subscriptions.create permission on the project for your service account?

Where can I do this? I don't see anything in the console that corresponds to what you are asking me to do. Is there a gcloud command that you have in mind?

Here are the current set of permissions that I have on the service account:

[stephenplusplus]

According to this table, pubsub.subscriptions.create is a permission under the Editor role. So from the drop down, there should be a Pub/Sub Editor role which would include that permission. Hopefully that helps, thanks for trying this stuff out!

[embatbr]

I have the same issue. I have two service accounts: one for publishing and other for subscribing. The 1st is OK and I can publish to a topic. The second always returns me { [Error: User not authorized to perform this action.] code: 403, metadata: Metadata { _internal_repr: {} } }.

The same happens with the editor role.

I really don't know what's happening, since I did it equal to the 1st one.

publisher: https://github.com/embatbr/livermore-collectors
subscriber: https://github.com/embatbr/livermore-ingestors

[embatbr]

@stephenplusplus

[embatbr]

I tried with PubSub Admin role. There was some errors due to node version (4 instead of 6). I updated and it was running. However, the PubSub Subscriber role still has authentication problems.

[bkoski]

This isn't immediately clear from most examples, but the topic.subscribe() call attempts to create a new subscription each time it runs (see discussion here: #1257).

Here's a pattern that worked for me based on this comment after adding "Pub/Sub Viewer" and "Pub/Sub Subscriber" IAM permissions directly on the subscription (no project-wide permissions necessary):

const pubsub = require('@google-cloud/pubsub')({ projectId: PROJECT_ID }) 
const topic = pubsub.topic(TOPIC_NAME)
const subscription = topic.subscription(SUBSCRIPTION_NAME)

subscription.get((err, subscription) => {
   // topic.subscribe() handlers go here

   // opts that you might have set as part of the subscribe() call
   // need to be set directly on the subscription in this scheme
   // subscription.autoAck = true
})

[ScottBrenner]

Confirming that if I create an IAM service account with the subscriber role I am unable to subscribe to a topic in Pub/Sub - need to use the admin role.

[thylong]

This issue doesn't seem to be specific to Node, I've got the problem as well with Go...

[samuelmanzer]

I also found this thread via hitting the same problem using the Go SDK - this misleading role seems to be a common point of confusion.

[sebastienvas]

I am having the same issue. It seems that editor role does not work to create topics or subscription in golang. Also one the error was telling that another service account that's the one I am using did not have permission, which confused the hell out of me.

[thedmi]

You need to distinguish between using an existing subscription (pubsub.subscriber role is enough) and creating a new subscription for an existing topic (pubsub.subscriber is NOT enough).

For the latter case, the problematic permission (not role) is pubsub.subscriptions.create. Unfortunately, only the pubsub.editor role has that permission, and it needs to be granted on project level. This means the user has then permission to modify all topics and subscriptions in the project!

To work around this issue, just create a custom role. Put the pubsub.subscriptions.create permission in that role. Then assign the custom role to the user (project level) and give the user the standard subscriber role on the specific topic.

[vizsatiz]

I am facing the same issue. I am not creating a new subscriber rather listening to an existing subscriber that I created on the GCP console. Only service accounts with Pub/Sub Admin privileges are able to subscribe, all others are getting a "User not authorized to perform this action."

SDK used: NodeJs
Permissions Tried: Pub/Sub subscriber, Pub/Sub editor

It only worked with Pub/Sub Admin. @thedmi Are you sure it worked for you?

[vaijab]

Why is this issue closed?

I am hitting the same exact problem. I am trying to attach to an existing subscription and getting PermissionDenied error using roles/pubsub.editor.

I use GO sdk, but that seems to not matter, as this seems to be a problem in Google's Pub/Sub IAM.

[inge4pres]

For me it turned out I had to assign ALL pubsub roles to the ServiceAccount to have it publish/subscribe messages. Literally just assing all the 6 or 7 roles/pubsub.* to the user and it will work, but that's a pity...

[kylieCat]

I realize that this is closed but leaving this here anyway for anyone else who finds it. Sorry the test code is in Go but it should be easy enough to translate to Node or just test using the CLI.

Bug Report

NOTE: Not really sure if this is a "bug" as it seems to be working the way it was designed but it's very unclear on how to use the standard roles provided to subscribe and consume messages from a topic.

NOTE: All roles were granted to the service account from the subscription not from the project

When creating service accounts to use our Pub/Sub topics I gave the accounts that only need to read from topics the Pub/Sub Subscriber role. The name of the role implies that any account that has that role should be able to read from a topic. This however wasn't the case, I encountered PermissionDenied errors. So I began experimenting with different combinations of standard roles trying to find the least permissive set of roles that would allow this.

The least permissive standard role that works is Pub/Sub Editor. Just looking at the name, this role would not be my first choice for a read only action. This role also contains pubsub.subscriptions.delete which is a dangerous permission to have for a service account meant to be "read" only (it does actually need to update the topic to ack the message which is actually an update to the topic but it's expected for a consumer in this context).

The least permissive set of permissions to achieve the goal of reading form a topic, processing and acking can be accomplished with the following custom role:

$ gcloud iam roles describe --project my-project my_custom_role
description: 'Created on: 2018-08-09'
etag: ...
includedPermissions:
- pubsub.subscriptions.consume
- pubsub.subscriptions.get
- pubsub.subscriptions.update
name: projects/my-project/roles/my_custom_role
stage: ALPHA
title: my_custom_role

Repro Steps

• Create a topic, my-topic
• Create a subscription for my-topic named my-topic-subscription
• Create a service account my-svc-acct
• Grant the Pub/Sub Editor role to my-svc-acct via the subscription, not via the project.
• Attempt to do a receive

Expected

A message is received

Actual

An error is received:

rpc error: code = PermissionDenied desc = User not authorized to perform this action.

Minimum Perms Needed to Subscribe

After some testing, I determined this is the minimum set of permissions needed to subscribe and process messages. Code used for testing provided below.

$ gcloud iam roles describe --project my-project my-custom-role
description: 'Created on: 2018-08-09'
etag: ...
includedPermissions:
- pubsub.subscriptions.consume
- pubsub.subscriptions.get
- pubsub.subscriptions.update
name: projects/my-project/roles/my-custom-role
stage: ALPHA
title: SRE Service Subscriber

$ GOOGLE_APPLICATION_CREDENTIALS=./creds.json go run tester.go my-topic-subscription my-topic

Testing topic perms projects/my-project/topics/my-topic
	NO PERMS!
================================================================================

Testing subscription perms projects/my-project/subscriptions/my-topic-subscription
	Allowed: pubsub.subscriptions.consume
	Allowed: pubsub.subscriptions.get
	Allowed: pubsub.subscriptions.update
================================================================================

Starting Receive...
hello

---

Test Code

Pushing data via CLI:

gcloud pubsub topics publish my-topic --message "hello"

package main

import (
	"context"
	"fmt"
	"os"
	"errors"
	"strings"
	"cloud.google.com/go/pubsub"
)

const (
	project = "my-project"
)

var (
	topicPerms = []string{
		"pubsub.topics.publish",
		"pubsub.topics.delete",
		"pubsub.topics.update",
		"pubsub.topics.getIamPolicy",
		"pubsub.topics.setIamPolicy",
	}

	subPerms = []string{
		"pubsub.subscriptions.consume",
		"pubsub.subscriptions.get",
		"pubsub.subscriptions.delete",
		"pubsub.subscriptions.update",
		"pubsub.subscriptions.getIamPolicy",
		"pubsub.subscriptions.setIamPolicy",
	}
)

func hr() {
	fmt.Println("")
	fmt.Println(strings.Repeat("=", 80))
	fmt.Println("")
}

func printPerms(perms []string) {
	if len(perms) > 0 {
		for _, perm := range perms {
			fmt.Printf("\tAllowed: %v\n", perm)
		}
	} else {
		fmt.Println("\tNO PERMS!")
	}
}

func main() {
	subName := os.Args[1]
	topicName := os.Args[2]

	client, err := pubsub.NewClient(context.Background(), project)
	if err != nil {
		fmt.Println(err.Error())
		return
	}

	topic := client.TopicInProject(topicName, project)
	sub := client.SubscriptionInProject(subName, project)

	hr()
	fmt.Println("Testing topic perms " + topic.String())
	tperms, err := topic.IAM().TestPermissions(context.Background(), topicPerms)
	printPerms(tperms)

	hr()
	fmt.Println("Testing subscription perms " + sub.String())
	perms, _ := sub.IAM().TestPermissions(context.Background(), subPerms)
	printPerms(perms)

	hr()
	fmt.Println("Starting Receive...")
	err = sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
		fmt.Println(string(msg.Data))
		msg.Ack()
		return
	})
	if err != nil {
		err = errors.New(fmt.Sprintf("error receiving from %s: %s", subName, err.Error()))
		fmt.Println(err)
		return
	}
}

[psucodervn]

I was faced the same problem when subscribing to an existed subscription. I received PermissionDenied with Pub/Sub subscriber, but OK with Pub/Sub Editor. Have no idea why :|

[vaishalig12]

Same issue in python. I had to create a sepeate PubSub API key and export its location to environment variables for it to function.

[bgbraga]

@Conky5 sorry, Is it fixed? Same problem here with Node @google-cloud/pubsub 0.22.2

[Conky5]

@bgbraga if you're asking if my pull request fixed the underlying google pubsub iam subscription issue, no, that change was for a different purpose...

[dxs516]

@Conky5 : Is this fixed in the latest version?