Support EventGrid triggers in callback functions

[mikhailshilkov]

I am spinning this out of #218 because EventGrid is a bit different from other triggers and warrants some extra considerations on the design of functions.

Azure Event Grid

Azure Event Grid is the latest messaging service in Azure and is positioned as a massively-scalable event routing service to power event-driven and serverless apps. Event Grid subscriptions are basically webhooks, where an HTTP request is sent to the specified endpoint for each event in a given topic.

One of the unique features of Event Grid is the existence of many built-in topics with events from native Azure services. For instance, each Storage Account has an endpoint to subscribe for events in that account: e.g., a call would be made every time a new blob is uploaded. Similar endpoints exist for events from subscription, resource groups, container registries, media services, maps, and so on, increasing over time.

Event Grid trigger

Azure Functions have a dedicated Event Grid trigger. Effectively, it's an HTTP function with some extra wiring to handle authentication hand-shake initiated by Event Grid.

The trigger definition is very simple:

{
    "type": "eventGridTrigger",
    "name": "eventGridEvent",
    "direction": "in"
}

The function binding does not define the events to subscribe to. It's the Event Grid subscription which makes it all work.

Event Grid subscription

Event subscription is mainly a combination of three things:

• Event Grid scope (the broad "topic" to subscribe to, the event source)
• Webhook URL (A function webhook URL)
• Optional filter to narrow the type of events to receive

Multiple subscriptions can be created for the same scope and/or for the same webhook (Azure Function).

Use cases

1. Custom topic callback

One can define a custom Event Grid topic and subscribe on it with a callback. This looks very similar to our usual scenario of callback functions, so the code could be:

const topic = new azure.eventhub.EventGridTopic("bla", {
    resourceGroupName: resourceGroup.name,
});

eg.onEvent("onBla", async (context, event) => {
    console.log(JSON.stringify(context));
    console.log(JSON.stringify(event));
});

2.1. Built-in topic subscriptions

A major, and arguably even more useful, use case is subscribing to built-in events of Azure. For instance, to get events from a storage account:

const storageAccount = new azure.storage.Account("play", {
    resourceGroupName: resourceGroup.name,
    accountReplicationType: "LRS",
    accountTier: "Standard",
});

We can provide two types of API here. One is defining a subscription resource:

new azure.eventhub.EventGridCallbackSubscription("MyStorageHandler", {
    resource: storageAccount,
    callback: async (context, event) => {
        // prints "Microsoft.Storage.BlobCreated" when a blog is uploaded
        console.log(event.eventType); 
    }
});

the resource property could be of type type AzureResource = pulumi.Resource & { resourceGroupName: pulumi.Output<string>, id: pulumi.Output<string> }; or similar.

2.2. Built-in topic with callbacks

We can also support a callback form

storageAccount.onGridEvent("MyStorageHandler", 
    async (context, event) => {
        // prints "Microsoft.Storage.BlobCreated" when a blog is uploaded
        console.log(event.eventType); 
    });

This implies some extra work because we need to identify all extension points (modules) where Event Grid is applicable and add a callback function in each of them. Note that things like Blob Storage and Event Hubs support both native events and Event Grid events, so I suggest naming such callback function onGridEvent or similar to minimize confusion.

3. Multiple functions in an app

MultiCallbackFunctionApp should support Event Grid too. The function itself is super simple:

const func = new azure.eventhub.EventGridFunction("MyHandler", {
    callback: async (context, event) => {
        console.log(event.eventType); 
    }
});

const app = new azure.appservice.MultiCallbackFunctionApp("ManyFunctions", {
    resourceGroup,
    functions: [func, /* other functions ...*/],
});

But there's no subscription at this point yet. There must be yet another resource to create one:

new EventGridFunctionSubscription("MySub", {
    resource: storageAccount,
    functionApp: app.functionApp,
    handler: func,
});

This guy is a bit tricky because it needs to make a POST call to management.azure.com to retrieve Azure Function's system key (Terraform doesn't have it) and then define a webhook URL. That's why it needs both a Function App and a Function (name).

This way, we can also define multiple Event Grid subscriptions for a single function.

Obviously, EventGridFunctionSubscription can be reused internally in EventGridCallbackSubscription.

Next steps

I've got a working prototype of this. Before I spend time on ironing it out and opening a PR, I'd love to get feedback on the design. It contains quite a few elements and might be confusing to newcomers if we don't present it properly. At the same time, Event-Grid-as-a-callback opens up some really cool scenarios and is very demo-able.

[mikhailshilkov]

tagging @CyrusNajmabadi

[CyrusNajmabadi]

@mikhailshilkov Overall i love this. The part i didn't quite get was this:

This guy is a bit tricky because it needs to make a POST call to management.azure.com to retrieve Azure Function's system key (Terraform doesn't have it) and then define a webhook URL. That's why it needs both a Function App and a Function (name).

Can you clarify it more?

In terms of priority, i think i woudl 2.1, 2.2, 3, then 1. But it all sounds super useful and powerful to have these abstractions from different directions.

Anything in particular you have specific design questions around?

[mikhailshilkov]

@CyrusNajmabadi Great!

Can you clarify it more?

This is my current code to create a subscription:

new EventGridEventSubscription(name, {
    webhookEndpoint: {
        url: pulumi.all([this.functionApp.id, this.functionApp.defaultHostname]).apply(async ([id, hostname]) => {
            const creds = await AzureCliCredentials.create();
            const client = new AzureServiceClient(creds);
            const url = `https://management.azure.com${id}/host/default/listkeys?api-version=2018-02-01`;
            const response = await client.sendRequest({ method: 'POST', url });
            const systemKey = response.parsedBody.systemKeys.eventgrid_extension;
            return `https://${hostname}/runtime/webhooks/eventgrid?functionName=${name}&code=${systemKey}`;
        }),
    },
    scope: resource.id,
}, opts);

There's no way to get systemKey property from Pulumi resources/terraform (that I know of).

Anything in particular you have specific design questions around?

1. Whether we want 2.2 and/or 2.1. I guess you already said both.
2. Naming, because we have many things with Event Grid and Subscription in it: the currently existing resource, the callback, the thing which creates the subscription from a function app and a function, ...
3. Whether we have to create an access token for the snippet above with AzureCliCredentials and other authentication classes from @azure/ms-rest-nodeauth library or can we somehow get an access token from Pulumi?
4. Whether you see a better way for API in (3) and EventGridFunctionSubscription.
5. I'll have to figure out types for events but that's details.

[ChristianEder]

Hi @mikhailshilkov , to keep it consistent with other trigger types, AND to avoid confusion with the EventGridFunctionSubscription in (3) I guess that option (2.2.) is preferrable to (2.1.). Also, for any of the proposed solutions, you should keep in mind that there might be resource types that offer more than just a single type of event grid event, i.e. making the syntax of both options (2.1.) and (3) not specific enough, since they only specify the resource.

[ChristianEder]

Also, regarding the webhook URL: If the event grid subscription is essentially just calling a webhook, why is there even a need for a specific event grid trigger as opposed to an HTTP trigger (this is more an Azure question than related to your proposal, I guess).

[mikhailshilkov]

why is there even a need for a specific event grid trigger as opposed to an HTTP trigger

It's a convenience: it parses the workload, converts exceptions into proper HTTP response codes, plus it's doing the handshake automatically. One can use HTTP triggers too, it even has some benefits, e.g. ability to switch to CloudEvents standard format.

[mikhailshilkov]

The basics were implemented with #302, so I'm going to close this issue. There are several other event sources, we'll add them on demand.