Automatic Subscription of Azure Functions to Event Grid

• Download EventGridSubscription.zip - 18.8 KB

Introduction

With the introduction of Event Grid to market the new possibilities appear to distribute events faster, to eliminate polling of events as in Service Bus. Now latency and cost could be cut as Event Grid executes simple HTTP-based event delivery itself when the message meeting subscription filter appears. To this point, everything works perfect, but to hook the function to Event Grid requires a lot of manual work, many clicks to be done in Azure portal. Taking into consideration many functions on few different environments could lead to many errors during deployment. But even after deployment, who will maintain it? How many clicks do you need to do in order to introduce a change in your subscription?

Background

"Let's use Event Grid in our project" - I proposed. This is how this has started.

Looking back at the existing solution that we had in those days for Service Bus subscriptions - each one created manually, maintained manually, no history of change. This was a "no-go" for me. I expected to have a definition of consumer (subscription definition) as close to the consumer as possible and have a clear view of how the subscription has changed over time in our code repository.

The idea behind this is very simple:

1. Create consumer function as Azure FunctionApp.
2. Create consumer subscription definition in the same Azure FunctionApp.
3. Hook the function to Event Grid during CI/CD process with PowerShell script.

There will be many components with FunctionApps. Each component will be enclosed in the Resource Group. Each FunctionApp can have many functions and each of these functions can have many subscription definitions.

-- Resource Group
|-- Function App
   |-- function
     | -- subscription filter 
     | -- ...
   | -- ...
| -- ...

Using the Code

I have created two functions in my FunctionApp 2.x project. The first one is the standard EventGridTrigger and is responsible for consuming incoming events. The later one holds the definition of subscription.

Consumer Function

using Microsoft.Azure.EventGrid.Models;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.EventGrid;
using Microsoft.Extensions.Logging;
using System.Threading.Tasks;

namespace EventGridSubscription2xFunctionApp
{
    public class MyFunction
    {
        [FunctionName("MyFunction")]
        public static async 
               Task Run([EventGridTrigger] EventGridEvent eventGridEvent, ILogger log)
        {
            ...
        }
    }
}

Remarks for FunctionApp 1.x

Please be informed that to make it work for Azure Functions 1.x, you must use the specific NuGet package versions.

And binding is also different.

    [FunctionName("MyFunction")]
    public static async Task Run
    ([EventGridTrigger] JObject eventGridEvent, TraceWriter log)
    {
        ...       
    }

Subscription Definition Function

Subscription function has to follow the naming convention to allow later in an easy way to be discovered and hooked on EventGrid component. In our case, we choose to use "_Subscription" postfix for the function name containing EventGrid subscription definition.

 [FunctionName("MyFunction_Subscription")]
        public static async Task<HttpResponseMessage> Run(
            [HttpTrigger(AuthorizationLevel.Function, "get")] HttpRequest req,
            ILogger log)
        {
            var subscription1 = Create("subscription-name-1", 
            "my-event-type-11", "my-event-type-12");
            var subscription2 = Create("subscription-name-2", 
            "my-event-type-2");

            var subscriptions = new[] { subscription1, subscription2 };

            return new HttpResponseMessage(HttpStatusCode.OK)
            {
                Content = new StringContent(JsonConvert.SerializeObject
                  (subscriptions, Formatting.Indented), 
                   Encoding.UTF8, "application/json")
            };
        }

The function will be triggered by http request and in response returns Json formatted subscriptions definitions. As one function can be triggered by different subscriptions, for each of them, separate definition will be created. In the example above, they are subscription1 and subscription2.

   private static EventGridSubscription Create
           (string subscriptionName, params string[] eventTypes)
   {
       var subscription = new EventGridSubscription
       {
           Name = subscriptionName,
         Prefix = "prefix",   // Subject Begins With
      Postfix = "postfix",    // Subject Ends With
           EventTypes = eventTypes
       };

       return subscription;
   }

The model of response with subscription definition is below. The response is Json formatted.

public class EventGridSubscription
{
    private readonly List<string> _eventTypes;

    public EventGridSubscription()
    {
        _eventTypes = new List<string>();
    }

    [JsonProperty("name")]
    public string Name { get; set; }

    [JsonProperty("eventTypes")]
    public string[] EventTypes { get; set; }

    [JsonProperty("subjectBeginsWith")]
    public string Prefix { get; set; }

    [JsonProperty("subjectEndsWith")]
    public string Postfix { get; set; }
}

PowerShell

The central component of this solution is PowerShell script that gets subscription definition from "_Subscription" function and hooks WebHook on EventGrid creating or updating subscriptions. In our current solution, we have many components with its FunctionApps. Each component is in a separate Resource Group. Therefore, script requires as input parameters component resource group name as well as resource group with EventGrid.

param (
    [string][Parameter(Mandatory=$true)]$EventGridTopicName,
    [string][Parameter(Mandatory=$true)]$EventGridTopicResourceGroupName,
    [string][Parameter(Mandatory=$true)]$FunctionAppsResourceGroupName
)

The logic is straightforward:

1. Find all FunctionApps
2. Find all functions with "_Subscription" postfix
3. Get function keys and subscription definition
4. Register the function on EventGrid

And now step by step.

Get All FunctionApps in Resource Group

Just get all resources that are Kind of "functionapp".

{
    [CmdletBinding()]
    param
    (
        [string] [Parameter(Mandatory=$true)] $ResourceGroupName
    )

    $azureRmResourceParams = @{
        ResourceGroupName = $ResourceGroupName
        ResourceType      = 'Microsoft.Web/sites'
        ApiVersion        = '2018-09-01'
    }
    $resources = Get-AzureRmResource @azureRmResourceParams

    $functionApps = @()
    
    foreach ($resource in $resources) {
        if($resource.Kind -eq "functionapp"){
            $name = $resource.Name
            $functionApps += $name
        }
    }
    return $functionApps
}

Then we iterate through FunctionApps and get all functions that have "_Subscription" prefix. Only those will be processed.

$azureRmResourceParams = @{
    ResourceGroupName = $ResourceGroupName
    ResourceType      = 'Microsoft.Web/sites/functions'
    ResourceName      = $FunctionApp
    ApiVersion        = '2015-08-01'
}
$functions = Get-AzureRmResource @azureRmResourceParams

foreach ($function in $functions) {
    $functionName = $function.ResourceName
    if ($functionName.endswith("_Subscription") )
    {
        ...
    }
}

And this is the core logic, to get master key for function, get subscription and finally group them as metadata.

           if($masterKey -eq $null){
                $functionAppMasterKeyParams = @{
                    ResourceGroup = $ResourceGroupName
                    FunctionApp = $FunctionApp
                }       
                $masterKey = Get-FunctionAppMasterKey @functionAppMasterKeyParams
            }
        
            ## get subscription parameters
            $subscriptionFunctionName =  $functionName -replace "$FunctionApp/", ""
            $subscriptionApiUrl = 
            'http://'+$FunctionApp+'.azurewebsites.net/api/'+$subscriptionFunctionName +
            '?code='+$masterKey
            $subscriptionResult = Invoke-WebRequest $subscriptionApiUrl | 
                                  ConvertFrom-Json
            
            ## get function endpoint
            $functionAppName = $subscriptionFunctionName -replace "_Subscription", ""
            $eventGridApiUrl = 'http://'+$FunctionApp+'.azurewebsites.net/
                               admin/host/systemkeys/
                               eventgrid_extension?code='+$masterKey
            
            $sysKeyResult = Invoke-WebRequest $eventGridApiUrl | 
                            select -Expand Content | ConvertFrom-Json | select value
            $sysKey = $sysKeyResult.value
             
            $eventGridSubscriptionEndpoint = 'https://'+$FunctionApp+
                                             '.azurewebsites.net/runtime/webhooks/
            eventgrid?functionName='+$functionAppName+'&code='+$sysKey
            foreach($result in $subscriptionResult)
            {
                $subscription = @{
                    name = $result.name
                    endpoint = $eventGridSubscriptionEndpoint
                    eventTypes = $result.eventTypes
                    subjectEndsWith = $result.subjectEndsWith
                    subjectBeginsWith = $result.subjectBeginsWith
                }
                $subscription
            }

Remarks for FunctionApp 1.x vs 2.x

There are differences in API for FunctionApp 1.x and 2.x. The ones concerning the described case are below.

The SystemKey API:

1.x 'https://'+$FunctionApp+'.azurewebsites.net/admin/host/systemkeys/
eventgridextensionconfig_extension?code='+$MasterKey
2.x 'http://'+$FunctionApp+'.azurewebsites.net/admin/host/systemkeys/
eventgrid_extension?code='+$masterKey

The Azure function WebHook for EventGrid URL:

1.x 'https://'+$FunctionApp+'.azurewebsites.net/admin/extensions/
EventGridExtensionConfig?functionName='+$functionAppName+'&code='+$sysKey
2.x 'https://'+$FunctionApp+'.azurewebsites.net/runtime/webhooks/
eventgrid?functionName='+$functionAppName+'&code='+$sysKey

Remarks for FunctionApp 2.x

Secrets for FunctionApp in version 2.x must be stored in storage file not in blob. As blob is the default value, it should be explicitly set. Access to FunctionApp secrets is needed by PowerShell script to get FunctionApp keys. It could be setup manually through Azure portal or automated in ARM template. I strongly discourage you from doing anything manually.

Glue It Together in Azure DevOps

The standard release pipeline of Azure DevOps can be used to deploy FunctionApps as well for PowerShell script. Just deploy FunctionApp and then run Azure PowerShell task, which will do the magic - hook EventGrid to functions.

Points of Interest

ARM Template Approach

There is also other way to do this. You can use this ARM template. But... the flow with ARM template will be:

1. Deploy FunctionApp
2. Gets SystemKey for FunctionApp using PowerShell
3. Deploy ARM using SystemKey

This is an egg-chicken problem. You need SystemKey before deploying FunctionApp. You can not extract it during FunctionApp deploy and pass to EventGrid template in the same ARM. The second drawback (in my opinion) in this approach is keeping subscription definition in ARM templates. I wanted to have filter definition close to consumer code.

History

• 19^th November, 2018: Initial version
• 20^th November, 2018: EventGridSubscription.zip added