Use Fusion Development to Level Up AI Apps Part 2: Integrating with Form Recognizer

• Download project source - 62.6 KB

In the previous article, we created a simple Power App to upload (or delete) images in Dataverse, just like a citizen developer. Our users can now upload scanned or photographed invoices or other documents and store them in Dataverse. We’ll add more advanced functions as a developer in this article.

We’ll use Event Hubs to send uploaded images from Dataverse to an Azure Function. Azure Functions will send the image to Azure Form Recognizer, which will use OCR to extract data, such as invoice details.

This approach makes AI accessible to everyone, helping the accounting department automatically extract relevant details from documents, which saves time.

Creating an Event Hub

We’ll start by creating an Event Hub. Go to all services in Azure and find Event Hubs. Creating a new Event Hub also creates a namespace.

In Create Namespace, first, pick your subscription and resource group. You can choose an existing resource group or create a new one at the same time as your namespace by entering a new name. For this tutorial, you should create a new Resource group called "powerplatform" that you can quickly delete later.

Also specify the Namespace name, which must be unique across all Azure, for example, "img-file-powerplatform." Choose a Location close to you, such as West Europe. You can pick the Basic Pricing tier, which has everything we need at the lowest cost.

Leave the other default values, click Review + create, then Create.

Deploying your Event Hub namespace may take a few minutes. When it’s ready, find your Namespace.

Creating an Event Hub is easy. Simply go to Event Hubs in the left-hand menu, click + Event Hub, enter a name (such as "forms"), and you’re done. The other values can remain default.

We’ll need an Event Hub connection string in the following steps. So, in your Namespace, go to Shared access policies and click RootManageSharedAccessKey. You can then copy the connection string for your primary key.

Connecting Dataverse to Event Hub

Next, we’ll connect our Event Hub to Dataverse using the Plug-in Registration tool (PRT).

Microsoft released the PRT on NuGet, so get the nupkg file. The easiest way is to download it directly from NuGet. Open the nupkg file using your favorite zip tool, like WinZip, 7-Zip, or WinRAR, and unzip the tools folder. In this folder, find PluginRegistration.exe.

Once you have the PRT, open it and click + CREATE NEW CONNECTION to log in. Choose "Office 365" as your deployment type. You probably want to show a list of available organizations and select "Show advanced" so you can enter your credentials (rather than use your computer’s credentials). Pick your region and enter your username and password.

After logging in, click Register, then Register New Service Endpoint. Pick Let’s Start with the connection string from the Azure Service Bus Portal… and enter your Event Hub connection string. In the following form, change the Designation Type from "Queue" to "EventHub" and enter "forms" as the Event Hub name. Then, save the endpoint registration.

With the service endpoint selected, click Register, then Register New Step. In the following popup, we'll tell Dataverse when to send a message to our EventHub.

The form’s fields will auto-complete when you type at least one letter yourself, so it’s helpful to know what you need to enter. For Message, set "Create." Your Primary Entity is your Dataverse table’s generated name in the format [prefix]_[table name in lower letters], for example, "cra6e_form." You can optionally change the Step Name and Description. For Execution Mode, pick "Asynchronous." Then click Register New Step.

You can test the app by adding a new form in the Power Apps app and checking your Event Hub. You should see some requests coming in.

Creating a Form Recognizer

So far, we’ve uploaded some random images, but the goal is to use OCR to extract data from forms, like receipts and invoices. We’ll need Azure Form Recognizer to do this.

Azure Form Recognizer uses machine learning to extract key-value pairs, text, and tables from documents. Its APIs recognize critical data from invoices, receipts, IDs (mainly US driver’s licenses and international passports), business cards, and custom documents. We can also train our models, but that’s out of this article’s scope.

To create a Form Recognizer, go to Azure and search for "form recognizer." Click + Create. Pick your resource group and a unique name (which will also be your custom domain name in the API endpoint), such as "powerapp-form-recognizer."

Choose the free tier, which is good enough for this example. Leave all other defaults, click Review + Create, then click Create. When the resource is ready, go to Keys and Endpoint to get both for the next section.

Accessing Dataverse

The first thing we’ll need to do in our Function is access Dataverse. The Event Hub message will not include the uploaded picture, so we need to fetch the image ourselves.

We'll register an Azure Function Application in Azure Active Directory, give the Azure Function Application access to Power Apps, then use it in our code. So, go into the Azure Portal of the account containing your Power Apps. Go to Azure Active Directory (Azure AD) then to App registrations.

Click + New registration. Then, name your application something like "Dataverse sample app" and register the application. Next, go to Certificates & secrets and create a secret. The secret won't be visible again after creation, so write it down.

Next, we need to give this application access to Dataverse. So, go to the Power Platform admin center to view a list of your environments. Click on the three dots next to your environment and click Settings.

In Settings, go to Users + permissions, then Application users. Then, click + New app user, find the app you just registered and select your business unit. We'll give it the "System Administrator" role, giving the application all the access it needs (and more), which is nice when testing.

Creating an Azure Function

Now that everything is in place, we'll create an Azure Function to glue it all together.

Open Visual Studio 2019 or 2022 and create a new project. Pick the Azure Functions template with an Event Hub trigger.

Enter a connection string name like "EventHubConnString." Enter your Event Hub’s name, "forms." By default, the storage account is an emulator, which is fine for this example (we don’t have to deploy the Function App to Azure).

After scaffolding the Function, open local.settings.json and add EventHubConnString: [your connection string] to the Values node. Verify everything works by starting the Function App, adding a file to your Power App, and confirming that your Function’s console has logged it.

We’ll need some information before we can access Dataverse in our code. We know the table name and can find the table prefix in the Dataverse table ID.

We can also find the environment ID in the Power Platform admin center. Click on the environment and view the environment URL. You’ll need the "org[…]" part before ".crm."

You’ll also need your Azure tenant ID, which you can find in your Azure AD. Plus, you’ll need the secret you generated earlier and the client ID, which you can find on your registered app’s overview page.

Accessing Dataverse in Code

We should already have some code that Visual Studio generated in the previous step, and we’ll add to it. First, we’ll get the uploaded form’s entity ID from the Event Hub message.

Now, fill in the values from the previous section:

string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
string primaryEntityId = JsonConvert.DeserializeObject<dynamic>(messageBody).PrimaryEntityId;

// Dataverse table
var environmentId = "<...>";
var prefix = "<...>";
var tableName = "forms";
var imageColumnName = "image";
var resource = $"https://{environmentId}.api.crm4.dynamics.com";
var apiVersion = "9.2";

// Azure app
var tenantId = "<...>";
var clientId = "<...>";
var clientSecret = "<...>";

var authContext = new AuthenticationContext("https://login.microsoftonline.com/" + tenantId);
var credential = new ClientCredential(clientId, clientSecret);
var authResult = await authContext.AcquireTokenAsync(resource, credential);
var accessToken = authResult.AccessToken;

// Get the uploaded image from Dataverse.
var client = new HttpClient
{
    BaseAddress = new Uri($"{resource}/api/data/v{apiVersion}/")
};
var headers = client.DefaultRequestHeaders;
headers.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);
headers.Add("OData-MaxVersion", "4.0");
headers.Add("OData-Version", "4.0");
headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

var response = await client.GetAsync($"{prefix}_{tableName}({primaryEntityId})?$select={prefix}_{imageColumnName},{prefix}_name");
var content = await response.Content.ReadAsStringAsync();
string image = JsonConvert.DeserializeObject<dynamic>(content)[$"{prefix}_{imageColumnName}"];
var bytes = Convert.FromBase64String(image);

First, this code uses the AuthenticationContext to fetch a token from Azure, using the app we registered. The token should have access to our Dataverse environment.

After that, it creates an HttpClient, adds some default headers, and adds the authorization header using the token and Bearer scheme.

The code can then Get our image using OData syntax. The returned image will be base64 encoded, so we’re decoding it to a byte array we can use with the Form Recognizer.

Getting Data from Form Recognizer

Last, we’ll need to send the image to Form Recognizer and get the results. We can use the Azure.AI.FormRecognizer NuGet package. Go to the Form Recognizer in Azure and find the key on the overview page since you’ll need it next.

Then, put this code in your application’s code:

// Form recognizer
var formRecognizerName = "powerapp-form-recognizer";
var formRecognizerKey = "ced734f796c7406cbe2cad5ec9c044e6";

// Send the image to Form Recognizer.
var endpoint = $"https://{formRecognizerName}.cognitiveservices.azure.com/";
var recognizer = new FormRecognizerClient(new Uri(endpoint), new AzureKeyCredential(formRecognizerKey));
using var stream = new MemoryStream(bytes);
var operation = await recognizer.StartRecognizeContentAsync(stream);
var result = await operation.WaitForCompletionAsync();

We can simply pass the bytes we got from Dataverse into the StartRecognizeContentAsync function, and we’ll get a result. The result will contain all the words Form Recognizer has recognized. We’ll use these words in the following article.

You can find the complete code on GitHub.

Next Steps

We’ve created our Azure environment and an Azure Function in this article. We also arranged access to and from Dataverse.

We used Form Recognizer to intelligently extract data from the documents, making robust AI solutions accessible to everyone. This AI-based approach saves the accounting department time they would otherwise spend manually copying information from invoices and other documents. Now the department can focus on higher-value work.

Continue to the final article, where we’ll send the results from the Form Recognizer back to Dataverse so we can use them in our Power App.

To learn more about build a complex, fully functional solution that combines Power Apps with Azure services, check out the e-book Fusion development approach to building apps using Power Apps.