AI Form Recognition in Java, Part 1: App Creation and Deployment via Azure App Service

• Download project code - 724 KB

Synopsis

This three-article series demonstrates how to use Azure Form Recognizer to build a realistic end-to-end form recognition app using Java and Spring Boot.

• Part 1 — App Creation and Deployment via Azure App Service
• Part 2 — Adding Image Upload to the Spring Boot App and Processing via Form Recognizer
• Part 3 — Making Practical Use of Data Returned by Form Recognizer

The complete code repository is available on GitHub.

Introduction

A typical business trip ends with reimbursement. Generally, you need to collect all of your receipts, describe them, and then pass them to the accounting department. This is a process seemingly destined for automation.

Thanks to Azure Form Recognition, we have all the tools needed to do just that. We can capture an image of our receipt, upload it to the web application, and then invoke a machine learning model to recognize all of the fields in the receipt. The web application can then extract all the necessary fields for reimbursement and automatically send it to the accounting department.

In this series, we’ll demonstrate how to implement a pipeline like this using a Java Spring Boot MVC application deployed to Azure App Service. The selected fields will be stored in a database provisioned using Azure Database for PostgreSQL.

Our users will be able to upload images of their receipts. Images will then be sent to the Azure Form Recognition service to extract the necessary fields. We'll store the fields in the database and present them in tabular and graphic forms.

To follow along, refer to the companion code.

In the first article, we'll prepare the application infrastructure, including our database, data access layer, models, and hosting. Finally, we will deploy the entire solution to the Azure App Service. Our finished product will look similar to this:

Prepare the Project

Let’s start by preparing the project. I decided to use Visual Studio Code with the Extension Pack for Java, along with Spring Initializr Java Support. For more information, refer to the Visual Studio Code documentation.

After preparing the environment, open the Command Palette by selecting View > Command Palette..., and start typing “Create Java”. Select Java: Create Java Project....

This opens a wizard.

In the wizard, follow these steps:

1. Select Spring Boot and then Maven project.
2. In Specify Spring Boot version, choose at least 2.6.4.
3. Select Java as the project language.
4. Set the Input Group Id to com.ai.
5. Set the Artifact Id to formrecognizer.
6. Use Jar as the packaging type, and set the Java version to 11.
7. Under Choose dependencies, select the following options:
   • Spring Web
   • Spring Boot DevTools
   • Spring Data JAP
   • PostgreSQL Driver
   • Thymeleaf
8. Specify your destination folder.

Visual Studio Code then generates the project template for you.

We now have our project template. Let’s proceed to the next step and provision the Azure Database for PostgreSQL.

Azure Database for PostgreSQL Server

We’ll start by provisioning the flexible server we need. To create the server, you can either use the Azure CLI or Azure Portal. We'll use Azure Portal.

Launch Azure Portal, then use the search bar to search for the Azure Database for PostgreSQL servers.

Next, select either Create Azure Database for PostgreSQL single server or the + Create link in the top panel.

The Azure Portal displays a few options. Select Flexible server. Then, select your subscription and resource group (here, we use the recognition resource group).

Provide a name for the server and select your Azure Region. We set our server name to form-recognizer and our Azure Region to East US. For the Workload type, select Development to reduce your cost. Finally, set your admin username and password.

We now need to ensure that the PostgreSQL server is accessible from our development machine and hosting environment (Azure App Service). So, let's configure our networking.

Select Next: Networking > to open the Networking tab.

On the Networking tab, follow these steps:

1. Select public access to set allowed IP ranges.
2. Select the Allow public access from any Azure service within Azure to this server checkbox to ensure the app deployed to App Service can connect to the server.
3. Select Add current client IP address to enable connections from your development machine.

Finally, select Review + create. Azure Portal now displays the summary of your configuration. To provision the database, select the Create button. It takes a few moments to provision the server instance.

Creating a Database

Now that we’ve created the server, we need to provision the database.

Open the Cloud Shell (the first icon in the top-right corner of the Azure portal) or use your local Azure CLI installation. Then, enter the following command:

az postgres flexible-server db create -g <RESOURCE_GROUP_NAME> -s <SERVER_NAME> -d recognitions

We have set the database name to recognitions. We’ll use this later on.

After the database is created, open the application.properties file and add the following code:

spring.datasource.url=jdbc:postgresql://<YOUR_SERVER_NAME>.postgres.database.azure.com:5432/recognitions
spring.datasource.username=<YOUR_ADMIN_USERNAME>
spring.datasource.password=<PASSWORD>
 
spring.jpa.hibernate.ddl-auto=update
server.port=80

Now, let’s build and run the project to ensure that our application can connect to the database.

Enter the following commands in the terminal:

mvn clean install
mvn spring-boot:run

If you receive connection errors, ensure that the Azure Firewall is not blocking the connection. To do this, change your Azure Firewall rules by selecting Azure Database for PostgreSQL > Networking > Firewall rules. You may also need to restart your PostgreSQL server.

Model and Repository

We've connected our application to the database, so let’s now add a model and implement the database access layer. If you need to, consult the list of all available fields that the Azure Form Recognizer extracts.

Typically, we can extract very detailed information from the receipt, including an itemized list of its contents. For simplicity, we’ll only use three fields: MerchantName, TransactionDate, and Total.

We’ll store these fields in our database along with the file name and identifier. The file contains the image of the receipt.

You can refer to the companion code for the full contents of the src/main/java/com/ai/formrecognizer/RecognitionResult.java file. The model we use looks like this:

package com.ai.formrecognizer;
import org.springframework.data.annotation.Id;
import java.util.Date;
import javax.persistence.*;
 
@Entity
@Table(name = "recognitionResults")
public class RecognitionResult {
    @javax.persistence.Id
    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;
 
    @Column(name = "receiptFileName")
    private String receiptFileName;
 
    @Column(name = "merchantName")
    private String merchantName;
 
    @Column(name = "transactionDate")
    private Date transactionDate;
 
    @Column(name = "total")
    private float total;
 
    // Getters and setters
}

For this model, we supplement the project with the following auto-implemented JPA repository:

package com.ai.formrecognizer;
import org.springframework.data.jpa.repository.JpaRepository;
 
public interface RecognitionResultsRepository extends JpaRepository <recognitionresult, long=""> {
    
}</recognitionresult,>

Controller

After implementing the data access layer, we need to create the controller.

You can refer to the companion code for the full contents of the src/main/java/com/ai/formrecognizer/FormRecognitionController.java file. Our controller looks like this:

@Controller
public class FormRecognitionController {
 
    @Autowired
    private RecognitionResultsRepository resultsRepository;
 
    @GetMapping("/")
    public String index() {
        return "index";
    }
 
    @GetMapping("/upload")
    public String upload() {
        return "upload";
    }
 
    @GetMapping("/results")
    public ModelAndView results() {
        List<RecognitionResult> recognitionResults = resultsRepository.findAll();
        
        ModelAndView modelAndView = new ModelAndView("results");
        modelAndView.addObject("recognitionResults", recognitionResults);        
 
        return modelAndView;
    }
}

The controller has one field of type RecognitionResultsRepository, which is automatically injected by Spring’s @Autowired annotation. Additionally, the FormRecognitionController has three methods: index, upload, and results. The index and upload methods simply return view names. The results method reads all the recognition results from the repository and then passes them to the results view.

Views

Next, let’s implement all three views referenced by the controller in resources/templates. All our views use the Thymeleaf rendering engine and utilize W3.CSS from W3Schools.

The index.html file looks like this:

<html lang="en">
<head>
    <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">
</head>
 
<body class="w3-black">
    <header class="w3-container w3-padding-32 w3-center w3-black">
        <h1 class="w3-jumbo">Form recognizer</h1>
        <p>Azure AI</p>
 
        <h3> 
            <a th:href="@{/upload}">Upload image</a> | 
            <a th:href="@{/results}">Recognition results</a>
        </h3>
    </header>
</body>
</html>

At this point, upload.html and results.html are very similar. They both simply render the static strings:

<html lang="en">
<head>
    <link rel="stylesheet" href="https://www.w3schools.com/w3css/4/w3.css">    
</head>
 
<body class="w3-black">    
    <header class="w3-container w3-padding-32 w3-center w3-black">
        <h1 class="w3-jumbo">Form recognizer</h1>
        <p>Upload a new file</p>
    </header>    
</body>
</html>

We’ll extend the upload.html and results.html views in the second and third parts of this tutorial series.

For now, when you run the application, the output looks like this:

Deployment to Azure App Service

We’ve created our application, so let’s deploy it to Azure App Service. We can do it almost automatically using Visual Studio Code. We just need to install the Azure App Service extension.

To find the Azure App Service extension, navigate to View > Extensions.

When you install the extension, it appears in the left pane of Visual Studio Code. Select the extension, then sign in using your Azure account.

Next, select Deploy to Web App..., which appears as a cloud icon to the right of APP SERVICE on the AZURE panel.

Alternatively, you can use the Command Palette. To do so, search for and select the Azure App Service: Deploy to Web App... command.

Both methods open a wizard to configure the deployment. In the wizard, follow these steps:

1. Click Java SE (Embedded Web Server) Create new Web App… (Advanced).
2. Set the name of your app. We set our app name to form-recognizer-82. It's important to note that this name must be globally unique. The App Service uses this name to create a unique URL for your application.
3. Choose or create a new resource group. We set our resource group to recognition.
4. Select Java 11 as your runtime stack.
5. Select Java SE (Embedded Web Server) as your Java web server.
6. Select Linux as your OS.
7. Choose an Azure Region for the web app. We set our region to East US.

Then, in the wizard, select Select a Linux App Service plan and follow these steps:

1. Click Create new App Service plan.
2. Set the name to form-recognizer-plan.
3. Select Pricing tier: Free (F1).
4. Click into the Select an Application Insights resource for your app field and select Skip for now:

Visual Studio Code now calls the Azure Resource Manager to provision resources for us. It first creates the App Service plan, which represents the workload and hardware on which our app runs.

After a short delay, Visual Studio Code asks for the application port. Enter “80” to match your application properties.

Next, the App Service Visual Studio Code extension packages our application and creates the Web App instance.

After a short delay, you should receive the URL for your app in this format:

https://<WEB_APP_NAME>.azurewebsites.net

Select the link to see your application running in the Azure cloud.

Summary

In the first part of this tutorial, we prepared our application and all the infrastructure it needs to run. We then deployed the app to Azure App Service. Currently, our views are simple and only render static strings. In the next part of this tutorial, we’ll implement file upload functionality and connect it with Azure Form Recognition.

To learn more tips for the easiest ways to deliver Java code to Azure and other clouds, check out the webinar Azure webinar series - Delivering Java to the Cloud with Azure and GitHub.