In this tutorial, I discuss how to load data in batches and display using UI-Grid in an AngularJS based application.
Introduction
Since the start of the new year, I have been thinking about a technical issue, and its solution. Imagine this, I have a list of data. I need to load these data and display them on pages. When the number of data items is small, like 100, 1,000, or 10,000, loading them all at once wouldn't take a lot of time. When there are a lot of data items, like 800,000 or so, or each data item has a lot of content, loading them all at once would be a time-consuming process. And users hate sitting in their comfortable chairs and waiting for the data loading to complete.
This problem has been puzzling me for a while. I like the idea of loading all the data at once, then breaking them into smaller sets as pages on the front end. This approach would make the pagination process very smooth. But the problem of letting users wait for a long time bothers me a lot. This wait time could stretch longer as the data grows. One solution is that I can read all the data in fixed-size batches. As soon as one batch of data is ready, add them to the end of the list of existing data. Then let the list component break the long list into pages. This is an idea. In order to realize this idea, I need a mechanism to do the batch read, and some type of front-end component that can automatically break the list into pages of data and display the data correctly.
It turned out the batch read of data can be implemented without much trouble. There is also a component called ui-grid for AngularJS that can easily break a large list of data and display them correctly on pages. I decided to try out my implementation using ui-grid and creating a general-purpose batch-read loop mechanism. This tutorial will discuss how this is done.
Overall Architecture
My sample application is done as a Spring Boot based web application. There is no fancy authentication mechanism. There is only a RESTFul API controller that takes care of the batch data read. The front-end application is an AngularJS-based single-page application. All it does is load the data by calling the RESTFul API controller, and continue doing so until all the data are loaded. The one page will host a ui-grid component that displays all the data as soon as the data are available.
The mechanism I have implemented would be general-purpose. That is, I can copy the interfaces and classes to a different project, and add some project-specific implementation code, and the mechanism would work as expected. To utilize the mechanism, I had to implement a recursion on the front end so that it can continuously load the data until there is no more data to load. The mechanism itself is done with a strategy pattern. This is the only way of ensuring I can move this mechanism into other projects and be re-purposed for use.
Let's begin the tutorial with the design of request and response objects. Then, I will discuss the batch-loading mechanism in detail. After that, I will show you the AngularJS code that utilizes the batch read mechanism, and how to run the sample application.
The Design of Request and Response Objects
Let me start with the request and response object types for the back-end API code. For the old approach, where I load everything at once, no specific request object is needed, and the response would be the list of data items. Since we know this approach is not optimal, and we have to break the long list into smaller parts to be fetched one at a time, we need a request that specifies which part of the long list to fetch, and we also need the response that represents the part of the long list which we requested. And yeah, what I described is paged data fetching. The request contains the page of data items I wanted, the response will contain the page of data items I requested. Here is the request object type:
package org.hanbo.boot.rest.models;
public class BatchReadRequest
{
private int startIdx;
private int batchItemsCount;
public int getStartIdx()
{
return startIdx;
}
public void setStartIdx(int startIdx)
{
this.startIdx = startIdx;
}
public int getBatchItemsCount()
{
return batchItemsCount;
}
public void setBatchItemsCount(int batchItemsCount)
{
this.batchItemsCount = batchItemsCount;
}
}
This class is very simple. The object will have two properties. The first one specifies the starting index of the data item on the long list. The second one specifies the number of data items to be fetched in this request. Yep, as I said, this is paged data fetching, there is no secrecy about it.
Next, I want to show you the response object type:
package org.hanbo.boot.rest.models;
import java.util.List;
import java.util.ArrayList;
public class BatchRetrievalResponse<T extends Object>
{
private List<T> listOfItems;
private int totalCount;
private int pageItemsCount;
private int recordStartIdx;
private int recordEndIdx;
private boolean moreRecordsAvailable;
public BatchRetrievalResponse()
{
setListOfItems(new ArrayList<T>());
}
public List<T> getListOfItems()
{
return listOfItems;
}
public void setListOfItems(List<T> listOfItems)
{
this.listOfItems = listOfItems;
}
public long getTotalCount()
{
return totalCount;
}
public void setTotalCount(int totalCount)
{
this.totalCount = totalCount;
}
public long getRecordStartIdx()
{
return recordStartIdx;
}
public void setRecordStartIdx(int recordStartIdx)
{
this.recordStartIdx = recordStartIdx;
}
public long getRecordEndIdx()
{
return recordEndIdx;
}
public void setRecordEndIdx(int recordEndIdx)
{
this.recordEndIdx = recordEndIdx;
}
public int getPageItemsCount()
{
return pageItemsCount;
}
public void setPageItemsCount(int pageItemsCount)
{
this.pageItemsCount = pageItemsCount;
}
public boolean isMoreRecordsAvailable()
{
return moreRecordsAvailable;
}
public void setMoreRecordsAvailable(boolean moreRecordsAvailable)
{
this.moreRecordsAvailable = moreRecordsAvailable;
}
}
This class is slightly more complicated than the request object type. It contains the following properties:
listOfItems: The returned list of data itemstotalCount: The total number of data items in the whole listpageItemsCount: The data item count of the returned list. Usually, it is the same as the items count in the request. If the response contains the end of the whole list, then this property contains the actual item count of the list.recordStartIdx: The starting index of the data item in the whole listrecordEndIdx: The end index of the data item in the whole listmoreRecordsAvailable: A flag variable that can indicate whether there are more items in the whole list, or not
I created this response type as a template data type. The template type extends from the Object class, hence the actual data type used in this class has to be an object, not a primitive type. By defining this class as a template class, I can reuse this object type in any project it can fit. I take advantage of it whenever the opportunity presents itself.
In the next section, I will discuss the design of the RESTFul API that can handle the paged data fetching using these request and response object types.
Fetching Data One Page at a Time
To fetch data in batches and make the mechanism as general as possible so I can reuse it in other projects, I have to use interfaces and implementations, then make them as loosely coupled as possible. I need three different pieces:
- The first piece is a converter. When I fetch a list of data from the backend, the data type may be different from the data type that was used at the front end. Hence, a general purpose data converter is needed.
- The second piece is a data loader. This data loader must have a project specific implementation, so it can load the data based on page calculation (i.e., the start index and number of items to load).
- The third piece is a general purpose orchestrator that can coordinate the first two pieces to load the data, convert the data, and then package the data list into the response object.
If you read through the description, you know you are looking at the strategy pattern. Now, let's take a look at the interfaces and implementations of these "pieces". I'll begin with the easiest of the three -- the data converter.
Here is the interface of the data converter class:
package org.hanbo.boot.rest.services;
import java.util.List;
public interface DataObjectConverter<T extends Object, K extends Object>
{
T convertValue(K origVal);
List<T> convertValues(List<K> origVals);
}
This interface is very simple. It uses two template data types, T and K. Both are inherited from the Object class. The interface defines the contract of converting from one type of object to the other type of object. There are two methods. The first one converts one object from one type to the other. The second one converts a list of objects of one type to the other. Now let's take a look at the implementation. The implementation is project specific. In this sample application. The data types that are used are just the String type. The implementation class looks like this:
package org.hanbo.boot.rest.services;
import java.util.ArrayList;
import java.util.List;
import org.springframework.stereotype.Service;
@Service
public class SampleObjectsConverter
implements DataObjectConverter<String, String>
{
@Override
public String convertValue(String origVal) {
return origVal;
}
@Override
public List<String> convertValues(List<String> origVals) {
List<String> retVal = new ArrayList<String>();
if (origVals != null)
{
retVal.addAll(origVals);
}
return retVal;
}
}
Now, we move on to the next interface. This one is also pretty simple -- the data loading object type. This interface looks like this:
package org.hanbo.boot.rest.services;
import java.util.List;
public interface BatchObjectsFetcher<K extends Object>
{
int getTotalItemsCount();
boolean batchLoadItems(int startIdx, int itemsCount, List<K> retList);
}
And here is the project specific implementation of this data loading object type:
package org.hanbo.boot.rest.services;
import java.util.ArrayList;
import java.util.List;
import org.springframework.stereotype.Repository;
@Repository
public class SampleBatchObjectsFetcher
implements BatchObjectsFetcher<String>
{
private static final int TOTAL_ITEMS = 800399;
private static List<String> allDataItems;
static
{
allDataItems = new ArrayList<String>();
for (int i = 0; i < TOTAL_ITEMS; i++)
{
String itemToAdd = String.format("This is text line #%d.", (i+1));
allDataItems.add(itemToAdd);
}
}
@Override
public int getTotalItemsCount()
{
return allDataItems.size();
}
@Override
public boolean batchLoadItems(int startIdx, int itemsCount, List<String> retList)
{
if (retList == null)
{
return false;
}
retList.clear();
int tempStartIdx = startIdx;
if (startIdx < 0)
{
tempStartIdx = 0;
}
if (startIdx >= allDataItems.size())
{
startIdx = allDataItems.size() - 1;
}
int tempItemsCount = itemsCount;
if (tempItemsCount < 0)
{
tempItemsCount = 1;
}
int tempEndIdx = tempStartIdx + tempItemsCount - 1;
if (tempEndIdx >= allDataItems.size())
{
tempEndIdx = allDataItems.size() - 1;
}
for (int i = tempStartIdx; i <= tempEndIdx; i++)
{
String objToAdd = allDataItems.get(i);
if (objToAdd != null)
{
retList.add(objToAdd);
}
}
return true;
}
}
The implementation class is marked as a repository. My purpose for this class is to load data from the back end data store. For demonstration purposes, I created a large list of 800,399 sentences. The only method of this implementation is made to load the data by finding the item of a start index and the count of items to be loaded, which are the parameters of this method.
I want to point out some features of this method. It checks the input parameters, making sure that the start index is valid, and calculation to make sure the end index is also correct. After that, I use a for loop to copy the sentences from the original list into the returning list. Since this is a simple demonstration, the calculation and data copy is easy to do. For a real project, the back end data store can be simple or complex, and the data loading by start index and item count can be more complicated. I will cover this in the Summary section. At last, I will show you the data loading orchestration interface and its implementation.
This is where the strategy pattern comes into play. The two interfaces I have discussed above are the strategies, and the orchestration interface will utilize these strategies to process the request of loading paged data. The interface of this orchestration object looks like this:
package org.hanbo.boot.rest.services;
import org.hanbo.boot.rest.models.BatchRetrievalResponse;
public interface BatchLoadObjectsService<T extends Object, K extends Object>
{
BatchRetrievalResponse<T> batchLoadObjects(
BatchObjectsFetcher<K> objsFetcher,
DataObjectConverter<T, K> converter,
int startIdx, int itemsCount);
}
As shown, the interface has just one method. It takes the parameters of the fetcher object and the data object converter, and also the start index of the item in the original list and the item count to be fetched. The reason I choose parameters instead of auto-wiring the object fetcher and data converter as properties of the implementation is that I can create a general purpose implementation of this interface, which can be reused in other projects. If I force these two objects as auto-wired properties of the implementation class, not only do I have to create a new implementation for every new project, but also I might need to create multiple implementations of this interface within the same project. By moving these two into the method as parameters, I can create one implementation of this interface which can be utilized for almost all related design requirements. Here is my implementation:
package org.hanbo.boot.rest.services;
import java.util.ArrayList;
import java.util.List;
import org.hanbo.boot.rest.models.BatchRetrievalResponse;
import org.springframework.stereotype.Service;
@Service
public class BatchLoadObjectsServiceImpl<T extends Object, K extends Object>
implements BatchLoadObjectsService<T, K>
{
public BatchLoadObjectsServiceImpl()
{
}
@Override
public BatchRetrievalResponse<T> batchLoadObjects(
BatchObjectsFetcher<K> objsFetcher,
DataObjectConverter<T, K> converter,
int startIdx, int itemsCount)
{
BatchRetrievalResponse<T> resp = new BatchRetrievalResponse<T>();
if (objsFetcher == null)
{
throw new IllegalArgumentException("Objects fetcher cannot be null.");
}
if (converter == null)
{
throw new IllegalArgumentException("Objects converter cannot be null.");
}
if (startIdx < 0)
{
startIdx = 0;
}
if (itemsCount <= 0)
{
itemsCount = 1;
}
int totalItems = objsFetcher.getTotalItemsCount();
if (totalItems <= 0)
{
resp.setTotalCount(0);
resp.setPageItemsCount(0);
resp.setRecordStartIdx(0);
resp.setMoreRecordsAvailable(false);
resp.setRecordEndIdx(0);
resp.setListOfItems(new ArrayList<T>());
return resp;
}
resp.setTotalCount(totalItems);
resp.setPageItemsCount(itemsCount);
resp.setRecordStartIdx(startIdx);
int endIdx = startIdx + itemsCount - 1;
if (endIdx >= totalItems)
{
endIdx = totalItems - 1;
resp.setPageItemsCount((int)(endIdx - startIdx + 1));
}
resp.setRecordEndIdx(endIdx);
List<K> respList = new ArrayList<K>();
objsFetcher.batchLoadItems(startIdx, itemsCount, respList);
if (respList.size() == 0)
{
resp.setTotalCount(0);
resp.setPageItemsCount(0);
resp.setRecordStartIdx(startIdx);
resp.setMoreRecordsAvailable(false);
resp.setRecordEndIdx(0);
resp.setListOfItems(new ArrayList<T>());
}
else if (respList.size() > 0 && respList.size() < itemsCount)
{
resp.setMoreRecordsAvailable(false);
resp.setPageItemsCount(respList.size());
List<T> respList2 = converter.convertValues(respList);
resp.setListOfItems(respList2);
}
else
{
resp.setMoreRecordsAvailable(true);
resp.setPageItemsCount(itemsCount);
List<T> respList2 = converter.convertValues(respList);
resp.setListOfItems(respList2);
}
return resp;
}
}
Why do I want to create such an implementation to satisfy almost all the related design requirements? The reason is that once I create one type of design requirement. Some slight tweaks can be applied to the implementation to satisfy a different design requirement. I am sure all of us have done something like copying one implementation of an interface, doing the tweaking, then apply to a different part of the same project, or copying the tweaked implementation into a different project. This can result in a lot of duplicated code and create a nightmare for maintenance.
The method first checks the input parameters, to make sure the start index of the item and the item count are valid. The start index must be greater or equal to 0. The item count has to be greater than 0. The two strategy objects cannot be null. After the checking, the method will get the total number of items in the original list. It is not necessary to get the total items count, but this number can help us in several ways, such as double-checking the start index or giving the client side the ability to estimate the number of fetches needed to get all items, etc.
If the total number of items in the original list is 0, then the method will not do any fetch, instead, it will create a response object wrapping an empty list. If the total count is greater than 0, then the method will use the object fetcher to fetch the page of items specified by the start index and the items count. The fetching operation will have three different outcomes. The first can be that the result contains a list of items representing the page of data items, there are more items available. The second outcome can be that it was at the end of the list, the returning list would be the remaining items, and there will be no more items available. The third outcome can be that the list is empty, which is an indication that there are no more items in the original list. Those three if-else blocks handle the likely outcomes. And based on each outcome, the response object is created and returned to the caller. Once the returned list is available, the method uses the data converter object to convert the list for the response object. As long as the back-end data can be represented in the format of a list, this method should be able to fetch the items out of it in batched pages.
Next, we will use this strategy pattern in the RESTFul API controller. This is my controller:
package org.hanbo.boot.rest.controllers;
import org.hanbo.boot.rest.models.BatchReadRequest;
import org.hanbo.boot.rest.models.BatchRetrievalResponse;
import org.hanbo.boot.rest.services.BatchLoadObjectsService;
import org.hanbo.boot.rest.services.BatchObjectsFetcher;
import org.hanbo.boot.rest.services.DataObjectConverter;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class BatchLoadItemsController
{
private BatchObjectsFetcher<String> _batchObjsFetcher;
private BatchLoadObjectsService<String, String> _batchLoadSvc;
private DataObjectConverter<String, String> _dataObjConverter;
public BatchLoadItemsController(
BatchObjectsFetcher<String> batchObjsFetcher,
BatchLoadObjectsService<String, String> batchLoadSvc,
DataObjectConverter<String, String> dataObjConverter)
{
_batchObjsFetcher = batchObjsFetcher;
_batchLoadSvc = batchLoadSvc;
_dataObjConverter = dataObjConverter;
}
@RequestMapping(value = "/batchread", method = RequestMethod.POST,
consumes = MediaType.APPLICATION_JSON_VALUE,
produces = MediaType.APPLICATION_JSON_VALUE)
public ResponseEntity<BatchRetrievalResponse<String>>
batchReadRecords(@RequestBody BatchReadRequest req)
{
BatchRetrievalResponse<String> resp = new BatchRetrievalResponse<String>();
if (req != null)
{
resp = _batchLoadSvc.batchLoadObjects(_batchObjsFetcher,
_dataObjConverter, req.getStartIdx(), req.getBatchItemsCount());
if (resp != null)
{
return ResponseEntity.ok(resp);
}
else
{
return ResponseEntity.notFound().build();
}
}
else
{
ResponseEntity<BatchRetrievalResponse<String>> retVal =
ResponseEntity.badRequest().build();
return retVal;
}
}
}
This class is not complex. I have declared three properties for the data fetching operations. They are the interfaces I have discussed in this section. The constructor of the controller will auto-wire the instance for these properties. The only method in the class handles the incoming API requests. This method is the one that can fetch the data items by the page. Because the request is a JSON object, I had to make this method handle HTTP POST requests. The content of this method just calls the fetching orchestrator and uses the data fetcher and converter to do the heavy lifting.
All these are the back end designs. To demonstrate the back end work, I have an Angular application that can do it. The way it fetches all items in batches or pages will be discussed in the next section.
Batch Read all Data Items via AngularJS and UI-Grid
In order to demonstrate the batch loading of data, I need UI-Grid. UI-Grid is the best open-source component that can easily integrate with AngularJS. It provides a lot of out-of-the-box functionalities that can be easily configured for use. For this sample application, I added the bare bone set of files for UI-Grid. And it works. You can check the ui-grid folder in the assets folder and see how the files are added to this project.
As I mentioned before, the sample application has only one page. In it, there is the UI-Grid based table. As soon as the page renders, it will begin loading the data, and the data items will accumulate as long as each batch loading completes. This is the HTML markup page of the page:
<div class="row">
<div class="col-xs-12 text-center">
<h3>Bulk Load Lots of Items</h3>
</div>
</div>
<div class="row">
<div class="col-xs-12 col-sm-offset-1 col-sm-10 col-md-offset-2
col-md-8 col-lg-offset-3 col-lg-6">
<div class="panel panel-default">
<div class="panel-heading">All the Items</div>
<div ui-grid="vm.gridData" ui-grid-pagination></div>
</div>
</div>
</div>
This page is pretty simple, all it has is a Bootstrap panel and the UI-Grid based table. The UI-Grid takes in a scope variable called vm.gridData. This scope variable is initialized in the AngularJS controller. This controller is the place where the repeated batch data loading is done.
This is the AngularJS controller class:
export class AppController {
constructor($rootScope, $scope, $timeout, batchLoadService) {
this._rootScope = $rootScope;
this._scope = $scope;
this._batchLoadService = batchLoadService;
this._allItems = [];
this._timeout = $timeout;
this._gridData = {
paginationPageSizes: [25, 50, 75],
paginationPageSize: 25,
data: null,
columnDefs: [
{
field: "value",
displayName: "Sentence",
width: 550
}
]
};
this.loadData();
}
get allItems() {
return this._allItems;
}
set allItems(val) {
this._allItems = val;
}
get gridData() {
return this._gridData;
}
set gridData(val) {
this._gridData = val;
}
loadData() {
let startIdx = 0,
batchItemsCount = 80;
this.batchLoadData(startIdx, batchItemsCount);
}
batchLoadData(startIdx, itemsCount) {
let req = {
startIdx: startIdx,
batchItemsCount: itemsCount
}, self = this;
self._batchLoadService.batchLoad(req).then(function(result) {
if (result) {
if (result.listOfItems) {
let i = 0;
for (; i < result.listOfItems.length; i++) {
self._allItems.push({ "value": result.listOfItems[i] });
}
}
self._gridData.data = self._allItems;
self._timeout(function() {
if (result.moreRecordsAvailable) {
self.batchLoadData(startIdx + itemsCount, itemsCount);
}
}, 3000);
} else {
console.log("No result available");
}
}, function(error) {
if (error) {
console.log(error);
} else {
console.log("Unknown error occurred while fetch batch load.");
}
});
}
}
Here is the code that initialize the UI-Grid component:
this._gridData = {
paginationPageSizes: [25, 50, 75],
paginationPageSize: 25,
data: null,
columnDefs: [
{
field: "value",
displayName: "Sentence",
width: 550
}
]
};
The property _gridData for the controller is a JavaScript object, used as configuration data for the UI-Grid component. I use the minimum set of configurations. The first one, paginationPageSizes, is a list of options, that allow the user to set the number of items to display per page (on display). This, however, has nothing to do with the number of items to fetch from the back end. The second one, paginationPageSize, sets the default number of items to be displayed on the page. Again, it has nothing to do with the number of items to fetch from the back end. The third one, data, is the list of items to be displayed on the page. It is set to null for now. This property is updated whenever the batch data load completes. Lastly, there is the property columnDefs. This property is to set the display title and the value to display for the column.
The code that reads the data in batches is this:
...
batchLoadData(startIdx, itemsCount) {
let req = {
startIdx: startIdx,
batchItemsCount: itemsCount
}, self = this;
self._batchLoadService.batchLoad(req).then(function(result) {
if (result) {
if (result.listOfItems) {
let i = 0;
for (; i < result.listOfItems.length; i++) {
self._allItems.push({ "value": result.listOfItems[i] });
}
}
self._gridData.data = self._allItems;
self._timeout(function() {
if (result.moreRecordsAvailable) {
self.batchLoadData(startIdx + itemsCount, itemsCount);
}
}, 3000);
} else {
console.log("No result available");
}
}, function(error) {
if (error) {
console.log(error);
} else {
console.log("Unknown error occurred while fetch batch load.");
}
});
}
...
This method is not hard to understand, the parameters are the start index of the item in the large list, and the items count to be fetched. Entering the method, the first one you will see is the preparation of the request object. Then the method calls the service object _batchLoadService and invokes the object's batchLoad() method. Once the load succeeds, the then() method from the $promise will massage the data so that it can be displayed in the UI-Grid. Within the then() method, after data is added to the UI-Grid, the service object will invoke its batchLoad() again by changing the start index to the item count (going to the next page of data). Between two subsequent data reads, there is a 3-second wait. I used $timeout to create this wait operation. It will make the data read operations easier to follow. This is a recursion if you have not noticed. It will repeat until the indicator result.moreRecordsAvailable is set to value false.
You might wonder what the service class batchLoadService looks like. Here it is:
export class BatchLoadService {
constructor ($resource) {
this._loadSvc = $resource(null, null, {
batchLoad: {
url: "/batchread",
method: "post",
isArray: false
}
});
}
batchLoad(req) {
return this._loadSvc.batchLoad(req).$promise;
}
}
In this service class, BatchLoadService, I inject the $resource, and use it to call the backend batch read data API. In my past tutorials, I have explained this is done many times, so I won't waste words and explain how the code works.
Here is the code that sets up the page navigation routing:
export function appRouting($routeProvider) {
$routeProvider.when("/", {
templateUrl : "/assets/app/pages/index.html",
controller: "AppController",
controllerAs: "vm"
});
}
It is a method that can be added to the AngularJS application to help to configure the browser URL and what page to be displayed. This is where I associate the page and page controller.
The following code is the AngularJS module that contains this single-page application. It sets up the dependency injections which are used by various parts of the application:
import { appRouting } from '/assets/app/js/app-routing.js';
import { AppController } from '/assets/app/js/AppController.js';
import { BatchLoadService } from '/assets/app/js/BatchLoadService.js';
let app = angular.module('startup', [ "ngRoute", "ngResource",
'ui.grid', 'ui.grid.pagination' ]);
app.config(appRouting);
app.factory("BatchLoadService", [ "$resource", BatchLoadService])
app.controller("AppController", [ "$rootScope", "$scope",
"$timeout", "BatchLoadService", AppController ]);
What I have listed are the most essential parts of the front-end web application. I must say that the error handling part of this front application is very weak. But this is just a useless demo, so I am going to cut some corners here. Now that we have everything integrated, it is time to give this sample application a try. In the next section, I will describe how to build this sample application and run it.
How to Run the Sample Application
After you download the source code, unzip it into a folder of your choosing, please go through the project's subdirectories, and rename all the *.sj files into *.js files. Also this application uses Java 17, if you are using lower version of Java, please modify the pom.xml file to lower the Java version.
In the base directory of the project, you can find the pom.xml file. Use a console application and run the following command:
mvn clean install
Once the build process completes successfully, it is time to run the application:
java -jar target/hanbo-angular-batch-read-sample-1.0.1.jar
The application starts up and will spit out a lot of output on the console window. Once the start-up completes successfully, you can run the application using a browser, just navigate to the following URL:
http://localhost:8080/
As soon as the page displays, there will be data available showing in the UI-Grid table. And every three seconds after, more data would be added to the UI-Grid table. Since I hard-coded 800,000+ data items. And the hard-coded items count per batch is 80, so it will take a long time to load all that data. Still, you should be able to see the data loading in real-time. And if you turn on the browser's "Developer Mode", and do a profile on the memory usage, you will see that the continuous operation does not consume a lot of memory. Also, the UI-Grid would be able to properly break the data items into pages based on the specified page item count.
Summary
As my first tutorial for the year 2023, this has been a great success. In this tutorial, I have explained in detail how to load massive amounts of data in batches. The purpose of such a design is that I want to load a little data as quickly as possible so that the data can be displayed in the application. Then the application can take time and load the rest of the data and display them.
Compare to the straightforward approach of taking a long time to load everything all at once, then have them displayed in the application, this approach is much more complex. It is also a bit harder to implement. The changes must be done on all the layers of the vertical stack, the front-end application must be changed to do the batch reads. In the RESTFul API layer, we need to add some new APIs to handle the batch read. That means the service layer must also add new code or modify the existing code base to accommodate such change. Finally, on the data repository layer and the database layer, how the data is queried, ordered, and broken into pages must be done correctly. If you have a project with a messed-up code base, I advise you not to try retrofitting it with this new approach. And for an easy-to-maintained project, be very careful when you are trying to get this design added.
Anyways, I am very happy with how this tutorial has turned out. UI-Grid is truly an awesome UI component for AngularJS applications. Without it, it would be much hard to experiment with a design like this. This proves the old saying, "Don't create your own design, use a good alternative, instead..." I hope you have enjoyed my latest tutorial. As always, I enjoyed writing it. Hopefully, it can help you resolve similar problems. Good luck to you!
History
- 9th February, 2023 - Initial draft
