Building Offline Experiences with HTML5 AppCache and IndexedDB

Develop a Windows 8 app in 30 days

Users expect their Web sites and apps to work well even when the network isn't available. With data increasingly stored in the cloud, developers want to enable fluid experiences that allow access to data when there is no connectivity; when devices are disconnected from the network or when they encounter dead spots in coverage.

In this article, we show how to create well-behaved offline sites and apps using the following HTML5 features:

• AppCache to store file resources locally and access them offline as URLs
• IndexedDB to store structured data locally so you can access and query it
• DOM Storage to store small amounts of text information locally
• Offline events to detect if you’re connected to the network

Example: Offline Support for Anywhere Access

Let’s say you go shopping with a printout of a recipe from your favorite food site but when you’re at the market, you can’t find some key ingredients.

Imagine that when you were home using your mobile PC to browse the recipes site, a portion of the site was automatically downloaded for offline usage. This enables you to take your mobile PC to the store, access the site, and search for a new recipe at the market. The best part is that you can do this without being connected to a network. As a consumer, you’d appreciate the site more because it just worked when and where you needed it.

As a developer, you can enable these types of scenarios with a combination of offline technologies: AppCache, IndexedDB, DOM Storage, and WebSockets (or XHR). Before exploring the individual technologies, let's explore the benefits.

For Metro style apps and Web sites, offline technologies allow you to handle connectivity failures. Imagine that your user is filling out a form and he loses network connectivity. What should your Web site or Metro style app do? A connection free development mindset allows your app to work correctly independent of whether it is connected to the network or not. Your app will just work correctly.

In more sophisticated scenarios, Web sites and apps allow users to create new content and to store new data, even though the application is completely offline. Imagine, Outlook Web Access (OWA), Hotmail, or GMail working seamlessly when offline, as Outlook does today.

Offline technologies can also improve overall performance by serving cached resources locally, pre-caching future information, and shifting processing power from the cloud (or the network) to the client device. The more information you’re able to cache locally, search locally, and compute locally, the fewer resources are needed from the server and the faster your users' experience will be.

The expectations of having a Metro style app work offline are higher than having Web sites work offline. Because they're deployed using self-contained packages from the store, users expect them to have some type of offline functionality (e.g. games, books, recipes, etc.). Even if these apps are not able to create or access new content, previous content should be visible (e.g. Contacts, Meetings, feeds, magazines, etc.).

Cache File Resources Locally Using AppCache

AppCache enables you to create long lived local caches of downloaded file resources; resources you can access while offline or use while online to improve performance. Imagine that a three year old child uses a laptop to download an interactive Web game (KidsBook) from your home network. If the application's resources are stored locally, the child can continue to play the game in the car where there is no network connectivity.

If KidsBook was implemented using AppCache, the game would have cached the necessary resources (JavaScript, HTML, CSS, audio, video, etc.) for the game to be played when it was first downloaded and later while disconnected from the network. This allows the child to remain entertained, even though the device itself has no network connectivity.

To see how to enable an interactive Web game to work offline, check out the KidsBook example on the IE Test Drive site.

AppCache uses a manifest file to specify resource URI's in order to cache content from a Web site. The caching happens behind the scenes after the browser displays the Web page, which allows resources defined in the manifest file to be downloaded. This guarantees that resources are downloaded to the local machine, as a unit in a single transaction, to create a local cache. If a single resource fails to download, no cache is created. To update content stored in a cache, update the manifest file on your server. When the user next accesses the site, the browser compares the manifest file on the server with the last cached copy. If the cached copy of the manifest is different from the server copy, a new version of the cache is created using the content defined in the updated manifest file.

AppCache also allows Internet Explorer and Metro style apps to access cached resources offline using traditional URL's. This allows you to type a URL in the browser window and access this information without any network connectivity. In addition, offline pages can resolve URIs using the local cached information. For code samples take a look at the HTML5 Application Cache (“AppCache”) section in the IE10 Developer’s Guide.

Overall, AppCache provides some advantages over HTTP’s caching. HTTP caching doesn’t guarantee that cached resources will be available after the TIF (Temporary Internet Files) is cleared. Also, HTTP caching doesn’t correctly resolve URLs while offline. However, HTTP caching can be used to optimize AppCache behavior by specifying the lifetime of a cached resource. This will determine if a resource is downloaded from the Web or copied from the cache when a new version of a local cache is created.

Metro style applications can benefit from AppCache by locally caching Web resources that are accessed by iframes, which allows that content to be accessed offline.

Cache Large Structured Data Locally Using IndexedDB

IndexedDB is a local database designed to store JavaScript objects in the local machine, allowing fast indexing and searching of objects. The recipe site presented earlier includes a database with sixteen recipes extracted from a parent site. Imagine using an RSS feed, a WebSocket, or an XHR connection to periodically update this database. This would allow your users to have access to the latest recipes even when they have no network connectivity.

IndexedDB enables you to manipulate and index JavaScript objects directly. An advantage of using indexedDB to search for information locally is that it reduces your computing costs by not forcing you to always search in the cloud. This assumes you’re able to maintain the relevance of the data that is cached in the local system.

IndexedDB is a technology created around ISAM database concepts. Like many Web platform technologies, it is designed to provide a low-level API that can be used by various library abstractions built on top of it. The following table compares IndexedDB development concepts with similar concepts from the well understood Relational model.

When using IndexedDB, you’ll create databases which contain object stores (Contacts, Emails, Meetings, etc.). These object stores contain the JavaScript objects that are required by your application (Contacts – First Name, Last Name, Address, etc.). Each JavaScript object is expected to have a unique identifier accessible via a keyPath. In addition, object stores will contain indexes on properties that can be used to query the dataset (Emails – Subjects, Dates, etc.). Filters will be used to organize or reduce the result set via KeyRanges on indexes or object store.

The following code snippet shows how to read a book record from a “Library” database:

var oRequestDB = window.indexedDB.open("Library");
oRequestDB.onsuccess = function (event) {
    db1 = oRequestDB.result;
    if (db1.version == 1) {
        txn = db1.transaction(["Books"], IDBTransaction.READ_ONLY);
        var objStoreReq = txn.objectStore("Books");
        var request = objStoreReq.get("Book0");
        request.onsuccess = processGet;
    }
};

Information contained in object stores is always accessed for read or write in the context of a transaction. There are three types of transactions:

• VERSION_CHANGE – used to create or update object store and indexes. Because VERSION_CHANGE transactions lock the complete database and prevent concurrent operations, they are not recommended to read and write records into the database.
• READ_WRITE – Allows records contained in object stores to be added, read, modified, and deleted.
• READ_ONLY – Allows records contained in object stores to be read.

The asynchronous API model provided by IndexedDB leverages the request/response model supported by many Web APIs, such as XHR. Requests are submitted to the local IndexedDB process and results are handled by client’s onsuccess or onerror event handlers. In addition, there is no explicit mechanism to commit a transaction. Transactions are committed when there are no more pending requests on the server and no pending results on the client. Furthermore, it is up to your application to handle exceptions and error events. In many cases, if an exception or error event is not handled, the transaction is aborted.

In summary, IndexedDB is an optimized mechanism for querying data objects via indexes. It provides sites with the APIs to access large amounts of related data via cursors and to filter data using KeyRange objects. The pattern we believe developers will follow is to have a "master" database with all the user records living in the cloud and a local IndexedDB database with a subset of records to facilitate fast searches and offline data access.

Store Small Textual Data Locally with DOM Storage & Offline/Online Events

Sites can use DOM storage and connectivity events to handle small amounts of textual data and detect poor connectivity. Imagine a game that uses these technologies to track the user’s score while offline. Imagine what would happen if you got a high score when there was no network connectivity. Would the Web page hang or crash?

Because this data is textual in nature and it doesn’t require a lot of space, you can use DOM Storage to store the information locally, without requiring network connectivity, and upload it at a later time when the network is available. DOM Storage supports more data than cookies and doesn’t require data encoding. In addition, DOM Storage doesn’t send data to the server on each request, and can be scoped to domain or sessions access.

Using this technology is as simple as accessing the windows.localStorage object. In this object, you can check or add name/value pairs in the form of properties. The following code snippet shows how to store the game score locally using localStorage:

<script type="text/javascript">
    var lStorage;
    function init() {
        if (window.localStorage) {
            lStorage = window.localStorage;
            if (typeof lStorage.score == 'undefined')
                lStorage.score = 0;
            document.getElementById('score').innerHTML = lStorage.score;
        }
    }
    function save() {
        if (lStorage) {
            lStorage.score = getGameScore();
        }
    }
</script>
...
<body onload="init();">
    <p>
        Your last score was: <span id="score">last score insert in init()</span>
    </p>
</body>

In addition, the offline/online events will help you detect when you have network access so you can push the data to your server. For example, you can detect when you are online and update the database with content from the server using WebSockets or XHR.

This is as simple as checking for the state of the navigator.onLine property. The following code shows how to register for online and offline events:

function reportConnectionEvent(e) {
    if (!e) e = window.event;
    if ('online' == e.type) {
        alert('The browser is ONLINE.');
    }
    else if ('offline' == e.type) {
        alert('The browser is OFFLINE.');
    }
}
window.onload = function () {
    status = navigator.onLine; //retrieve connectivity status
    document.body.ononline = reportConnectionEvent;
    document.body.onoffline = reportConnectionEvent;
}

In Metro style apps, we’re providing an additional API, Windows.ApplicationData, that allows you to store more data types locally and enables them to roam across multiple machines.

The key point is to design your application or Web site with the idea that your connectivity may disappear at any time and you need to be able to handle that situation smoothly. Implementing a data pattern that stores information locally before sending it to the cloud will allow you to deal with problematic network connections.

Update Local Data with WebSockets and XHR

For some scenarios, your customer data will continue to live in the cloud in order to be easily accessible from any device. Therefore, you need to ensure that cached data remains relevant, current, and up-to-date. In order to do that you need to create channels to synchronize data between the cloud and your app. You can leverage WebSockets and XHR to facilitate this synchronization. This requires you to package your data into transferable formats (e.g. XML or JSON), use XHR or Websockets to transfer those resources to the client, and then use XML or JSON parsers to create JavaScript objects that will be stored in the IndexedDB database. This can also be used to upload information stored on DOM Storage to the server.

Conclusion

Network connectivity is not always reliable; however, your apps need to be. Using these offline technologies to anticipate network shortage will make your apps even better in many consumer scenarios and situations. Furthermore, you can take advantage of a huge opportunity to differentiate your site and Metro style apps by allowing them to work properly offline. This will increase their usage and create a larger opportunity for your service. Leverage the various offline technologies specified here (AppCache, IndexedDB, DOM Storage, and others) to cache as much information as possible locally.

For more information, check out the BUILD presentation Building offline access in Metro style apps and Web sites using HTML5, which also describes the role of the File API to handle offline scenarios.