Why HTML5? Why Not? UI / UX Versus Storage Challenge (Part 3 of N)

• Download source code (dotnetcore 6.x project) - 1.6 MB
• Download code from Github @ https://github.com/raddevus/LibreStore

Read the Previous Articles in this Series

• Why HTML5? Why Not? UI / UX Versus Storage Challenge (Part 1 of N)[^]
• Why HTML5? Why Not? UI / UX Versus Storage Challenge (Part 2 of N)[^]

Introduction

I've done quite a bit of research to discover a service which would fit the needs of our little app, but I've really come up empty. Yes, there may be solutions out there but:

1. They were too expensive -- I'm the ultimate cheapskate. I saw one service which required buying 50TB (Yes, TeraBytes) of storage minimum. It was over $3,000 / month.
2. They were far too difficult to even understand -- I just want to post data to a URL (web api) and have the data saved for me.
3. If I did understand how the service works, I didn't understand how much it was going to cost me. I've signed up for AWS service before and it ran wild and I racked up $100 charges before I noticed. I really don't want that.
4. What I really want is SaaS (Software as a Service) Data Store -- Why can't I have that? Why can't it be cheap? Why is it so difficult?

That last question is a little naive isn't it? But sometimes, you want to stay naive.

There are a few challenges (Security, Identity / Authentication, Protecting the Service, etc.) to making remote data storage easy, but if we ignore those challenges, maybe we can get somewhere.

Let's try!

Project Constraint

One of the main project constraints is that we must be able to do everything in...wait for it...

That constraint is simply because on the client side, it is the language we have available to us since we are building this app on HTML5 technologies.

I know that JavaScript feels like a punch in the stomach for a lot of devs. It means a lot of devs won't even read this article, but again, I'm attempting to consider fulfilling this whole Write Once, Run Anywhere pipe dream so bear with me. I learned to program using C/C++ and then moved to C# around 2000 so I understand the disdain for JavaScript, but a lot has changed over the years.

Additionally, I like to use the right tool for the job and in this case, it does seem that JavaScript is the tool we need.

Enter .NET and C#

Of course, the problem we are going to solve in this article is the ability to save data on the Server side (after posting data using the client / JavaScript).

The easiest way (in my mind) to build a Server-side WebAPI is using .NET Core and writing it in C#.

Main Point of the WebAPI

In this article, we will build a simple WebAPI which will allow a developer to:

1. post her user's data to a WebAPI endpoint
2. have it stored (at the server)
3. so she can retrieve it again at a later time for the user

Alarms are Ringing!

After reading those basic requirements, alarm bells may be ringing in your mind.

You may be shouting at this article things like:

"The user's data will be compromised!!"

"You can't just save data out in the clear like that! You are ridiculous!"

I will provide a solution for that in this article and I hope you will find it intriguing.

Spoiler Alert

It's spoiler, but here's how we're going to do it: AES256 Encryption

All data posted to the Server will be encrypted on the client side via AES256 encryption.

More on that as we get further along.

Setting Up WebAPI the Project

Keep in mind that I'm showing you how the WebAPI will be designed and built, but the real end-user developer will not have to know these details.

Target: JavaScript Developer

We are attempting to get to a solution that any JavaScript developer can use to safely post their application data while only having to understand client-side development. We want to make the WebAPI as simple as possible (no simpler) so the JavaScript dev can simply call the fetch api on a specific endpoint (URL) to save the user data and then call fetch on another endpoint to retrieve the data. That's it.

How I Work

I moved to Linux a few years ago so all my development is done using Visual Studio Code (VSC).

However, you will see that we will create a complete Visual Studio Solution so if you are using full-blown Visual Studio, you should have no problem following along.

Of course, you'll be able to simply download the completed solution and open it in Visual Studio or VSC anyways, so it'll all be good.

Project Creation with Unit Tests

I also want to include Unit Tests in my project because it will allow me to more easily test the WebAPI methods and my domain models. Nothing fancy, just the ability to run a method without having to do a lot of set up work, but instead just have the Xunit runner do it for me.

Because I want to include Unit Tests, I've discovered that creating a good clean .NET WebAPI project is a bit of a pain, but I've worked out the details so that we get a nice clean directory structure that looks like the following:

All my projects are in an outer folder named dotnet.

The target folder for a project is named MainProject and will contain two main subfolders:

1. MainProject -- contains the WebAPI code
2. MainProject.Tests -- contains the Unit Tests for the project.
3. MainProject.sln will reference both of these projects.

This makes it quite easy to create Unit Tests that reference our Models & Controllers which will be in the WebAPI and wraps up everything in one nice outer folder named MainProject.

Naming My Project: LibreStore

Now, let's see how this will work with my WebAPI project which I will name LibreStore.
I am naming it LibreStore because Libre means free in Spanish and this project is an attempt to create Storage freedom. I also own the web site domain NewLibre.com, so eventually it'll be a nice project under that domain.

Getting to that Project Structure

To get to that project structure, we will run the dotnet commands shown below:

Please note: I'm assuming if you want to follow along, you have .NET Core 6.x installed in your machine and set up ready to go. See Official .NET Core Installation docs for more.

Here are the commands to create our initial project with the structure previously shown:

command	explanation
$ mkdir LibreStore	Creates outer folder
$ cd LibreStore	Move into folder
$ dotnet new mvc -o LibreStore	Creates MVC Project (WebAPI with MVC controllers)
$ dotnet new sln	Creates new Solution File
$ dotnet sln add LibreStore	Adds Main project to SLN file.
$ dotnet new xunit -o LibreStore.Tests	Creates new XUnit test project and all dependencies
$ dotnet sln add LibreStore.Tests	Adds Unit Test project to SLN file.
$ dotnet add LibreStore.Tests reference LibreStore	Adds Reference to main project (LibreStore.dll) into the XUnit test project so it can be used.
$ cd LibreStore.Tests	Move into Test project folder
$ dotnet test	run unit tests

If you followed all of those steps and everything worked, then you should see something like the following, showing that 1 test was run and succeeded.

If we had decided on running the main target (our LibreStore MVC / WebAPI), then we would move to the LibreStore project folder and run the following command:

$ donet run

This builds the WebAPI project and starts up the web server on specific port (yours may be different).

Once it starts up, you can navigate to the site using your web browser.

You'll probably be warned by your browser because the certificate the web server uses is unsigned.

If everything went well, you should see:

That's the HomeController returning the Home View to the browser.

Gathering Our Most Basic Requirements

That's all good, but we just want to get to where we can:

1. Post data
2. Data is saved on server
3. Retrieve data

That leads us to a few requirements:

1. Need a method that accepts posted data -- a URL
2. Need a bucket to store the posted data into
3. Need a way to identify the bucket that the dev-user is saving the data to

Thought About This for a While

I'm trying to keep this article short so you may find some details are missing. I thought about my next step and the next step is to create a class which will help us save our data. I'm calling this class MainToken.

MainToken Class

I'm calling it that because all you will need is this String-based Key to store your data in the remote data store.

Here's the class:

public class MainToken{
    public int Id{get;set;}
    public int OwnerId{get;set;}
    public String Key{get; set;}
    public DateTime Created {get;set;}
    public bool Active{get;set;}

    public MainToken(String key, int ownerId=0)
    {
        OwnerId = ownerId;
        Key = key;
        Created = DateTime.Now;
        Active = true;
    }
}

It's just a class with a few properties and constructor.

In an attempt to keep this purposefully simple, I am only requiring that the user provide a MainToken.Key value which will be her GUID to store her data in the remote storage.

This field will be required to be unique. Later, I'll require that it is at least 10 chars long also -- in an effort to ensure that it is at least somewhat difficult to guess.

Security Doesn't Come From Obscurity

However, security in LibreStore won't come from Obscurity. Instead, it will come from always ensuring that the data is password encrypted AES256 data.

If someone guesses the users Key however, they would be able to retrieve her data. Of course, it will be AES256 encrypted so it _should_ not matter. Unless there is some unknown hack that has defeated AES256 -- not known at the time of this writing.

Data Could Be Added By Others

Also, if the Key were guessed, then someone else could maliciously insert data using the key and the real owner of the key may have data they don't want.

The Point: Make A Strong Key

Making a strong GUID Key is very important here so you can ensure only you can post data to your LibreStore.

This is a Novel Idea

I know that at this point, many readers may be balking at this whole idea. This idea is that all you need is:

1. GUID MainToken.Key
2. Encrypted data
3. LibreStore enpoint (URL)

and then, you will be able to post data that can be retrieved by anyone.

When you post this data, it will store your GUID Key as the Key to the data in the data table.

If the user posts data to the data table with a MainToken.Key which has never been created, then it will be created at that time.

How Will Data Be Stored?

We haven't discussed how the actual data will be stored? It could be stored in one or more text files on the Server or in a database. In our case, we will start out with something in between those two options.

SQLite

We are going to use the text-based SQLite database.

This provides us with an easy way to create the database and an easy way to start inserting data.

I will wrap the SQLite calls in some kind of Data Access wrapper so that later, when we want to switch the implementation to SQL Server or MySQL database, we can do so without any impact to users.

Why Sqlite?

Cool Thing About Sqlite DB - If you get the code, build & run it, you will see that the first time you post data that a file named librestore.db will be created. That is the database and it is created from the DDL (data definition language) code in the SqliteProvider class. It's kind cool because you don't need any Database Server installed.

I can't believe how amazing Sqlite is and how easy it is to use. Writing up this article and really delving into Sqlite has opened my eyes to what a great technology this little database is. It is a robust system that is amazingly simple to use. Check out more about this great project (which is FOSS Fully Open Source Software) at: https://sqlite.org/.

About this Article: I Went On My Merry Way

I wrote the previous part of this article a few days ago and then I left on my journey to create an ASP.NET Core 6.x MVC / WebAPI that would write / read from a sqlite database.

I return now having successfully accomplished the task.

Completed WebAPI, But I Took Liberties With Design

Getting everything working the way I wanted it to took quite a bit of work and the design suffered from the fact that I wanted to get to a working prototype. I just wanted to see it work.

Instead of detailing everything I did to create the initial Web API, I am instead going to list as many of the challenges as I can and provide a few details of what those challenges looked like.

I'll also talk about what I wanted to accomplish and how that seemed to alter the design and the code.

Database Snapshot

Here's a quick look at the three tables that currently make up the Sqlite db.

Three Tables

1. MainToken - MainToken.Key is the secret key you will use as your master identifier into the system (more later).
2. Bucket - Yes, this table can store anything in the Data field (up to 8,000 bytes). However, you will need your master identifier to get them back. And of course, you will AES256 encrypt the data.
3. Usage - This table allows me to track IPAddresses and the Actions (GetData, SaveData, etc.) so I can understand how my service is being used and if it is being attacked. I figure I may have to disallow some IPs in the future. There is nothing else identifying in there though.

You can also see that I have an Active field in each of the tables. This field will allow me to set a MainToken to inactive so it cannot be used any longer -- in case someone is hacking me or whatever.

How to Use the Web API

Here's how easy it is to use the web API.

Where Is It Hosted?

https://newlibre.com/LibreStore[^]

That's my website and it is HTTPS enabled so anything you post is protected.

That means your MainToken.Key won't be compromised when it is passed along in the URL.

Two Endpoints that Help you Store App Data

There are only two main endpoints that you can use at this point.

SaveData

To save your data, you just:

1. Create a (String) Key that is at least 10 bytes long & <= 128 bytes long. Store it in a place where you won't lose it, because without it, you're not getting your data back.
2. Send your data (right now, I just have it set up to accept HTTP Get commands) using the following URL: https://newlibre.com/LibreStore/Data/savedata?key=<your-key-here>&data=<your-data-here>

The easiest way to try it is to use the JavaScript fetch API (Using Fetch - Web APIs | MDN[^]).

Try It In Your Browser Dev Console

Here's how you can send data using your Browser's developer console.

1. Open your browser's dev console (F12 in most web browsers).
2. Paste the following code and alter to include your Key & Data.

fetch("https://newlibre.com/LibreStore/Data/
       SaveData?key=FirstOneForTest&data=First post to data for test.")
  .then(response => response.json())
  .then(data => console.log(data));

I already posted using the Key which means the MainToken record has already been created.

If you post using a previously used Key, it just means the data will be bound to that Key.

What Happens When Calling SaveData?

When you send the data using the fetch above and your own key, then here is what will happen.

1. A new unique entry (row) will be created in the MainToken table.
2. The data will be inserted in the Bucket table's Data field and will be tied to the MainToken.ID that was generated when the Key was created.

At this point, your Bucket Data is stored.

What Is Returned?

When it completes, the API returns:

JSON including two fields:

1. success: (true or false)
2. bucketId: id of the row just inserted into the bucket table

It looks something like the following:

If you save that into an object, then you will have an object with those two properties (success & bucketId) and you'll be able to re-use the values.

GetData

Now, when you want to retrieve your data, you send to the following URL:
https://newlibre.com/LibreStore/Data/GetData?key=<your-key>&bucketId=<your-bucket-id>

Again, you can use the Fetch API to get the data back.

Use the bucketId that was returned to you when you saved the data.

fetch("https://newlibre.com/LibreStore/Data/GetData?key=FirstOneForTest&bucketid=2")
  .then(response => response.json())
  .then(data => console.log(data));

It'll look something like the following in the browser console:

There are Two Major Rules For Using LibreStore

1. Never let your MainToken.Key out into the wild - I already broke this one with the example above. It means that anyone can attempt to use my Key to retrieve data now. Of course, when I save my real data, I will create a long random Key to store my data.
2. Never Store Unencrypted Data - I also broke this rule so I could show you how it works, but in the future I will encrypt all my data using AES256. If you encrypt your data properly, then you don't really have to worry about Rule #1 because no attacker should be able to decrypt your data.

Can You Use LibreStore Right Now?

Yes, you can post to it and retrieve data, but I haven't showed you how to easily encrypt your data yet so for now you (obviously) shouldn't post anything that really matters.

Next Article: Encrypting Data Via AES256

Since I'm just trying to get the Web API and this article out there and in an effort to keep this article shorter, I will write up how to encrypt your data using the AES256 encryption algorithm via JavaScript.

Sneak Peak: AES256 Encryption

However, since I've already done the research and figured out how to encrypt & decrypt data using AES256 via JavaScript, you can take a look at the code running at my Codepen.io and try it out if you like: https://codepen.io/raddevus/pen/VwMXawY[^].

To try it:

1. Type in your password
2. Type in your data
3. Click the [encrypt] button

You will see a string of Base64 encoded bytes appear. That is the encrypted bytes converted into Base64 data.

It is not the cleartext bytes. It is the cipher bytes converted to Base64.

If you click the Decrypt button, the data will be:

1. Base64 decoded
2. Descrypted using the password
3. Clear text will be added to a div at the bottom so you can see it -- it will match the original text.

If you change the password before decrypting, then it will not be able to decrypt the bytes and you will see nothing.

List of Things I Had to Learn, Research, Develop

Sqlite

• How to create a database
• How to get last inserted row in table (@@IDENTITY is in SQL Server but you use SELECT last_insert_rowid() in Sqlite
• How to connect and query via ASP.NET Core Data Provider -- add via nuget $ dotnet add package sqlite
•  

ASP.NET Core

• How to get around the CORS (Cross-Origin Resource Sharing) so anyone can hit the Web API from any other site (see code snapshot below). AllowAnyOrigin() resolves it, but it feels dangerous, doesn't it?
• How to build and deploy ASP.NET Core to my (limited) hosting site -- I can't control the web server completely and it can be difficult to get things running properly, but I figured out what I needed to do for my host in the web.config & how to get the app to start up.
• I also had to figure out how to build it as a "single exe" that is deployed out to the hosted web site: LibreStore.dll & LibreStore.exe which runs in the web server. Quite a challenge.

Run It on Your Own Web

Of course, if you get the code and run it on your own web site, then you can use CORS to limit where the data can be accessed and make it so only your web apps can read / write to your local database.

This is How We Create Simplicity

Please keep in mind that this entire idea is a novel idea and is meant as a way to:

Original Purpose:

Make it easy for a web developer to save and retrieve data in their app

All the work behind the scenes (building the Web API to do this) wasn't so easy, but now the Web App Developer really does have an easy way to save data via JavaScript Fetch.

See You Next Time

Next time, we will update our ImageCat app to store its data in the LibreStore so it can be retrieved whenever the app is run.

I can think of a few challenges we are going to encounter, but those will just lead to use learning more.👍🏽

Add the Ability to View Usage Stats

We will also add the ability to view the Usage stats so we can keep an eye on how the Web API is being used and ensure that it isn't being abused.

Later on, I will add the Owner table so that people can register the use of their MainTokens via email address which will keep their data from being deleted. In the future, I will have to ensure that data that is not being used (from people just trying out the service) will be removed periodically. More on that as we turn this into more of a true SaaS (Software As A Service).

Points of Interest

I have always wanted to include something under the Points of Interest but never really had anything much to add until now.

My Key Is Emojis

While testing, I discovered that I can use a MainToken.Key that is made up of emojis. This is very cool.

I ran a Fetch against my dev environment like the following:

fetch("https://localhost:7138/Data/SaveData?key=🤬🤖🤢🤫😊🥴😨🤭😁🤓&data=
       I'm trying to save emojis, but???")
  .then(response => response.json())
  .then(data => console.log(data));

It returned fine and I did a select against my local sqlite database on the MainToken table and saw the following:

Next, I did a Fetch call to GetData and saw the following:

I was able to successfully use emojis as the String Key and then retrieve using that same Key.

That just made me smile.

Notes about the Code

• I didn't show a lot of the code in this article because I was trying to keep the article shorter & the code is pretty basic.
• The DataController is basically where everything begins & ends in this. It's a big Controller that just let me get this thing going.
• I wanted it to be a nicer design and was moving toward a better DAL (data access layer) but in the end, it was kind of ugly. But, it works. And, it is easy enough to extend so I'm ok with it.
• You will find a couple of extra methods in the DataController which I didn't mention. One named GetAllTokens() can only be run if the user provides the correct password (which will match a SHA256 hash).

History

• 6^th January, 2022: First published

Note: Original fist image is royalty-free image from Pixabay (here).