Complex File Systems in your Application

Content storage seems like such a simple thing.

Unfortunately that assumption has led many individuals and companies down a dangerous path of poorly utilized content. And when the need arises to make that content flexible and integrate with other applications, the complexity of building one’s own cloud storage can be quite daunting. There are content management solutions built to help organizations with this problem, but they are often heavy, expensive, hard to implement and rarely adopted.

The modern approach is with cloud solutions and the next evolution of electronic file systems is platforms that are vigorous enough to integrate with anything.

Solutions that have robust enterprise-level content features, but take the end user’s need for simplicity seriously. Solutions that are not only useful for business, but also as a component in modern consumer mobile and web applications. Solutions with powerful functionality that you do not need to write, and flexible enough to integrate with any application via an API.

Bitcasa offers just such an API.

My Experience with Bitcasa CloudFS APIs

Bitcasa has done something very unique: they took a consumer product and turned it into an enterprise platform. They took what they know best about content storage, turned it into an API anyone can use, and emerged with a file system platform that can integrate into any application - whether for consumer or enterprise use.

Content storage is not just about serializing files to a drive. It’s about creating proper versioning, management, metadata, and disposition functionality, on top of the file storage. This is all stuff that you could write on your own, but you would be wasting your time if you did and it’s certainly not as easy as you think.

I recently had a chance to spend time with the Bitcasa CloudFS APIs, but before I share more about my experience I first want to tell you a bit about my background.

I am a coder turned enterprise process automation specialist. I have a deep background in enterprise information management and this is important because my enterprise focus, and knowing the downstream processes that file storage influences, influences my position on Bitcasa API functionality.

The application I used to test the API was a small content syndication web application. My goal was to create an easy-to-use publishing application that would enable you to publish individual documents to a team of field staff and external contractors. A real-life scenario I’ve faced several times in my industry.

In addition to basic publishing, the app also needs to be able to manage and retract content. It’s very important you know what version is out in the wild. The process is simple: upload, publish, and track.

I initially wanted the application to be a .NET one, but there is only one .NET SDK and it uses an old version of the API. I didn’t have time to create my own, and am also quite comfortable developing in PHP, so I decided to go with a more comprehensive Bitcasa SDK for PHP to write my application.

Getting started was a little rough for two reasons.

1. The PHP SDK had a bug. It took some time to find it, but when we did the Bitcasa team fixed it within two days, which was very impressive.
2. Another small bump I had to get over was the authentication model. The documentation on this is concrete, but it doesn’t exactly explain why. I finally pieced it all together and it makes more sense, but to save you some time when you’re trying yourself here are two things that you might not expect:
   1. The first thing is you can only access individual user content via the API. I assume this is to protect user content - even from even the vendor (a common issue/question many organizations have with cloud providers).
   2. The second is how users are set up. You have a master account login that tracks your API, storage, and bandwidth. It also provides you your endpoint for your code, but doesn’t give you application users. You can create test users, but for a full-blown application you are expected to take on the user management and authenticate them via OAuth. If you think about it, this makes sense. If you are building an application you already have users, and for security’s sake it is these users that you would want to authenticate with.

Once I wrapped my brain around these points and got going, everything else was pretty fast.

My application is basically made up of an upload form and then a "send-to location" for once the upload is complete. Once uploaded, the file is sent to the logged-in user’s storage location. A share is created for which I get a shortened URL that is used to publish out to external users. The application also keeps track of version updates and always provides the latest version to the user. It also has the ability to retract versions.

Observations and Suggestions

Here are some of the key points I found when working with the Bitcasa CloudFS APIs.

1. Once I understood everything the code was pretty straight forward, although there is some terminology unique to Bitcasa that I had to learn. For example "endpoint" was not quite what I expected it to be.
2. Sharing was a very important and nice feature and definitely not one that I’d want to write on my own. Since the links and the share/file locations can spider out of control quickly with larger sets of documents, the ability to create shares, delete shares, and have direct URLs is great.
3. Versioning is fantastic. Versioning is something that organizations don’t always take seriously. But in a publishing scenario like this, having the wrong version out there is a legal risk. I also liked that the API handled version conflicts. However, there are a few ways that versioning could be improved (which I outline below).
4. The fact that Bitcasa provides a trash bin is very nice. For me the trash bin was more than just a container to throw away stuff and avoid ‘oops’ moments. It was a retention tool. For example, in my application the retract function actually trashed that version of the document, which helped avoid duplicates after the document was retracted. I can then throw away versions and have a set period of 30, 60, or 90 days that they are kept.
5. The PHP SDK still needs a bit of work. Some of the implementations of the functions such as versioning are not quite as expected.

Here are some things I would like to see in future versions:

1. Custom metadata values. The APIs have metadata functionality, but I want to take it further and add things like terms, functions etc. The way it works today I would have to store this in a separate place if I needed it for my application.
2. At least early on I would like the premium accounts to have the ability to create test users and the ability to easily navigate and see the test user accounts file system. For me this was a faster validation than assuming there is a bug in my code or checking the storage usage and doing the math.
3. The ability to lock documents would be great so that no new version can be added. Even if it’s for an individual user who owns the documents and doesn’t want to accidently supplant an approved version of a document with an unapproved one.
4. It would be nice to have a way to global purge the version history. There is a feature where you can limit version depth. It requires more code, and I do not like to limit versions. But it is still a good alternative to purging.
5. Analytics. I would love to see who accessed files and where etc. This does not seem to fit well in the current information architecture of the API, but it sure would be nice. One of the use cases where this would be helpful is when I might want to block access to shares from certain regions etc.

Summary

I certainly could take this application further. And it would be more powerful if the consumption end were a mobile application instead of a web one. The use case might be field guides or pricing sheets for partners.

My application was highly business focused, but it does not take much imagination to picture consumer productivity applications such as a family recipe app, a family-critical documents app, or basically any use case where you need to store documents and share them with family members.

Though it took me a while to get started, the effort in no way compares to the amount of time it would have taken me to create my own cloud file system. Creating functionality like sharing and versioning, but also maintaining and creating the backend to make sure it is scalable, stable and secure would have taken a huge effort.

Bitcasa has already done all this work for me at a massive scale and turned it into a full-fledged enterprise-level platform. I’m very excited about how this will open the door to more powerful content driven applications. But don’t just take my word for it; give the Bitcasa CloudFS APIs a try for yourself.