This article documents the Cloud Over IFileSystemInfo Extension 3 (COFE3), a framework library for designing RESTful home cloud services using FSI like interface.
Introduction
Cofe3 is a set of libraries that enable developers to build their home cloud services for their specific kind of entries (e.g., IFileSystemInfo/CofeIO is just an implementation), which enables users to host them on their own computer, and using it to serve their own thin client computer like smart phone and tablet via the web browser or
apps. This library is released under the LGPL license.
Cofe3 is a developer friendly implementation, the Core library is similar to
System.IO.FileSystemInfo, except the entry is now an interface called
Cofe.Core.ICofeSystemInfo (entries). Developers can choose to develop their own entries (derived from
ICofeSystemInfo), or they can add new properties or actions to existing entries (e.g.,
ExifProperties to JPG IFileInfo), either way will make them
storable in the database, searchable and accessible on the web, like the IFileSystemInfo in the Cofe.IO library. But please keep in mind that behind the scenes,
the implementation of Cofe3 is much different from System.IO, one should not think
about doing a custom implementation as effortless as inheriting a class.
CofeWs3 uses ASP.NET MVC3 to host the server, which provides Authentication support, accessing the API requires at least a user ([Authorize] tag), and there are volume permissions and user role checks on the Cofe side. The configuration is and will be developed as
an MVC web page, but accessing
the CofeSystem will not. The communication between CofeWs3 and the user thin client is
a REST based API, which is mainly through JSON, and Atom10 for events, instead of ViewBag. This allows third-party
apps to be developed.
CofeDb3 is Entity Framework Code First based, this allows third-parties to plug in their entry table for CofeSystemInfos;
the developer can choose to save properties in a PropertyPairs table (e.g.,
PropertyPair_Int32 for int values) as well. Regardless
of which way we cache data, both ways allow the user to
search their entire collection using a FilterString, which is similar to searching
a string in a search engine.
Background
This project has went through a number of iterations.
What Cofe1 does was to mash archives (supported by Cake and then SevenZipSharp) to
a file system (see DirectoryInfoEx), which allows DirectoryInfo like browsing for archives; it's one of the major parts of my zip-ware program. Then after I completed FileExplorer2, I started working on Cofe2, which is actually my dissertation project. The earlier version of Cofe2
was very primitive, it used hard coded SQL to communicate with the database, and my own JSON to XML (and vice versa) converter. The final version of Cofe2 uses Entity framework model first, supports files only, with only text based security support.
The main reason for scrapping Cofe2 for Cofe3 is to use the latest framework to do something that I couldn't, namely the security in MVC3 and non-hardcoded-entries using EF Code First, to support actions/tasks as properties, which allows calling in WebAPI, using GET (e.g.,
CofeDirectoryProperties.List), or DELETE (e.g., CofeProperties.DeleteAsync) action, and supports non-file entries through Code First.
I wanted to emphasis that since the earlier stage of Cofe2, the project is released under LGPL license on CodePlex. Home cloud storage does access user's most sensitive data, so my initial thought is that it has to give confidence, confidence for the user to know what they are running, and it shouldn't be used to take unfair advantage of the user.
I have tries to make the web service as secure as I can, but as I am not an expert on this, there may be some insecure code, please report
them to me if you find any.
Uncompleted items
Even after long development, there's still some important but uncompleted items, here's a list:
Separate database updating code in Web.Services to a Windows Service - important
Currently, Cofe.Web.Services creates three threads when started, two ActiveUpdater for updating the expired entry and listing in
the database, and one+ event source to listen
System.IO.FileSystemEvents and trigger Cofe.Core.CofeSystemEvents. The problem is that when ASP.NET refreshes the AppDomain, for various reasons, it destroys my threads. To solve the problem I will need either to separate the database updating code and make it
a Windows Service or Console Application, or use the WebAPI Self-Host.
Fix the WinRT implementation
The earlier build of Cofe had WinRT support in Cofe.Core and Cofe.IO (no EntityFramework.RT, so no Cofe.Data.RT), but I have stopped updating
it after working on WebServices. Currently the build is not compilable. Hopefully I can make both libraries work again, and add
the Cofe.Media.RT library.
Improve HTML5 client / Add WPF client
Currently the HTML5 client does not automatically update when there's a change in
the cofe system. It can receive the cofe system change events from feeds, but it doesn't update the web
UI.
Script support for database update and entry actions
Cofe does provide IScriptServices to parse and run a script, but its only use is creating
an events feed in CofeWs. I am looking to extend it to support the CofeSystem.
Index
This article is divided into two parts, the first will be a quick look for the features Cofe3 provides
and how to use it.
Part II will try to explain briefly how it works internally.
Part I
Click here to go to Part II
Libraries
- Cofe.Core - Main library, provides all services so CofeSystem can work, it also includes the root interface for all entries (
ICofeSystemInfo), and several other interfaces (ICofeItemInfo,
ICofeDirectoryInfo). - Cofe.IO -
System.IO implementation for Cofe, added support to
IFileSystemInfo. - Cofe.Media - Media implementation for Cofe, currently only parses
EXIF tags from JPG images, provides
IMediaInfo and IImageInfo interfaces to any entry
that has a name ending with jpg and supports
CofeStreamProperties.OpenStreamAsync. - Cofe.Data - Caches entries and their properties in the database, translates search string (
EntryFiltersString) to SQL statement to do the query.
- Cofe.Web - Provides the implementation of Web Services, including event updating
and entry formatting.
- Cofe.Web.Services - ASP.NET application, includes the APIController
to provide the Web API and the JavaScript to access the API.
Cofe as a general library
LinqPad
There's a number of LINQ files in \doc\LinqPad\Tutorial for demo, which I mainly use for debugging and testing.
The database related
LINQ files complain about migration, it's caused by Cofe.Media not linked in those LINQ
files, run LINQ again to fix the problem.
Configuration
There are two things that you have to configure
in Cofe3 before using, one is determining which library to use, and then registering the volume.
Register Library
To register a library, we can use ModuleBootStrapper. The library contains a
RegisterModule class in its namespace; for example, Cofe.IO.RegisterModule in
Cofe.IO, you have to include all RegisterModules in the constructor of
MofuleBootStrapper, e.g. (the sequence of registerModule doesn't matter):
new ModuleBootStrapper(true,
new Cofe.Core.RegisterModule(),
new Cofe.IO.RegisterModule()
).Run();
The above examples include only Cofe and IO, this is minimal for Cofe3 to work; you can register
the volume provided by IO, but there will be no database support in this case, searching will be done by iterating sub-directories.
new ModuleBootStrapper(true,
new Cofe.Core.RegisterModule(),
new Cofe.Data.RegisterModule(true, CacheUpdateOptions.Manual),
new Cofe.IO.RegisterModule(),
new Cofe.Media.RegisterModule()
).Run();
The above examples added Data and Media, you can now "cast" files as
IImageInfo using the .As<I>() method, and the database support is now included.
Notice that CacheUpdateOptions is set to Manual.
CacheUpdateOptions
- Manual (Test) - User has to manually call
DatabaseProperties.WriteRepositoryAsync to update the database. - Passive (Desktop) - Update automatically when the user parses or lists an entry.
- Active (WebServices) - Create background threads to look for the expired entry or listing and updates in background.
Register Volume
Volumes are provided by registered IVolumeFactorys. Currently
the only usable one is CofeIO provided by Cofe.IO.
To register a CofeIO volume, you have to specify the VolumePath parameter, which is the root directory for
CofeIO to use (you can map multiple volumes at a time):
await VolumeFE.MapAsync("CofeIO", "cofe", Tuple.Create("VolumePath", "c:\\cofe"));
This will give all users permission to CRUD the directory; for a web user, you should specify the permission, e.g.:
VolumeFE.Unmap("cofe");
await VolumeFE.MapAsync(PermissionType.None, "CofeIO", "cofe", Tuple.Create("VolumePath", "c:\\cofe"));
VolumeFE.GrantPermission(new string[] { "Admin" }, "cofe", PermissionType.All);
VolumeFE.Factories displays a list of registered factories, while
VolumeFE.Volumes returns a list of registered volumes.
This can be annoying to setup volume and permission each time when the application starts, so one can use
the CofeSettings class when you update the volume and permission,
it updates CofeServices.CofeSettings.Permissions/Volumes as well, and these settings can be saved to
an external file (default external storage). You can then restore it when the application is restarted.
await CofeServices.SaveAsync();
await CofeServices.RestoreAsync();
Helper classes
There's a number of useful methods in different classes, which can be hard to remember. Cofe3 provide a number of static helper classes, which called those frequently use methods. These helper class all ends with FE.
- Cofe.Core.CofeServices - CofeServices is a central place to access all services in Cofe.
ICofeServices is implemented by all services, so developers can call
CofeServices.ServicesLocater.FindServices<>() to find the desired service interface. Some of the services included ServicesLocater, EntryConstructor, PropertyDictionary, PathParser, and EntrySerializer. - Cofe.Core.PathFE - Some path related utilities. Cofe path contains {} letters, which made it incompatible with
System.IO.Path (equivalent to System.IO.Path).
- Cofe.Core.EntryFE - Entry (
ICofeSystemInfo) related helper functions, include path parsing (TryParsePath/Exists), Transfer (Move/Copy/Rename/LinkAsync) and Search (SearchAsync) - Cofe.Core.VolumeFE - List and register volume, assign permission to roles (equivalent to
System.IO.Drive). - Cofe.Core.UserFE - Assign role to user.
- Cofe.Core.EntryListFE - EntryList related helper functions, includes path parsing and EntryList constructing.
- Cofe.Core.LinkFE - Get or Set entry links.
- Cofe.IO.FileFE - File (
IFileInfo) related helper functions (equivalent to
System.IO.File). - Cofe.IO.DirectoryFE - Directory (
IDirectoryInfo) related helper functions (equivalent to
System.Directory).
Entries
Although the smallest piece of data in the Cofe system is a property pair, an entry is the business entity inside
the Cofe system. Dependent
on what you are developing, it can represent an IO item (file/directory), a contact of a person, or perhaps even a section of a
Word document.
Once you have registered a volume in a previous section, it's in the Cofe system and you can parse it using the path:
IDirectoryInfo rootDirectory = rootEntry.As<IDirectoryInfo>();
rootDirectory = await EntryFE.TryParsePathAsync<IDirectoryInfo>("{cofe}");
rootDirectory = DirectoryFE.FromDirectoryPathAsync("{cofe}");
IFileInfo file1 = await FileFE.FromFilePathAsync("{cofe}\\testParse1.txt");
Console.WriteLine(file1.Length);
If an entry is parsed from EntryFE.TryParsePathAsync(), which returns
ICofeSystemInfo, rootEntry is not necessary already implementing the
IDirectoryInfo class, so you have to call the cast As<T>() method, Cofe checks if the current implementation supports the requested interface, and re-creates the class that supports the interface if needed. You can use
the Is<T>() method to see if it can be "cast" as a specified type.
Cofe.IO.IDirectoryInfo inherits from ICofeDirectoryInfo/IFileSystemInfo and then
ICofeSystemInfo, while Cofe.IO.IFileInfo inherits from ICofeItemInfo/IFileSystemInfo, and then
ICofeSystemInfo, and System.IO has an implementation of
FileInfo and DirectoryInfo. So to allow one to access both
IDirectoryInfo and IFileInfo properties of an archive file,
we have to use the
As<T>() method to switch between both interfaces.
All interfaces implement ICofeSystemInfo. ICofeSystemInfo has some important properties,
Properties and
EntryTypeInfo.
The Properties property is a IPropertyHost, which is responsible for accessing all other properties (e.g., length, md5) and actions (e.g.,
DeleteAsync, List) of an entry. When you call DeleteAsync you actually get a
Func<pd, Task> from a set of PropertyProviders and execute it, this screenshot will give you some idea.
PropertyHost will be explained in the next article.
EntryTypeInfo is a property provided by IPropertyHost, it provides type information for the entry. In old days, file type information is not available in
FileSystemInfo, and because WinAPI is involved, it's very resource consuming to parse the type information of every entry, so type information like description and icon are stored in
an external dictionary. External memory based cache does work on a desktop application but not web services, I decided to include an implementation of
IEntryTypeInfo, which is cacheable in the database.
The binary digit (byte[]) in the above screenshot is for EntityFramework to cache the icon, its
IEntryTypeInfo.LargeIcon returns a BitmapContainer object, which contains
the
BitmapSource that allows you to use in WPF applications.
Directories
ICofeDirectoryInfo supports listing and creation of sub-entries. For Listing,
ICofeDirectoryInfoExtension provides a range of EnumerateCofeSystemInfos() and
GetCofeSystemInfosAsync() methods, which takes FilterCollectionss as parameters.
FilterCollection is used to specify what to return. Although we can use
FilterCollection.FromParseName(), FromName(), etc., to produce
a simple query, you can use multiple entry and option filters in a
FilterCollection. To create those filters, we can use the
FilterCollections.FromFilterString() method, which converts a filter string to
Entry/OptionFilter objects, for example:
var filterStrList = rootDir.EnumerateCofeSystemInfos(
FilterCollections.FromFilterString("filetype:txt size:>0").AndProperty(
CofeItemProperties.Length, "<100")).Select(e => e.ParseName);
For IDirectoryInfo, we can use the EnumerateFileSystemInfos() and
GetFileSystemInfosAsync() methods, which, like DirectoryInfo.EnumerateFileSystemInfos(), takes file name patterns and SearchOptions.
Searching
In the last section we used EntryFE.TryParsePathAsync() to
return an entry, used
Get/EnumerateCofe/FileSystemInfos to list sub-contents, with the use of
the database. We can skip the hierarchical lookup and search for an entry
easily. Again, FilterCollections is important for searching, all EntryFilters and
OptionFilters are converted to one EntryFilterExpression, which is an
Expression<Func<CofeSystemInfo, bool>>. It can then be used to query for the requested entry in Entity Framework Code First.
Search is done by using EntryFE.SearchAsync() with a FilterString:
IAutoEntryList el = await EntryFE.SearchAsync("name:testParse* filetype:txt size:<100 root:{cofe}");
foreach (ICofeSystemInfo e in await entryList.GetCofeSystemInfosAsync())
String.Format("{0} ({1})", e.ParseName, e.GetType().Name).Dump();
Organizing entries
The information described above mainly works with entries of the original hierarchy.
The user may want to have another way to categorize items. Currently Cofe3 does provide a number of ways to organize entries in CofeSystem, tags, links, and
a custom entry list.
Tags (Require DB)
Tags are hash tags applicable to any database entry, the user can use it to categorize their entries. It requires
a database because the implementation is a one-to-many table that links from items to tags.
Once an entry is in the database, you can call the entry's .AddTags(), RemoveTags(), and
GetTags() extension methods (in CofeDB3) to add, remove, and access tag information.
The tags parameters are comma separated, so you can add/remove multiple tags at a time.
Search is done by using the tags property, please note that comma is not supported at this time.
Links
IEntryLink enables any entry to link to another entry, by calling
the LinkFE.AddLinks(entry, entryToAddLink) helper method. It tells Cofe3 to create a new
IEntryLink entry that links from the first to the second entry. Below is how the database looks after the command is called.
The LinkFE.EnumerateLinks() method returns a list of entry links for a particular entry. They are all IEntryLinks so to get the actual entry you will have
to access its IEntryLink.Entry property, or you can call the IEntryLink.As<T>() method cast to
a particular class.
To remove a link, call the link's DeleteLink() method. If you call the DeleteAsync() method, it will delete your actual entry.
CustomEntryList
Another way to organize entries is CustomEntryList; like IAutoEntryList, it contains a number of entries, but CustomEntryList is not built from a search string. Users build the list themselves, by linking them.
The difference of CustomEntryList from other directories in Cofe is that, it's
a virtual item in the database or memory, and most importantly, its
ListCore method returns added links described in the last section, while other directories returns
their sub-items but not their links. Because these links can be cast as
ICofeDirectoryInfo using the .As<T>() method, a multi-level of
CustomEntryList can be used as a hierarchical directory structure, based on
the database.
The following examples demonstrate how to create a custom entry list as a volume, and how to add items and sub-CustomEntryList and change
the position of a particular item:
ICustomEntryList cel = EntryListFE.NewEntryList("cel");
cel = await cel.MakeParsbleAsync();
await cel.AddLinksAsync(file2add);
IEntryLink subdirLink = cel.CreateLinkFolder("sub");
ICustomEntryList subCel = subdirLink.As<ICustomEntryList>();
subdirLink.Position = 1; //Change position for an entry.
await subCel.AddLinksAsync(file2add);
It will look like this in the database (noted that those ID fields should be in GUID format):
EntryListFE.FromPathAsync() calls EntryFE.TryParsePath() with
a lookup parameter, can parse a custom entry list that made as volume:
Cofe as a web
service
Admin
There are two things you can admin right now, user roles and volume information. Editing them requires admin
permission, you have to create a user, then run Cofe.Web.SecurityConfig.RegisterAsAdmin("your-user-name"); in
ImmediateWindow.
User roles
Because the software is designed for home use, I have created a number of roles - Admin / Me / FamilyMember / Friend / Other.
The server address is {server}/admin/permission, you can change the role of any registered user except
the admin.
Volume
Volume is the VolumeInfo in the previous section, which defines the root directories. The page is just an online version of
the
VolumeFE.MapAsync() method, with permission support. Note that you can set different permissions for different roles.
RESTful APIs
Besides the IFileSystemInfo interfaces for desktops, Cofe3 can be accessed through RESTful APIs, these APIs are
not SOAP based, they use HTTP's GET/PUT/POST/DELETE actions to retrieve and manipulate the
Cofe system. The following is a brief list of the implemented APIs:
- /api/entry - Access properties, resources, and actions of an entry using GUID.
- /api/entryList - Construct a group of entries (not completed).
- /api/parse - Using a parse path, redirect to appropriate /api/entry path.
- /api/search - Search using filter string.
- /api/events - Return Cofe system changes using ATOM RSS feeds.
Entry APIController - /api/entry/
Entry API controller is the main way to access particular entries in Cofe3, their metadata (properties), resources, and actions; these category types are described in the following table:
One may wonder how Cofe3 distinguishes properties of different categories. Actually they are defined exactly in the property, for example,
CofeProperties.RefreshAsync has its alias set as refresh, and it's an action and web action. In other words, they are coded in a manner that you can change what to show in what category and what alias any time.
The following is a sample output of an /entry/{guid}, you can see the information
is formatted in JSON. Metadata is serialized as a JSON property, while Resource and Action are serialized inside the links array. The Links node allows
the user of JSON to discover the available services.
We should use the rel property to find the appropriate link of action. The
rel property in a link represents its relation with the current entry, except self, which represents the
URL to access the entry itself; other links are Resources and Actions, and rel is actually its alias. The mediaType property represents the mimetype of what to return. There's no indication for distinguishing Resources and Actions (former requires GET, while latter requires POST action), or what parameter is supported at this time.
The entry and its resource supports HTTP level cache, entry is based on the LastModified/IfModifiedSince HTTP header (which loads from
CacheProperties.LastCachedTimeUtc or LastListTimeUtc), while the resource is dependent on the properties'
WebResourceAttribute.CacheMethod (None/LastModified/MaxAge).
Delete - DELETE /api/entry/{guid}
This is not advertised in the entry's JSON, but you can use the DELETE action to remove an entry.
Create - PUT /api/entry
Create entry is done by using the PUT action with a query string or JSON data, the newly created entry will be returned. For file,
the user will then have to POST on its stream address (/api/entry/{guid}/stream) with the stream content.
Form based upload - POST /api/entry/{guid}/upload
Some may want to use form's file input to upload files, this is possible through POSTing
/api/entry/{guid-of-root}/upload or /api/parse?path={cofe}&rel=upload.
List - GET /api/entry/{guid}/List
The List resource returns its sub-contents, they are in JSON too, with all entries inside the
entries array. To reduce bandwidth, not all metadata/resource/actions are displayed in this entry,
you have to call its self link for all of them.
EntryType - GET /api/entry/{guid}/typeInfo
All entries that have type info will have its TypeInfo link exposed. TypeInfo are properties that can be shared with many entries, like its
MIME and file type shown below, but most importantly its icon resource. You can access its icon in various sizes using the appropriate link, because the icon
URL of the same type of file is the same, this allows cached icons (on browsers) to be reused.
Update Tag - POST /api/entry/{guid}/addtag
This is the same command you can find in a section earlier this article, the
AddTag() method calls UserFeedbackProperties.AddTag, with a
TagNames tag. In the web services, you can call the AddTag properties directly, and the
TagNames parameter is embedded in the query string. Because it updates the
Cofe system,
UserFeedbackProperties.AddTag is a WebAction and POST is required, and the updated entry is returned. The RemoveTag command is similar.
You can also replace the Tags property using - POST /entry/{guid} with just the tags value in it, CofeWS will try to update every property in your upload except
ID and parsename.
EntryList APIController - /api/entryList
Execute action - POST /entryList/{action}
Most of the time we have to handle multiple entries instead of one. Instead
of running a request for each entry, we can group them in a JSON entry list with just IDs (other properties are discarded), and POST. The screenshot above actually uses
/entryList instead of /entry.
Get Properties - GET /entryList
Similarly, if you want to get properties of multiple entries, you can create a JSON entry list with
IDs, the web services will return the completed entry.
Update properties - POST /entryList
Update properties is done by POSTing a JSON entry list with IDs and properties you want to change.
Parse APIController - /api/parse
Using EntryApiController can handle most of the work of an entry, but to access any entry
we must have its GUID. ParseApiController takes a ParseName and redirects to the respective location in
the EntryApiController. To reduce complexity, parameters are attached in the query string:
- path - The entry path to the entry (e.g., path={cofe}\testParse1.txt)
- rel - If you want to call a particular resource or action, you can specify the rel (relation) to that entry, the list of
rels can be found in the
entry's links.
Please note that only GET is fully supported and redirects to the appropriate URL in EntryAPIController. While POST may work, it invokes the resource or action directly in
the
ParseAPIController.
Search APIController - /api/search
All metadata of an entry is stored in a database and is searchable, and the easiest way to search it is to use
a search string. Searching the
APIController takes the
filterStr parameter and returns a list of entries that match the criteria.
Except a few, most properties are supported through a PropertyEntryFilter, for those properties you have to use its alias (the property without alias, like file attributes shown below, does not work).
You can use the paginate (page) filter to control how many to show, SortResult (sort) to control how to order your list, and
searchOption (subdir) filter to control whether to search a subdirectory.
Events APIController - GET /api/events
When there's a change in the Cofe system, it generates events, which is then transported to
an EventHub and then to the feed
writer in Cofe3.Web. The feed writing code is taken from the book REST in
practice, but I have rewritten the code a bit to fit the existing classes in Cofe3.
The web service has a thread to update the feed once in a while, and when a feed is full,
a new one is created. When you call /api/events, you are redirected to the latest feed in memory, or you can call with
a query string page={page#} to specify the page of events you want to show.
Explore
The web interface of the explore page (/explore) is fairly basic, I am not a web designer so this is what I could create in a limited time.
It's divided into several sections:
- Directory Tree - which shows subdirectories, automatically updates when
a directory changed.
- Entry List - displays all entries in a directory or search, shows 20 items per time, paginate refresh is activated when user scrolls to bottom. Upload files using drag and drop.
- Search - allows user to input a filter string, web interface calls
SearchAPIController, and returns result. - Metadata - when an item is selected, display all metadata found in the entry, otherwise display "x items selected".
If the selected item is a photo with a GeoTag EXIFf data, a
Google map is displayed with the location marked. While the metadata is
generated from JSON, the actions are hard-coded.
The underlying components are written with SpineJS, CoffeeScript, and jQueryUI, they talk with the server only with the WebAPI described in
the above sections.
Conclusion
It may be one of the dumbest projects ever imagined, to create a web service over the web to serve all personal data, in a world where online companies provide
APIs for everyone to store and process all types of data, with more space than your local hard drive, more fast, for free. But I think, no matter how
non-evil a company is, at the end of the day, whether and how to use your personal data, whether to continue
to provide access to certain services or API, how long they keep the data after deletion
is requested, and most importantly, how often they change the license agreement, is a decision of business, not
always ethical.
Many factors drove me to switch the development from Cofe1, a pure desktop component, to Cofe3, a desktop serving as
a web service. Among them, the interesting book about RESTful web services, the availability of new technologies like WCF, MVCWebApi, and EntityFramework, the dissertation project, and the change of atmosphere of the online environment
were the main reasons.
I never imagined Cofe will become a web service when I was developing Cofe1 and DirectoryInfoEx. This took a long time,
and throughout the process there was a lot of new technologies learned, and I enjoyed
all that. I hope development can continue and those who need such services can benefit.
Although my project is not as feature rich, does not have as good performance, and
is even not as stable and may not even be as secure as the online options, hopefully after reading this article, I can convince you that your own home web service can do much more than some kind of file serving server.
References
