DeveelDB - A Brief Introduction

Introduction

Data management is the key component for the success of software products and, at higher levels, for the enterprise resource planning. This problem has been faced by developers and engineers in earlier times of IT: the maintenance of a state for applications was firstly approached in the mid ages of the past century and was solved with the implementation of several solutions (database management systems). The growing adoption of these systems by organizations required the establishment of standards and communication protocols, helping developers to easily interface with systems dedicated to the storage of information.

The most important and most long-lived language for communication between the storage systems and application is SQL (Structured Query Language), which during the years have been revised officially (ISO ANSI standards) and non-officially (database vendors dialects), but always used as the base model for data structuration and management (Data Definition Language), modifying the contents of the database (Data Modification Language) or simply querying the data stored.

Database management systems (DBMS) have been deployed for years as complex systems over internal or public networks, to be consumed by applications in a determined enterprise-level context: the introduction of reduced versions of these systems have modified the approach that developers and projects have taken during the architecture and implementation of applications.

This article will describe the functionalities of DeveelDB, a fully-functional embedded relational database management system, based on the ANSI/ISO SQL model, designed for the adoption by .NET/Mono applications aiming to store and manage large amount of data in a secure and robust way.
Several solutions already present for the .NET and Mono frameworks (eg. MySQL, SQLite, MSSQL) suffer from the difficult portability of the libraries and performances: actually the original code of the applications is not managed and the bindings produced to interface with the underlying functions require the marshaling of the data passed during the conversion from managed types to unmanaged. Furthermore, the limitations given by the nature of interaction between managed and unmanaged code are easily imaginable.
DeveelDB aims to provide developers with a robust management system which is natively written in managed code and run into the same environment as the applications consuming it.

The current development of servers and services to dispatch the data management functionalities of the project to a wider audience will probably spread the original definition provided above, but won't restrict it (since the embedded support will always be assured).

DeveelDB is the result of a collaboration with McKoi team: actually a considerable portion of the code of the project was ported from the first version of the Java database management system.

Using the Code

Using DeveelDB in an embedded context is very easy and immediate, especially if you're already an expert SQL developer.

To start managing a database system within your application, you can start by simply referencing the namespace Deveel.Data.Control, contained into the DeveelDB library (libdeveeldb.dll): this contains two small, but very important, classes DbController and DbSystem (don't confuse this with DatabaseSystem present in the parent Deveel.Data namespace), which are used to start interacting (whether creating or opening) with a database.

The class DbSystem, which is returned by a call to StartDatabase or CreateDatabase will provide the key method (overloaded) GetConnection, which will establish a new ADO.NET connection to the database, that you can use to pass SQL commands to be processed (whether creating database objects, querying data, or modifying the contents of databases: refer to SQL standard for a complete list of commands).

using System;
using System.Data;

using Deveel.Data.Control; 

namespace Foo.Bar {
    public class Program {
        [STAThread]
        public static int Main(string[] args) {
            // let's create a default configuration for the system...
            DbConfig config = DbConfig.Default;
            // ... now create a new database named "Test" and administrated by
            // "SA" identified by "123456" password ...
            DbSystem dbSys = DbController.Current.CreateDatabase
				(config, "Test", "SA", "123456");
            // ... and retrieve an IDbConnection instance to communicate to 
            // the newly created "Test" database
            IDbConnection conn = dbSys.GetConnection("SA", "123456");
            // at this time the connection is already opened and you must not 
            // call 'Open' method from IDbConnection...
            [...]
            
            // ... and close the interaction with the database system once
            // finished: this will also close all the connections opened.
            dbSys.Close();
            return 0;
        }
    }
}  

Creating a Connection using DeveelDbConnection

Although the use of DbController and DbSystem is highly encouraged to open ADO.NET connections, an alternative way to do this is by instantiating DeveelDbConnection class which can be found in Deveel.Data.Client within the libdeveeldb.dll): actually the IDbConnection instance returned by DbSystem is an instance of DeveelDbConnection which connects directly to the local database system, bypassing the connection string parsing and using the IDbConfig instance for configuring the connection.

using System;

using Deveel.Data.Client;

namespace Foo.Bar {
    public class Program {
        public static int Main(string[] args) {
            // create a connection to the database "Test" which will dictate
            // the system to create it for the administrator "SA" and return an
            // IDbConnection instance to interface the users with the newly created
            // database...
            DeveelDbConnection conn = new DeveelDbConnection
		("Host=Local;User=SA;Password=123456;Create=True;Database=Test");
            conn.Open();
            
            [...]
            
            conn.Close();
            return 0;
        }
    }
} 

Transactional System

Note that DeveelDB is a natively transactional system: which means that you don't need to begin a transaction after a connection is opened to start a transactional executing context. Anyway, to accomplish with the ADO.NET specifications, the BeginTransaction method defined in IDbConnection will return an instance of IDbTransaction which will be registered and prevent the creation of further shadow transaction objects: calling Commit or Rollback methods from this object will act as a callback to Commit or Rollback methods into DeveelDbConnection class.

When a COMMIT command is issued to the system, all the modifications done to the database are flushed and the state of the transaction is reset to begin journaling the modifications until the next commit or rollback.
The same logic is used for ROLLBACK which will void any modifications made to the database until the issue of this command.

When a connection modified the state of a database and the COMMIT command is not explicitly issued before closing, the connection will rollback any modification.

Large Amount of Data Managed

DeveelDB supports the management of large objects of binary format (BLOB) and strings (CLOB). The nature of the system is to be employed within embedded systems and applications, which don't require handling a large amount of data, such as in enterprise contexts (think about indexing a memo of 120 pages in Word format): the reason to implement the support for *LOBs is the forecast of the creation of a wider infrastructure to dispatch the storage system within private and public networks.

At the present moment, each database created with DeveelDB will implement a sub-storage for handling large objects in the context of tables.

Pluggable Architecture

The whole architecture of DeveelDB is defined pluggable: this means that, with a small development effort, you can implement your own handlers for every component of the project.
An example of this open architecture is the storage system: currently the built-in types of storage systems are HEAP and FILE storages, where the first one is used to keep in memory a database and discard every structure and data at Garbage Collection, while the second one handles a journaled system which handles the state of a database into files.

If you need to implement your own storage system (e.g. a network based storage, a one-file storage system, etc.), you can implement the interfaces provided in the Deveel.Data.Store namespace and configure the system to use it instead of a built-in one.

For a complete guide on the storage implementation, please refer to the DeveelDB Wiki site.

(This paradigm can also be applied to several other components in the system: functions, fsync, diagnostics, regular expressions processors, etc.)

Open Source

Deveel Relational Database Management System (DeveelDB) is released under the open source license GPLv3: this is one of the more restrictive licenses around and probably the most radical. Unfortunately, since a large portion of the code of DeveelDB was ported by another open-source project, we were forced to apply the license to the rest of the code.

A considerable amount of members of the open source community complain about the restrictiveness of the GPL license and I personally stand on this position: as said, the choice was forced.

Although in the current situation, it is possible to indirectly link the project within other open source projects, without violating the license, with a code as the following one:

using System;

namespace Foo.Bar {
    public class Program {
        public static int Main(string[] args) {
            string connTypeString = "Deveel.Data.Client.DeveelDbConnection, libdeveeldb";
            Type connType = Type.GetType(connTypeString, true, true);
            IDbConnection conn = (IDbConnection)Activator.CreateInstance(connType);

            [...]

            conn.Close();
        }
    }
} 

An LGPL version of the client is under development: this will permit the use of DeveelDB in more complex contexts, such as distributed networks where applications are developed and released under different licenses.

The use of DeveelDB in commercial projects is subjected to agreements between the user and Deveel.

Points of Interest

The DeveelDB system is said to be robust for data management and this is given by the following features already built-in:

• Natively transactional system: By nature, the system encapsulates connections into Serializable kind of transactions; this means that every modification of the structure or the contents of a database are registered and will be flushed into the database only at COMMIT or discarded at ROLLBACK.
  This key component of the system prevents unwanted and unprivileged modifications to the database, which could corrupt the quality of data stored.
• Large amount of data: Although DeveelDB was born to be used within embedded contexts, where the amount of data managed is small, it was also designed to manage very large amount of data (BLOBs and CLOBs), with an optimized access level to stream in and out the system binary and textual amount of data.
• ADO.NET Interface: The implementation of a standard ADO.NET client, an open interface for database communication, eases the use of the system to developers with very small knowledge requirements, in terms of code.
• Pluggable architecture: Every component of the system is designed to be extended by developers to fulfill even advanced requirements.
• Open Source: The open-source license under which the project is released ensures users of the quality of the code.

Further Reading

For further reading about the functionalities, configuration and consuming DeveelDB, please refer to the website.

History

• 22^nd October, 2009: Initial post