Introduction
Object Relational Mapping is the process of mapping data types between an object-oriented language as C++ to a relational type system as SQL. So what is the challenge? C++ has different types of primitive types such as int, char, float, double and variations of that. So it's a real challenge to map all these to an actual SQL type. There may or may not be an exact type that is similar to the C++ types. Say for float, C++ and SQL may support different kind of standards too. So there are different tools to do this job. There are a lot of matured libraries too out there in the market. ODB is one that is really nice.
To help me in my daily work, I have created a simple C++ library called as Bun.
What's New?
- * Bun 1.5.0 Convert an object with vector to JSON and Msgpack and create an object from a JSON containing a vector. NOTE: It does not include vector persistence yet. Its still in development.
- Bun 1.4.0 Has support for converting objects to JSON and create Objects from JSON. It has the capabality to convert Objects to Message pack and construct object from message pack.
- Bun 1.3 Has support for Object lazy iteration and ranges based for loop support. Same is supported for the key-value store too.
- Bun 1.2 has support for embedded key-value store. But default, the key-value store is based on Unqlite.
Features
Easy to Use- Work with Plain Old C++ Objects (
POCO) - Object persistence - You can persist
C++ objects directly Not intrusive - You do not have to modify the classes to make it persistent- Constraint Specification in plain C++
- Persist
Nested Objects EDSL Object Query Language (No SQL Query needed)Compile time EDSL syntax check for type safety - Catch bugs before the execution starts- Multiple database support - SQLite, Postgres, MySQL
- Easy to use
embedded key-value store - Convert C++ objects
to JSON and create C++ objects from JSON. - Convert C++ objects
to Message Pack and create C++ objects from Message Pack. STL Friendly. These are regular C++ objects. So can be used in C++ STL algorithms.
Who is using Bun?
This section describes who all are using Bun and in what context. If you find Bun useful and use it please let me know I will add it here.
- A little adventure with PI.
Background
In a lot of my tools application, I use SQLite as the primary db. Every time I use SQL queries, I feel like wasting a lot of energy in the task that is not really related to my actual use case. So I thought of creating a framework for the automated mapping of these types. The criteria for the library is as follows:
- Free to use for any kind of project (BSD License)
- Easy to use (no SQL query knowledge needed)
- Provide constraint like unique key constraints for fields
- No SQL queries needed. EDSL query.
- Non intrusive
- Expressive
- Should be DSL for C++ so the queries syntax can be checked by the C++ compiler
- No customization compiler needed (C++11 and above)
- Performant
- Support for multiple database backends like SQLite, Postgres, MySQL
- Easy embedded key-value Store
All of these haven't been met till now. Going on eventually, I will be addressing all these issues. Currently, only a basic version of the library has been developed.
Using the Code
Bun Object Store Interface
Before we get into the gory details of the internals, in this first article, let's see how to use the library.
Bun has a BSD 3-Clause License. It depends on the following opensource and free libraries:
- boost (I have tested on 1.61 version, Boost License)
- fmt (Small, safe and fast formatting library, BSD License)
- spdlog (Fast C++ logging, MIT License)
- SQLite (Self-contained, serverless, zero-configuration, transactional SQL database engine, Public domain)
- SOCI (C++ database layer, BSL License)
- JSON for modern C++ (C++ JSON and Message pack utility, MIT License)
- Rapid JSON (Fast C++ JSON library, See License)
The GitHub page contains all the dependencies needed. It contains a Visual Studio 2015 solution file also for ease of use. Boost and SOCI are not included. To download the project, put the boost headers under the "include" directory or change the solution file path in the solution file. Build SOCI (very easy to build using cmake) and link the libraries with Bun.
#include "blib/bun/bun.hpp"
namespace test {
struct Person {
std::string name;
std::string uname;
int age;
float height;
};
}
struct Child {
int cf1;
Child(const int cf = -1) : cf1(cf) {}
Child& operator=(const int i) {
cf1 = i;
return *this;
}
};
struct Parent {
int f1;
std::string f2;
Child f3;
Parent() :f1(-1), f2("-1"), f3(-1) {}
};
SPECIALIZE_BUN_HELPER((Child, cf1));
SPECIALIZE_BUN_HELPER((Parent, f1, f2, f3));
SPECIALIZE_BUN_HELPER( (test::Person, name, uname, age, height) );
int main() {
namespace bun = blib::bun;
namespace query = blib::bun::query;
bun::connect("postgresql://localhost/postgres?user=postgres&password=postgres");
using PersonFields = query::F<test::Person>;
blib::bun::Configuration<test::Person> person_config;
person_config.set(PersonFields::name = blib::bun::unique_constraint)
(PersonFields::uname = blib::bun::unique_constraint);
bun::createSchema<test::Person>();
bun::Transaction t;
for (int i = 1; i < 1000; ++i) {
bun::PRef<test::Person> p = new test::Person;
p->age = i + 10;
p->height = 5.6;
p->name = fmt::format( "Brainless_{}", i );
const bun::SimpleOID oid = p.persist();
bun::PRef<test::Person> p1( oid );
}
t.commit();
const auto person_oids = bun::getAllOids<test::Person>();
const auto person_objs = bun::getAllObjects<test::Person>();
using FromPerson = query::From<test::Person>;
FromPerson fromPerson;
auto valid_query = PersonFields::age > 10 && PersonFields::name != "Brainless_0";
std::cout << "Valid Grammar?: " << query::IsValidQuery<decltype(valid_query)>::value << std::endl;
auto invalid_query = PersonFields::age + 10 &&
PersonFields::name != "Brainless_0";
std::cout << "Valid Grammar?: " <<
query::IsValidQuery<decltype(invalid_query)>::value << std::endl;
const auto objs = fromPerson.where( valid_query ).where( valid_query ).objects();
const auto q = PersonFields::age > 21 && PersonFields::name == "test";
const auto objs_again = FromPerson().where( q ).objects();
const auto objs_again_q = FromPerson().where( PersonFields::age > 21
&& PersonFields::name == "test" ).objects()
std::cout << fromPerson.query() << std::endl;
blib::bun::createSchema<Child>();
blib::bun::createSchema<Parent>();
std::cout << "How many objects to insert? " << std::endl;
int count = 0;
std::cin >> count;
for (int i = 0; i < count; ++i) {
blib::bun::l().info("===============Start===================");
blib::bun::PRef<Parent> p = new Parent;
p->f1 = i;
p->f2 = i % 2 ? "Delete Me" : "Do not Delete Me";
p->f3 = 10 * i;
p.persist();
std::cout << "Added to db: \n" << p.toJson() << std::endl;
blib::bun::l().info("===============End===================\n");
}
std::cout << "Get all objects and show" << std::endl;
auto parents = blib::bun::getAllObjects<Parent>();
for (auto p : parents) {
std::cout << p.toJson() << std::endl;
p.del();
}
return 0;
}
So this is how we persist the object. After running this, the following list is created in the SQLite database:
Now let's have a deeper look at few elements here. The DDL for the schema is as follows:
CREATE TABLE "test::Person" (object_id INTEGER NOT NULL, name TEXT, age INTEGER, height REAL);
This schema is created internally by the library. I am just showing it here for reference.
The data is as follows:
Persistent Store
| oid | name | age | height |
90023498019372 | Brainless_1 | 11 | 5.6 |
90023527619226 | Brainless_2 | 12 | 5.6 |
90023537497149 | Brainless_3 | 13 | 5.6 |
90023553459526 | Brainless_4 | 14 | 5.6 |
90023562946990 | Brainless_5 | 15 | 5.6 |
Range Based iteration
Bun also supports the iteration of objects using the range based for loop in C++. The following gives a simple example of how this is going to work.
using FromParents = query::From<Parent>;
using ParentFields = query::F<Parent>;
FromParents from_parents;
auto parents_where = from_parents.where(ParentFields::f2 == "Delete Me");
for(auto v : parents_where) {
std::cout << v.toJson() << std::endl;
}
JSON and Message pack conversion (To Object and From Object)
Now we can convert C++ objects to JSON and create C++ objects from JSON. We can even convert C++ objects to Message Pack and create C++ objects from message pack
Its very easy just specialize the bun helper then its a childs play.
namespace dbg {
struct C1 {
int c1;
C1() :c1(2) {}
};
struct C {
int c;
C1 c1;
C(const int i = 1) :c(i) {}
};
struct P {
std::string p;
C c;
P() :p("s1"), c(1) {}
};
}
SPECIALIZE_BUN_HELPER((dbg::C1, c1));
SPECIALIZE_BUN_HELPER((dbg::C, c, c1));
SPECIALIZE_BUN_HELPER((dbg::P, p, c));
int jsonTest() {
namespace bun = blib::bun;
blib::bun::PRef<dbg::P> p = new dbg::P;
p->p = "s11";
p->c.c = 10;
p->c.c1.c1 = 12;
blib::bun::PRef<dbg::C> c = new dbg::C;
c->c = 666;
const std::string json_string = p.toJson();
blib::bun::PRef<dbg::P> p1;
p1.fromJson(json_string);
const auto msgpack = p1.toMesssagepack();
blib::bun::PRef<dbg::P> p2;
p2.fromMessagepack(p1.toMesssagepack());
std::string msgpack_string;
for (auto c : msgpack) {
msgpack_string.push_back(c);
}
std::cout << "1. Original object Object:" << json_string << std::endl;
std::cout << "2. Object from JSON :" << p1.toJson() << std::endl;
std::cout << "3. Object to Messagepack :" << msgpack_string << std::endl;
std::cout << "4. Object from Messagepck:" << p2.toJson() << std::endl;
std::cout << "=== Vector JSON Conversion ===" << std::endl;
blib::bun::PRef<bakery::B> b = new bakery::B;
b->j = "test";
b->i.push_back(12);
b->i.push_back(23);
std::cout << "5. Object with Vector: " << b.toJson() << std::endl;
blib::bun::PRef<bakery::B> b1 = new bakery::B;
b1.fromJson(b.toJson());
std::cout << "6. Object copy with Vector: " << b1.toJson();
return 1;
}
Key Value Store
Bun has an embedded key-value store. The default implementation is based on Unqlite.
template<typename T = DBKVStoreUnqlite>
class KVDb {
public:
KVDb(std::string const& param);
KVDb(KVDb const& other);
~KVDb();
bool ok() const;
std::string last_status() const;
template<typename Key, typename Value>
bool put(Key const& key, Value const& value);
template<typename Key>
bool get(Key const& key, ByteVctorType& value);
template<typename Key, typename Value>
bool get(Key const& key, Value& value);
template<typename Key>
bool del(Key const& key);
};
Following is the way that we can use it:
int kvTest() {
blib::bun::KVDb<> db("kv.db");
db.put("test", "test");
std::string val;
db.get("test", val);
std::cout << val << std::endl;
const int size = 10000;
for (int i = 0; i < size; ++i) {
const std::string s = fmt::format("Value: {}", i);
db.put(i, s);
}
for (int i = 0; i < size; ++i) {
std::string val;
db.get(i, val);
std::cout << val << std::endl;
}
return 1;
}
Range based iteration of the Key values
Bun supports the range based iteration of the key values of the elements in the kv store. This iteration is like the iteration of maps. The key and value both are returned as a pair. If you see below kv is a pair, the kv.first carries the key value and the kv.second carries the value. The kv.first and kv.second has values as a vector of bytes.
blib::bun::KVDb<> db("kv.db");
const int size = 3;
for (int i = 0; i < size; ++i) {
const std::string s = fmt::format("storing number: {}", i);
db.put(i, s);
}
std::cout << "Start iteration Via size "<< std::endl;
for (int i = 0; i < size; ++i) {
std::string val;
db.get(i, val);
std::cout << val << std::endl;
}
std::cout << "Start iteration via foreach "<< std::endl;
count = 0;
for (auto kv : db) {
int key = 0;
blib::bun::from_byte_vec(kv.first, key);
std::string value;
blib::bun::from_byte_vec(kv.second, value);
std::cout << count++ << ")> key: "<< key << "\n Value: " << value << std::endl;
}
Internals
Some of the internals of the ORM are as follows.
Reflection
Bun internally uses simple reflection to generate take care of compile-time type information. There is a plan to extend it a little so it can be more useful.
SPECIALIZE_BUN_HELPER
This macro will generate all the binding for the objects at compile time. All the template specialization is created using this macro. It should be safe to use the macro in multiple headers or CPP files.
The following should be passed to the macro:
(<Class name, should include the namespace details too>, Members to persist ...)
The member list can be partial class members too. Say we have a handle on one of the objects we use, there is no point to store it in the DB. In this case, we can omit the handle and persist all the other features. This way, only the given fields will be populated.
Constraint
Applying constraint is easy in Bun. The following example explains it.
using PersonFields = query::F<test::Person>;
blib::bun::Configuration<test::Person> person_config;
person_config.set(PersonFields::name = blib::bun::unique_constraint)
(PersonFields::uname = blib::bun::unique_constraint);
As you can see its very easy to create unique constraints. As given above we can club together multiple constraints using the overloaded () operator rather than call set multiple times.
Things to remember:
- For now, constraints can be applied only before the table is created. The statements have no effect after the table is created.
- The only unique key is supported.
In further releases, I will be removing these limitations.
PRef
PRef is one of the central elements in the library. It holds the object that needs to be persisted. It also contains the oid of the object, which is independent of the actual object. Few rules to make an object persistent:
- The member that needs to be persisted has to be
public. PRef maintains the ownership of the object and deletes the object when it goes out of scope.- If we assign a
PRef to another, then PRef the former loses the ownership of the object. Just like a unique_ptr. Actually, PRef stores the object in a unique_ptr underneath. - Before persisting objects, we have to create the schema (using
blib::bun::createSchema<>()) and generate the bindings (using SPECIALIZE_BUN_HELPER( (test::Person, name, age, height) );) - It also contains the md5 sum of the object at a particular instance. So if there is no change in the object, then it won't persist it. I have it as in my own use, I keep a timestamp of the update. I do not want to update the object every time. For this public release, I am omitting the time stamp.
Insert or Update
How does the library know if we want to insert or update the database? This happens with the md5 of the object. If the md5 has some value, then it is an update else it's an insert. The following query is automatically generated for the insert:
INSERT INTO 'test::Person' (object_id,name,age,height) VALUES(91340162041484,'Brainless_4',14,5.6)
Search
Searching in Bun is quite easy. There are different mechanisms to search.
- Oid Search: We can get all the
Oids using the method:
const auto person_oids = blib::bun::getAllOids<test::Person>();
- Search all objects of a type: We can get all the objects in the database as a vector of objects:
const auto person_objs = blib::bun::getAllObjects<test::Person>();
- Object EDSL: We can search through the EDSL query that Bun provides. The EDSL is implemented using boost proto library. The query is checked in compile time by the C++ compiler. When
SPECIALIZE_BUN_HELPER is called, it creates some special variables.
For example: For the Person class, the SPECIALIZE_BUN_HELPER generates the following:
bun::query::F<test::Person>::name
bun::query::F<test::Person>::age
bun::query::F<test::Person>::heigh
The bun::query::F class of Bun will be specialized with all the fields of Person class.
To apply any kind of filters, you just need to use the "where" function like:
const auto objs_again = bun::query::From<test::Person>().where( valid_query ).objects();
const auto objs_again = bun::query::From<test::Person>().where( valid_query && valid_query ).objects();
Discussion Forums
- Gitter.im: here you can ask question for quicker answers or chat with us if we are available
- Github issues: Create an issue here
History
- Alpha 1 (16th May 2016)
- Initial version of the library
- Alpha 2 (2nd July 2016)
- Implementing the Bun EDSL
- Alpha 3 (14th March 2018):
- Integrated SOCI as the database interaction layer. This makes the library use any SQL database as SQLite, Postgres, MySQL. It mostly supports other databases that SOCI supports but it's not tested yet.
- Use of Boost Fusion. The code is much cleaner, fewer preprocessor macros The code is more debuggable.
- Support for transaction handling using the
Transaction class - Better error handling and error logging
- Added a lot of comments to help users
- Alpha 4 (5th March 2018)
- Support for nested objects
SimpleOID now uses boost UUID to generate a unique identifier- Additional comments
- Small performance enhancements
- Alpha 5 (19th May 2018)
- Support for constraint before table creation
- Alpha 6 (18th July 2018):
- Adding key value functionality to bun
- Alpha 7 (11 August 2018):
- Added range based for loop support for object iteration.
- Added range based for loop support for key-value store iteration.
- Both the iterations are lazy iterations.
- Alpha 8 (19 October 2018)
- Added support to create C++ object from JSON string
- Added support to create Message Pack from C++ object
- Added support to create C++ object from Message Pack
- Alpha 9 (13 January 2018)
- Added support to convert C++ object containing vector to JSON and Msgpack
- Added support to convert a JSON or Msgpack containing a vector to C++ object.
Next Features
- Adding C++ vector persistence
Iterator based lazy data pull- Custom
Oid class support - Support for ElasticSearch
- Improved Error handling
- EDSL query language enhancements
- Constraint modification after table creation
- Support for other constraint
- Index support
- Support for pre and post hooks for processing objects
- Persisting
std::vector members - Unit test implementation
- Support for Leveldb
Key Value iteratorSupport for Composite types. (Done)
Help Needed
Hi All,
Considering the work needed to make this library further enrich I will be needing any help needed. Help is needed in the following areas.
- Enhancement
- Fix bugs.
- Restructure and cleanup code.
- Enhance documentation.
- Constructive criticism and feature suggestions.
- Write tests.
- Use Bun
Any small or big things is appreciated.
