CPM - A C/C++ Package Manager

• Download source - 2.7 KB

Introduction

Today, software development world has pretty much standardized on using Git as version control system. Unfortunately, when it comes to reusing code between different repositories, there is far less agreement on how it should be done. Just Google for Git submodules or Bitbucket subtrees and you will find references exhorting you to use one or the other and immediately after, other references warning you to avoid those as plague. After trying different solutions, I ended up with my own system that I described it in a previous tip (Eat Your Own Dogfood). It shows how you can organize multiple C/C++ projects using symbolic links. The feedback received was very positive.

Now I decided to take this one step forward and provide a tool that automates this process. The result is CPM (C Package Manager). It is a simple tool that does only a very specific job:

• It builds a C/C++ project that depends on many other C/C++ libraries.

It doesn't do many other things including, but not limited to:

• french fries
• your bed in the morning
• ... well, you get the idea :)

In using this tool, there are two parts: one is projects' layout that must follow a certain pattern and the second is to describe projects' dependency tree using a small JSON file for each project.

As an example, let us consider two libraries, cool_A and cool_B that need to be used in an application super_App. Both cool_A and cool_B use code from another library utils. Each one for these has its own Git repository.

Project Layout

You must adhere to the principles shown in the tip mentioned before.

RULE 1 - All projects have their own folder and all project folders are in one parent folder. The environment variable DEV_ROOT points to this root of development tree.

Here is some ASCII art showing the general code layout:

DevTreeRoot
   |
   +-- cool_A
   |    |
   |    +-- include
   |    |      |
   |    |      +-- cool_A
   |    |            |
   |    |            +-- hdr1.h
   |    |            |
   |    |            +-- hdr2.h
   |    +-- src
   |    |    |
   |    |    +-- file1.cpp
   |    |    |
   |    |    +-- file2.cpp
   |    +-- project file (cool_A.vcxproj) and other stuff
   |
   +-- cool_B
        |
        +-- include
        |      |
        |      +-- cool_B
        |            |
        |            +-- hdr1.h
        |            |
        |            +-- hdr4.h
        +-- src
        |    |
        |    +-- file1.cpp
        |    |
        |    +-- file2.cpp
        |
        +-- project file (cool_B.vcxproj) and other stuff

RULE 2 - Include files that need to be visible to users are placed in a subfolder of the include folder. The subfolder has the same name as the library.

If users of cool_A can refer to hdr1.h file like this:

#include <cool_A/hdr1.h>

An additional advantage of this organization is that it prevents name clashes between different libraries. In this case, if a program uses both cool_A and cool_B, the corresponding include directives will be:

#include <cool_A/hdr1.h>
#include <cool_B/hdr1.h>

RULE 3 - Include folders of dependent modules are made visible through symbolic links

In the structure shown before, the application that uses cool_A and cool_B will have an include folder but in this folder, there are symbolic links to cool_A and cool_B include folders. The folder structure will look something like this (angle brackets denote symbolic links):

DevTreeRoot
  |
  +-- SuperApp
  |      |
  |      +-- include
  |      |     |
  |      |     +-- <cool_A>
  |      |     |      |
  |      |     |      +-- hdr1.h
  |      |     |      |
  |      |     |      +-- hdr2.h
  |      |     |
  |      |     +-- <cool_B>
  |      |     |      |
  |      |     |      +-- hdr1.h
  |      |     |      |
  |      |     |      +-- hdr4.h
  |      |     other header files
  |      |
  |      +-- src
  |      |    |
  |      |    +-- source files
  |      other files
  ...

RULE 4 - All libraries reside in a lib folder at the root of development tree. Each module contains a symbolic link to this folder.

Without repeating the parts already shown of the files layout, here is the part related to lib folder (again, angle brackets denote symbolic links):

DevTreeRoot
  |
  +-- cool_A
  |     |
  |    ...
  |     +-- <lib>
  |           |
  |           all link libraries are here
  +-- cool_B
  |     |
  |    ...
  |     +-- <lib>
  |           |
  |           all link libraries are here
  +-- SuperApp
  |      |
  |     ...
  |      +-- <lib>
  |            |
  |            all link libraries are here
  +-- lib
       |
       all link libraries are here

If there are different flavors of link libraries (debug, release, 32-bit, 64-bit), they can be accommodated as subfolders of the lib folder.

Dependency Description Files

To describe the relationship between projects, each project uses a file called CPM.JSON.

The CPM.JSON in super_App folder has the following content:

{ "name": "super_App", "git": "git@github.com:user/super_App.git",
  "depends": [
      {"name": "cool_A", "git": "git@github.com:user/cool_A.git"},
      {"name": "cool_B", "git": "git@github.com:user/cool_B.git"}
  ],
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["super_app.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

For each dependent project, there is a line describing the dependent and giving the address of the Git repository. The build section specifies the command used to build the package for each operating system.

Similarly, the descriptor in cool_A folder looks like this:

{ "name": "cool_A", "git": "git@github.com:user/cool_A.git",
  "depends": [
      {"name": "utils", "git": "git@github.com:user/utils.git"},
  ],
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["cool_a.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

And in cool_B:

{ "name": "cool_B", "git": "git@github.com:user/cool_B.git",
  "depends": [
      {"name": "utils", "git": "git@github.com:user/utils.git"},
  ],
  "build": [
      {"os": "windows", "command": "msbuild", "args": ["cool_b.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

Finally, in utils, there are no dependencies; only the build rules:

{ "name": "utils", "git": "git@github.com:user/utils.git",
  "build": [
      {"os": "windows", 
      "command": "msbuild", "args": ["utils.proj"]},
      {"os": "linux", "command": "cmake"}
  ]
}

Operation

Once you have created the dependency description files and setup your DEV_ROOT environment variable, you just have to clone the topmost repository (super_App) and invoke the CPM utility with a command like:

cpm -v super_app

(The -v flag specifies verbose mode and lets you see the intermediate steps.)

It proceeds to read the CPM.JSON file and recursively fetches each dependent package and creates the required symbolic links. It then initiates the building of each dependent package starting with those at the bottom of the dependency tree.

Final Thoughts

I maintain a moderately sized codebase with dependencies running up to 10 levels deep and this tool makes it easy to ensure that all required projects are up to date and built in the proper order. The program is written in Go because I was just learning Go and what better way to learn a new language than to use it for a small project. It can be downloaded from its GitHub repository as a precompiled binary for Windows or Linux or you can compile it from the attached source.

History

• 26^th November, 2021 - Initial version