Using ClearCase winking mechanism with new Visual Studio versions

• Download source code - 5.24 KB

Introduction

Working with ClearCase could be as useful in some cases as frustrating in others. Our team was in one of such frustrating moments when Microsoft stopped supporting the "export to makefile" feature in Visual Studio. The project we've been working on was written in Visual Studio 6, and used the ClearCase build system based on makefiles. We kept using Visual Studio 6 for the project for years. The major motivation for that was not to confront the makefiles issue and keep using the ClearCase winking mechanism. Recently, we finally decided to catch up with the rest of the world and move to Visual Studio 9. But doing it without ClearCase winking would have turned builds of twenty minutes into six hours and the software team into choir practicing the "Help!" song. We had to find a solution for the matter. In the end, it took close to two weeks to develop scripts and makefiles which provided winking for the new Visual Studio. This article shares this experience, and presents a detailed process for smooth transition of legacy Visual Studio 6 projects into newer Visual Studio versions without giving up on makefiles and ClearCase winking abilities.

Background

Before striking you with technical details, please allow me to discuss the ClearCase building system. Building of a big scale project is not an easy task, and it may take a lot of time and resources. ClearCase allows you to speed up the building process with the so called "build avoidance" system. Every build step is audited and recorded by the ClearCase system into configuration records. The recorded data is used by the system at every successive build to avoid unnecessary build steps. Let's say we need to compile a file. The file won't be compiled unless its bill-of-materials has been changed since the last successful build. Instead, ClearCase will copy the resulting obj file into your ClearCase view. This mechanism is called "winking", and works through all dynamic views in the ClearCase system. The only limitation is that the winking mechanism works only if your projects are built with makefiles.

Starting from Visual Studio 7, Microsoft removed the ability to export projects into makefiles. To build a Visual Studio solution, you must use Microsoft building tools such as MSBbuild. The problem is that this is not fast enough for big projects. Even if you spend a decent amount of money to buy a compile server, its speed will never beat the ClearCase winking mechanism. So if the build times are important, ClearCase winking is the right choice. Since winking does not work without makefiles, there is not much to do, but to implement the "export to makefile" functionality yourself.

Of course, ClearCase winking has its own flaws, so you might want to examine alternatives for your project. Let's take a brief look at these alternatives. The major problem with winking is that it is available only in dynamic views. Dynamic views are slow, and relay on network and ClearCase server performance. So you may think of using snapshot views instead. The problem with snapshot views is their creation and update time. If you have a big project with multiple dependencies, it could be simply unfeasible to create a snapshot view. Also, for a big project, the building time will rise accordingly. So for efficient usage of snapshot views, you must keep your projects small. Despite the configuration management advantages, this "small projects" approach implies high modularity which, in turn, brings better design into your system. The only problem with this approach is that it requires a build system which should track versions of the small projects and bring them all together into the final product. Investing in such a system for legacy projects is definitely not cost-effective. As for new projects, you are highly encouraged to adopt this approach and design your system to be highly modular. You should also think of hiring professional PR to convince your management to invest in build systems that will support this modularity. This is because such a system will payoff only after a couple of years and only if your project will turn into a real product with multiple versions and dozens of developers working on it. In other words, it will turn into a monster that eats your brain cells at every feature design and drinks hours of your life at every merge.

The above is true not only for the ClearCase environment, but for any other source control system. Luckily, in ClearCase, we have the winking trick that allows us to build projects with bad modularity.

Building Makefiles for Vcproj Files

Let's assume that you are working on some huge legacy project (or not so legacy), and high modularity approach is not an option for you. As it has been stated already, you should think of implementing the "export to makefile" functionality yourself. In general, creating such a makefile is not such a difficult task, and ends up in writing some scripts and makefiles. The rest of this article intends to help you in completing this task fast and with good quality. First of all, let's define our goals:

1. Have full winking - meaning your build will use ClearCase winking abilities for all the so called derived objects in your project.
2. Fast development - you do not want to spend too much time in writing scripts for this task.
3. As less maintenance as possible - you want even less to chase after changes in your project and adjust your scripts to the changes.

One of the important outcomes of the above is that we are not going to build a script that will convert any possible Visual Studio project into a makefile, but concentrate on a specific project. Still, the resulting system must be robust enough to handle future changes.

To speed up the process, the script will only parse project (vcproj) files and will not parse solution (sln) files. Parsing solution files is possible, but may require some sophistication, which would be better avoided. For a start, it is enough to parse projects and set dependencies between different projects in makefiles. More than that, if you upgrade your project from Visual Studio 6, then you probably used makefiles to define dependencies between Visual Studio 6 project (dsp) files. Indeed, Visual Studio 6 workspace (dsw) files could not be saved in ClearCase because they are altered by Visual Studio 6 during its work. So in case you already have makefiles which work with Visual Studio 6 dsp files, moving to vcproj files will be a simple substitution in the makefiles. This will preserve project dependences as before.

Whether you decided to work with project (vcproj) files or solution (sln) files, you will end up with some makefile containing rules for your vcproj files. Let's say you named this file "makefile.mak" and it contains the following rule:

Project_1.vcproj : Project_2.vcproj Project_3.vcproj

For the sake of simplicity, assume that:

1. All the project files are given with their full path. So the rule above contains full paths (up to the drive letter of ClearCase view).
2. All the projects in the dependency tree are built with the same configuration. Configuration of the project is specified in the parameter CONFIG, and contains one of the following values: D - for Debug, and R - for Release.
3. All the projects have been created in Visual Studio 9.

Of course, the above assumptions are far from being valid for most of the projects, but they are good enough to explain the technique.

Now, let's add to makefile.mak a generic rule for vcproj files, so this rule will apply to Project_2 and Project_3, and then to Project_1 vcproj files as the rule above suggests:

%.vcproj .ALWAYS : 
       %chdir $(.TARGET,D)

       # Run makefile generator
       python.exe -m GenerateMakefileFromVCPROJ $( .TARGET,B) \
                         $(CONFIG) $(.TARGET,B)_$(CONFIG).vcproj_mak

       
       # Call makefile that handles VisualStudio-9 projects build
       omake -f makefile_vs9.mak build_vs9_project_target \            
                         VCPROJ_MAK=$(.TARGET,B)_$(CONFIG).vcproj_mak

       %chdir $(MAKEDIR)

The rule above applies to all vcproj files, and does two things. First, it calls the Python script that parses the vcproj file and puts all the relevant definitions into the file with the extension "vcproj_mak". For example, for the project Project_1.vcproj, in Debug configuration, it will create the file Project_1_D.vcproj_mak. We assume that the path to your Python executable and the PYTHONPATH environment variable have been set before getting into the rule script.

The second thing that the script does is recursive call to omake with the makefile_vs9.mak makefile. This makefile is responsible for building Visual Studio 9 projects, and the call passes to its parameter with the recently created vcproj_mak file. This is crucial to separate Visual Studio 9 related stuff to a separate makefile and recursively call to omake for several reasons.

First, it is better not to mix all the rules in one file, and make a logical separation between rules. If you have had rules for Visual Studio 6, it would be a good practice to put them away to makefile_vs6.mak. Of course, the logical separation does not require an omake recursive call, but it makes makefiles more clear and modular.

The second reason is more technical. Recursive calls to omake positively affect bill of materials that omake records for derived objects. Let's say, you need to add the Python executable path to the PATH variable. Then the vcproj rule may look like this:

%.vcproj .ALWAYS : 
       %set OLD_PATH = $(PATH)
       %setenv PATH=$(PYTHON_PATH);$(PATH)
       %chdir $(.TARGET,D)

       # Run makefile generator
       python.exe -m GenerateMakefileFromVCPROJ $( .TARGET,B) \
                         $(CONFIG) $(.TARGET,B)_$(CONFIG).vcproj_mak

       
       # Call makefile that handles VisualStudio-9 projects build
       omake -f makefile_vs9.mak build_vs9_project_target \            
                         VCPROJ_MAK=$(.TARGET,B)_$(CONFIG).vcproj_mak

       %setenv PATH=$(OLD_PATH)
       %undef OLD_PATH %undef OLD_PATH 
       %chdir $(MAKEDIR)

The above is valid, but there is a catch. ClearCase records everything you do in the rule script. So all the macros you reference in the script (R-values) are added into the configuration record of the derived objects. In the example above, values of PATH and OLD_PATH macros will be part of configuration record of the $(.TARGET,B)_$(CONFIG).vcproj_mak file. This is clearly a problem since the PATH value may vary from one computer to another, and omake will rebuild the vcproj_mak file every time it encounters a different configuration record for the file.

There are two ways to overcome the problem. First is to update the path variable somewhere in the makefile before the rule script is executed. For example, you can put the following directive at the beginning of the makefile:

%setenv PATH=$(PYTHON_PATH);$(PATH) 

Another, more elegant, way to handle this situation is to use recursive calls to omake. Such a call invokes a new instance of omake that inherits the environment from calling omake including the newly defined PATH variable value. The new omake does not know anything of the script that called it, so whatever R-values were accessed in the calling script, they will not get into the configuration records of the new omake instance.

The third reason to use a recursive call to omake is the most important one. The Python script has just parsed the vcproj file and created the vcproj_mak file. The vcproj_mak file contains definitions which determine the rules behavior in makefile_vs9.mak. So reloading omake with makefile_vs9.mak at this point is crucial for correct processing.

To illustrate the last point, let's take a closer look at the vcproj_mak and makefile_vs9.mak files.

VS9_INTERMEDIATE_DIR=Debug 
VS9_SOURCES=file1.cpp file2.cpp file3.cpp ... 
VS9_COMPILER_OPTIONS=-I. /D DEBUG /Z7 ... 
...

# Include makefile_vcproj.mak
%include $(VS9_MAKEFILE) 

# Create list of object files
VS9_OBJECTS=$(VS9_SOURCES,B,>.obj,<${VS9_INTERMEDIATE_DIR}/) 

# Compilation rules for object files 
%foreach VS9_SOURCE in $(VS9_SOURCES)  
${VS9_INTERMEDIATE_DIR}/$(VS9_SOURCE,B,>.obj) : $(VS9_SOURCE)
         # Compile 
         cl.exe $(VS9_COMPILER_OPTIONS) /Fo"$(.TARGET)" $(.SOURCE) 
%end 
...

As you can see in the above code, the Project_1_D.vcproj_mak file contains no rules but only definitions extracted from the corresponding vcproj file. The makefile uses definitions from the Project_1_D.vcproj_mak file to create a list of object files and rules to compile them. Without a recursive call to omake (omake -f makefile_vs9.mak), it would be impossible to generate these rules.

Now we are ready to complete makefile_vcproj.mak. Again, for the sake of simplicity, we assume that vcproj contains static library definitions. The above rules take care of the library sources compilation. Archiving and the build_vs9_project_target rules will be as follows:

...
VS9_TARGET= Project_1.lib 
VS9_ARCHIVER_OPTIONS=
VS9_POST_BUILD_COMMAND_1=echo "Build has been finished"
VS9_POST_BUILD_COMMANDS_COUNT = 1 
...

# Rule to build lib 
%.lib : $(VS9_OBJECTS) 
         # Archive 
         $(VS9_ARCHIVER) $(VS9_ARCHIVER_OPTIONS) $(VS9_OBJECTS) /out:"$(VS9_TARGET)"  

# Build target 
build_vs9_project_target .ALWAYS : $(VS9_TARGET)
         # Post build steps
         $(VS9_POST_BUILD_COMMAND_1)
         $(VS9_POST_BUILD_COMMAND_2)
         ...
         $(VS9_POST_BUILD_COMMAND_10)
         %if $(VS9_POST_BUILD_COMMANDS_COUNT) > 10
                 %error "Too much post build commands"
         %endif 
...

Again, the Project_1_D.vcproj_mak file contains only simple definitions and no rules. The linkage rule is straightforward, but the "build_vs9_project_target" rule requires some explanation. This rule is the main entry point to the makefile. The post build commands in it are always performed (.ALWAYS directive). This is done on purpose. Indeed, post build commands may contain some esoteric operations that ClearCase may find hard to track. For example, post build may copy files to local disk to create an installation later on.

We are almost done with makefiles. The last important issue to resolve is compiler and linker "incremental" features. These features pass data through file between successive calls to the compiler or linker. For example, if you turn on the minimal rebuild (/Gm) option, the compiler creates an .ibd file during the first compilation, and uses it in successive compilations to determine whether the source file should be recompiled. In case the source file is recompiled, the configuration record of its object file will contain a reference to the .ibd file. Clearly, this will ruin the winking process and lead to rebuild in other ClearCase views. So as a general guidance, you should avoid incremental features whatsoever. Still, there is one feature that you can not just drop; it is a debug information (pdb) file. This one is tough and has only a partial solution. During compilation, you can use the /Z7 flag to put debug information into the obj file, but for the linker, there is no such option, and using the /DEBUG option in the linker implies the /INCREMENTAL option as well. So if you intend to create a PDB file along with your EXE or DLL, be prepared to link them in every build.

Vcproj Files Parser

Now let's examine the script that parses the vcproj file. The vcproj file is an XML file, and it is structured in such a way that every configuration has its own XML sub-tree that in turn is divided into general and tools settings. In order to navigate to the correct sub-tree, the script should get the configuration on its input. Extracting definitions from a vcproj into the corresponding vcproj_mak file is a simple task that requires basic scripting skills. The full example of the vcproj parsing script is attached. It is written in Python, and could be used as a basis for your own script. Here are some rules that will help you to speed up script coding and reduce future maintenance:

Rule 1: Support only settings which are relevant for your projects.

Do not try to create the most generic script ever that will parse all possible settings. For example, if you are using the "Maximize Speed (/O2)" compiler optimization in your project, there is no need to support any other setting. In contrast, if some of your projects do not use this setting, the script will discover it and report an error. You may consider then to change the setting to comply with the rest of your projects. Usually, this will be the right thing to do because such differences in settings are most likely a result of lack of attention to the matter.

Rule 2: Avoid special settings for specific files.

Project files are listed in the "Files" XML sub-tree. Each file may have specific settings for every possible tool. For example, you may want to allow usage of intrinsic functions (/Oi) in some files. This is important ability, but it makes the script and makefile rules creation much more complex. So if you have a bunch of files that require intrinsic functions, consider to separate them into a static library instead of supporting per file settings.

Rule 3: Find a right balance between vcproj advanced features and script complexity.

vcproj files have some sophisticated features which could be hard to support. Try to find out what features are essential, and code your parser to support them. Non-essential features should be removed or changed to simplify your script. For example, the "InheritedPropertySheets" property allows inheriting of settings from some default configuration. Supporting this vcproj property will require recursive parsing of vsprops files. So you should carefully examine whether you really need this feature in your projects. For example, if you upgrade projects from Visual Studio 6, the "InheritedPropertySheets" property will point to "UpgradeFromVC60.vsprops" which in turn adds a macro (_VC80_UPGRADE=0x0600) into your compiler settings. So in this specific case, you can bypass parsing of vsprops files in your script and simply add the _VC80_UPGRADE macro to the compiler settings.

Summary

This article shows an easy way to achieve full winking for Visual Studio projects. At this point, you should be able to write your own makefile and vcproj parser. You will need to add rules for DLLs, EXEs, and other targets to your makefile. The vcproj parser will probably require adjustments as well. Hopefully, the techniques described in this article will help you to handle these tasks and other challenges.