Introduction
Visual Studio 2015 supports old-fashioned *.vdproj projects for generating MSI installation files. However, there is not a visible way to introduce custom dialogs, as it is possible with WIX and other similar tools. This article explains step-by-step how to do it and what tools to use.
This is an advanced topic and it is assumed that you already worked with Setup projects in previous versions of Visual Studio.
Required tools
In order to create Setup project, it is needed to install an extension from Microsoft, by using the Extensions and Updates tool option in Visual Studio. Search for the word 'installer' for easy location, then download and install.
The second required tool is Orca, which comes in the Windows Platform SDK. Orca is a generic tool for opening several "database" files used by the Windows infrastructure. I have tried with the Windows Server 2003 SP1 Platform SDK but I assume it may work with other SDKs. The link I used was: http://www.microsoft.com/en-us/download/details.aspx?id=6510
It is not needed to install the whole SDK, but only a couple of components, as shown in the following dialog.
After the SDK installation is completed, the Orca installation file (Orca.msi) can be usually located at: C:\Program Files\Microsoft Platform SDK\Bin
As an alternative, there is an independent product called SuperOrca MSI Editor, from Pantaray Research Ltd. It's documentation file may give some additional insights about the MSI internals.
Creating a Setup project (vdproj)
After a succesful installation of the Visual Studio extension and reinitialization of the IDE, a Setup project can be added to an existing solution containing some application. In the Solution Explorer, the project can be added as any other, as shown in the following screenshot:
It is recommended to use the Wizard option for the first time, as it will gather information from the existing project in the solution and fill up whatever is needed.
At this point, I will assume you will spend some time researching on how to add files from other solution projects and organize them in output folders. This topic is out of the scope of this article.
Creating a custom dialog file
The setup project comes with few default dialogs that can be customized by changing labels and setting up default values for the controls. The existing controls for a given dialog cannot be removed, relocated, neither is possible to add more controls.
The User Interface view shows what standard dialogs have been already included by the Wizard, as shown:
At this point we will detour from the normal workflow and proceed with the customization.
The standard dialogs are stored in binary files with extension *.wid, and are usually located in the folder: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\VSI\bin\VsdDialogs\, which is the root of several language/codepage folders. This article will center on the 0 (Neutral) and 1033 (English) sub-folders.
The first step to create a custom dialog is to take another one as a base, and place it in the same folder. In this article the chosen name is MyCustomDialog.wid, copied from VsdCustomText3Dlg.wid
Editing the custom dialog
The WID files can be opened by the Orca tool, although that extension is not suggested in the Open dialog. It would be useful to open the WID file in the Windows Explorer, in order to associate it with the application. Once opened, it will show a list of tables with several rows each.
It would take thousands of lines to explain how the WID file is organized and how to edit or add more controls. I would suggest to open several WID files in parallel in order to learn by comparison. It is also possible to copy rows from one file and paste on the other.
As a hint, it is important to preserve the same dialog name after pasting rows, as well as pay attention to the sequence of controls, which are chained by name in some columns. Also it is important to edit the DisplayName column in the ModuleDialog table.
A good literature regarding the MSI internals, including WID, can be found in "Easy Msi", by Robin Fischer.
Usage from Visual Studio
The file dropped in both 0 and 1033 folder shall already appear in the Visual Studio dialog for adding a new dialog. The DisplayName will be used instead of the filename:
If the WID file is succesfully loaded, its properties will appear in the Properties window, and can be edited as with other standard dialogs. Also the Setup project shall compile without issues.
Usage in a Build engine
As the Visual Studio IDE has been customized by introducing a new WID file in the VSDialogs\1033 folder (or some other folder, depending on the culture), this approach wouldn't work in a Build engine, like Visual Studio Online, because the build engine doesn't contain that file referenced from the vdproj file.
Although it is possible to install the required WID file into the build engine with a script, an easier approach would be to copy the WID file into the Setup project folder, and add it to the source code repository. In order to be sure the file is in the source code repository, it would be good to browse it from the web interface.
After that, the vdproj shall be manually edited in order to reference to the local WID file, instead of the one inside the VsDialogs folder, by removing the <VsDialogsDir> tag, as shown:
If the Setup project compiles correctly after this modification, it should work in a Build engine.
