Writing a Single-File Generator

• Download LLLPG-SFG.zip
• Download LLLPG-SFG-noexe.zip

Introduction  

A Single-File Generator (SFG), also known as a "Custom Tool", is a simple code-generation mechanism that has existed in Visual Studio since Visual Studio 2003 (I think).  It takes a single file as input, and generates a single file as output. Here's a screen shot showing my Custom Tool "LLLPG" in action:  

SFGs are normally quite limited in what they can do. They can only produce a single output file, Visual Studio itself is in charge of creating it, and all you are allowed to do is tell is what file extension to use for the output file. To make your options even more limited, you have to choose the file extension before you are given the input file name or input file contents. Apparently it is possible to write a SFG that behaves like a multiple-file generator, but I haven't figured out how to do that yet.

Background   

I recently wrote a Parser Generator called LLLPG, and I wanted users to have an easy way to use it in Visual Studio. There's a "Custom Tool" option in the properties box of every file, so I searched for a way to use that mechanism.

Originally, I found a nice CodeProject article about Custom Tools. I'm sure it was a useful article in 2008, but today it's somewhat of a blind alley.

Single-file generators work differently than most other Visual Studio extensions. Modern versions of Visual Studio use "MEF" (Managed Extensibility Framework) for most extensions, and although Visual Studio's core is unmanaged code, MEF extensions are .NET code. In MEF extensions, you use an enormous number of "magic" attributes: in order to communicate with visual studio, you place a ton of attributes on your classes and fields, then cross your fingers and pray that Visual Studio pays attention to them (because if it ignores them, how can you possibly find out why?) MEF extensions only work in Visual Studio 2010 and later. 

SFGs, in contrast, are COM-based, so you could theoretically write one in unmanaged code or .NET 1.0 rather than having to use the same version of .NET that Visual Studio is using. COM is an old mechanism from the 1990s for cross-language interoperability. Compared to .NET it is very clunky, and also difficult to research because when you search for "COM", Google recognizes "COM" as the last three letters of most website names; you might as well be searching for "THE". Luckily, though, .NET makes it relatively easy to create (not to mention consume) COM components. 

I have found that writing Visual Studio extensions that other people will actually use is usually an enormous pain in the butt, because: 

1. You have to have a paid (non-Express) edition of Visual Studio to write one. 
2. You have to install the Visual Studio SDK that matches your version of VS. For example, if you have Visual Studio 2013, you need the Visual Studio 2013 SDK. Without the exact SDK expected by a project, you cannot even open that project in Visual Studio. 
3. Given Visual Studio SDK version X, you apparently cannot use it to write an extension for Visual Studio X-1. If you only paid for Visual Studio 2013, you can't create an extension that supports Visual Studio 2012 or 2010; in my case, I found that the Visual Studio 2010 SDK refuses to install on a machine that has VS 2013 Professional and VS 2010 Express. 
4. You can't assume that an extension written for VS 2010 will work in VS 2012 or VS 2013. Separate testing is required. 
5. Visual Studio may decide to ignore your extension, not use it, and give you no reason why. 

And it is important to note that most kinds of VS extensions do not work in Visual Studio Express editions. That's not news you want to find out after you've purchased and installed Visual Studio 2010, 2012 and 2013 and installed the three matching SDKs!

The Microsoft Single-File Generator sample is subject to these limitations. It requires Visual Studio 2010 Pro (and it must be 2010, samples for 2012 or 2013 do not exist) with the VS2010 SDK, and it is unclear whether it can be made to work in Express Editions (let me know if you figure that out.) When I tried the sample, it didn't work for me, due to some sort of problem with registry settings:  

 

Not understanding the mechanism by which Microsoft expects registry settings to be created for an SFG, I spent over three weeks waiting for answers on StackOverflow and the Visual Studio Extensibility Forum, occasionally trying a new tweak to my "vsix" project, until I finally gave up and tried something completely different.  

The best thing I've discovered about SFGs is that it is possible to bypass all of this nonsense. I've found a way to write a SFG project that   

• Can be opened and built in VS 2010, VS 2012 and VS 2013, including Express editions
• Does not require any Visual Studio SDK  
• Simultaneously supports all Visual Studios from 2008 and 2013 (it might support VS 2005, too, I just don't have a copy to test with.) Your extension can work in VS 2008 even if it uses .NET 4, which VS 2008 is not aware of.   
• Works in Visual Studio Express Editions.

There is also a couple of disadvantages to my technique: you won't have access to all the facilities of Visual Studio, and you can't easily bundle other features (e.g. syntax highlighting extensions) with the SFG. But hey, you probably didn't come here to think about installation woes. Let's talk about how to actually write a Custom Tool!

Writing a Single-File Generator

Before you can write an SFG based on my technique, you must

• add a reference to Microsoft.VisualStudio.Shell.Interop.dll, which is a standard part of all Visual Studios. 
• Copy the source files CustomToolBase.cs and CodeGeneratorRegistrationAttribute.cs from the demo project to your project. CustomToolBase takes care of implementing the IVsSingleFileGenerator interface. 

Then all you have to do is write a derived class that overrides the DefaultExtension() and Generate() methods (plus some attributes, see below). That's it, you're done! 

For reference, here is the code of CustomToolBase:

public abstract class CustomToolBase : IVsSingleFileGenerator
{
	protected abstract string DefaultExtension();
	public int DefaultExtension(out string defExt)
	{
		return (defExt = DefaultExtension()).Length;
	}
	
	protected abstract byte[] Generate(string inputFilePath, string inputFileContents, 
		string defaultNamespace, IVsGeneratorProgress progressCallback);

	public virtual int Generate(string inputFilePath, string inputFileContents, 
		string defaultNamespace, IntPtr[] outputFileContents, 
		out uint outputSize, IVsGeneratorProgress progressCallback)
	{
		try {
			byte[] outputBytes = Generate(inputFilePath, inputFileContents, 
				defaultNamespace, progressCallback);
			if (outputBytes != null) {
				outputSize = (uint)outputBytes.Length;
				outputFileContents[0] = Marshal.AllocCoTaskMem(outputBytes.Length);
				Marshal.Copy(outputBytes, 0, outputFileContents[0], 
					outputBytes.Length);
			} else {
				outputFileContents[0] = IntPtr.Zero;
				outputSize = 0;
			}
			return 0; // S_OK
		} catch(Exception e) {
			// Error msg in Visual Studio only gives the exception message, 
			// not the stack trace. Workaround:
			throw new COMException(string.Format("{0}: {1}\n{2}", 
				e.GetType().Name, e.Message, e.StackTrace));
		}
	}
} 

The Generate method returns a byte[] rather than a string or a StringBuilder; this allows you to select the encoding to use for the output file. Here's a very simple SFG implementation that simply converts a file to uppercase:   

// Note: the class name is used as the name of the Custom Tool from the end-user's perspective.
[ComVisible(true)]
[Guid("91585B26-E0B4-4BEE-B4A5-12345678ABCD")]
[CodeGeneratorRegistration(typeof(ToUppercase), "Uppercasification!", vsContextGuids.vsContextGuidVCSProject, GeneratesDesignTimeSource = true)]
[CodeGeneratorRegistration(typeof(ToUppercase), "Uppercasification!", vsContextGuids.vsContextGuidVBProject, GeneratesDesignTimeSource = true)]
[CodeGeneratorRegistration(typeof(ToUppercase), "Uppercasification!", vsContextGuids.vsContextGuidVJSProject, GeneratesDesignTimeSource = true)]
//[ProvideObject(typeof(ToUppercase))]
public class ToUppercase : CustomToolBase
{
	protected override string DefaultExtension()
	{
		return ".txt";
	}
	protected override byte[] Generate(string inputFilePath, string inputFileContents,
		string defaultNamespace, IVsGeneratorProgress progressCallback)
	{
		return Encoding.UTF8.GetBytes(inputFileContents.ToUpperInvariant());
	}
}
  

Visual Studio Custom Tools are language-specific (that is, they are available, or not available, depending on the programming language associated with the project that wants to use the custom tool). Therefore, you must specify a CodeGeneratorRegistration attribute for each language you want to support; the third argument is a GUID that identifies a project type in Visual Studio. The GUIDs for C#, VB and J# are

vsContextGuidVCSProject = "{FAE04EC1-301F-11D3-BF4B-00C04F79EFBC}";
vsContextGuidVBProject = "{164B10B9-B200-11D0-8C61-00A0C91E29D5}";
vsContextGuidVJSProject = "{E6FDF8B0-F3D1-11D4-8576-0002A516ECE8}"; 

When writing a SFG that is installed "normally", I have the impression that a ProvideObjectAttribute (from Microsoft.VisualStudio.Shell.10.0.dll) is also required, but it's not needed here. [ComVisible(true)] is required and important, though.

The CodeGeneratorRegistrationAttribute is provided in source-code form in Microsoft's Single File Generator sample. That version of the attribute depends on the Visual Studio SDK, however, so my sample includes a modified version that does not require the SDK. My version of the attribute will not work if you are trying to install the SFG the "normal" way, it only works for installing "my" way. So what is my way? Read on...

The real problem: Installing a SFG

As I was saying, installing an SFG the "normal" way using a .vsix file (Visual Studio's extension installer mechanism), has several disadvantages. My alternative technique is to reject the straightjacket of the vsix file and write my own "installer":  

This installer is designed to work with any SFG. You can take the installer Form from the sample project and add it to any other SFG project and it will simply work, no need to change anything (oh, wait, actually I used an abstraction called IMessageSink from Loyc.Essentials.dll, but it's simple to replace IMessageSink with MessageBox if you prefer not to reference Loyc.Essentials.dll.) 

This installer does two things: 

• It creates a few registry entries under the Generators subkey of the each Visual Studio registry "hive" (key). 
• It registers associates the COM classes in LllpgForVisualStudio.exe. The ability to create a COM object (not just in Visual Studio, but everywhere in Windows) depends on certain registry settings existing. There is a utility called regasm.exe for creating these registry settings for .NET assemblies, and there is also a BCL class called RegistrationServices  for doing the same thing (the equivalent for non-.NET COM DLLs is called regsvr32, by the way).
  
  One hiccup I noticed is that apparently separate COM registration is required for 32-bit and 64-bit apps. Although a program written in .NET for "AnyCPU" works in both 32-bit and 64-bit, I use the RegistrationServices class to register the assembly with COM. Unfortunately, if the installer is built as AnyCPU, RegistrationServices will only register the module as 64-bit on 64-bit systems, not 32-bit. Since Visual Studio is 32-bit, it won't be able to find the SFG. To fix this problem, build the installer as "x86" (32-bit) instead of AnyCPU.

Sadly, my solution is not yet perfect. Under certain conditions (which are not clear to me), Visual Studio may not recognize the custom tool after it is registered (you'll get the dreaded "Cannot find custom tool 'LLLPG' on this system" message). The problem is that there are not one but two registry "hives" per copy of Visual Studio. One is located at   

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\VisualStudioEditionName\NN.0 

(where "Wow6432Node" only exists on 64-bit systems) and the other will be at  

HKEY_CURRENT_USER\Software\Microsoft\VisualStudioEditionName\NN.0_Config 

Aaron Marten says "Do not ever edit a key that ends in _Config" because the second hive is supposedly "blown away" and recreated when Visual Studio starts. That is, the "Config" area is for the running copy of Visual Studio, and when you start Visual Studio it is supposed to copy the stuff from the non-Config area to the Config area on startup. For some reason, this does not always happen.

If VS starts refusing to copy the registry settings over to _Config, the end-user will have to close VS and run devenv.exe /setup with Administrator permissions. To do this, create a shortcut to Visual Studio and change the shortcut's "Target" to something like this (the path varies depending on Visual Studio version; this example is for Visual Studio Express, while Pro editions use devenv.exe):  

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\WDExpress.exe" /setup 

Next, click "Advanced" on the Shortcut editing tab, and check "Run as administrator".

Obviously, this is a cumbersome thing to ask end-users to do, and this need to run devenv.exe /setup was exactly the thing I was trying to avoid when I changed from using a "standard" vsix installer to this new installer (fairly early, I got a vsix installer working in VS2013, save for the need to run devenv /setup). Perhaps this problem could be avoided by editing the installer to install registry keys in both hives. But I haven't tried it; for now, I leave that idea as an exercise for the reader.

Good luck, SFG authors, and good night!

History 

• Nov. 22, 2013: Initial release