Getting Quick Turnaround for Production Issues

Introduction

One of the biggest problems the developer faces is when a production issue occurs. The developer has to quickly find the root cause of the problem or replicate the same in the development environment to resolve it. Since time is critical here delay in getting the root cause can lead to monetary loss to the client or leading to reputation loss of the company. There are few ways to resolve the production issue efficiently and with less turnaround time.

Ways to Debug an Issue

1. Go through Logs for Call Stack
2. Enable Tracing and checking the Trace Output
3. Replicating Scenario in Development Environment and debugging using Visual Studio
4. Using Program Database (PDB) Files
5. Remote Debugging

1. Go through Logs for Call Stack

This is the most common way of debugging the issue, i.e., getting the user log file and checking the exceptions in it. If the exception is known and handled with custom message, then it is useful but if the exception is unhandled then it is difficult to find out how the exception has occurred. Call Stack comes in handy in this approach but still call stack gives the method level information. The method level information does not show which line is exactly causing the problem. To get exact line information, we need to use Program Database (PDB) Files along with the application. PDB are covered in detail below. Another disadvantage of Logs is we cannot have extensive logging in our application especially if logs are written in output file. This can tremendously slow down the application performance. So logging should be kept limited and on need basis.

2. Enable Tracing and checking the Trace Output

Trace can be enabled through configuration and application can be re-run to check the trace output. If trace is disabled, the program ignores the Trace statements. Thus one can have Trace statements in the code with Trace Switches. But still this approach has limitations since it needs to be written for every logical code which is a bit tedious to maintain.

3. Replicating Scenario in Development (Dev) Environment and debugging using Visual Studio

Since a common practice is to deploy only assemblies (compiled code) and Framework on Client’s Machine, it is obvious that Client’s machine will not have application code or Visual Studio IDE / Debugger for debugging purpose. So to get the root cause, the developer would require to setup the Dev environment with similar scenario and debug it on developer’s machine.

This is the most sure shot way of getting to the root cause of the issue, but it has serious limitations.

1. The data cannot be same on Dev and Production Environment so data related issues are difficult to recreate.
2. It is a time consuming task and looking at the criticality to replicate the issue in Development could be difficult to setup depending on the multiple systems involved for the application to work.

4. Using Program Database (PDB) Files

This is the easiest approach to debug an issue on client’s machine. But what are PDB files? We normally see them in bin folder when the application is compiled. The program can run without them existing so what is the information they are carrying.

• PDB Files: Program Database files also called as symbol files consist of debugging information of the code, e.g.:
  • Global variables
  • Local variables
  • Function names and the addresses of their entry points
  • Frame pointer omission (FPO) records
  • Source-line numbers (Ref MSDN)

Due to this, the stack trace can be enriched more. When we include PDB files along with the EXE/DLL and then replicate the scenario, the current stack trace will also provide the line number where the error has occurred. This helps us understand where exactly the error has occurred and then remediate or provide a fix for the same. Let’s take an example to show the use of PDB File.

Let us create a simple console project named “ProdProject” and a Class Library named “DataLayer”. The DataLayer Class Library has DatabaseOperations class which has a method named SaveIntoDatabase.

DatabaseOperations.cs

namespace DataLayer
{
    public class DatabaseOperations
    {
        public void SaveIntoDatabase(string Name,string MathsMarks,string ScienceMarks)
        {

            int Maths = Convert.ToInt32(MathsMarks);
            int Science = Convert.ToInt32(ScienceMarks);
            // Code to insert values into the database 
        }       
    }
}

Program.cs

namespace SymbolProject
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                string[] arrValues = new string[3];

                Console.Write("Name:");
                string Name = Console.ReadLine();
                Console.Write("Maths Marks:");
                string MathsMarks = Console.ReadLine();
                Console.Write("Science Marks:");
                string ScienceMarks = Console.ReadLine();

                DataLayer.DatabaseOperations dbOperation = new DataLayer.DatabaseOperations();
                dbOperation.SaveIntoDatabase(Name, MathsMarks, ScienceMarks);
                Console.Write("Data inserted successfully");
                Console.ReadLine();
            }
            catch(Exception ex)
            {
                Console.WriteLine("Message:" + ex.Message + 
			Environment.NewLine + "StackTrace:" + ex.StackTrace);
                Console.ReadLine();
            }
        }
    }
  }

Before building the application in Release Mode, we need to enable creation of pdb files.

This can be done by Right Click DataLayer Project on Solution Explorer => Properties => Build => Advanced => Debug Info => Pdb only.

There are 3 options here:

• None – It will not create pdb files
• Pdb only – The symbols will be stored only in pdb file
• Full – The symbols will be stored in pdb and the assembly

Inside \bin\Release, you will see the following files created:

1. DataLayer.dll
2. DataLayer.pdb
3. ProdApplication.exe

Now you can deploy the DataLayer.dll & ProdApplication.exe to the client.

(Without PDB file) If the error occurs on the client machine, the following output will be seen:

You will see the stack trace is showing the method name where error has occurred but still where in method is unknown. i.e. DataLayer.DatabaseOperation.SaveIntoDatabase.

Now, if you provide the PDB file (DataLayer.pdb) at the EXE location and re run the application at the client machine, you will find the below output:

Here, it shows the method name, i.e., DataLayer.DatabaseOperation.SaveIntoDatabase as well as mentions the line number on which the error has occurred which is line number 15.

If you go to the source code and check for this line, you will see which code is giving the issue and providing a fix for this issue.

Very Important: PDB files are build specific of a DLL/EXE. They contain GUID which links to respective DLL/EXE. So a PDB file of one build cannot be used with DLL/EXE of another build even if there is no change in the code. So as soon as you create a build in Release Mode, you need to store that PDB in some repository namely “Symbol Server”. It is optional to provide PDB with Release Build. But there is not much threat to providing it since your assembly and EXE already have your source code in it. To reverse engineer your executable is possible and probably similar efforts can be put for PDB files also. So in all, there is not much of an issue unless your install package is size specific since pdb files can take up a lot of space.

5. Remote Debugging

PDB files are good way to start debugging but if you can connect to your client’s machine you can debug the application from your machine. Remote debugging is fairly simple to implement. You just need to do the following:

You Need to Setup Remote Debugging Tool on Client's Machine.

The tools can be found on Microsoft website, e.g., https://www.microsoft.com/en-us/download/details.aspx?id=48155. The URL might change so it better to search for “Remote Tools for Visual Studio 20XX”.

1. To install remote debugging tools, the user would require admin privileges.
2. Once installed and open the application and you will get the following window. The window is to ask you to configure remote debugging to allow firewall to have connections to this machine. Click on Configure Remote Debugging.

   

   After clicking, you will see:

   

   “METADOR” is my server host name. 4020 is the port number on which this service is hosted.

3. Click on Tools => Options => Permissions and setup user if both the machine share same active directory or select “No Authentication”.

   

4. Now run the application on the client’s machine. Do not proceed with the scenario but doing that we need to attach this process on your Visual Studio debugger.
5. Once done, you have setup Remote Debugger on Client’s machine. Now it is time to discover client’s machine and attach your Visual Studio debugger to the application process running there.
6. To do that, open Visual Studio with the application code you want to debug. Go to Debug=>Attach to Process.
7. Select Transport as “Remote (no authentication)” [If there is no authentication as mentioned in step 3] and click on find.

   It will search for the server/client’s machine with the host name displayed. Here it is “METADOR” and click on Select.

   

8. Select the application in the process window and click on Attach. Now the process is attached and you can add up the debug point on your code. The inputs entered by user can now be step by step debugged from your machine

   

9. After attaching the process, you need to check whether symbol files are properly loaded or not. This can be observed when you try putting breakpoints and you get the below error:

   

   To resolve this, go to Debug=>Windows => Modules

   You can see ProdApplication.exe and DataLayer.dll symbols are not loaded. If symbols are not loaded, then you cannot debug using breakpoints.

   To resolve this issue, you can right click on DataLayer.dll in Modules and select option Load Symbols. This will throw a dialog box to search pdb file. Find the PDB file for that release and load it.

   Similarly do this for ProdApplication.exe also if you have created one for that. Once loaded, you will see Symbol status in modules as “Symbols Loaded”.

   

   On Client’s Machine (METADOR): Input the values that can replicate the scenario.

   

   On Programmer’s Machine:

   

   Press F11.

   

   

This is how a developer can speed up the process of debugging the application and providing a quick turn around. There are few other ways also such as analyzing the Application memory dump and finding out the reason for the crash or memory leak. But I am keeping it out of the purview of the current article, probably I could cover it up in the next part.

References

The below references are pretty comprehensive and really good, if you get time please do read them:

• http://www.codeproject.com/Articles/37456/How-To-Inspect-the-Content-of-a-Program-Database-P
• http://blogs.msdn.com/b/visualstudioalm/archive/2012/12/10/creating-ngen-pdbs-for-profiling-reports.aspx
• https://msdn.microsoft.com/en-IN/library/y7f5zaaa.aspx
• http://www.wintellect.com/devcenter/jrobbins/pdb-files-what-every-developer-must-know
• http://www.codeproject.com/Articles/6084/Windows-Debuggers-Part-A-WinDbg-Tutorial

Coding is Simple