Table of Contents
In the last fifteen years, a huge change has been seen in the Information Technology industry in India. Many Indian companies undertook quality journey. India emerged as the country having the largest number of level V certified companies on the CMM framework. Needless to say, CMM based Metrics based management at level IV and above. Many companies interpreted the provision loosely and achieved level V certification without having internalization of Metrics in the organization for managing and improving software projects. When Ron Radice had participated in the QAI seminar in India, he asked the audience to raise their hands if they were from a company which was at level IV or higher. 50 odd hands went up. The second question was how many of them followed the control chart; about 8-10 hands went up. That was an interesting observation. Technically, all those who had raised their hands earlier should have raised their hands again. This clearly shows the state of Process and Metrics implementation in many organizations.
Since then, Metrics based management has come a long way. Many companies have adopted Metrics for reporting their performance. I have noticed a very interesting scenario. Metrics is collected, and presented with beautiful charts, and trends are shown but the project still limps. Many buzz words like “Defect Prevention” are a part of regular quality discussion in the companies. I have noticed that the Metrics program, very often, are not connected with a set of parameters which can be used to meet customer expectations. I have seen that many times a schedule variance is computed in product release scenario with zero deviation. Whenever product releases are planned, the release date is announced. If the release progress is not in line with the release date, the scope is adjusted to meet the release date. In such a situation, going with the traditional definition of schedule variance is elusive. What is required here is a Metric which points out the problem and leads towards the possible solution. In software companies, you can find many such instances where the buzz words keep floating without any real benefit to the organization.
A set of Metrics is put in place and the same is collected and tracked for all similar projects. There is no customization to suit the requirements of customers, employees and shareholders.
This paper attempts to identify the specific needs of these stakeholders and suggest a framework for developing Metrics to suit specific needs of various stakeholders. In the process, the organization will have holistic dashboard representing health of the project from different perspectives.
Let us look at the state of Metrics implementation in the software industry. In the last decade, Metrics implementation has matured to a great extent. Few companies are more mature in terms of Metrics implementation compared to the others. They have dash boards; they keep track of the trends and use Metrics for process improvements. They have automated systems to capture and analyze the Metrics. Others have a long way to go. Irrespective of the maturity level, these Metrics generally have limited coverage. They represent engineering parameters very well but invariably lack in giving a holistic view of the project vis-à-vis organizational goals.
A software company uses Balanced Scorecard Framework, in limited extent, to measure the monthly performance of various projects. The metrics were grouped in four categories as mentioned below. In each category, some Metrics were collected and reported.
Learning & Innovation
- Training hours per resource per year
- Attrition
Internal Processes
- Delivery timeliness
- Efforts
- Quality
- Productivity
- Process compliance
- Resource utilization
Customer
- Customer satisfaction
Financial
- Revenue per head
- Cost per head
- Gross margin
They had a target for each Metric and the actual was compared to the target value. Variances were discussed in a monthly review by management team.
Many companies follow a very similar framework. A lot of effort is put in collecting and analyzing the Metrics. But the results are not always consistent.
The Metrics in the foregoing example is elaborate, but has the following limitations:
- The Metrics is not aligned to the company’s goals and objectives. Some aspects are covered randomly and some are missing.
- It primarily focuses on the performance of the project. For example, schedule variance, effort variance, defect density, etc. They represent the engineering parameters for a project performance. They do not represent the holistic view of project success in the organization.
- It also focuses on financial performance of the project. The Metric revenue per employee talks about overall revenue performance of the organization. But there is absolutely no indication as to how these Metrics represent the organizational goals.
- All stakeholders and their interests are not covered.
- The framework is similar to the one used by Kaplan and Norton in the famous book “Balanced Score Card” but it emphasizes only on postmortems i.e. lag indicators.
- The framework treats all projects in a similar fashion. The customer expectation variation is not accounted for. The fact of life is – each customer is different and their expectations are different.
Before we go ahead with a suggested framework for setting Metrics in a software organization, it is important to look at various stakeholders and their expectations.
- Owner/Shareholders: Shareholders’ goal is represented by financial performance of the company like gross margin, revenue per employee ($/ per person), cost per employee ($/ per person) etc. There may be additional goals depending on the intent of the organization like creating a learning organization.
- Customers: Each customer is different. One should have a process of understanding customer success factors explicitly.
- Employees: They play a very important role in delivering the services. It is important to note here that the employee cost in the software industry is in the range of 50% compared to a manufacturing company where it might be 2-5% of the revenue. Therefore, employees need a special address while considering metrics in an organization. This aspect is generally ignored.
Kaplan and Norton suggested Balanced Scorecard Framework to align organizational goals to the working level and four categories were created such as Financial, Customer, Internal Process and Learning & Innovation. Under each group, the authors have suggested to have lead and lag measures.
We are in the knowledge industry. Our raw material is the people. Their involvement is important for long term success of the organization. There is a need to add a separate category for employees. I will be discussing the suggested framework in subsequent paragraphs.
As discussed earlier, an employee plays a significant role in the success of software companies. Therefore, I am suggesting a customization of Balanced Score Card model. The critical success factors for Company, Customer and Employees should be identified explicitly. Learning and Growth and Internal Processes are supporting factors for the above three. It goes without saying that the same should be aligned downwards from Strategic Goals/Initiatives. The CSFs for each group, i.e., Company, Customer and Employee can be chosen from the broader list of Key Performance Indicators maintained for the company. At each level CSFs should have both, lead and lag indicators, included in measurements (metrics). A pictorial representation is given below:
The above framework for identifying relevant Metrics is very powerful and helps in giving the overall health of a project from various perspectives as listed above. A project may be doing very well from a shareholders' perspective but it may be poorly positioned from employees’ perspectives. The overall health dashboard helps management to trigger necessary actions to improve the overall heath and move towards bringing balance among the various perspectives.
Having discussed about the overall framework to set up Metrics, it is important to discuss the steps involved. The following steps may be used to establish the Metrics for a given project.
- Establish the organization level Key Performance Indicators. The set of indicators should cover all aspects of performance management like Financials, Employees, Customers, Growth, Learning, Internal Processes, etc. A sample set is given at Annexure I.
- Discuss with each of the stakeholders and identify Critical Success Factors for them. You might come across difficulties in establishing the CSFs. For example, while working with customers, it may not be a bad idea to suggest some customer related CSFs to help him choose a set to represent his case. Ideally the number of CSFs should be kept three or less for each of them.
- These CSFs should be a subset of Organizational KPIs. In case you come across a new factor, update the Organizational KPI with the same.
- Once you arrive at CSFs, let us define one or more Metric to represent each of them. Ideally a mix of lag and lead indicators may be a good combination.
- After defining Metric, levels should be decided for them. For instance:
- schedule variance of +- 3% Green
- Schedule Variance 3-6% Amber
- Schedule Variance >+- 6% Red
A level has to be developed for each selected metric.
Diagrammatically, the relationship with organizational level KPIs and each CSFs are shown below:
Once the CSFs are established and KPIs are mapped to them, you have a ready set of Metrics against Shareholders, Customer and Employees to be tracked and controlled through Dashboard.
Based on the above framework, a Metrics reporting system was developed for a company. The reporting was done regularly. A sample dashboard is given here for reference.
If you examine the dashboard, you come across the following observations:
- Customers Perspective: All customer related CSFs are green. The project is doing great from customer perspective.
- Shareholders Perspective: The project is not in a good shape from Shareholders' perspective. RMI is a proxy for the Margin and the same is amber. So is the case with billable utilization. Even the component development which is set as a companywide goal is not under control.
- Employee Perspective: The project is certainly under threat from employee perspective. The ESAT score is low and there are no employee development opportunities on this project. This might lead to high employee attrition in months to come.
Therefore, looking from all three dimensions, you get a holistic view of the project. This framework is very powerful and can be utilized to drive the improvement actions in the organizations.
Having set the Metrics program congruent with expectations of all stakeholders, it is important to look at the dash board, verify whether it is a meeting goal of the organization. If necessary, additions, deletions should be made in the Metrics to truly reflect the fulfillment of CSFs for various stakeholders.
Metrics is not a destination, it is a journey. It needs continuous improvement.
Although the Metrics program is useful irrespective of whether you follow a systematic approach or not, the suggested framework brings the following advantages to the organization:
- It aligns the organizational goals with the Metrics on the floor.
- It aligns customer expectations with the measurement in place and helps to grow customer engagement.
- It gives holistic view of the status of the work. It brings success in the long run.
- Employees’ expectations are aligned to the organization.
- It is a custom implementation for each project. The same medicine is not administered to all. The system is in place, congruent with the requirements.
This is a Balanced Scorecard Framework for a knowledge industry giving proper emphasis on “EMPLOYEE” the most valuable resource in the game.
A Sample KPI indicating few areas:
