Read this first, please
This is the first in a series of articles that hopefully will be educational
to novices and helpful to more advanced developers alike. I hope to learn
a lot from doing this also. I am taking the approach of maintaining a
journal of the day-to-day process of software development starting from a
concept and going to a product. I hope you enjoy this series of articles.
Introduction
I had surgery on Friday June 6 and as a result am stuck at home until at
least the 16th. I could be reading, watching TV or doing just about
anything a person can do while sitting on his/her butt. Being a
programmer, I have decided to take this time to develop a software application.
I have no shortage of ideas for an application, but there is one that I have
wanted to write for a long time. One of the problems I have at work is
keeping track of change requests, program changes, ideas, etc. What I need
is an issue tracking system. Actually, I have an issue tracking package
called DevTrack which is not bad and in fact has many great features, but for
many reasons I do not use as I should. The primary reason I do not use it
is because I feel that it just adds work, instead of augmenting my work. I
have worked with a few other issue tracking systems and they all seem to have
this same common problem. For this reason, I have chosen to develop my own
issue tracking system that I will be less of a burden and more of a benefit.
What I want is a tool that is almost transparent which allows me to record
issues, change requests, bugs, defects, ideas and to-do lists quickly or ideally
automatically. I don't really care if it is integrated with my development
tool as much as I care that it is virtually transparent. I want to be able
to easily put data into this tool without jumping through hoops and I want to
get it out just as easily. I don't want a conventional MDI/SDI/whatever
application, I want a service that I can call upon to take my requests and deal
with them accordingly. I want to be able to make changes to the system
while making requests, instead of having to jump to some other application, do
something, then come back and finish my request. Basically, I want a
system that can read my mind, do what I want, when I want and interfere with my
life as little as possible while making my life simpler. Sounds like what
most users want, doesn't it.
I am creating a short series of articles entitled "The Life of a
Project" to detail my journey from here to the end of this project. I
hope to use these articles as a means to teach some valuable programming
techniques (and learn some for myself) as well as to deliver a product that
others can use and benefit from. This is the first in that series and so
will probably be somewhat more verbose than most. I hope you enjoy reading
them.
My little diatribe
Somehow, when most applications are designed a process is followed that
inevitably leads to one or more nice and neat application programs. These
may be Windows GUI applications, web apps, or whatever, but the scientist in us
developers likes to see straight lines and nice and neat boundaries between
systems with well defined lines of intersection, union or whatever. Part
of our job is to take what users (managers, bosses, etc) request and turn that
into code. Often this is very simple and we can fit all of the requests
into a nice and neat package, but with more complex requests, we end up with
more complex designs where the lines are not so neat and the boundaries not so
clear. The reason for this is obvious, computers don't talk human and
humans don't talk computer. Until this changes, there will always be a
challenge in software development.
Some people will argue that we are making real progress in this area because
we are adding more and more layers of abstraction which make it easier for
developers to make it appear that computers indeed do talk human. They may
be right, maybe abstraction is all that is needed, but we all know that all of
this abstraction is adding risk to. What happens when there is a serious
flaw in an abstraction layer way below our applications? The eternal
constant (change) takes over and we are forced to either redesign our
applications or pigeon-hole them to work with the underlying changes, or worse
some abstraction layer below ours adjusts itself to accommodate the change which
means either that later was pigeon-holed or redesigned, forcing more changes
down the line. Where does all of this end? I guess you could say
that we will reach a point where the abstractions will become so stable and
standardized that change will not be an issue, but I doubt we are anywhere near
that point now.
So where does that leave us? Well, it leaves us right back where we
started, we must take a set of requirements and design an application that
leverages as many or as few of these shaky abstractions as we feel necessary to
fulfill the requests while trying to balance the risks. We can choose more
control by using fewer abstractions, but this often forces us to either reduce
the feature set, increase the amount of code we write, or invent our own
abstractions to replace the ones we avoided earlier. We could take the
risks of increased abstractions and hopefully benefit from the productivity
gains, code reduction and feature enhancements that these abstractions promise,
but we must be careful to choose abstractions that we feel carry the least risk.
OR, maybe the pipe dream that is object oriented programming is just plain
wrong. Maybe it is a house of cards that will ultimately collapse upon its
own weight. OOP has been around a long time and has promised quite a bit,
and in truth has delivered some of its promises, but I always have a nagging
feeling in the back of my head that says "it just isn't right."
Maybe I misunderstand OOP and maybe all of these abstractions are just illusions
any my concerns are misplaced, but I doubt this too. How can we have a
system dependent on so many chances of failure? How can we have a system
that relies on a lack of change when we know that change is not only inevitable,
but a daily occurrence?
Then, a little voice in the back of my head says, "what about all those
mainframe applications that have been running unchanged for 20 years?".
Hmmm...could it be that these systems prove that OOP is and always has been
wrong? Could it be that COBOL is indeed the language of kings? Could
it be that the evil empire (Microsoft) created OOP to drive sales of its
software for the next 100 years? I doubt it, I suspect that the only
reason any of these applications continue to run is because they are
batch-oriented systems with well defined inputs that for the most part don't
change ever. I suspect that the interactive portion of these applications
has indeed changed quite a bit, that the portion that deals with communicating
information (either receiving or sending) has been changed many times while only
the core processing engine has remained unchanged.
So where does that leave us? Well, it leaves us right back where we
started, only more confused than before. Should we use OOP techniques or
should we use some other programming technique such as functional programming or
structured programming or should we give up and find a nice easy career as an
accountant? (Hmmm. Maybe all of this is just the effect pain medicine has
on a programmers mind?)
I always hate reaching this point in the application design stage because its
like walking on a treadmill, you can go a long way and never get anywhere.
The reality is that in today's world, to write an interactive application with
any-kind of human usable interface, OOP is the only way to go. We must
accept a certain amount of risk, in-fact we must accept inevitable and
uncontrolled change, and we must cope with at least some abstractions. Our
job here is to wisely choose the abstractions we rely upon.
Getting started
Before we can choose what abstractions we want to use, we need to do a little
requirements gathering, some specification writing and put on our creative hat.
We need to understand the project in detail, from the users perspective and from
a business needs perspective. Then we must create a design that takes into
consideration all of these needs and solves all of the problems. So,
without further ado, lets take a look at our requirements.
Requirements
The users of this system have been interrogated and here are their demands:
- Want to record user change and feature requests
- Want to record defects and track their progress
- Want to record Ideas for existing projects, new projects, old projects,
etc
- Want to record to-do lists
- Want to record change history for projects
- Want to report on outstanding changes, features, defects
- Want to update the contents of various items with ease
- Want to hide old data, but be able to access it when needed
- Want to report on past changes, defects, etc.
- Want to organize items for personal ease of navigation
- Will only use this app if it is dead-easy
- Will only use this app if they can customize everything as needed, when
needed
As usual, the users don't tell us everything we need to know. There are
questions that must be asked and answered. Here are some of them:
- Q: What information will be gathered for a change or feature request?
- A: I told you, I want to customize everything.
- Q: OK, What do you want the reports to look like?
- A: I told you, I want to customize everything.
- Q: That's great. How do you see this information being entered?
- A: I don't know, but I know I don't like any of the tools I have used so
far.
- Q: Define "dead-easy"?
- A: I don't want to do anything, I just want it to do everything for me.
- Q: OK, but obviously you will have to do something. How much do you
see yourself doing?
- A: I see myself clicking a button, entering the information and clicking
OK.
- Q: Do you want to be notified by email when someone-else puts an issue in
to the system?
- A: What? I don't want anyone but me to use this system. Anyone
else will screw it up.
That's about all we are going to get out of the users for now. This may
seem funny, but is not too far from the truth. Users sometimes know what
they want, but not what they want. Specifics are hard to come by and
sometimes can't be found without a little (lot) more research. That does
not mean that talking to the users was a bad idea, only that we now realize that
users aren't software developers, and typically don't think the way we do.
Trying to get a user to decide between a tree control, a list control, or a
group bar is like trying to get a 2 year old to pick what career he wants to
take when he/she grows up. It's not that they aren't smart or don't
understand their needs, but the question is beyond their scope of understanding.
Requirements - Take 2
We have spoken with the users, asked our follow-up questions and come to the
conclusion that we need to do a little more research. We know that our
requirements list is a little light on details so we need to fill it out a
little with some educated guesses combined with some well placed research.
First, lets take a few educated guesses and get them out of the way.
- Lets assume that keeping track of when items are added/changed is
important
- Lets assume that keeping track of who adds or changes items is important
- Lets assume that there is some kind of workflow process involved since
most jobs have some workflow to them. This workflow will probably
start with opening and issue and end with closing the issue.
- Lets assume that users should not be allowed to permanently delete
anything.
- Lets assume that a typical GUI application is not enough to satisfy this
crowd.
- Lets assume that certain fields are common such as Title, Comments, Date
encountered and Status.
- Lets assume that we can't assume what other fields the users will want.
- Lets assume that getting data out will be important so we need to make
reporting as powerful as possible with printable as well as viewable reports
- Lets assume some form of logon is desired
Now, lets do a little research and see what we can come up with. We are
going to look at what the competition has to offer and try and glean some ideas
from each of them. Here is a list of the products we looked at.
We could have looked at many more, and it is clear that this is a market
littered with applications. There are several things we learned from doing
this research. Some of the things we learned are:
- All of these take an application approach to the problem. What I
mean is that all of these is an application which a user must launch (either
as a standalone program or a web app) in order to perform usual tasks.
- Many are project based allowing for the creation of "projects"
to help organize issues.
- All provide a pre-defined set of fields with pre-defined value lists.
- Most allow for a high level of field/value list customizations.
- All provide some form of email notification of new issues/changed
statuses.
- All are clearly targeted at team development.
- Priority is a common field
- Attaching of files appears to be common
- Embedding images is often possible
- All provide some form of list-view based approach to listing issues.
- Common statuses include Open, Closed, Under Review, Fixed - Awaiting
Review, Cancelled, On-Hold
Requirements - Take 3
We are now armed with some more information on the subject as well as some
solid ideas of what features users will really want and use (based on what other
applications provide). Lets go back to our users and examine their
workflows and behaviors a bit. We want to learn how they do their job so
that we can best tailor the application to them. We are going to
look over their shoulders for a while to passively observe and we are going to
share what we have learned with them to get some feedback from them. Since
I am the only developer available at this time, I am going to examine my own
workflow and process.
Some basic details:
- I work in the R&D department of a market research firm
- Most of my work is done in Visual C++ 6.0, though Visual Studio .NET is on
the horizon
- I support several web based (ASP) applications
- I support several GUI applications
- I support several Windows NT Services
- I typically leave Outlook, Developer Studio and IE open at all times.
- A large part of new code is done in C++. The rest is ASP and HTML
applications.
- I use Visual Source Safe
How projects/issues/change requests arrive at my desk:
- We sell a client something which we don't already posses and a sales
person passes on that information to me through paper or email.
- We come up with an idea and choose to build it. This usually
involves discussions over lunches, and during breaks.
- We discover a problem in an existing application which usually occurs
because I review a log and find an issue, or because someone else in the
company encounters something and emails or calls me with the issue.
- A client asks for a change in an existing system which is passed down to
me usually through email.
How I currently manage issues, change requests, ideas, to-do lists:
- I have a product (DevTrack) which I keep long-term issues in such as bugs
I don't intend to fix soon, but I don't use it regularly.
- Typically ideas are passed between myself and my boss through email.
- When I finish a change, fix a bug, or whatever I put the change into
production and notify appropriate people.
What can I tell you about the other issue tracking products that I looked at:
- The "application" approach works for managing issues, but not
recording them which is where I struggle most
- Project based systems are fine, but I would like to be able to have issues
that exist across projects and occasionally are assigned to No project
- Different types of items require different fields (ie. A Change requests
will require fields that a Feature request will not, and idea will require
only a few fields, etc)
- Priority is important, but difficult since everything is a high priority.
I would like a better way to assess the real priority of an item.
- Attaching files and images is a must.
- I will allow others to put issues in IF I can control what they can do and
where it goes and WHEN I see it.
Vision
Having been through the requirements analysis up-to-this point, I have
decided that I need an innovative solution to issue tracking if I (and other
developers) are going to use it as an every day tool. So, here is my
vision for this tool.
I see a link on the Start menu (and a tray icon with a popup menu) with
specific tasks I want to perform such as "Record change request",
"Record bug", "Review open bug list", "Print bug
list", "Close issue".
I see these same links available as toolbar icons in my favorite
applications such as Developer Studio, Outlook and IE.
I see a dynamically generated form created for each type of task.
These forms should be customizable on-the-spot and I should be able to add new
task-types on-the-spot.
I see a special email-recipient which I can forward emails to and have them
automatically cataloged within this system. I would like to have
addresses like "issues@mycompany.com",
"ideas@mycompany.com",
etc" and I would like to be able to forward any email I get or send (or
CC) to one or more of these addresses and have it inserted appropriately.
I want to be able to create logical groupings of issues which can overlap
and I want to create my virtual email recipient addresses when defining these
groupings.
I see an IE namespace extension where I can drag-drop files to that will
automatically create entries and attach them to issues.
I see a journaling tool that runs in the background monitoring file changes
to .cpp, .h and .asp files (and others) that records day-to-day changes and
provides a way for me to comment these changes and to automatically extract
comments from the source code itself. These journaling entries should be
part of my issues DB.
I see an application interface that allows for management of issues, but is
not used much.
What's next?
Tomorrow, I will review where I am and begin to write a specification for
this new application. I may also create a simple prototype of some of the
features, but I will wait and see how the specification goes first.
