Read this first, please
This is the second in a series of articles that hopefully will be educational to novices and helpful to more advanced developers alike. I hope to learn a lot from doing this also. I am taking the approach of maintaining a journal of the day-to-day process of software development starting from a concept and going to a product. I hope you enjoy this series of articles.
Day 1 can be found here.
Where are we?
Yesterday we were handed a project (an Issue Tracking system) and investigated the basic requirements of this application. We interviewed one of our users (twice), reviewed existing solutions and finally formulated a basic vision for our application. We need to take that vision and requirements list and produce a solid specification from which we can create a design.
Often, software developers take the requirements and jump right into coding. We do not want to do this for many reasons. First, while jumping into the coding may work for simple and small projects it is doomed to failure for anything of any scale. Next, jumping into the code presumes to know what the best tool for the job is without investigating the real specification or having a solid design. Even more seriously, jumping into the code ignores the complexity of the project and proves that all we have at our disposal is a hammer, and we are treating this project like a nail.
We must take every project seriously and apply a solid methodology to that project to achieve desirable results. Every project inevitably presents unexpected complications that are best discovered early in the process before coding has begun. We want to flush those complexities out by analyzing the requirements, assessing our risks and doing some research as needed. Then we can create our specification, prototypes, and design. Only after all this work is done can we get to the actual coding of the project.
Requirements analysis
Yesterday we gathered a list of varied requirements from several sources and ultimately confirmed that requirements list with our user. Lets take a look at the requirements we gathered and draw some conclusions from them. One thing we want to formulate from this analysis is a good philosophy for our application.
Putting aside our vision, for a moment, lets take a look at our requirements list and make some "big-picture" observations, not so much about the application, but about our users.
- We can clearly see that our user wants a tremendous amount of control over this tool, and not just toolbar and menu locations and whether or not the status bar shows up.
- Our user appears to want a task-oriented (activity bases) system, not a monolithic application.
- Our user does not want a "bug" tracking system. He wants a complete project lifecycle tool.
- Email is a significant part of the workflow for our user.
- Typically hard coded structures will not work here.
- Creating customizable groupings of issues is important to the workflow.
- Our user does not want an application, he wants services that do what he wants.
- Integration with our users workflow is critical to user buy-in and acceptance.
In many ways, our requirements list is driving us away from a conventional application approach. We need to think in terms of services to deliver what our user wants. This is conceptually similar to the "digital personal agents or digital personal assistants" approach that has been talked about for years. There will be background tasks that are constantly trying to fulfill the users wishes and others that are waiting to take his requests.
OK, now we need to take some of our observations and compare them to our vision. Do they fit? What have we missed in our vision? in our observations? ... OK, I think we covered everything. It is important to remember that we may have missed something at this point, so we must keep our eyes out for missed requirements as we progress through our project.
Risk assessment
We have reached a point in our project where it is important that we consider what the risks to project success are. We could wait to do this until after the rest of our pre-coding activities are fished, but I prefer to do this early as opposed to late so that every stage of development can benefit from this assessment. Project risks are potential problems that may arise that have the potential to derail our project either to completely kill it or force ugly changes on our schedule, requirements list, vision, design or worse, our code. Every project has risks, we just typically ignore them. We want to be open-eyed about our project so we can head-off potential problems before they happen and keep our schedule and design in-tact.
So, what are the risks we need to watch out for and how can we reduce these risks.
- User buy-in is critical to every project, but even more-so to this one. We are dealing with software developers, a group of people typically quite peculiar in their demands and often quite picky about what software they use. If our users do not like our application they may take the attitude of "I'm not using this crap. I can do a better job than that."
- Some parts of this application require knowledge we don't already possess. (ie. Explorer namespaces, dynamic email accounts). We know that these are doable things, but we need to take some time to offset this risk.
- Pesky users again. Software developers want to accomplish their primary job which is writing software. Our application can't interfere with this. This means that our applications memory foot-print, CPU usage, etc must be carefully balanced to insure it does not slow things down for them. We also must be careful that whatever we do we don't mess with their coding, compilations, testing and debugging.
- Data structures for this application are going to be run-time configured which is going to make for some interesting problems. Lets examine our options carefully and pick an appropriate solution.
Risk reduction and research
We must quickly assess our risks and where needed take additional steps to limit their impact on our project. Often, in evaluating risks, we uncover new requirements and issues that impact our specification and design.
Explorer namespace extensions
I have no experience with these beyond just a passing exposure to a few remarks and an occasional article in a magazine or on a website. I have a basic understanding of what they are, but I need to learn more about them before proceeding with development. My first step (as always) is to see if there is anything useful on Code Project. So, I go to the CP home page and enter a search for "namespace extension". The first page of results doesn't seem to hold any clues, but the second and fourth pages reveal several interesting looking articles. These are:
Namespace extensions - the undocumented Windows Shell by Henk Devos
Namespace Extensions: the IDelegateFolder mystery by Henk Devos
The Complete Idiot's Guide to Writing Shell Extensions - Part IX by Micheal Dunn
I book marked these articles and jumped to Google and did the same search. I quickly located a couple of interesting links which are:
http://archive.devx.com/free/mgznarch/vcdj/1998/dec98/namespace1.asp
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/shellcc/platform/Shell/programmersguide/shell_adv/
namespaceextension/namespace.asp
My first goal is to skim these articles to determine if a Namespace extension will indeed do what I want for this application. After skimming these articles, and taking 2 aspirin, I begin to get the feeling that I may have bitten off more than I can chew. All I can say is "Henk and Micheal - Well Done!". OH, and Micheal - "Part 9!, you've got to be kidding. Where do you find the time?"
This namespace stuff is cool, and indeed looks exactly like what we want. I need to go back to these articles and read them more thoroughly before implementing this part of our application. For now, I have enough information to feel comfortable that this will do what the application needs and I feel comfortable that I have enough resources to get started. I may need to buy a book to augment these articles, but that won't be necessary just yet. (Actually, usually I would go look for a good book at my local Borders and Barnes and Nobles, but unfortunately, I am confined to the house this week due to my ACL surgery, so that will just have to wait for later.)
Dynamic email accounts
Our vision statement included a reference to being able to dynamically create new email addresses and have our system automatically pick up incoming email to these addresses and insert appropriate records into our issue database. How the heck are we going to do this!!? My first though was: "This will be easy, I'll just write an Exchange Server extension to do this". After some thought, I realize that this may not be a good solution since not everyone uses Exchange and I suspect that standalone developers are even less likely to use Exchange. So, back to the drawing board. How the heck are we going to do this?
I left this issue alone for a while and went looking into some other aspects of this project. While eating lunch, I had an idea. "What if I just create an email server of my own?" After some thought, this seems to work quite well, except for one potential problem. What if the user already has an email server running? Ideally, this email server will run on a server PC, but what if they only have one? Could I put my email server on a different port? After some research, I think I could put it on a different port, but I don't think it would work too well. Another option would be to, in the case where the user already has an email server, require the creation of a second IP address on the email server PC and assign our new email server to only respond to that IP address. OK, this sounds like a plan.
Crap! I don't know anything about writing an email server. Time to put Code Project to use again and see what I can come up with. A quick search of "email server" reveals that the CP search engine needs more work. BUT, on page 3 and 6, we find 2 interesting articles. These are:
SMTP and POP3 Mail Server by Ivar Lumi
CPop3Connection v1.19 by PJ Naughter
Unfortunately, neither of these articles seem to provide many answers, though the code the articles are based upon appear to be quite useful. Thanks Ivar and PJ. I was hoping I could get the answer from just the article, but I guess I need to download the code and see what we have. So, I download the mail server first and what do I find but an explosion of source code. I know now why the article is light on words. The code is awesome and explains more than any article ever could. Well done Ivar! Unfortunately for me, I am not as familiar with .NET and C# as I would like to be, but reading the code is not too bad, and actually makes a lot of sense.
After reviewing the Mail Server by Ivar Lumi, I feel that this is exactly what I am looking for. I will need to add some code to allow me to dynamically (and remotely) add user accounts, but I don't think this should be too bad. I think I will add a thread and a socket listener on a new port that accepts specialized commands such as "ADD USER Sombody". At this point, I was thinking about just enhancing this server application to also deal with the issue of parsing the incoming emails and inserting them into our issue DB, but after some thought, I changed my mind. I like to keep separate systems separate and this is a case of extending the email server too far. Maybe if I was going to write my own mail server and not use an existing one, I would do this simply because my email server would have no other uses, but to bastardize this excellent piece of code seems like a bad idea.
After some more thought, separating the email server and the code to parse emails makes perfect sense. By doing this, it doesn't matter what email server I use (or the users use for that matter). I just need to write a mail reader that can connect to any standard mail server and get email for a given user. This way, if I want, I can change out the email server for another, or allow the user to use Exchange, or do just about anything else. Perfect!
Now, I need to take a look at the class provided by PJ Naughter to see if it will do what I want. So, I download the code and open it in Visual Studio. I must say, I am impressed. The code is not overly commented, but is well organized and easy to follow. Well done PJ! It is always nice to find what you want right here on Code Project. It really is an excellent resource.
Great! After researching the dynamic email accounts issue, I feel that I have a good grip on how to do this. All the pieces of the puzzle are falling together quite nicely. <INSERT EVIL LAUGH>. World domination can't be too far away.
Run-time customized data structures
Our pesky users want to be able to customize everything, including what data they enter. This means that we are not going to be able to use hard-coded data structures or table layouts. (Oh sure, we could just hard code a bunch of extra fields in our data structures, but this is just plain stupid.) We need a way to allow our application to work with data it knows nothing about. How the heck are we going to do this!?
Actually, I am being a little overly dramatic here, because I have spent a large part of my programming career coping with exactly this problem. In fact, I have developed a system for my employer and previous employer that does exactly this and much more. So, I guess it's not that great of a risk, but it still deserves mentioning. Unfortunately, I cannot give away the system I have developed already, but I can use some of what I have learned, and I am going to share some of that knowledge here on CP.
I have already posted one article called "The Entity Design Pattern" which you may find interesting. This design pattern is something I wrote myself and at the time I wrote it, thought was fairly original. As it turns out, my ideas were not really all that original (typical), but that is not really a bad thing. An anonymous user (I wish I could give proper credit) posted some information which stated that "User Defined Product Framework or Active Object Models" were similar concepts that had been around for quite some time. I jumped to Google and did a little searching on these terms. Below are the links to some articles I found very interesting and I believe you will too.
http://st-www.cs.uiuc.edu/users/johnson/papers/udp/UDP.pdf
http://users.pandora.be/michel.tilman/Publications/OOPSLA98WS15B.pdf
http://www.adaptiveobjectmodel.com
There is quite a bit of other information on the web on these topics, in-fact there is a conference held yearly on some of these topics (who would have thought it). The 3rd link above contains many links to other useful resources, so read at your leisure. I may just try to join some of these groups myself. (Hmmm.. I wonder if I had attended college if I would have learned these concepts there?)
Concerns about interference with users workflow
In identifying risks, we realized that one of the hurdles we must face is insuring that our software does not adversely affect our users daily workflow. Our users are software developers which mean they are very possessive about their source code (in other words, no deleting source files), and don't like anything that slows down compilation, source editing or any of their other activities. How are we going to design our software so that these concerns don't become full scale problems?
Lets start with the issue of memory footprint. This is a big deal. Most developers have PCs with lots of memory on them, but not all. I would risk a guess that standalone developers are more poorly equipped than those working for large corporations on larger teams. Regardless of how much memory they have available, they are probably using a lot of it. I know that I do. So, we need to be conservative in our memory usage on the client side. This is a risk because it is going to impact our choice of development tools for our client-side tools.
Now, lets think about the issue of CPU usage. This is also a big deal. We can't have our applications stealing precious cycles from compiler time, and more importantly the source editor. Our application needs to be user-friendly in the way it uses the CPU. I can't stand having an app (yes, I'm shouting at you Microsoft) that steals my CPU while the compiler is running and I especially can't stand it when I'm typing happily along and my PC hangs momentarily for some stupid back-ground process.
We need to be careful here, because no matter what we do, our client apps are going to consume some memory and use the CPU. We need to offset our memory and CPU usage with our development schedule, feature list and our other issues. We also need to consider the fact that there will probably be more than one client-side application. We will most likely have a data-entry application, some form of journaling service and a full-fledged issue GUI. Each of these has different needs and may require a different approach.
Data Entry Applications
There is going to be some form of data entry application which allows the user to quickly record new issues/changes/etc. At first, I was thinking of making this part of the larger GUI, but this seems like a bad idea when considering the memory issue. Memory usage will probably be the biggest concern for these applications for 2 reasons. First, these applications need to have quick startup time. Second, due to users workflow, they will probably be used allot which means NO MEMORY LEAKS.
As I see it, our choice of development tools includes .NET, MFC, WTL and Win32. (I could mention VB, but I don't like having tomatoes thrown at me, and my shrink tells me that if I ever touch it again I just may go through with my suicide plans.) I think .NET is out because although my experience with it is light, I have seen it in action and memory footprint is not one of its glowing features. MFC is not as bad as .NET but still requires quite a bit. We can offset some of MFC's footprint by assuming that users will probably have some app running that is using MFC and so the DLLs will already be loaded, but relying on this seems like a bad idea. Win32 would certainly deliver our memory requirements, but would certainly require more time to develop. I guess that leaves us with WTL. Unfortunately, I am not terribly familiar with WTL, but that's too bad. WTL fits the bill, so we can plan on using it for our data entry applications. I guess I need to learn that somewhere along the way.
Journaling Application
Our vision foresaw the existence of a journaling application that monitored changes to files and automatically recorded these changes. This application will most likely be some form of windows service that runs in the background. I think it's safe to say that .NET is out for this one. MFC really doesn't offer very much for this either. WTL is primarily a GUI tool, so its out. That leaves us with Win32. No problem, I have a library of classes that are written in Win32 to make working in Win32 easier. Since this is not a GUI application, allot of the problems of Win32 development don't really exist.
Task Bar Application
To put an icon and menu system on the toolbar, we need to have an application that provides this facility. The question is what library should we build this on. .NET is definitely overfill for this. MFC is overkill too. I am not really sure what WTL brings to the table, so it's out too. That leaves us with Win32. No problem, there are many classes here on Code Project and other places that do exactly this.
Management GUI
There is going to be some form of management GUI similar to what other issue tracking provide, in-fact it's the only thing most other issue tracking systems provide. With this application, we need to add an additional possibility to the list of libraries to use: ASP.NET. This could definitely be a web based application and might even benefit from it. I don't think we can make the decision between a web app and a typical windows GUI app at this point. So, lets think a little about what would work.
Win32 is out because I am not a masochist. WTL is out because I don't know it well enough and I already have a large set of MFC based GUI controls which this application could benefit from. MFC is a possibility especially since I am highly familiar with it and I have a large set of GUI controls and other useful classes. .NET is in also a possibility since it offers numerous benefits to this type of application, though I am not confident in its viability for this type of GUI user-interface. I would say that .NET is at best a long shot for this one. ASP.NET is definitely in since it is the best way to develop a web app, if we make this a web app.
Risk review
OK, I feel pretty good about our risks and where they leave us. It's kind of scary looking at the list of applications we are going to have and the way they are going to be developed, but that is not atypical in software development. We have a .NET based email server, an ATL COM based namespace extension (could be done in .NET though), a pure Win32 applications, a WTL application, a windows service in Win32 and another application that could be .NET, MFC or ASP.NET based. Not to mention, the need for a server-side state management and persistence service.
Some of you out there are no doubt thinking that it's too early to be deciding some of these things since we don't have a complete specification yet, or a design document. You are at least partially right, many of these things we have talked about today are preliminary, but before you can design something, you must have an understanding of what it is you are designing. It is important to remember that the typical development stages of requirements gathering, requirements analysis, risk management, specification creation, design creation are not always linear processes. There is overlap at all stages along the way.
Requirements - Take 4
Damn it! We haven't even started coding yet, and we have more requirements to add to our list. Looking at our risk assessment, it is clear that one of our most important requirements is that our application not interfere with the users daily workflow. Lets add these items to our requirements list now.
- Must maintain a small memory footprint
- Must maintain a small CPU footprint
- Must not interfere with developer tools, compilers, debuggers
I know many developers (including myself) who won't run certain virus scanners because they cause problems with development like slowing down compilations, locking source files unexpectedly, crashing DevStudio, etc. We must be 100% certain our application doesn't cause these problems, or our app is sunk, especially since it's shaping up to run something like a service.
What about our specification?
Yesterday I said that today we would write a specification and maybe even develop some prototypes. Well, I leaped before I looked. How can I write a specification when I haven't even analyzed the requirements list and performed a risk assessment and done some more research? My ego says to tell you guys that I intentionally said that and I am teaching a lesson by not dong that today. In reality, I just spoke too soon. This is often a mistake developers make and one which I myself have made all too often. Always remember that development is more involved than we give it credit for and respect the process for its complexity and importance. If we can learn from our mistakes and not trip over our egos too much, we might just deliver an amazing product that meets and exceeds our users expectations.
What's next?
Tomorrow, I will again review our requirements, review our analysis, review our risks and our research and finally write a specification. There may or may not be time to do some prototypes, but we'll just have to wait and see.
