Software Artisan

This is in preparation for a newbie series on the Ruby language; however, it’s included for your benefit here. There are certain things that any project needs to keep track of if it is going to be successful:

• What needs to be done?
• What was done?
• And how close are we to being done?

I’m going to avoid all the project management and tech speak as much as possible. For you college students, pay attention, if you aren’t doing these things you are writing a recipe for disaster. So let’s look at these things one by one.

What needs to be done?

To answer the question you need to ask yourself, “if I get nothing else done what will make me successful?” These things are called requirements and every project has a different focus, so the requirements will not be the same. The next thing you have to ask yourself, “what is standing in between me and something I am proud to show off?” These things are called issues and they can be your own doing, or they can be external forces. Now, you can lump everything together, because you have to address both your requirements and your issues in any given release. A release is a version of what you are doing that addresses a list of things—whether they are requirements or issues. It’s kind of a package deal.

I’ve managed a few projects where we put everything in an issues list, and assigned version numbers to those things. The issues could be further classified into requirements, enhancements (aka new requirements), bugs (mistakes we made), configuration problems (environment problems), and risks (something that may or may not happen that we need to be aware of). The important thing is that you assign these things to a version so that you can have an idea of what the package is. The complete list of things to be done is your release, and as you knock them off, you can see how close you are to making a release. The list can grow as your are working and new bugs come up, or the people you are writing the application for come up with something new.

What was done?

To track what was done, you not only need to track how you are doing in your issues list, but you also need to make sure that your workspace (source code, etc.) matches what you said is done. You need to make sure that when you have a number of people working together on the same project, that no one accidentally deletes someone else’s work or added something extra. For software projects, the chief tool to keep track of what was in older releases and merging multiple people’s work together is called a version control system. Some common free version control systems include CVS and Subversion. For software projects, there really is no excuse for not using a tool for version control. The two listed are free open source projects, and the second one is a little better to manage. There are commercial software projects, but the important thing is that you are using something other than people to manage the process. Your version control software has to support the following functions:

• Tagging (marking a set of code at a version in history so you can say what was done for a previous release)
• Merging (when more than one person is working, their work should be merged into the controlled version and not just replacing the whole file)
• Logging (every commit should have a message summarizing what was included in the change)

Document management systems usually don’t support tagging and merging. They just keep a history of versions of a document. They also usually require you lock a file so that no one else can touch it while the changes are happening. While that may be necessary when you are working with word processor documents, with source code it is an unnecessary restriction that gets in the way of doing work.

So how do we make sure we did what we said we did? You test. Testing is essentially checking the application is doing what we expect it to do. You can do testing manually, automatically, or some combination of the two. The best option is to combine automated testing with manual testing. The automated testing will always check the “blessed path” or expected way to use an application. That way you make sure you don’t accidentally break something with new changes. The manual testing (people actually using the application) will have people trying to break it or abuse the system. If they succeed in breaking it in an unexpected way you have a new bug and a new automated test to deal with.

How close are we?

Hopefully, the list of open issues is getting smaller and the number of passing tests is increasing. Eventually, the number of open issues for a release will reach zero, and all the new stuff will be fully tested and given the green light. That’s when you know you are ready to cut a new release. It helps to chunk up the work so that you can get a feel for how close we are getting a little at a time. Let’s say you have a long list of things for a release, and you think it’s going to take about six months to get it all done. If you wait until the end of those six months to begin checking your work, you are probably going to get overwhelmed with the list of things that went wrong. It’s better to break up the work into smaller chunks.

I find that one or two week cycles give a good balance between getting the work done and testing early. If you do too much work before testing, it will take a lot of work to fix the problems. If you do too little work before testing the work becomes tedious and unpleasant. It’s best to have a good battle marching speed to do little mini releases so that people testing will be able to catch and report problems before they get too big.

For any project where money is involved (i.e. just about anything other than an open source project) it’s important to see how close you are to your estimate. Unexpected problems can really affect when you get the work done. Let’s say you are marching along really well and it looks like you might even beat your 6 month deadline, but suddenly the database is imploding under the work load. Now you have to find out why it’s misbehaving and fix it. The problem might be a configuration problem, or it might be that you have too little database for your application, or it might be that you are doing too much with the application. Finding the problem and fixing might set you back a month, so meeting that six month deadline is starting to look like a pipe dream. Now you have to figure out how you are going to handle the schedule problem. You can move the schedule, change what’s required for the release, change the way you are doing things to speed it up, or make people work longer. There’s a physical limit to how long people can work, and the longer the work the exponentially more inefficient they become. If you can work smarter instead of harder, go for it. It will help you with the next release.

With open source projects, the schedule is less of an issue, and things are released when they are ready. However, if someone wants to help out and start helping, it’s always best to give a clear answer of what you are trying to do. If they can see the list of issues and help get it done quicker, more power too them.

Process . It’s a dirty word around some people. All it means is “A way to get something done”. Nothing more, nothing less. However, we have things like ISO 9001, CMM, Agile, XP, Scrum, etc. all presenting themselves as the way to develop your software. Each has it’s benefits and drawbacks, and each works with a different kind of client. If you hear anyone tell you that their way to do things is better, faster, etc. than anything you are doing without knowing the details you know you are dealing with a snake oil salesman. Even within the CMMI framework, there is a lot of leeway for you to have a heavy process or an agile process.

Process is just a way to get things done.

I’ve heard of some people’s software development process as “write a little code, run the app, write a little more code, and so on.” That process can work for some people, but not for everyone. I happen to be a fan of Test Driven Development coupled with Continuous Design—a fairly Agile approach. However, I know there is a lot of people who just can’t think that way. That’s fine. We just probably won’t be working on the same projects.

Manual Process vs. Automatic Process

The first aspect software development process is finding the right ratio between things you do manually, and things that the process takes care of for you—as a side effect of just doing the process. We programmers tend to be lazy, and so we do whatever is easiest on us to do. Ironically, we are so lazy sometimes that we would rather not invest the time to set up the automation unless there is a clear reason to do so. It’s actually a good thing because too much automation can get in the way of making changes. Of course, manual processes don’t scale very well.

Anything you do manually is something you have to remember to tell someone else.

When you are working alone, you can take all the shortcuts you want because you don’t have to explain what you are doing. The problem comes when you add someone to your team. All the things you were doing you now have to explain to someone else and get them to do it, or it won’t get done. Heck, sometimes you either forget a step even when you are by yourself. It’s for that reason we even consider automation. The places where we have the best benefit from using tools include configuration management and creating builds. I wouldn’t imagine writing any code without version control software of some sort even if I’m by myself. The versions are managed for you so you can go back to a release to reproduce issues if necessary, the code is backed up and managed beyond your own hard drive, and as soon as you add more people to the team it takes care of merging code for you. The benefits far out way the alternative. Creating a build might be easy or difficult depending on the structure of the complete application. The build process will create your distributables and run any tests you have included. Tools include Make, Rake, Ant, etc. and have differing levels of complexity. The important aspect is that you aren’t relying on someone else to make the identical configuration settings in their IDE to get the same executable. It’s all taken care of by the script.

There are other areas where automation can help out, but it does depend on the culture of the project, and the ease of setting it up. How do you manage your issues? Many tools generate reports just from how you use the application. You can show how quickly you are burning through your issues, and you can see how fast or slowly you are approaching the completed release. Sometimes you can add flags to the tool to notify you if values are outside a certain set of parameters.

Testing your Process

In any software project there are number of things you have to keep track of, and your process (the way you get things done) should make it easier. So how do we know if the process is working or not? You have to start by identifying what’s important. What is going to make your client lose confidence in you? If you do nothing else, what has to be done? Do you really need full Earned Value Management (EVM) or can you get away with proving you are making progress at the expected rate. Many times it’s enough just to know when you are falling behind. One thing that’s a must in any process is that you produce the deliverables.

If anything important slips through the cracks, your process is broken. Period.

Once you’ve identified the important stuff you can’t miss, you keep track of it. So how do I do that? You do your process. If you can ensure your process takes care of tracking the important stuff, all the better. Keeping track of something is what some people call “metrics”. The thing I don’t like about metrics is that you are essentially associating numbers to parts of your process. Numbers people like metrics because they have something to play with. The reality is that numbers don’t mean anything at all. Trends mean something, numbers don’t. If you are getting better, (i.e. your trend is going in the right direction), your process is working. If you are getting worse, (i.e. your trend is going in the wrong direction), your process needs to be fixed.

So how do you track trends, and when does a change really mean something? The tricky thing with trends is that if your sample is too short you might make adjustments that are not warranted, but if the sample is too long you won’t make the adjustments soon enough. There is no substitute for your gut feeling when you are just starting to track how well your process is working. The trick is not to be alarmist, and make proper adjustments. It’s like learning to drive for the first time. You either make too many large adjustments and overcompensate, or you make too many small adjustments. The good news is that you aren’t going to crash right away.

There are a few things that process people (you know those heavy process proponents) forget to look at while they are testing their process. How much work is needed to feed the process, vs. how much work is spent actually making deliverables? If you need 5 people that do nothing other than track the project’s progress when you have a team of 3 developers generating code, there’s something a little off-kilter. Notice that I said the 5 people are feeding process. Ensuring you have a quality product is work that’s done for the deliverable. Ensuring you have a quality code review is not. That’s “meta-work” or work about work. It’s stuff done to feed the process. Even the discipline of code reviews can be brought into question as long as you gain the benefit another way. A cod review is merely a means to both mentor other team members and ensure the quality of the product. If your process handles those benefits in other ways, then you don’t need a formal code review.

Fixing a Broken Process

If you are like me, then you really won’t wonder about how you fix a broken process. You just make the changes on the fly and the process evolves. You know these “lessons learned” meetings people have that dig up all the surprises and issues that we ran across? Are you merely identifying lessons, or are you actually learning them? If a technology choice turned out to be bad, how can you make sure the same mistake doesn’t happen again?

You haven’t really learned a lesson unless you did something to correct it.

If you discover that something you are doing isn’t lending any value whatsoever, you shouldn’t be doing it anymore. The advantage of changing your ways vs. logging issues is that it actually fixes the cause of your heartburn. Logging an issue helps you vent and get things off your chest, but it’s still there until you actually do something about it. That’s why the traditional “lessons learned” meetings don’t work. They let people vent, and then nothing is done about it. “Yeah, that was painful. We’ll be doing it again next go ‘round too.”

One of the biggest lessons I learned is that waiting until a “lessons learned” meeting to vent is waiting too long. If you are experiencing some heartburn now, fix it now. If you spend more time collecting numbers than you do using those numbers (um, I mean metrics), then just how important are those numbers in the grand scheme of things? Can you find another way of collecting those numbers? It’s funny, but machines are much better about collecting them than people are. There’s probably some way of making those numbers be a byproduct of how you do things. In essence, if you can make your tools support the way you do things and keep track of the numbers for you, you are better off. As long as you are depending on people to actually log numbers, you are depending on the most unreliable source for numbers. It’s a pain to keep track of them. Even if we only need to type them into a form, it’s still a pain to keep track of them. Let your tools do all the data entry for you.

The bottom line is prove you need to change, change, and then prove your change is helping. It’s TDD for process. If you see a bad trend and the process is the culprit, prove it. Then decide what a good trend would be. Make your changes and see how close you are to the results you want. You may find that your change is “close enough”, in which case you are done. Of course, you may find that you just introduced more work than the way of doing things, in which case you revert back or make a different change.

How often do you see something cool, and take a picture of it? How often does it resurface 60 years later? Yep, this picture is another from the set of pictures my grandparents took around 1949. I think it is quite telling how no one thought it strange to take a camera to a pool back in the day, but now it’s an international crime.

No doubt this gentleman was showing off for the ladies. No matter how strong you are, you can only hold a pose like that for so long. Without all the modern gadgetry of today, my grandma was able to take this picture quickly enough. When photographing people you do have to be ready. The camera was loaded, and probably set for a reasonable exposure given the time of day. All that was needed was to cock the shutter (if it wasn’t a press type shutter that was self cocking) and release it. Advance the film and be ready for the next one.

Is there anything that could have done better? Absolutely, but what’s done is done. I think in this instant age where everything is done for you, the effort to retry is so little that people tend not to be as critical as they should be before pulling the trigger. The really good picture is lost in a sea of mediocre pictures, and will likely never see the light of paper.

Quite frankly, quantity is the enemy of excellence. I’ve taken literally thousands of pictures, and more than half of them are really good—some excellent. However, the pain of going through them all will likely cause them to never really get any special treatment. Wall space is more precious than album space, which is in turn more precious than hard drive space or negative storage space. While I publish far fewer pictures and take far fewer pictures now, the ones I do take are on average better than when I took several at a time. This fact doesn’t change whether I shoot digital or film.

Don’t be afraid not to take a picture if you can’t make it special. Have you ever taken a picture of something you thought was amazing, but after you took the picture it looked so-so? I know I have. Have you ever taken a step to either side or gotten lower or higher to reveal on film (or digital frame) what you were experiencing? Do your walkabout quickly to get in position if you can. A mediocre picture of a spectacular event is is more frustrating to me than not having the right picture of the event. I’ve not taken pictures because the moment passed before I found the right position—but I’ve also learned from that experience. I can better anticipate the right perspective now.

Don’t be afraid to make your own decisions about what you like and don’t like. There’s lots of opinions and guidelines about what makes a good picture. However, if you don’t like a picture that follows the rules then you don’t like it. Break the rules and make something you do like. That’s when you develop your own style.

Don’t be afraid to try something different. Whether it’s going old school or new school, you can only grow by incorporating new tools or processes. Just make sure you don’t try too much too quickly. Introduce yourself one bit at a time to the new stuff. Build on what you’ve learned so far and go a step further. Don’t spend thousands of dollars on equipment you’ll never get around to using. As you add to your equipment, learn to use the new stuff properly before you get the next thing.

Lastly, don’t be afraid to enjoy what you do. If all you are doing is worrying about how to make your equipment pay for itself with jobs then it’s only a matter of time before you get burned out. Find something that you enjoy, that relaxes you, and don’t worry about trying to make it pay for itself. Just have fun.

Out of only three hours of photography, I got four pictures with my large format and a roll of small format pictures. The trip was primarily to support my son’s basketball team, but I was able to get some time aside. Even in the winter you can find some truly wonderful pictures. You just need to be able to look properly.

This particular subject was a bit challenging. If I went to the beach my view was obstructed by dense dead trees. However, here I had the sun coming through gray clouds, almost backlighting my subject. The bright spot in the upper right corner is where the sunlight was coming from. My Rodenstock 26cm super-aplanat lens was made in 1935, so it doesn’t have all the improvements in lens manufacturing made since then. It still resolves an incredible amount of detail.

This particular picture is not a scan of a print, but of the negative. In order to overcome some of the problem areas of the picture I had to do some digital dodging and burning. I have to say that I don’t enjoy doing these things digitally because I think in terms of stops, not percentages. Nevertheless the light refractions in the lens obscured the top of the lighthouse so I had to do something. I dodged it about 30% which was enough to make it more substantial without making it look fake. I dodged the edges of the picture slightly differently for each edge due to the way the tones lay. The right side with the tree didn’t need as much burning as the left side. Additionally the sky required more dodging than the ground. Lastly, the sky where the sun was peaking through required some more control so I burned that top corner in an additional 40%. The tree in the middle had a bit too much weight, so I had to dodge it. Unfortunately, the scan didn’t have any more detail to give me. An optical print will help me out there.

I played around with different toning options and I found that selenium just didn’t suit it very well. I wanted a cooler look to it, so I went for a simulated gold tone. I did try a sepia tone as an alternative, and I was pleasantly surprised at how nice that would look—but went with a straight gold tone.

Sometimes playing with a scan of a negative digitally will help with visualizing what you can do with the optical print. Looking at this picture, I can see how I should have changed my angle a little bit. I wanted the large tree on the right to help frame the picture, but the tree in front of the light house is not at a good angle. I probably should have set up in the brush on the right side of the large tree. Sometimes you have to choose between personal safety and the perfect shot. Traversing snow and ice while carrying several pounds of equipment isn’t the best idea.

When I print this optically, I will definitely have to put it on 11×14 or larger paper. It needs the breathing room of the larger area so you can get up close and see the detail. The 11×14 also might cut off the little bit of the sign on the right.

My wife and I have done our share of cooking, me as a short order chef and she as someone who just enjoys it. Both of us are a “clean as you go” type of chef (loosely speaking here). My friend has the attitude that if I cook it, you clean it. We can all create good food after our style, but the state of the kitchen and the ability to do more is very different by the time we are done. By the time my wife and I are done, the used dishes are either in the sink or the dishwasher and the surfaces are wiped clean. By the time my friend is done, it looks as if hurricane Katrina had her way with the kitchen. Either way, there is some finish work that needs to happen but the amount of work is drastically different.

There’s a subtle psychological difference and a big difference in the time to completion. If you clean up your code as you go, the effort is much smaller and the number of loose ends are far fewer. If you wait until you are done with a cycle, you’ll end up with several methods that accomplish the same goal with only slight differences. The effort to consolidate them can be pretty heinous. Particularly if you have to adjust the client code to be consistent.

In both cases the underlying philosophy is “make it work, then make it better”. I’ve found that making the cycle shorter between making it work and making it work better keeps the code cleaner. It also makes it done. If you hand a customer a working application, but there are a “few things to clean up” then the customer will only be so patient. A lot of the things you may want to clean up will just not get done because the client doesn’t want to pay for it. They need to be done, and those who need to maintain the app after you will curse under their breath if they aren’t done—but it’s a hard sell. What’s worse is that the time to clean up becomes longer than if you’ve been cleaning as you go. There’s no baked on grease, or staining things that have been left on the cooking surface too long.

Keep things simple, and keep them clean as you develop. Create your test proving you have work to do, make the test pass, then make the code cleaner. You’ll find that not only does it make it easier to move on to the next thing, but you end up working quicker overall. It takes me about the same time to clean as I go as it does to “make it work” and not clean as I go. The hacks you did along the way accumulate and dry and harden, and make it generally unpleasant to fix.

In martial arts, we perform kata (also known as Hyung or just simply forms). Contained in the kata are a selection of techniques, stances, transitions, etc. to help train your body to move properly. There’s usually a general focus of the kata, and as you progress each kata builds on the lessons from before. According to the rules of progressing through the belt ranks in Tae Kwan Do, they explicitly say you should have performed the kata 300 times that are prerequisites for the next belt. That’s a lot of repetition. Essentially, you really have to know the kata inside and out before you can progress.

If you do it right, you learn the areas you have problems with and you work on fixing those issues. Each repetition of the kata shouldn’t be a mindless action because you will never have reached perfection. There’s always something to improve. You learn subtle nuances as you slow down and visualize your opponent attacking. You learn that minor changes in your stance, shifting of your weight, etc. will have a profound impact on the power you can deliver and the ease of moving into the next technique. You’re constantly learning and improving.

As software writers, we tend to have to solve the same solutions over and over again. Each time through, we learn from our past experience and seek to improve on what we’ve done in the past. It’s one of the reasons why I think pattern chasers have missed the boat. If all you do is mindlessly slap together some patterns you learned from a book or an online resource, you never learn. You never can appreciate what the technique buys you, and how much (or little) of it you have to do for the application at hand. Some parts of the problem are so ingrained you really don’t have to think about it. That’s OK, and that’s expected. However, the corner cases, the challenging parts are where you have to focus a little extra mental effort. Following a recipe book isn’t going to get you over the hump.

In martial arts, physics is our friend. When we apply force to one part of the problem (the body), it will move in predictable ways—even if that problem is resisting. When we’ve gone through the repetition of performing the forms, one-steps, etc. we can instinctively apply the right force to get the problem to move in a way that we want. If we do our job right, that problem will be in subjection in short order.

In software, logic is our friend. When we apply control to one part of the problem (the software), it will move in predictable ways—even if system events try to throw a monkey-wrench in it. Each time we deal with the issues that a machine can throw at us, we can instinctively know which system events we can ignore and which we need to control. If we do our job right, the application will be finished in short order.

Unit tests are the best way we have to understand how the code works in an application. The goal of a good unit test is to express in as few lines as possible how the code should behave under specific circumstances. If you put too much in your unit tests, you never know just quite what might be going wrong. With unit tests, you can ensure a certain amount of predictability in the system.

Integration tests are the best way we have to understand how the application works in its environment. The goal of a good integration test is to express in as few lines as possible how the system should behave as a whole under specific circumstances. The integration tests should simulate or force certain system events to ensure the application handles it robustly.

The best way to describe it with a good mental picture is again with the forms. As we are performing our forms, we can focus on each technique one at a time. Some techniques are going to be particularly troublesome for you (one I’m learning right now is kicking my but with some jumping techniques). However, you can isolate parts of the whole kata and focus on just that technique (or two if you are having problem with a transition from one technique to the other). These are like unit tests. However, your training is incomplete if you never put it all together and perform the kata as a whole. Your sensei may have people attacking you as the kata is designed, but it doesn’t change the form itself. The practice of performing the whole kata is like the integration tests.

Repetition provides you with several advantages over cookie cutter approaches: deeper understanding of the problem, better understanding of your weaknesses, and more confidence in the solution you chose. Another, more subtle, advantage is that for the parts you know well your brain does disengage allowing you to move “in the flow” for longer stints.