Just a Thought...

.Net 4.0 has some new features that will make it easier to work with multiple threads. This will in turn make it easier to use up all those cores on your multi-core processors. The usual warnings and caveats apply with ensuring your classes are thread safe. So, what is there to get excited about?

First, .Net finally gets it’s Barrier class. A barrier allows all the threads running in parallel to complete at the same time. Java’s had theirs since Java 5 (more than a couple years old). I find it most useful when I am using multiple threads to do some number crunching, but I need to make sure I’ve incorporated all their work before I go on. The initial purpose is to have the multiple threads sync up before doing the next round of processing. Using multi-threaded ray tracer problem from yesterday, let’s look at the problem it will solve:

ThreadPool.SetMaxThreads(screenWidth,25);

for (int y = 0; y < screenHeight; y++)
{
    for (int x = 0; x < screenWidth; x++)
    {
        int cx = x;
        int cy = y;

        ThreadPool.QueueUserWorkItem(state =>
        {
            Color c = RenderColor(cx, cy, scene);
            RenderPixelDelegate dl = RenderPixel;
            pictureBox.Invoke(dl, new Object[] { cx, cy, c });
        }, (y+1) * (x+1));
    }
}

Now, the RenderPixel(x,y,color) method takes care of plotting the pixel on the bitmap and telling the screen to redraw. The problem is the screen redrawing. That takes a lot of time, and the whole program performs better when you only redraw when the whole raster line is done. Problem is, you can’t be sure that the raster line is complete just because the last pixel in the row finished when you are computing the pixels in parallel. Truly we only care about the last line. In Java we could poll the ThreadPool to see if there are any remaining workers. Unfortunately .Net doesn’t give us that option. This is where the Barrier comes in to play. By using a Barrier we can ensure all the pixels are rendered before issuing the final redraw command:

ThreadPool.SetMaxThreads(screenWidth,25);
Barrier barrier = new Barrier(screenHeight * screenWidth + 1);

for (int y = 0; y < screenHeight; y++)
{
    for (int x = 0; x < screenWidth; x++)
    {
        int cx = x;
        int cy = y;

        ThreadPool.QueueUserWorkItem(state =>
        {
            Color c = RenderColor(cx, cy, scene);
            RenderPixelDelegate dl = RenderPixel;
            pictureBox.Invoke(dl, new Object[] { cx, cy, c });
            barrier.SignalAndWait();
        }, (y+1) * (x+1));
    }
}

barrier.SignalAndWait();
barrier.Dispose();

pictureBox.Refresh();

A couple things I’ll point out here. When you want the barrier to force the main thread to wait, you have to include it in the number of Barrier participants. I added a participant for each thread. The Java variation allowed for a thread to signal that it got to the synchronization point but not pause. That’s more useful in a situation like this where we only need to be notified that the thread is done. Having too many open Barriers may have bad effects on the ThreadPool because the worker queue items can’t officially close until all the barriers are synced up. Unfortunately I do not have .Net 4 installed on my machine so I can’t say for sure. It’s my suspicion based on my experience with the Java equivalent.

Another key threading improvement is the Parallel class. In some ways I think it is a step up from the Java Fork/Join Task architecture. The nice thing about the Parallel class is that you can use the Action delegate instead of the WaitCallBack delegate. The WaitCallBack item passes in an object for the thread state, however you rarely need it. The Parallel class will let you invoke the same delegate several times, or perform a parallel loop for you. Essentially, the code block I’ve been using can look like this now:

Barrier barrier = new Barrier(screenHeight * screenWidth + 1);

Parallel.For(0, screenHeight, y =>
{
    Parallel.For(0, screenWidth, x =>
    {
        int cx = x;
        int cy = y;

        Color c = RenderColor(cx, cy, scene);
        RenderPixelDelegate dl = RenderPixel;
        pictureBox.Invoke(dl, new Object[] { cx, cy, c });
        barrier.SignalAndWait();
    }
}

barrier.SignalAndWait();
barrier.Dispose();

pictureBox.Refresh();

Of course, I’m assuming that the Parallel For loop has the same mutable integer problem that I experienced with the delegate BeginInvoke problem. The problem still exists for the ThreadPool worker pool, so there is no reason for me to assume any different here. I like the simplicity and lack of clutter that the Parallel class provides.

Please do note that I chose a highly parallelizable problem to show off these features. Not all complex actions are as responsive. Below is a checklist to determine if a type of problem/algorithm can be ridiculously parallel:

• No state needs to be shared between threads
• All state necessary for the function was passed in as parameters
• There are no reference or output parameters

Ray tracing fits this bill well because each pixel can be computed and plotted completely independent of the other pixels. If we added anti-aliasing to the mix, we would need to create more samples than we display. Web applications also fit this bill well because HTTP is a stateless protocol. Each request is handled independently from the others. There are several other problems that fit this bill.

When you have a set of data that is shared between threads and it has to remain consistent (i.e. race conditions would cause major problems or unstable behavior), you have to be more careful with your threads. Sure you have locks, mutexes, etc. but they essentially turn a multi-threaded application into a single threaded application and carry more overhead than if you never dealt with threads in the first place. By rethinking the problem a bit, you might be able to make the overall solution a bit more friendly to multiple threads. Below are some ways of making room:

• Copy all data into each thread
  • Avoids synchronizations because the data is being used only in one thread at a time
  • Adds memory overhead and requires efficient and safe copy routines
• Don’t do micro threads
  • The ThreadPool and Parallel classes handle micro processing needs, reusing threads as necessary
  • The costs outweigh the benefits. Always look for the major points that can be parallelized before looking at smaller items.
• Understand the goals you have for multi-threading
  • The raw processing time might be shorter if you did everything in one thread
  • You can process more things at once if you have multiple cores, but you won’t gain much from having more threads than processing units (in fact you might lose some)
  • Is it the data or the process that can be run in parallel? Sometimes it’s not recommended to split an algorithm up into multiple threads, but you still might be able to process the data in chunks.
  • Stop if you can’t understand what the code is supposed to be doing. Debugging parallel code is very difficult, but if you have no working mental model for how the code is supposed to work it is impossible.

The trick with multi-threaded programming is minimizing the dependencies between the threads. The less they have to coordinate with each other the more efficiently the threads can use your computer. That’s a lesson from Erlang.

Disclaimer: I’m originally a Java guy, and I know all kinds of asynchronous patterns and approaches that work in the Java world. The .Net approach to multi-threading and asynchronous applications is quite a bit different. In some ways the way that Microsoft approached the problem is more problematic. I stumbled across someone’s code for a small C# ray tracer with a fixed scene. I figured that this would be the perfect place to start playing with asynchronous code. Microsoft likes to hide things from you. I find this a bit problematic while trying to find out what is going wrong. Case and point: my assumptions about the way lambdas and closures ought to work (from my Ruby background) turned out to be wrong. Consider this code snippet:

for (int y = 0; y < screenHeight; y++)
{
    for (int x = 0; x < screenWidth; x++)
    {
        RenderColorDelegate px = RenderColor;
        px.BeginInvoke(x, y, scene, ar =>
           {
                Color c = px.EndInvoke(ar);
                RenderPixelDelegate dl = RenderPixel;
                pictureBox.Invoke(dl,new Object[]{x,y,c});
            }, null);
    }
}

Before I go into the surprise, let me tel you what this code is doing. This code is launching the rays from the view port one pixel at a time to calculate the color for that pixel. The px.BeginInvoke call is done on a delegate I had to create to alias the method I wanted to make asynchronous. I needed to pass in the variable “x”, “y”, and the scene that was passed in to the Render method. The original code makes use of synchronous calls, and behaves exactly as you would expect. X is always between 0 and one less than screenWidth. Y is always between 0 and one less than screenHeight. Now here is the surprise: when calling the methods asynchronously X can become greater than screenWidth!

Let me state that again in other terms: the values you pass in to the asynchronous method are not the values that end up being used. It behaves as a reference to the integer you pass in, and any call to x++ will increment the value on any asynchronous calls that have not been executed yet. That should alarm you. This can be (and was in this case) the source of serious and difficult to trace errors. With Ruby and Java anonymous classes, the values for x and y are frozen at the moment the asynchronous block is created. However, with .Net it is not. In order to fix the problem we have to copy x and y to new integers strictly for the purpose of passing in to the asynchrnous call:

for (int y = 0; y < screenHeight; y++)
{
    for (int x = 0; x < screenWidth; x++)
    {
        // Necessary to work around bug
        int cx = x;
        int cy = y;

        RenderColorDelegate px = RenderColor;
        px.BeginInvoke(cx, cy, scene, ar =>
            {
                Color c = px.EndInvoke(ar);
                RenderPixelDelegate dl = RenderPixel;
                pictureBox.Invoke(dl,new Object[]{cx,cy,c});
            }, null);
    }
}

The “pictureBox.Invoke” method should be familiar to .Net and Java UI developers. Essentially screen drawing and refreshing has to be done within the GUI thread. The “Invoke” method makes that happen when you are in another thread. The style of asynchronous call that I used included the callback function. This makes the redraw happen after the calculation is done.

When it comes to more traditional Thread manipulation, Java’s bag of tricks is a bit more complete. That may have changed with .Net 4.0 but I need to stick with 3.5. For example, the Future construct is really what I need. A Future is basically the promise of a value for when you need it later. It allows you to start processing a complex algorithm before you need the result, and pause for the result only when you really need it. It works for lazy initialization, but it’s more effective when you have a number of complex calculations you need to merge together. Java also has some good thread pool support, tasks, and other features. The Fork/Join approach in Java7 is also a very effective way to divvy up work. .Net 4.0 may have something similar to that, but both of those are taboo for new work for the time being.

All in all, I only scratched the surface. The delegate BeginInvoke() approach was designed for asynchronous event processing. It wasn’t really designed for more serious multi-threading. In fact, the way I used it for the ray tracer was the wrong tool for the job.

The bottom line is that I have to change the way I think about multi-threaded application design in the .Net platform. Some of the principles I’ve picked up from other languages still convey—but there’s a new twist on them. It’s to be expected. However, do be cautious of the delegate problem I ran into. It’s not good to have your values change underneath of you.

I’ll admit it, I’m a Ruby fan. There are certain aspects of programming with that language that are really fun. While I don’t have the privilege of working with it every day, I’m working with it enough. With yesterday’s look at the world of LINQ, I decided to play a bit more with delegates and extension methods. For the other Ruby fans out there, it is true that C# is both a statically typed language and those types are locked in at compile time. However, there are ways to make C# behave more like Ruby. Sure you can use the var keyword which just means that the type isn’t decided until the first assignment. But I’m going to look at imitating the way Ruby does looping in C#.

list = ["one", "two", "three"]

list.each {|item| puts item}

# Alternately:

list.each do |item|
  puts item
end

Perusing the .Net APIs, your IEnumerable<T> or List<T> objects don’t have an Each() method. Although it’s not difficult to add it after the fact. The process is not hard. The first step is to create an extension method. Extension methods are much like Ruby mixins, and are implemented very similarly. As long as our mixin class in in scope, so is our extension method. Let’s look at how it would look in C#:

string[] list = {"one", "two", "three"};

list.Each( item => Console.WriteLine(item) );

// Alternately:

list.Each(
    delegate(string item)
    {
        Console.WriteLine(item);
    });

The code to implement this really is not difficult to do, but it requires some understanding of the underlying mechanisms that make it possible:

public static class IEnumerableExtensions
{
    public static void Each<T>(this IEnumerable<T> list,
            Action<T> callback)
    {
        foreach(T item in list)
        {
             callback(item);
        }
    }
}

To better understand what’s going on here, I’m going to have to call out a few things. First, the mixin class IEnumerableExtensions must be static. Without the static keyword in front of the class the CLR will not be able to look inside for extension methods. That keyword also tells the compiler that all members of this class will also be static. The class cannot be instantiated in any way. Functionally, it behaves a lot like a Ruby module, which is why I’m calling it a mixin class. Note: the name of the class really doesn’t matter, but convention adds the word “Extensions” to the class or interface we are adding functionality to.

Next, the first argument of an extension method includes the keyword “this” and the type we are extending as the first parameter. Without the keyword “this” we would have just another random static method. Also note, this is a feature that is not possible in Java without bytecode manipulation or dynamic proxy chicanery. Perhaps that will change in the future. For those that used to know how C++ objects worked, the structure here makes a lot of sense. It’s similar to the idiom popular among C programmers to pass the struct being modified as the first parameter. I chose the IEnumberable interface because the same method will work on anything that implements that interface: which is exactly how Ruby attacks the problem. Essentially Ruby has an Enumerable module (mixin) that defines all the extra methods the collection classes can use.

We used generics here to provide the same idioms and levels of type checking that come with the .Net platform. Even when you are mixing in concepts derived from other languages, you should never completely throw away the principles in the language you are using. The generics allow us to use the method preserving all the type safety built into the language without requiring us to write a number of overloads. This helps using the method feel a little more Ruby-like without ignoring C# principles.

Lastly, I want to point out the type Action. It is a delegate defined by the .Net framework, along with its companion “Func”. The only difference between the two is that Action does not return anything and Func does. Instead of creating my own delegates, I simply used what was already available. With the pair of delegates supplied, we can have a lot of fun. For example, let’s say we wanted to derive a list of answers from executing the same function across all the members of a set of values. One example would be to derive the Root Mean Square (RMS) value on a set of numbers. Our extension method would look like this:

public static List<TResult> Collect<T,TResult>(
        this IEnumerable<T> list, Func<T,TResult> callback)
{
    List<TResult> answers = new List<TResult>();

    foreach (T item in list)
    {
        answers.Add(callback(item));
    }

    return answers;
}

The way you would use the “Derive” method we just defined to calculate the RMS would look something like this:

double[] values = {1.0, 4.0, 3.0};

double rms = Math.Sqrt(
    values.Collect(x => x * x).Sum() / values.Count());

Console.WriteLine("RMS is: {0}", rms);

The “Derive” method as it is written will also allow you to do a mass conversion of all the elements in one IEnumerable object into a new list. For example, if you had an array of numbers and you wanted a set of strings it can be done with just one line:

double[] values = {1.0, 4.0, 3.0};

List<string> conversion = values.Derive( item => item.ToString() );

conversion.Each( item => Console.WriteLine(item) );

Your creativity is bounded only by your imagination.

I know I just talked about the conceptual complexity of C# and Java, and C#’s Language Integrated Query (LINQ) is another example of why that is. However, it provides a feature that can potentially aid in properly unit testing database bound objects. The way Linq is implemented, it works on anything that is IEnumerable. That means you can have an List of objects or a database bound collection of objects and your interface to query the contents is identical. Put in other terms, you can write your queries once and they will function the same regardless of the back end implementation. Technically speaking, that means you can swap out databases fairly easily.

So, how does this work? You might ask (esp. if you are a Java developer). The book Professional C# 2008 from Wrox publishing does a good job of walking you through the transformation. Essentially, the query is performed in the object space rather than in the back end space which may not support queries anyway. First lets look at Java’s file filtering API:

File[] files = File.listFiles(new FileFilter() {
    public boolean accept(File file) {
        return file.getName().contains("like_it");
    }
});

This helps set the stage, because it’s the same principle. From here on out we are using C#. The equivalent C# code uses something called delegates which in some ways resemble Ruby’s code blocks you can pass in to certain functions. The starting code looks like this:

List<Widget> widgets = new List<Widget>(Repository.GetWidgets());
List<Widget> filteredWidgets = widgets.FindAll(
    delegate(Widget w)
    {
        return w.Name.Contains("like_it");
    });
filteredWidgets.Sort(
    delegate(Widget w1, Widget w2)
    {
        return w2.Name.CompareTo(w1.Name);
    });

As you can see it’s the same principle. Instead of anonymous inner class mess with Java, you are using delegates in C#. Very little difference here. Apparently C# has something called extension methods which I can only imagine they got the inspiration from Ruby or something similar (yes, Ruby is one of my favorite languages). Essentially the extension methods let you create static methods that accept the object you want to extend and lets you work with the public interface for that object. The twist is that you can call the extension method on your object as if it were part of the original API. This is how the core of LINQ is implemented. LINQ adds the following extension methods for the IEnumerable interface:

• Where
• OrderBy
• OrderByDescending
• Select
• ... and more …

Using the extension interfaces our code sample gets rewritten as:

IEnumerable<Widget> filteredWidgets = Repository.GetWidgets()
    .Where(
        delegate(Widget w)
        {
            return w.Name.Contains("like_it");
        })
    .OrderByDescending(
        delegate(Widget w)
        {
            return w.Name;
        })
    .Select(
        delegate(Widget w)
        {
            return w;
        });

It’s further refined by Lambda expressions. Lambda expressions have existed in venerable languages like Smalltalk and Lisp who claim the origin of all useful language features (tongue in cheek here). They also exist in Ruby. Essentially, they let you rewrite the whole delegate clause to take up much less space and read more elegantly:

IEnumerable<Widget> filteredWidgets = Repository.GetWidgets()
    .Where( w => w.Name.Contains("like_it") )
    .OrderByDescending( w => w.Name )
    .Select( w => w );

This is further simplified using new keywords. To the best of my knowledge the keywords map back to the extension methods with lambda expressions. I’m not positive about that. However, there is a key distinction here. Unlike the work we’ve done thus far, a LINQ query is nothing but an execution plan. Similar in concept to how your database server will take your query and create a plan for it. The query plan is not executed until you start iterating over it. The final results of the LINQ query are below:

var query = from w in Repository.GetWidgets()
            where w.Name.Contains("like_it");
            orderby w.Name descending
            select w;

OK. So let’s be straight about this. The concept of LINQ is pretty cool. It also helps simplify some otherwise complex filtering you would have to do on your objects. However, it appears to be strictly run in the object space. I can’t be certain because I haven’t read up on DataSource bound LINQ queries yet. There are some analogs to Hibernate’s HQL. If all the processing is done within the object space you run the risk of having very long running queries when dealing with large data sets. LINQ works with any object implementing IEnumerable, so there is nothing that says you have to keep all objects resident in memory. I’m pretty sure that the DataSource only loads what is needed at one time. Since the query is run as you iterate over it, you don’t run the risk of running out of memory when your query delivers a large result set.

For most CRUD based applications (like blogs, etc.) LINQ will work pretty well—particularly if you are dealing with limited result sets. However, if you deal with very large result sets involving joins you may have some performance problems. Those are the types of things that databases were designed to handle well. Like I said, I don’t know if there is a mapping to the underlying database querying through LINQ if it is working on DataSets. It’s still a potential performance problem that has plagued other languages and frameworks doing something similar.

All in all, I think LINQ has been done in a pretty slick way. What it allows you to do is focus on the logic of the query and how it is supposed to work for you. It is data agnostic, so it doesn’t care if it is working with lists, sets, databases, or a custom object that implements the IEnumerable interface. That really helps out when you need to mock up some base data to run your queries against in your unit tests. The fact that you can use this process with any list also opens up some unique possibilities for dealing with lists of data on screen. Hope you enjoyed this intro to a C# topic that piqued my interest.

I have an interest in language design, even though I have no direct outlet for it at the moment. So as I’ve been contemplating what I like and what I don’t like about the languages I have been exposed to, I’ve realized that both Java and C# are suffering from the same core ailment. That ailment is the conceptual complexity underlying these platforms. I have to say platform because both Java and C# use a virtual machine that has been used to host other languages as well. C# without the CLR is like Java without the JVM: useless. This is in stark contrast to the almost sublime conceptual simplicity of Lisp, Smalltalk, and even Ruby.

Both C# and Java have bolted on several different features to deal with the underlying complexities, much like the English language has imported words from several different languages. English, technically a Germanic language borrows significantly from Romantic languages like Latin, and even some Greek. We won’t mention some import words from vastly different languages like Japanese (kimono, karaoke, katana, kanji). So it is with Java and C#. A short list of concepts shared by both languages include:

• Autoboxing
• Attributes/Annotations
• Dynamic binding (.Net 4.0 has a DLR and Java 7 has new JVM opcodes for this purpose)
• For each style iterating
• API document generation
• and more…

The problem isn’t so much the features in and of themselves. The problem is more subtle than that. In order to deal with the complexity of the language itself, these features are necessary. In some ways, a language like Lisp has conceptual appeal, even though its syntax is hard to wrap your head around. If everything is a list, from parameters passed in to a function to data values, and the language is built around set theory, it maps pretty well to a discipline of math. Heck, with Lisp a function is just a list of operations. Although perhaps in some ways Lisp is too conceptually simple.

The problem I’m getting at is being able to form a reasonable hypothesis of how the software is addressing your problems. I remember reading a PR piece on how Java was better than C# that had a small snippet of code asking how many method invocations there were. The two or three line snippet actually ended up invoking an unexpectedly large number of methods, from attribute accessors to delegates and some other magic. The intent of the developer was clear, although the impact of the code was unexpectedly complex. That’s not to say that C# is bad. The article was a PR piece to help Java developers still feel good about themselves. However, Java is just as guilty. Have you ever tried to debug dynamic proxy code? Have you worked with features that injected functionality into your code for you (Spring/Hibernate comes to mind)?

Other than the general second law of thermodynamics, what is it that drives languages to be more complex? Rather than truly seeking simplicity, both Java and C# have progressively moved toward sweeping the inherent complexity under the rug. Essentially moving the problem from something the developer has to worry about to something the platform has to worry about. To paraphrase my wife’s favorite movie:

There are three kinds of pipe. You have nickel, and you can see where that’s gotten you. You have bronze, which is very good… until something goes wrong. Something always goes wrong. And then you have copper, which is the only kind I use. from Moonstruck

Programming is a complex process. Translating the sometimes conflicting desires of a human into something a computer can understand is not easy. That complexity is further compounded by the moving parts we need to work together to accomplish our goals. My goal in exploring the world of language design is to find the right path for true simplicity. While we are approaching on that ideal from different programming paradigms, we haven’t quite reached the ideal yet. It feels like we live in a world where there is only nickel and bronze, and copper has yet to be discovered. I’m not the only one thinking about this for sure.

In my transition to learning C#/.NET I’ve run into what is my biggest hurdle: culture shock. The technology behind Java and the JVM and the technology behind C# and the CLR are becoming more similar than different. However the culture behind the technologies are like night and day, oil and water. I supposed if we were going to liken it to eras gone by, the Java culture would be more like the 70s hippie culture and .NET would be like the 80s yuppie culture.

One of the things I liked about Java was the share and share alike mentality. There are thousands of open source projects in Java, many of them with free integrations into your IDE. If you needed help with anything, there was someone with a clue that could help you—and they would. Much of what I expect out of an IDE came from the Java world. The concept of refactorings included and automated in the IDE was a major breakthrough, and now no Java IDE wanting to be taken seriously can exclude that feature. Thank you JetBrains for introducing the world to the way it should be. When .NET first came on the scene I don’t think Microsoft was ready in this regard. When the MS peddler came to my company I asked about refactoring tools in Visual Studio (this was around 2003), and the guy looked at me sheepishly. “But with .NET you can design one interface and use it on the web or on a desktop…” he fumbled. When I said I don’t do that every day, and I need something that helps me do my job better every day he wrote it down. I don’t think I was the only one to raise that objection because by the next release of Visual Studio they had the beginnings of refactoring tools included.

The Java culture works well with venerable organizations like the Apache Software Foundation. In some ways the Java culture mirrored the meritocracy already ingrained at the ASF. However, the one thing that hurt Java in the long run is also the one thing that made it better as a first language to learn. It’s that just about every major infrastructure piece has been freely distributed under open source projects. Sure, you get what you pay for in terms of set up and configuration, but even that got better with time. The free aspect is what undermined the ability of companies to make money. Why spend tens of thousands of dollars on a license when the free option was there? Just put an intern on it and it will cost less than the commercial option. Of course, that meant that it was equally easy to play with these tools yourself and make yourself more valuable to the company.

The .NET culture is a pay and share alike mentality. While there are a number of open source projects, they are fewer and farther between. Even so, plugins we expect to be included with the IDE like JUnit integration or ANT/Maven/Boost integration for builds either are non-existent or require you to pay a hefty fee. There’s an NUnit for .NET projects, and by the looks of it is a bit better off than the JUnit 4 equivalent. However, unlike JUnit 4, Visual Studio doesn’t have a plugin for it. In fact, it never will because Microsoft has its own testing features it’s wanting to push. You can incorporate NUnit, but it’s a bit more involved. Or you can use TestDriven.NET, which is not cheap.

The pay and share alike culture extends to the community surrounding .NET as well. When you need help, it’s hard to find what you want online. The only people volunteering free advice shouldn’t, and I’m not convinced that I’d be getting my money’s worth if I paid someone either. While the MSDN has gotten better, it still has a long ways to go before it is truly usable. Part of the problem is that it is so big, trying to find the answer you need is quite difficult. Even when you find it, there’s rarely enough depth to be able to put it to good use. Many .NET books are well in excess of 1000 pages. Rather than focusing on one corner of the technology and bringing the user through the process of solving a problem, the books focus on comprehensively covering all of the API and assume that the reader has more knowledge than they do. Or they are written down to a third grader, and the happy medium is no where to be found. I’m sure the Head First book is good, as they usually are.

Bottom line is that there are obstacles in the .NET world that impede self learning. It’s not insurmountable, and many of the obstacles are cultural in nature. The Job market for Java still doubles .NET, but enough key customers insist on the technology you can’t completely ignore it. Additionally, any new languages you learn can open your eyes to new and better ways to solve problems. I’m still hunting for my go-to resources in the .NET space. It’s going to be an interesting ride, and I’m actually looking forward to it.

With my new job comes a new language and platform to learn: C#/.NET. All in all, it’s not a bad language but there are some design decisions that are making things more difficult than they probably should be. The problem stems from the interoperability problem of .NET and some COM interfaces. The core issue is the “Variant” type that is used in COM. So, a quick history lesson:

Once upon a time, there was a set of separate platforms that used COM to tie them together: C/C++ and Visual Basic. VB was the limiting factor, so COM’s design was modeled after VB features. One such feature was the evil “Variant” type. Honestly, I think Microsoft is trying to live this one down because it was thoughtfully removed from .NET. In fact, Microsoft has come up with a new component model, using the CLR as the unifying feature. Unfortunately, COM is still the bridge between managed and unmanaged code. And COM still includes the Variant type in some APIs I absolutely have to use.

Problem: C# does not have a variant type. The interop library does not have anything to wrap variables into a variant type, at least that I can see. The method call that I need to use requires two output parameters of the Variant type (specifically VT_BOOL). The marshaller is not converting the bool type for me properly. Every resource I can find through Google is missing key information I need to pull everything together. I’m supposed to sacrifice three chickens, howl at the full moon, and dance the watoozi. So here I am, three chickens less, neighbors complaining about the noise, and thoroughly embarrassed about my dancing—but no closer to a solution.

I need this call to work:

void IsPortAllowed(string imageFileName, NET_FW_IP_VERSION ipVersion,
                   int portNumber, string localAddress, NET_FW_IP_PROTOCOL ipProtocol,
                   out object allowed, out object restricted);

The “out object” parameters (the last two) are a VT_BOOL type, which at its roots is still a variant. So I need to allocate a Variant object in COM space, and get results out of it in C# space. So far I have been frustrated. Please help me come up with something that works, and doesn’t involve sacrificing chickens if possible.

A while back, before they had such a wide variety of shoe sizes to fit everybody’s foot we had this tool called a shoehorn. The shoehorn was there to help force your foot into a shoe that no longer fit. Well, it’s purpose was to keep the heel of the shoe from collapsing—but back in the day we used it to get a little more life out of a shoe that was too small. Buying a new pair of shoes was out of the question, because it was a choice between spending the $30 on shoes or some other need that we had. Times have changed, and so has my budget.

Background

Software languages have undergone many different fundamental shifts in thought over the years as well. Originally we had machine language and its symbolic cousin Assembly Language. Originally they had little structure and GOTOs (AKA JMP instructions) were written to specific addresses. Some natural evolutions happened as processors became more complex and subroutines were born, enabling some of the more Calculus like breakdown of applications. The first thought shift was Structured Programming which replaced hard addresses with symbols and registers with variables, and enabled an arbitrary number of variables—despite the limitations of the processor underneath. Then came Procedural Programming with C and its collection of similar languages. You’d think that Object Oriented Programming came next, but that was just one of several other shifts in thought. You also had declarative languages, logic languages, etc. For a longer breakdown you can visit Wikipedia’s Programming paradigm page. Each programming train of thought lends itself to a particular set of problems.

The ultimate sophistication is simplicity. Leonardo da Vinci

There are many aspects to simplicity, and it’s easy for me to throw this term around and add the disclaimer about how it’s easier said than done. Instead, let’s explore what makes software simple. What goes into this word simplicity that I keep harping on?

1. Economy of concepts. There should be no more than one or two fundamental design principles that are used consistently, and without exception. If there are two principles, the choice between them also has to be consistently applied and clear when to use which principle.
2. Consistency. As soon as you add one exception to any rule, you’ve just made it a lot more difficult to understand. For you math happy folks think of the complexity in terms of R^e+1 where R is the number of rules and e is the number of exceptions (in total). Smaller numbers are more simple and easier to understand. For example, if you have 9 rules and rule 1 has 2 exceptions and rule 3 has 4 exceptions the complexity is 9⁶⁺¹ or 4,782,969. That’s bad.
3. Predictability. Every action that you can make has to have a clear and predictable outcome. To understand the impact of unpredictability, treat the math for predictability like the math for consistency. For everything that can go wrong, you’ve just made the system that much harder to comprehend.
4. Self evident. Based on the consistently applied principles and the predictability of actions, the course of action for the user to take should be self evident. In short, once you’ve learned how things work you shouldn’t need anyone to hold your hand when you have to make a change.

Now you get an idea of what I mean by simplicity, and how it is actually hard to pull off for anything but simplistic systems. I mean if you only have two rules and two exceptions you still have a relatively simple system. Of course, once you start adding more rules and actions the complexity increases exponentially.

Annotations Are Mixing Concepts

I’ll be the first to say that Annotations are not completely useless, and they can help address some otherwise potentially sticky problems. However, as soon as you use Annotations to perform a programming task you are using declarative programming. Java and C# are supposed to be Object Oriented Programming Languages. Build languages like ANT and Make are declarative languages. You declare some tasks and their dependencies and allow the build process to sort it all out. Java wasn’t designed like that, but because there were some aspects of programming that are too hard to do in Java’s version of OOP we introduce annotations.

Annotations provide a way to mix concepts in a declarative fashion.

All the Annotation does is mark something in the class with some sort of tag. In essence it’s like a marker interface for the class, method, attribute, or parameter. The most innocuous use of Annotations is simply to mark a method for documentation purposes. It doesn’t take up compiled class file space or runtime memory, just gets added to the documentation. That’s a reasonable and predictable use for annotations.

The tricky part is when we change what the application does when we use annotations. For example, marking methods as a web service endpoint changes how the method is used. Marking a method as a test method affects how that method is called. Or in the case of Java ActionPack, marking a method with @Filter has more side effects. That method will now never be used for an Action, and the filter will be called before/after all the actions that it applies to. I recently added a convenience method so that you can query whether an action is marked with an annotation or not.

I used this solution so that I could mark all my AJAX support methods with the @AJAXSupport annotation. Why would I want to do that? Two reasons: for documentation purposes, and so that I don’t have the Single Sign On (SSO) service bring me back to an AJAX support page. The system would automatically set a callback URL in a filter applied for every action in the system. If someone called an AJAX support action that returns a partial page result, and then clicks on the SSO link the return URL from the SSO service would be for the partial page result. I had to do something (other than manually listing the actions and testing each one of them) to prevent that from happening. So I added an annotation so that I could check if the action has that annotation or not.

When I explained that to someone on the team, they responded with “I would never have guessed that”. Of course they wanted the equivalent of a modal dialog box on the web—which I believe is bad even in desktop application design. Another complaint was that I was mixing application logic with display logic. To be fair, it is navigation logic which is sometimes considered application logic and sometimes considered display logic in different situations. However, from a pure “separation of concerns” principle I was now mixing concerns in my code.

Does that mean that the solution was bad? It works, and it was the simplest thing to maintain and perform. It does not mean it is the most elegant solution or the purely simple solution. For one thing, I’m mixing declarative logic with object oriented logic. Another problem is that I’m mixing navigation logic with application logic. To some extent you can’t get around that, but that can be a symptom of design smell as well.

So am I saying that Annotations are evil, and because they are mixing concerns and programming paradigms we should avoid them at all costs? No. I’m saying that they trade code complexity for conceptual complexity. The complexity is not really gone from the system, it’s just that it has been moved to a different form of complexity. You might be able to grasp each concept in isolation, but mixing the two will probably make it hard to maintain in the future.

Part of the reason that Ruby on Rails works so well is that they did keep the concepts as simple as they could. There are relatively few rules to learn, and the rules are applied consistently. It was also built on a language that was flexible enough to keep everything object oriented, so there is no need to resort to the declarative annotation model.

Justification for Oⁿ Complexity

I chose the formula because as soon as you introduce one exception, the human mind has to sort out whether or not that exception applies to this rule. Even if there is one exception to a list of 10 rules, we have to recall whether the rule we are using has the exception or not, and if it does whether the exception is in effect. A list of rules with no exception still has the complexity of deciding which rule (or set of rules) applies so it has a base exponent of 1. You add one exception, and you’ve squared your complexity. For example a set of ten rules with no exceptions has a complexity score of 10, and as soon as you add one exception it becomes 100.

The complexity values have no units and no real meaning other than to provide a relative indication of what you are getting yourself into. It helps you understand the number of decisions the brain has to sort through before it is satisfied that it can predict that you are doing the right thing. Sometimes a little change can provide a huge impact on the relative complexity. The idea is to keep the numbers as small as you can.