D. Patrick Caldwell on Software Engineering

Mobile Magic Developers Co-founded by Autopilot Founder

2011-11-26T12:29:00.001-08:00

Autopilot Consulting, LLC founder Patrick Caldwell has joined forces with long time friend and coworker James Brechtel to found Mobile Magic Developers, LLC.

James and I worked together for a few years several years ago and have been close friends ever since. It was an easy decision to get together and start developing mobile applications.

Mobile Magic Developers focuses on developing fun and exciting mobile products like educational mobile applications for kids, amusing home screen widgets, and mobile utilities. Autopilot Consulting focuses on … well consulting.

When your company is ready for your own custom mobile application, come talk to Autopilot about how we can help you wow your customers and outshine your competitors with a mobile application of your own. We'll apply our mobile experience as well as the talents of Mobile Magic to make sure you get the biggest mobile bang for your buck.

Mobile applications come in several varieties. Whether you're interested in native mobile applications or mobile skins for your web applications, we have the experience to get you there, save you money, and help you make the best decisions for your business, your customer, and your future.

Mainframe Migrations: Source Code Translation Tools

2011-11-16T05:35:00.001-08:00

I was working on a mainframe migration project a while back to move an application from Adabas and Natural to Sql Server and C#. Software AG developed Adabas and Natural back in the 70's so I was very pleased to have some Software AG developers and managers on the team.

Shortly into the project, one of the managers recommended a product Software AG has that translates Natural code into C#. Much to the surprise of my teammate, I recommended strongly against the software. In fact, if you've read any of my other mainframe migration articles, you know I was recommending against a "migration" strategy altogether (a lift-and-shift my teammate called it) vs. a guided redevelopment strategy.

In the process of making my argument, I spent a fair amount of time positing that a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap. The customer did decide not to use the translation tool, but in the process of making my argument, my curiosity was piqued (and I do have something of a strong research background) so I put together the What Really Makes a Good Programmer Survey.
For an in depth discussion of the results, see my Results of the "What Really Makes a Good Programmer" Survey post.

For this post, we'll be looking primarily at these data:
If you can't view this chart, I managed to keep a copy of the What Really Makes a Good Programmer Chart from the legacy charts API.

Because most of my respondents were programmers, I like to look at the group averages to eliminate selection bias. I group the "traits" into the following three categories:

Traits that contribute to being a good programmer
1. Has good problem solving skills
2. Learns from experience
3. Interested in learning
4. Passionate
5. Sees the big picture
6. Recognizes patterns
7. Communicates effectively
8. Tries new approaches
9. Detail oriented
10. Seeks help when needed
11. Interested in helping others
Traits that are nice to have if you can get them
1. Fast
2. Co-located
Traits with no effect on programmer quality
1. Has a college degree
2. Has a computer science degree
3. Cheap
4. Has certifications

So, back to my original stipulation that "a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap." While I still believe that statement to be true, when I conducted my survey, I asked the question in a more positive way so I can't actually claim that the tool is a bad programmer. I believe that it is analogous to a bad programmer; I just can't support that claim with these data. I'll explain why, but first, for the pedants, statements like "the community represented in my responses appears to believe that having good problem solving skills contributes to being a good programmer" will be abbreviated to "having good problem solving skills contributes to being a good programmer."

That being said, the data suggest that having good problem solving skills contributes to being a good programmer. If that stipulation is true, then the contrapositive must also be true that not being a good programmer means not having good problem solving skills. The inverse, however, is not necessarily true so I cannot claim that not having good problem solving skills contributes to not being a good programmer. I'll let you decide on the validity of that statement, but for the purposes of this post, I'll change my argument to, "using a source code translation tool is little more than hiring a programmer, who lacks the traits of a good programmer, simply because he's fast and cheap."

Think of a translation tool and go through the list of traits. The translation tool does not have problem solving skills at all, can't learn from experience and certainly isn't interested in doing so, has no passion, and doesn't even know what a big picture is let alone the big picture with your application in your environment. It may be able to recognize patterns technically, but it likely won't recognize patterns that are meaningful to your application. It's also unlikely that the tool will communicate effectively, create and try new approaches, or be interested in helping others.

I suppose you could say it's detail oriented (as long as the details are explicit and don't require problem solving) and it seeks help when needed (likely by way of errors). It's probably faster than a good programmer, though this may be negated if it's lack of the aforementioned traits produces less than useful results; that is to say as long as you don't need a programmer to fix the translation once it's finished the speed may be beneficial. Another good thing about the tool is that it's possibly cheaper than a good programmer (see previous caveat) and it's arguably well educated, but that doesn't seem to constitute a good programmer according to the survey.

Software development is an art almost as much as it is an engineering field. You hear many people talk of software craftsmanship. Allowing a craftsman to rewrite your application in the new framework from scratch using the old application as a guide will allow her to provide a unique expertise that translation tools currently don't have. She'll be able to apply a high level human analysis that the tool cannot.

For example, imagine there's a bug in the old code? The programmer can spot and fix the bug, but the tool will translate it into a prettier and newer bug. What if the old code is doing something unique to that language because the language doesn't have the features of the current language. A programmer can take advantage of these features and the tool cannot. A programmer will naturally analyze your business process as the application comes together and will make recommendations that may make your new application even better than the old one was in its heyday; the tool will not.

I can think of dozens of examples like this and I'm sure you can as well. Ultimately, I believe that allowing a tool to convert your legacy code into newer legacy code is a waste of time, money, and opportunity and if it doesn't cost more in the short run, it will cost more in the long run.

Switching on Enums: A Style Observation

2011-11-15T17:43:00.001-08:00

I was digging through the framework today looking for another good dotPeek of the Week topic. I was perusing the Lazy<T> class and found an interesting snippet.

This post isn't quite like my previous dotPeek of the Weeks insofar as this is more of an example of what not to do. This is certainly merely my opinion, but one rule I try to follow when writing code is that more expressive is almost always better than less expressive. When I was looking at the Lazy class, I found a great example of this.

Here's the code (abridged for clarity … and also because the threading in this class will make for better discussion later):

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    default:
      // lock and set values
      break;
  }
}

Is there anything you notice about this code? Perhaps any unanswered questions as you read it and try to figure out what it does? Specifically, what exactly constitutes the default case?

As I read through this code, learning about some of the interesting thread safety techniques, I found myself pondering, "why would locking be the default behavior? In fact, what is the default case? Do the default values have something in common?"

I looked at the enum LazyThreadSafetyMode and found this:

public enum LazyThreadSafetyMode
{
  None,
  PublicationOnly,
  ExecutionAndPublication,
}

That's when I decided that, in most cases, when you're switching on a reasonably small set of values, it's best to express these values explicitly so that the people (including you) who have to maintain the code can better understand why the default case is the default case … even if there are no comments.

For example, the following code is functionally equivalent:

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    case LazyThreadSafetyMode.ExecutionAndPublication:
      // lock and set values
      break;
  }
}

Personally, I find the latter example much more expressive. It's obvious to me what the three cases are and I don't have to wonder what possible values can become the default case. In fact, I may even go so far as having the default case throw an exception in this class. I'd do this so that, for whatever reason, if someone were to change the LazyThreadSafetyMode enum and not implement that case for the new values in Lazy<T>.LazyInitValue(), they'd get an exception in testing instead of incorrectly using the default functionality.

Elon University has #1 MBA Program

2011-11-12T06:23:00.001-08:00

Elon University's part-time MBA program deserves a lot of credit for the success of Autopilot Consulting. Bloomberg Businessweek recently gave the Martha and Spencer Love School of Business (LSB) much deserved recognition.

In The Best Business Schools of 2011, Elon's MBA program was ranked first among competitors like UCLA, Carnegie Mellon, UC-Berkley, and Rice.

When I was looking for an MBA program to attend, I compared Elon's program with Duke University's and UNC's programs and was blown away by the comparative quality of the LSB program; the experience, friendliness, and helpfulness of the staff and faculty; and the extraordinarily low price tag.

Congratulations to the Elon LSB Faculty, Staff, Students, and Alum on a job well done.

Cow Says Moo for Android - Teach Your Kids Farm Animal Sounds

2011-09-21T09:22:00.000-07:00

James, my partner in Mobile Magic Developers, and I have a handful of Android applications under our belts now. We loved working on Chinese Whispers, an idea my wife gave me for a new twist on The Telephone Game. James wrote an immensely handy application for getting your WiFi Passwords from your phone.

I wrote the Chuck Norris Fact of the Day Widget and the Jane Austen Fact of the Day Widget, and James wrote a really nice widget that shows you your last few installed applications called Latest Apps.

But… Then we Wrote Cow Says Moo

Cow Says Moo is definitely my favorite so far. Many of my friends and family are having babies now and my wife and I have one on the way. I noticed that kids are growing more and more interested in technology and are remarkably capable with it. My niece loved playing with my Motorola Xoom and I loved the way her face lights up when she touches the screen and something happens. That's where we got the idea for Cow Says Moo.

Cow Says Moo is an educational game to teach young kids and toddlers about farm animals and the sounds they make. Each picture is hand illustrated by Emily Upchurch Robinson and when pressed plays farm animal sounds. You can use the front facing camera on your device (or the rear camera if you prefer) to record video and sound while your child interacts with your mobile device or while you simply delight him or her with the sounds. You can use the built in media gallery to share your videos with friends and family. You'll be surprised how much fun you'll have playing Cow Says Moo with your kids and you'll get a lot of cute video footage like I did "User Acceptance Testing", i.e., playing Cow Says Moo with my godson AJ.

Check out Cow Says Moo, and as always, feel free to contact us with any comments or bugs.

7-bit Encoding with BinaryWriter in .Net

2011-09-02T08:18:00.000-07:00

At work this week, I had the need to serialize objects and encrypt them while trying to keep the smallest data footprint I could. I figured the easiest thing to do would be to binary serialize them with the BinaryFormatter. It was, indeed, the easiest thing to do; however, the BinaryFormatter seems to come with a fair amount of overhead. By the time the BinaryFormatter was finished listing all of the necessary assembly-qualified type names, the 60 bytes of data I wanted to preserve were more than a kilobyte!

I needed another way so I extended BinaryWriter (mostly to get it to serialize the types I needed it to) and now my 60 bytes take 68 bytes to serialize. In the process of writing this class, I looked through the disassembled BinaryWriter using JetBrains's dotPeek and found this little gem (Write7BitEncodedInt(int)) and decided it'd make a great dotPeek of the Week:

protected void Write7BitEncodedInt(int value)
{
  uint num = (uint) value;

  while (num >= 128U)
  {
    this.Write((byte) (num | 128U));
    num >>= 7;
  }

  this.Write((byte) num);
}

This is how the BinaryWriter.Write(string) encodes the length of the string. Thus, if you have a string with fewer than 128 characters, it only takes one byte to encode the length. In my implementation, I used this to specify the length of collections too. In fact, when writing a 32 bit signed integer, you should be able to save space for all positive numbers less than 2 ^ 22 and break even (require four bytes) for 2 ^ 29. Encoding an Int32 that is greater than or equal to 2 ^29 will require five bytes. Thus, if your integers tend to be smaller than 536,870,912, you'll probably save space encoding this way. A similar function could be used for a long where all positive values less than 2 ^ 57 will result in at least breaking even.

Here's how it works:

Convert the number to an unsigned int so you can do arithmetic on positive numbers (after all, you're just writing bits)
While the converted value is greater than or equal to 128 (i.e., 8 bits)
1. Write low 7 bits and put a 1 in the high bit (to indicate to the decoder more bytes are coming)
2. Shift the 7 bits you just wrote off the number
When the loop finishes, there will be 7 or fewer bits to write so write them

I think this is really clever and very easy. Reading the data back is a smidgen more complicated (Read7BitEncodedInt()):

protected internal int Read7BitEncodedInt()
{
  // some names have been changed to protect the readability
  int returnValue = 0;
  int bitIndex = 0;

  while (bitIndex != 35)
  {
    byte currentByte = this.ReadByte();
    returnValue |= ((int) currentByte & (int) sbyte.MaxValue) << bitIndex;
    bitIndex += 7;

    if (((int) currentByte & 128) == 0)
      return returnValue;
  }

  throw new FormatException(Environment.GetResourceString("Format_Bad7BitInt32"));
}

Here's how this works:

Set up an int to accumulate your data as you read them
Set up a place to keep track of which 7-bit block you're reading
While your bit index is less than 35 (i.e., you've read no more than 5 bytes comprising 1 "more bytes" indicator and 7 data bits)
1. Read a byte
2. Take the byte you just read and logical conjuction (bitwise and) it with sbyte.MaxValue (127 or in binary 0111 1111)
3. Left shift those 7 bits to the position in which they belong
4. Use a logical disjunction (bitwise or) to write those seven bits into your accumulator int
5. Add 7 to your bit index for the next 7 bits you read
6. If the current byte you read does not have a 1 in its most significant bit (i.e, byte & 1000 0000 == 0)
  1. There are no more bytes to read so just return the current accumulator value
If you get to this point, you've read a 6th of 5 bytes and it's time to let the user know there was a formatting problem

.Net GetHashcode Functions

2011-08-25T19:09:00.000-07:00

Last week, I wrote a post about a .Net method to combine hashcodes. In the process of designing that method, I looked into dozens of .Net's GetHashcode implementations.

If you'd like to know the principles of hashcodes, take a look at the aforementioned article. This one is part of the dotPeek of the Week series so I'll just be sharing the insight I got from the framework's implementations here.

First, some really basic ones:

// from Int32
public override int GetHashCode()
{
  return this;
}

// from Int16
public override int GetHashCode()
{
  return (int) (ushort) this | (int) this << 16;
}

// from Int64
public override int GetHashCode()
{
  return (int) this ^ (int) (this >> 32);
}

The Int32 implementation easily meets the hashcode requirements by returning itself. The Int16 copies itself to the top half of the Int32 and returns that. The Int64 takes the bottom half of itself and XORs it with the top half.

This XOR is the first valuable piece of information. If you look at the logical functions, the XOR produces the best bit twiddling for hashing. Basically, the XOR produces a relatively even distribution of bits as opposed to the OR and the AND operations which will be biased 3 to 1 in one direction or the other.

Some more complicated implementations:

// from String
public override unsafe int GetHashCode()
{
  fixed (char* chPtr = this)
  {
    int num1 = 352654597;
    int num2 = num1;
    int* numPtr = (int*) chPtr;
    int length = this.Length;
    while (length > 0)
    {
      num1 = (num1 << 5) + num1 + (num1 >> 27) ^ *numPtr;
      if (length > 2)
      {
        num2 = (num2 << 5) + num2 + (num2 >> 27) ^ numPtr[1];
        numPtr += 2;
        length -= 4;
      }
      else
        break;
    }
    return num1 + num2 * 1566083941;
  }
}

// from Tuple<>
internal static int CombineHashCodes(int h1, int h2)
{
  return (h1 << 5) + h1 ^ h2;
}

These two methods have some really good information in them. The String.GetHashcode implementation has what's called a rolling hash. It loops through the characters, does a barrel shift by 5, and then XORs with the next character.

While I liked this, I preferred the simplicity of the Bernstein Hash. The main component of the Bernstein Hash is the (i << 5) + i. i << 5 == i * 32 (except that bit shifting is much faster than multiplying).

Thus, i << 5 + i == 32i + i == 33i. The Bernstein Hash just takes the current hash value, multiplies it by 33, and XORs in the new value.

I didn't use 33 in my hash function because I feel like using prime numbers is healthy. Thus, instead of adding I subtract so my hashcode method is (hash << 5) - hash ^ value (or 31 * hash ^ value). This, by the way, is the way Java tends to do it.

Here are some interesting ones from System.Drawing:

// from Size
public override int GetHashCode()
{
  return this.width ^ this.height;
}

// from Rectangle
public override int GetHashCode()
{
  return this.X ^ (this.Y << 13 | (int) ((uint) this.Y >> 19)) ^ (this.Width << 26 | (int) ((uint) this.Width >> 6)) ^ (this.Height << 7 | (int) ((uint) this.Height >> 25));
}

// from Point
public override int GetHashCode()
{
  return this.x ^ this.y;
}

.Net Method to Combine Hash Codes

2011-08-18T15:26:00.000-07:00

A while back I developed a helper method that I've been using to aid me in computing good hash values from sets of properties for an object. I was writing a dotPeek of the Week entry about .Net GetHashcode implementations and I decided to spruce up my implementation and post it on my blog.

I know It's pretty uncommon that you need to override the Equals method, but when you do, you have to take particular care in considering overriding the GetHashCode method as well. To make this a little easier, I developed a method I call CombineHashCodes().

Using CombineHashCodes, I can easily compute a well randomized composite hash code based on several parameters from an object. This curtails a lot of the work involved when dealing with objects that have complicated .Equals overrides.

Microsoft's documentation on Object.GetHashCode() lists the following rules:

A hash function must have the following properties:

If two objects compare as equal, the GetHashCode method for each object must return the same value. However, if two objects do not compare as equal, the GetHashCode methods for the two object do not have to return different values.

The GetHashCode method for an object must consistently return the same hash code as long as there is no modification to the object state that determines the return value of the object's Equals method. Note that this is true only for the current execution of an application, and that a different hash code can be returned if the application is run again.

For the best performance, a hash function must generate a random distribution for all input.

Thus, if you override the .Equals method such that it's no longer doing the default reference equality, you probably need to change your GetHashCode implementation. I've found this is usually a very simple task. I whipped up a quick example in LINQPad:

void Main()
{
 var lauren1 = new Person { FirstName = "Lauren" };
 var lauren2 = new Person { FirstName = "Lauren" };
 
 Console.WriteLine(lauren1.Equals(lauren2));
 Console.WriteLine("{0}.GetHashCode() = {1}", "lauren1", lauren1.GetHashCode());
 Console.WriteLine("{0}.GetHashCode() = {1}", "lauren2", lauren2.GetHashCode());
 
 if (lauren1.Equals(lauren2) && lauren1.GetHashCode() != lauren2.GetHashCode())
  Console.WriteLine("This is bad.  This is very bad.");
}

public class Person
{
   public string FirstName { get; set; }
}

/*
False
lauren1.GetHashCode() = 47858386
lauren2.GetHashCode() = 20006478
*/

As one would expect, lauren1 does not equal lauren2 and neither do the hashcodes; however, if we were to override the .Equals method on the Person object to compare just the first name, we'll find that lauren1 and lauren2 will be equal but will have different hashcodes! This violates the first rule of GetHashCode Club. Here's an example:

public class Person
{
   public string FirstName { get; set; }
  
   public override bool Equals (object obj)
   {
  var otherPerson = obj as Person;
  
  if (otherPerson == null)
   return false;
   
  return String.Compare(this.FirstName, otherPerson.FirstName, StringComparison.OrdinalIgnoreCase) == 0;
 }
}

/*
True
lauren1.GetHashCode() = 33193253
lauren2.GetHashCode() = 37386806
This is bad.  This is very bad.
*/

That's why we override GetHashCode as well and something like this will work:

public override int GetHashCode()
{
 return FirstName.GetHashCode();
}

/*
True
lauren1.GetHashCode() = 2962686
lauren2.GetHashCode() = 2962686
*/

That's all well and good (except the .Equals() does a case insensitive compare so "Lauren" and "lauren" will be equal but will result in different hash codes unless you do something like return FirstName.ToLower().GetHashCode(). Why'd I leave that error in the post just to parenthetically correct it . . . because I think it's a good demonstration of how easy it is to screw this up :)). Consider what happens when you end up with a more complex class that looks like this:

public class Person
{
   public string FirstName { get; set; }
 public string LastName { get; set; }
 public DateTime BirthDate { get; set; }
  
   public override bool Equals (object obj)
   {
  var otherPerson = obj as Person;
  
  if (otherPerson == null)
   return false;
   
  return String.Compare(FirstName, otherPerson.FirstName, StringComparison.OrdinalIgnoreCase) == 0
   && String.Compare(LastName, otherPerson.LastName, StringComparison.OrdinalIgnoreCase) == 0
   && BirthDate.Equals(otherPerson.BirthDate);
 }
}

You could leave the GetHashCode function the way it was previously, but then all Lauren's will be lumped in the same bucket together which violates the third rule about a random distribution. I've seen people combat this problem by concatenating the string values of the equals fields delimited by some extremely unlikely string and getting the hash code of that.

I felt like that was not really the best way to do it in case someone decides a colon is a good character in a first name or something worse.

Thus, I wrote something like this (which has recently been modified to what you see now based on my explorations for my dotPeek of the Week series). This is CombineHashCodes:

public static class HashCodeHelper
{
 public static int CombineHashCodes(params object[] args)
 {
  return CombineHashCodes(EqualityComparer<object>.Default, args);
 }

 public static int CombineHashCodes(IEqualityComparer comparer, params object[] args)
 {
  if (args == null) throw new ArgumentNullException("args");
  if (args.Length == 0) throw new ArgumentException("args");

  int hashcode = 0;

  unchecked
  {
   foreach (var arg in args)
    hashcode = (hashcode << 5) - hashcode ^ comparer.GetHashCode(arg);
  }

  return hashcode;
 }
}

With this method, you can get a nice pseudo-random distribution of hash codes without violating any of the GetHashCode rules as long as you include the values relevant to your Equals method in your call to CombineHashCodes:

public override int GetHashCode()
{
 return HashCodeHelper.CombineHashCodes(FirstName.ToLower(), LastName.ToLower(), BirthDate);
}

/*
True
lauren1.GetHashCode() = -891990792
lauren2.GetHashCode() = -891990792
*/

.Net Optimization for Int32

2011-08-15T14:14:00.001-07:00

Another entry in the dotPeek of the Week series here. I was digging through some .GetHashcode() implementations (expect that as the next dotPeek of the week) and noticed some interesting implementations of overridden .Equals() methods.

I found this in Int16:

public override bool Equals(object obj)
{
  if (!(obj is short))
    return false;
  else
    return (int) this == (int) (short) obj;
}

public bool Equals(short obj)
{
  return (int) this == (int) obj;
}

So, I checked Byte and sure enough:

public override bool Equals(object obj)
{
  if (!(obj is byte))
    return false;
  else
    return (int) this == (int) (byte) obj;
}

public bool Equals(byte obj)
{
  return (int) this == (int) obj;
}

Can you guess how Char.Equals() is implemented?

public override bool Equals(object obj)
{
  if (!(obj is char))
    return false;
  else
    return (int) this == (int) (char) obj;
}

public bool Equals(char obj)
{
  return (int) this == (int) obj;
}

Even the unsigned int gets cast as a signed int in UInt.Equals(). I was pretty curious, so I looked at some other methods:

public int CompareTo(short value)
{
  return (int) this - (int) value;
}

public int CompareTo(byte value)
{
  return (int) this - (int) value;
}

public int CompareTo(char value)
{
  return (int) this - (int) value;
}

I took note and discussed it with a few friends. It seems to make sense that on a 32 bit operating system, it'd be optimal to work with 32 bit integers. It turns out, even on a 64 bit architecture, .net is optimized for the Int32. I found this gem online:

Best Practices: Optimizing performance with built-in types

The runtime optimizes the performance of 32-bit integer types (Int32 and UInt32), so use those types for counters and other frequently accessed integral variables.

For floating-point operations, Double is the most efficient type because those operations are optimized by hardware.

MCTS Self-Paced Training Kit (Exam 70-536): Microsoft® .NET Framework 2.0—Application Development Foundation

So storing integer values, no matter the ceiling of your expected value, is best done using Int32, particularly if they'll be operated on heavily like an index to a collection or as a counter in an iteration control structure.

Enumerable.Any vs. Enumerable.Count

2011-08-08T15:58:00.000-07:00

This is the innagural entry into the section of my blog I'm calling "dotPeek of the Week." Workload permitting, it will be a weekly thing where I'll be using JetBrain's dotPeek - a free .NET decompiler to learn more about the .NET framework and share any interesting things I come up with.

In this post, I'll be sharing a little tip I picked up from my friend David Govek.

I can't .Count() the number of times I've seen a line of code like this one:

if (someEnumerable.Count() > 0) 
    doSomething();

I know I've done that myself a handful of times. I think it comes from the pre-.net 3.5 days when you were used to dealing with ICollections that had a .Count property that returned the value of a private field:

public virtual int Count { get { return this._size; } }

When 3.5 came out, the System.Linq.Enumerable class brought with it a Count extension method. I believe it was at that point that people started using .Count() everywhere, including on IEnumerables, which previously didn't have a method for getting the length of the enumerable.

Most of the time, there was very little pain because the engineers over at Microsoft were clever enough to help us out. The first thing they try to do in the .Count extension method is check to see if the IEnumerable is an ICollection. If it is, they just use the Count property which we already know is plenty fast; however, if it's not an ICollection, they have to iterate the Enumerable and count the elements.

Here's what that looks like:

public static int Count<TSource>(this IEnumerable<TSource> source)
{
  if (source == null)
    throw Error.ArgumentNull("source");

  ICollection<TSource> collection1 = source as ICollection<TSource>;
  if (collection1 != null)
    return collection1.Count;

  ICollection collection2 = source as ICollection;
  if (collection2 != null)
    return collection2.Count;

  int num = 0;
  using (IEnumerator enumerator = source.GetEnumerator())
  {
    while (enumerator.MoveNext())
      checked { ++num; }
  }
  return num;
}

If your source is indeed an Enumerable, this is still a fine way to find out how many items there are. The problem is, in the sample code where we're just simply checking to ensure that the Enumerable isn't empty, using .Count can prove costly. Checking that the count is greater than 0 has to enumerate the entire collection and count each item despite the fact that we know it's not empty as soon as we spot the first element.

Fortunately, the clever folks at Microsoft thought of this and also gave us .Any(). This extension method simply gets the Enumerator, calls .MoveNext(), and disposes the Enumerator. MoveNext tries to move to the next element in the collection and returns true until it passes the end of the collection.

Here's what the .Any() method looks like:

public static bool Any(this IEnumerable<TSource> source)
{
  if (source == null)
    throw Error.ArgumentNull("source");

  using (IEnumerator<TSource> enumerator = source.GetEnumerator())
  {
    if (enumerator.MoveNext())
      return true;
  }

  return false;
}

Thus, there's no need to Enumerate the entire Enumerable if you can use .Any() like this refactored code:

if (someEnumerable.Any()) 
    doSomething();

About dotPeek of the Week

2011-08-07T10:23:00.000-07:00

Yesterday, I posted the first post of a series I call the dotPeek of the Week. Why am I using a JetBrains product name in my blog? Well, 'cause I like JetBrains. I love ReSharper for C#, IntelliJ for Android development, and of course, dotPeek for decompiling .Net code.

I also want to support JetBrains by encouraging people to use dotPeek so that they'll keep it free, unlike that other one that got bought by that one company and now is a paid app. Hopefully, I won't have to change the name of this section any time soon :).

The other (and primary) objective of this exercise is to learn more about the .Net framework and how it functions. If I find something interesting, I'll share it here on my blog.

Latest dotPeek of the Weeks

All dotPeek of the Week posts

C# Yield Keyword, IEnumerable, and Infinite Enumerations

2011-07-25T08:00:00.000-07:00

Visual Studio 2005 came with a slew of .net and .net compiler features. One of those features that I particularly enjoy is the yield keyword. It was Microsoft's way of building IEnumerable (or IEnumerator) classes and generic classes around your iterator code block. I'm not going to discuss the yield keyword much in this post because the feature has been around a long time now and the internet is replete with discussions on the topic.

Despite the long life of the yield keyword, I still find it conspicuous when I see it in projects I work on. I suppose it's just rare that I find myself writing my own enumerable. As a result, when I see yield return, it tends to stand out. I started looking around to see how the rest of the programming world uses the yield keyword wondering if I was under-utilizing the flexibility provided.

Specifically, I wondered if I was missing out on the lazy nature of the iterator and the numerous linq extension methods optimized to take advantage of that aspect of iterators. What I mean by that is that an iterator doesn't need to store each sequential value in memory the way a collection would and thus you can use each value without necessarily increasing the memory overhead. Further, you can take advantage of calculations which tend to already be sequential in nature (like the Fibbonacci sequence for example).

The second thing I thought about was an infinite (well, sort of infinite) enumerable. I'm not sure if I feel like it's a bad idea or not so I wrote these examples to be unending? I may eventually find a use for such an iterator and then get burned when someone tries to call Fibbonacci.Min() and the application throws an overflow exception, but I suppose at that point I'll make it a method and take a sanity check variable.

In the meantime, here are a few examples of some iterators I thought were interesting and fun challenges:

static IEnumerable<ulong> Fibbonacci
{
    get
    {
        yield return 0;
        yield return 1;

        ulong previous = 0, current = 1;
        while (true)
        {
            ulong swap = checked(previous + current);
            previous = current;
            current = swap;
            yield return current;
        }
    }
}

static IEnumerable<long> EnumerateGeometricSeries(long @base)
{
    yield return 1;

    long accumulator = 1;
    while (true)
        yield return accumulator = checked(accumulator * @base);
}

static IEnumerable<ulong> PrimeNumbers
{
    get
    {
        var prime = 0UL;
        while (true)
            yield return prime = prime.GetNextPrime();
    }
}

static IEnumerable<List<uint>> PascalsTriangle
{
    get
    {
        var row = new List<uint> { 1 };
        yield return row;

        while (true)
        {
            var last = row[0];
            for (var i = 1; i < row.Count; i++)
            {
                var current = row[i];
                row[i] = current + last;
                last = current;
            }

            row.Add(1);
            yield return row;
        }
    }
}

Some of these methods use a few extension methods I adapted from places in the .net framework:

public static ulong GetNextPrime(this ulong from)
{
    for (var j = from + 1 | 1UL; j < ulong.MaxValue; j += 2)
        if (j.IsPrime())
            return j;

    return from;
}

public static bool IsPrime(this ulong value)
{
    if ((value & 1) != 0)
    {
        var squareRoot = (ulong)Math.Sqrt((double)value);
        for (ulong i = 3; i <= squareRoot; i += 2)
            if (value % i == 0)
                return false;

        return true;
    }
    return value == 2;
}

To keep the test console app clean, of course, I used my favorite IEnumerable.Each() extension method:

public static void Each<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    foreach (var element in enumerable)
        action(element);
}

Here's the sample code:

static void Main()
{
    Fibbonacci.Skip(10).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    EnumerateGeometricSeries(2).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    PrimeNumbers.Where(p => p > 600).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    foreach (var row in PascalsTriangle.Take(10))
    {
        row.Each(element => Console.Write("{0} ", element)); 
        Console.WriteLine();
    }

    Console.ReadLine();
}

// The second set of 10 elements in the Fibbonacci sequence
// 55 89 144 233 377 610 987 1597 2584 4181

// Base of 2 to the first 10 powers
// 1 2 4 8 16 32 64 128 256 512

// The first 10 prime numbers greater than 600
// 601 607 613 617 619 631 641 643 647 653

// The first 10 rows of Pascal's Triangle
// 1
// 1 1
// 1 2 1
// 1 3 3 1
// 1 4 6 4 1
// 1 5 10 10 5 1
// 1 6 15 20 15 6 1
// 1 7 21 35 35 21 7 1
// 1 8 28 56 70 56 28 8 1
// 1 9 36 84 126 126 84 36 9 1

.Net ObjectFormatter - Using Tokens in a Format String

2011-07-20T07:20:00.000-07:00

If you've already read this article and you don't feel like scrolling through my sample formats, you can jump directly to ObjectFormatter on github to get the source.

At some point in almost every business application, it seems you eventually run into the ubiquitous email notifications requirement. Suddenly, in the middle of what was once a pleasant and enjoyable project come the dozens of email templates with «guillemets» marking the myriad fields which will need replacing with data values.

You concoct some handy way of storing these templates in a database, on the file system, or in resource files. You compose your many String.Format() statements with the dozens of variables required to format the email templates and you move on to the greener pastures of application development.

Now, you've got a dozen email templates like this one:

Dear {0},

{1} has created a {2} task for your approval. This task must be reviewed between {3} and {4} to be considered for final approval.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

No big deal, everything is going swimmingly, and the application goes into beta. Then, it turns out, the stakeholders don't want an email template that looks like that. That was more of a draft really. Besides, you should've already known what they wanted in the template to begin with. After all, it's like you have ESPN or something.

It's important to add information about the user for whom this action is taking place, so this is your new template:

Dear {0},

{1} has created a {2} task for your approval regarding {5}({6}). This task must be reviewed between {3} and {4} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {7}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

So far so good. You've updated the template, updated your String.Format() parameters, passed QA and gone into production. But, now that users are actually hitting the system, it turns out that you need a few more changes. Specifically, you need to add contact information for the supervisor, remove the originator of the task, and by the way, what kind of sense does it make to put a low end limit on a deadline? Here's your new template:

Dear {0},

A {2} task for {5}({6}) is awaiting your approval. This task must be reviewed by {4} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {7} at {1}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

Now you have an email template format with various numbers all over the place, a String.Format() call with more parameters than there are tokens, and you have to go through the QA - deployment cycle again.

I've gone through this process on almost every application throughout my career as a software engineer. Hence the ObjectFormatter. Now, my email template looks like this:

Dear {Employee.FullName},

A {Task.Description} task for {TargetUser.FullName}({TargetUser.UserId}) is awaiting your approval. This task must be reviewed by {DueDate} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {Supervisor.FullName} at {Supervisor.PhoneNumber}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

I find that the ObjectFormatter makes my templating much easier to maintain and much more flexible. It also usually makes my calling code a lot cleaner. Here's an example of the approaches you could take to populate the sample templates:

// plain string formatting
String.Format(template, Employee.FullName, Supervisor.PhoneNumber, Task.Description, String.Empty, DueDate, TargetUser.FullName, TargetUser.UserId);

// if you have a dto already built
ObjectFormatter.Format(template, myDto);

// if you don't have a dto built
ObjectFormatter.Format(template, new { Employee, Supervisor, Task, DueDate, TargetUser });

I've found that most of the time they ask for template changes, they want me to add some value that is already a property on an object that's already in my object graph because of the current email template. That way, when they come tell me they want he target user's name formatted differently, I don't even need to recompile (well, sometimes I do . . . I mean, I can't predict everything). I can implement a lot of changes by using objects I already know I'm passing into the ObjectFormatter.Format() method. Here's the new template with the changes and I didn't have to change a line of code to make it work:

Dear {Employee.FullName},

A {Task.Description} task for {TargetUser.LastName}, {TargetUser.FirstName}({TargetUser.UserId}) is awaiting your approval. This task must be reviewed by {DueDate} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {Supervisor.FullName} at {Supervisor.PhoneNumber}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

If you'd like to check out the source or use the ObjectFormatter in your own projects, look for ObjectFormatter on github. If you make any cool changes, please let me know and I'll try to figure out how to merge them into the repository.

Extension Method to Replace foreach With Lambda Expression

2011-07-19T16:45:00.000-07:00

It's pretty often I find myself looping through some enumerable and performing an action on the elements. Sometimes it's just displaying results with Console.WriteLine. Other times I need to do something a little more complicated. In any case, every once in a while, I feel like the foreach statement and the for statement aren't really quite expressive enough.

That's why I have this little guy:

public static void Each<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    foreach (var element in enumerable)
        action(element);
}

It's pretty basic but I like the way it looks and feels. I used it in my blog post about a C# UpTo Extension Method a la Ruby's int.upto method.

Here's a simple demonstration I wrote in LinqPad:

void Main()
{
    Enumerable.Range(1, 5).Each(Console.WriteLine);
}

static class Extensions
{
    public static void Each<T>(this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
            action(element);
    }
}

In writing this post (and perhaps because I've been spending way too much time with jQuery lately), it occurred to me that I may want to be able to chain my actions with another Each() or with other extensions from Linq perhaps:

void Main()
{
    Enumerable.Range(1, 5).Each(Console.WriteLine).Each(Console.WriteLine);
    // 1 2 3 4 5 1 2 3 4 5
}

static class Extensions
{
    public static IEnumerable<T> Each<T>
        (this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
            action(element);
   
        return source;
    }
}

The problem is though, that generally you don't want your action to enumerate your enumerable until something is to be done with the results. Instead, you often want it to be executed during the enumeration of your enumerable, so you'd write it like this:

void Main()
{
    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Where(i => i <= 5)
        .ToList();
    // 1 2 3 4 5 6 7 8 9 10
    
    Console.WriteLine();
    
    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Take(5)
        .ToList();
    // 1 2 3 4 5

    Console.WriteLine();

    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Skip(5)
        .Take(5)
        .ToList();
    // 1 2 3 4 5 6 7 8 9 10
}

static class Extensions
{
    public static IEnumerable<T> Each<T>
        (this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
        {
            action(element);
            yield return element;
        }
    }
}

GiantDropdown jQuery Plugin for Styling Large Select Lists

2011-06-17T15:36:00.000-07:00

A few months ago, I was working on a performance management application for the Centers for Disease Control. They wanted a select box that contained a list of competencies from which they could compose a template. The problem was some of the items in the list were hundreds of characters long and, at the time of writing, most browsers would behave very badly with drop down boxes of that size.

I was sitting at home that night and decided I could probably come up with a jQuery plugin that would use the select list as the backing field and present a prettier display. I did and we ended up using it at the CDC. I decided I'd share in case anyone else runs into the same issue.

Here's an example of what one of the select boxes would look like:

It's not very pretty and it behaves unpredictably with other DOM elements. With the GiantDropdown plugin, we can style it any way we choose and still preserve all of the original behavior of the select list (meaning, it even works with multi-selects and option groups).

Here are some samples of the giant-ified dropdowns:

Sorry, I felt like I had to make it an iframe to get a reliable demo. If you can't see the samples, visit my blog for the giant dropdown demos at my old url. You can also check them out in git.

If you'd like to get the giant dropdown jquery plugin, the can check out the Giant Dropdown jQuery Plugin Git Repository. Feel free to make improvements.

Autopilot Consulting has a Logo!

2011-06-16T14:36:00.000-07:00

At long last, Autopilot Consulting has an official logo! It's exciting to see this dream of mine coming together and now there's a face to the name.

I know it seems slow going. After all, I posted the announcement of Autopilot at the end of May and here we are in the middle of June and I'm just getting my logo together.

Well, it's true. I'm trying to get all of my ducks in a row in my spare time while most of my efforts are still spent trying to do a great job here at the Centers for Disease Control. We're not doing the same mainframe migration project anymore (but I still intend to finish that series someday). We've actually started doing some pretty interesting asp.net projects. I usually prefer mvc.net, but they focus on very rich user experience which is a good experience for a nuts-and-bolts kind of guy like me.

I noticed that not only are my blog posts becoming fewer and further between, they're also pretty non-technical. I promise to spend a little more time writing about the things I'm learning out there in the thick of it (and maybe I'll have a little advice from time to time for those who are thinking about going independent).

The Beginning of Autopilot Consulting, LLC

2011-05-27T08:39:00.000-07:00

If you follow my blog, there are a few things I'd imagine you know about me:

I love writing software!
I love feeling like I'm improving the lives of my customers.
I love feeling like I'm doing my job well.
I love flying airplanes.

I Love Writing Software!

This is the very reason I got into the software industry in the first place. I get such a thrill out of a good days work programming. I love the challenge of solving problems as quickly, creatively, and efficiently as possible. A software engineer not only thinks about solving problems, but thinks about thinking about solving problems. Our craft is always changing and growing and it's great to be a part of this community.

Improving the Lives of my Customers

Whether I'm your employee, I'm a contractor for you, or you purchase my commercially available software, my goal is to find some way to improve your life. I've worked in paperless onboarding, business process automation, data warehousing, entertainment, e-commerce, insurance, pharmaceutical research, and academia.

There's one thing I've found in common no matter what kind of software I've been writing. That is that I find the most joy in my work when I can change peoples lives for the better. I like to think that some of the work I have done has bettered the human condition, and I am hopeful that my work will continue to do so.

Doing my Job Well

If I'm not doing my job well, I shouldn't be doing it. If I know I'm not doing a good job, I'm not satisfied with my work. If I'm not satisfied with my work, my customers can't be satisfied with my work. Sometimes you find yourself in a position where you can't meet your full potential. Other times, you don't know what your full potential is.

It is for this reason that I've decided to strike out on my own and form my own software company. I've been developing software for a long time working for various companies. I've decided that starting my company will allow me to touch the lives of more people.

I hope to continue consulting in the software industry while I build my library of commercial products including web based services and mobile applications.

Flying Airplanes

About the same time I got my first job with the word "programmer" in the title, I started flying airplanes. Since then I've flown tailwheel, multi-engine, upside down, through the clouds, and I can't get enough. One thing I've learned is that sometimes an airplane has so much going on, that you just can't do everything by yourself. That's why some planes require 2 pilots.

That's also why a lot of airplanes have autopilots. An autopilot is there to take care of going where you tell it to so you can focus on the important things. That's why Autopilot Consulting is here. We want you to focus on what's important &emdash; your business; we will handle the technology.

Together, we'll grow your business and improve the lives of your customers.

Do more. Work less. Put your business on autopilot.

Telephone Game for Android

2011-01-26T18:36:00.000-08:00

You may know it as Grapevine, Gossip, or the Telephone Game; we call it Chinese Whispers. No matter what you call it, this is a new take on the classic game and we hope you enjoy playing it as much as we enjoyed making it for you.

You start the game with a short story and pass it to a friend. The next player illustrates the story with our built in finger painting canvas and passes it to the next player who will provide a new caption. Several rounds later, hilarity ensues!

We hope you'll check the Android Market for the free Telephone Game for Android by Mobile Magic Developers.

WTF is WTF Ware?

2011-01-26T18:12:00.000-08:00

James and I worked together for about 3 years at Emerald Software Group. James went off to ThoughtWorks and I stuck with Emerald. We've both spent most of our careers writing business applications.

While we love leaving a big project knowing we've made a difference, we got to wondering how we could write software that enriches the lives of more people. We decided we'd get together and start writing mobile apps that are entertaining, useful, funny, amusing, and enriching.

We started writing apps on our own and have had a lot of fun doing it. We decided that we ought to join our efforts and see what we came up with. Thus, WTF Ware was born.

I wish I could say we had a good story for how the name came to be or at least something clever WTF Ware could stand for. Unfortunately, no such story exists.

We were just about to start writing our first app and we realized we needed some sort of organization name. James asked, "WTF is our company name?"

"Beats me dude. Just make something up for now and we'll come up with something later."

Well, WTF Ware is what James made up and it stuck. So, without further ado, I'd like to introduce ourselves. We're wtfware.com and thanks for checking out our applications.

Upgrade Chuck Norris Facts Free

2011-01-24T10:12:00.000-08:00

Initially, when I released Chuck Norris Facts, I published two versions. I had a free version which was limited in facts and functionality. Then I published a paid version that included all the facts and customization.

This proved difficult to maintain, so I decided to make the free version ad supported and (at the time of writing) fully functional. Unfortunately, it was still a little difficult to maintain. I still wanted to have only one code base so that I could release one app all the time. Another issue was that if you have the free version for a while and switch to the paid version, you lose your fact history and start over.

Thus, I switched upgrade paths. Now, I encourage everyone to try the Free Chuck Norris Fact Widget for a while until you decide that you like it. Then, you can purchase the Chuck Norris Fact Widget Upgrade for less than a buck. This will remove the ads immediately and you won't lose your current fact history.

If you are a user who has already installed the full Chuck Norris Fact Widget, you can download and install the free version and there will be no ads (as long as you don't delete the paid version). You'll have to start over with your facts (but they're random anyhow so it shouldn't be too bad). This way you'll continue getting updates with new facts and seasonal backgrounds.

If you really want to delete the previous paid version, send an email to me and I'm mail you a buck or something so you can download the paid key.

I'd also like to thank all of the users of both the free and paid versions for sticking with me through this learning experience. I hope it's been as enjoyable to use as it has been to write.

Sincerely,
Patrick Caldwell

Chuck Norris Facts Widget for Android

2011-01-06T07:12:00.000-08:00

A few weeks ago, I decided I'd like to try my hand at writing an Android application. One of my favorite features of Android over iOS (other than the fact that I think Apple lacks any form of ethics or sense of social responsibility) is that Android allows you to add widgets to your home screens. Sure, you can still have the half dozen views full of rows of boring icons, but the widgets give you a great deal of functionality without ever having to launch an app.

I decided a good first shot at Android development would be to write a Chuck Norris Fact of the Day widget my family and friends could stick on their home screens and enjoy a little daily pick-me-up. I knew that adding customizability would let me work with preferences and the nature of a daily fact would give me some Android data management experience.

I released the Chuck Norris Fact Widget on the Android Market and several of my family and friends downloaded it the first day. Over the next few weeks, about 1,500 of their family and friends downloaded it, so I've been working to improve it ever since.

If you're on your Android device now, you can download the free version called, Chuck Norris Facts Free or for just a buck, you can get the paid version called, Chuck Norris Facts with Widget. The icon should look familiar from above and the publisher name is, of course, D. Patrick Caldwell.

I hope you check it out and I hope you enjoy a little bit of my sense of humor.

Sincerely,
Patrick Caldwell

For the programmers who read my blog, I'll be posting some technical information I learned while developing this application. Specifically, how I managed to change the background image of my widget with remote views.

Improved UpTo Extension Method

2010-11-16T08:57:00.000-08:00

Back in May of '09, I wrote a post about an extension method to mimic Ruby's upto.

Today, I was hanging out at Visual Studio Live and had another idea. In Ruby, the upto method takes an end value and executes a code block. In C#, interpreted this as an extension method that takes an end value and an action. They syntax ends up being pretty similar and it works fine.

Lately, I've been using an Each extension method for IEnumerables. I know an Each extension method seems like a waste, but I like the fluency of it for basic actions. So, given that I have the Each method on IEnumerable<T> anyway, I decided UpTo should really return an IEnumerable as well.

Here's my new UpTo extension method:

public static IEnumerable<int> UpTo(this int start, int end)
{
    while (start <= end)
        yield return start++;
}

Now, instead of passing your action to the UpTo method like Ruby's upto, you get your enumerable and pass your delegate to your Each extension method. Here's what it looks like:

static void Main()
{
    //for (int i = 5; i <= 10; i++)
    //{
    //    Console.WriteLine(i);
    //}

    5.UpTo(10).Each(Console.WriteLine);
    
    Console.ReadLine();

    // output
    // 5
    // 6
    // 7
    // 8
    // 9
    // 10
}

If you're using .Net 3.5 or newer, Linq shipped with an Enumerator class which has a Range static method so your code could look more like this:

public static class NumericExtensions
{
    public static IEnumerable<int> UpTo(this int start, int end)
    {
        return Enumerable.Range(start, end - start + 1);
    }
}

class Program
{
    static void Main()
    {
        //foreach (var i in Enumerable.Range(5, 6))
        //{
        //    Console.WriteLine(i);
        //}

        5.UpTo(10).Each(Console.WriteLine);
        
        Console.ReadLine();

        // output
        // 5
        // 6
        // 7
        // 8
        // 9
        // 10
    }
}

Another reason to make UpTo return an IEnumerable instead of making it a void action executer is because sometimes you don't really want to pass a large anonymous method to the Each method. You can use the same UpTo method like this:

static void Main()
{
    //foreach (var i in Enumerable.Range(5, 6))
    //{
    //    // Do a lot of stuff!!
    //}

    //5.UpTo(10).Each(i =>
    //    {
    //        // Do a lot of stuff!!
    //    });

    foreach (var i in 5.UpTo(10))
    {
        // Do a lot of stuff!!
    }
}

Results of the "What Really Makes a Good Programmer" Survey

2010-10-28T20:31:00.000-07:00

Back in September I wrote a blog post called What Really Makes a Good Programmer? My goal was to ask various members of the development community what traits they thought contributed to the quality of a programmer. If you haven't taken the survey yet, I'd recommend you do that before reading the results. Wouldn't want to bias your opinions, right?

If you have taken the survey, but haven't told everyone you know to take it too, go ahead and do that; I'll wait.

Now that we've covered that, let's get on to the analysis. As you recall from taking the survey, there were 17 traits and you rated them on a scale of Very Important, Important, A little Important, Negligible, and Completely Unimportant. In order to do any analysis, I had to take these ratings and convert them to numbers.

I decided that A Little Important was the baseline and basically represented a lack of opinion on the topic. These traits are the ones most people feel are nice to have but aren't requirements. Basically, a good programmer often has these traits but not having them doesn't mean you're not a good programmer. I recoded these values with 0 points.

Very Important and Completely Unimportant are the ratings that people used when they felt strongly about the item. That means that, respectively, the trait is either absolutely necessary to being a good programmer or the trait has no bearing on the quality of programmer you are whatsoever (or perhaps a little bit of a negative indicator). These values were recoded as 10 and -10 respectively.

The central values of Important and Negligible are what I call the non-committal values. You feel that they make a difference but they're not quite important enough to bank on them. I gave these a 3 and -3 respectively.

For example, "Has good problem solving skills" was rated Very Important and "Cheap" was rated Completely Unimportant. One could say, "someone with poor problem solving skills would make a poor programmer." On the other hand, you cannot say, "someone who is not cheap is a poor programmer." Of course, the contrapositive could also be stated that, "someone who is a good programmer is also a good problem solver." The contrapositive "someone who is a good programmer is also cheap" is considered by the community to be a largely invalid assertion.

"Fast" and "Co-located" appear nearest to the baseline. This may be due to the fact that many of the respondents didn't know what I meant by co-located. In any case, one might say, "a good programmer is a good programmer whether or not she's in the same building," or that "just because you're fast doesn't mean you're good and just because you're slow doesn't mean you're bad."

"Communicates effectively" and "Interested in helping teammates" are moderately rated in favor of contributing to being a good programmer while the two "college degree" items are rated moderately against contributing to being a good programmer. For example, you might know a lot of programmers who are great developers but who lack the social skills or interest to become good communicators or team players. As a result, you might say that good programmers often communicate effectively and help teammates, but some good programmers cannot.

You might also say, "Many people without college degrees, let alone a computer science degree, are great programmers. Thus, while having a degree is helpful, you can learn to be a great programmer without it."

Now, I feel it is worth noting that "unimportant" does not mean "negative." Just because most of the development community feels that having a CS degree or other certifications is unimportant to being a good programmer, it doesn't mean that the degree itself is unimportant. It just means that having the degree won't make you a good programmer; you still have a lot of learning to do.

With notes on the analysis out of the way, what do you say we take a look at some data? Currently, the new Google charts won't let me force the display of the categories, so you'll have to hover over the dots you're interested in. Here's a legacy chart you can look at if the javascript version below is unsatisfactory. The legacy chart has current data; it's just not as fancy as this one:
I'm sure you noticed there are two series in this graph. All Responses represents the average responses for all respondents; however, as you noticed in the respondent histogram that I have a dramatically unbalanced sample with programmers outnumbering all other groups combined by almost 3 to 1. To get a more accurate representation of the "general feel" of the community, I included the Group Average measure. This is the average of the averages. It's sort of the electoral college of informal research.

This chart demonstrates the average responses for each group:

Improving Search Performance when using SQL Server 2008 Encrypted Columns

2010-10-28T12:27:00.000-07:00

This is a story of courage, honor, and data encryption. In addition to being something of a tribute to one of my favorite games of all time, I do feel I need to preface this post with a bit of a disclaimer.

Generally, when I describe this problem to someone, we go through pretty much the same conversation.

Why didn't you try Sql Server 2008 Transparent Data Encryption? Why don't you just update your ACL? Why don't you try this? Or that?

Well, sometimes the environment, either technologically or politically, precludes some of your better options. Ultimately, you have to go with your best permissible solution to solve the problem at hand. That being said, let's say you've been charged with securing your organizations PII. Your only option is to encrypt the column with SQL Server's column encryption.

You start looking at the impact this is going to have on your current and future applications. If you're like me, one of the first concerns you're going to have is performance. For example, let's say you have the need to search on the encrypted field. A good example of an encrypted field that you'll likely have to search is the Social Security Number.

In my original benchmarks, I found the following results:Knowing that I was going to have to improve this performance, I started playing with some searching alternatives. First of all, it's obviously much faster to search an indexed column (especially one that is clustered), so my first goal was to generate an indexed column.

Ryan McGarty (my best friend and a damn fine programmer) and I discussed and quickly ruled out a basic hash. Sure, it'd be fast to search an indexed column containing the hash value, but that opens you up to a relatively simple plain text attack (especially with a known set of possible values). I decided that you could reduce the threat and still speed up the search by using hash buckets a la hashtables instead.

I concocted a hash function that produced a reasonable distribution within the buckets. My good friend and esteemed colleague David Govek pointed out that an MD5 hash would produce a pretty effective distribution between buckets (with high cardinality data like the social security number. David was right and I ended up with this hash function:

CREATE FUNCTION [dbo].[GroupData] 
(
 @String VARCHAR(MAX),
 @Divisor INT
) 
RETURNS INT
AS BEGIN

  DECLARE @Result INT;
  
  SET @Result = HASHBYTES('MD5', @String);
  
  IF (@Divisor > 0)
    SET @Result =  @Result % @Divisor;
  
  RETURN @Result;

END

I created the test data just as I had before, except that this time I also added the hash bucket. I created a clustered index on the bucket and I changed my select statement a little:

-- original select statement
SELECT *
FROM PersonData
WHERE 
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

-- hash key select statement
-- @Divisor is the divisor for the modulus operation in the hash function
SELECT *
FROM PersonData
WHERE 
  SocialSecurityNumberGroup = dbo.GroupData('457555462', @Divisor) AND
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

Here are my results:

So, what of the known plain text attack then?

So, I don't want to gloss over the plain text attack issue. Given the possible set of socials, the hashes would be very unlikely to have a collision. Thus, for most people, you'd be able to get their socials easily by hashing every possible social and joining to that table. By modulo dividing the hash value, I'm able to evenly distribute social security numbers among a known set of buckets. That means, I can control the approximate number of socials in each bucket given my set of values. I generally aim for about 1000 socials per bucket.

For example, an MD5 % 100 has a possible set of values from -100 to +100. That's 201 buckets so if you have 2010 rows of data to hash, you'll have about 10 rows per bucket. The benefit is that you now only have to decrypt 10 rows to find the exact row you're looking for. The detriment is that you've narrowed your possible result set. Within your own data set, you'd have narrowed it to 10 possible plaintext values; however, given that these values are unknown, you then have to look at the set of possible values.

Social security numbers have a set of possible values less than 1,000,000,000. It's hard to say exactly how many of them are the possible set of values, so let's say we're using only those socials currently in use by living Americans. The population of the United States at the time of writing was about 312,000,000. As I said, I usually aim for about 1,000 records per bucket. If I have 1,000,000 rows of data, I would modulo divide by 500 (1,001 buckets). If you knew my set of values, then you'd only have 1,000 possible values to reduce. Given that you know when and where I was born, you could probably narrow it to 20 or so possible socials.

But, you don't know my set of values, so you really have 312,000 values to chose from. Even if you did know the first 5 digits of my social security number (based on my state of issuance and my date of birth), your set of possible options would be so large, you'd probably be better off just pulling my credit report and getting my social that way.

Thus, while it seems to introduce a weakness to plain text attacks (and with low cardinality data it would be an issue), in the case of social security numbers, I don't believe it to be a reasonable attack.

Column Encryption in SQL Server 2008 with Symmetric Keys

2010-10-28T08:40:00.000-07:00

Most of the time when I write blog posts, I do it to share ideas with my fellow developers. Sometimes I do it just so I can have a place to reference when I forget the syntax for something. This is one of those reference posts.

Recently I've been charged to column level encrypt some personally identifiable information. The present post is not intended to discuss the merits of column level encryption; rather, as I said it is to put a few code snippets up so that I can reference them later. If you should find yourself in a column level encryption predicament in a SQL Server 2008 environment, you may find these useful as well.

First thing's first. Get the database ready for column level encryption by creating a master key:

--if there is no master key create one
IF NOT EXISTS 
(
  SELECT * 
  FROM sys.symmetric_keys 
  WHERE symmetric_key_id = 101
)
CREATE MASTER KEY ENCRYPTION BY 
  PASSWORD = 'This is where you would put a really long key for creating a symmetric key.'
GO

Now, you'll need a certificate or a set of certificates with which you will encrypt your symmetric key or keys:

-- if the certificate doesn't, exist create it now
IF NOT EXISTS
(
  SELECT *
  FROM sys.certificates
  WHERE name = 'PrivateDataCertificate'
)
CREATE CERTIFICATE PrivateDataCertificate
   WITH SUBJECT = 'For encrypting private data';
GO

Once you have your certificates, you can create your key or keys:

-- if the key doesn't exist, create it too
IF NOT EXISTS
(
  SELECT *
  FROM sys.symmetric_keys
  WHERE name = 'PrivateDataKey'
)
CREATE SYMMETRIC KEY PrivateDataKey
  WITH ALGORITHM = AES_256
  ENCRYPTION BY CERTIFICATE PrivateDataCertificate;
GO

Before you can use your symmetric key, you have to open it. I recommend that you get in the habit of closing it when you're finished with it. The symmetric key remains open for the life of the session. Let's say that you have a stored procedure in which you open the symmetric key to decrypt some private data which your stored procedure uses internally. Someone who has access to the stored procedure can run it and then will have the key opened for use in decrypting private data. My point, close the key before you leave the procedure. Here's how you open and close keys.

-- open the symmetric key with which to encrypt the data.
OPEN SYMMETRIC KEY PrivateDataKey
   DECRYPTION BY CERTIFICATE PrivateDataCertificate;

-- close the symmetric key
CLOSE SYMMETRIC KEY PrivateDataKey;

Here's a little test script I wrote to demonstrate a few points. First, the syntax for encrypting and decrypting. Second, the fact that the the cipher text changes each time you do the encryption. This prevents a plain text attack.

-- open the symmetric key with which to encrypt the data.
OPEN SYMMETRIC KEY PrivateDataKey
   DECRYPTION BY CERTIFICATE PrivateDataCertificate;

-- somewhere to put the data
DECLARE @TestEncryption TABLE
(
  PlainText VARCHAR(100),
  Cipher1 VARBINARY(100),
  Cipher2 VARBINARY(100)
);

-- some test data
INSERT INTO @TestEncryption (PlainText)
SELECT 'Boogers'
UNION ALL
SELECT 'Foobar'
UNION ALL
SELECT '457-55-5462'; -- ignoranus

-- encrypt twice
UPDATE @TestEncryption
SET 
  Cipher1 = ENCRYPTBYKEY(KEY_GUID('PrivateDataKey'), PlainText),
  Cipher2 = ENCRYPTBYKEY(KEY_GUID('PrivateDataKey'), PlainText);

-- decrypt and display results  
SELECT
  *,
  CiphersDiffer = CASE WHEN Cipher1 <> Cipher2 THEN 'TRUE' ELSE 'FALSE' END,
  PlainText1 = CONVERT(VARCHAR, DECRYPTBYKEY(Cipher1)),
  PlainText2 = CONVERT(VARCHAR, DECRYPTBYKEY(Cipher2))
FROM @TestEncryption;

-- close the symmetric key
CLOSE SYMMETRIC KEY PrivateDataKey;

Overdraft Fee Survey

2010-09-21T20:18:00.000-07:00

OK, I know I've been big on the surveys lately, but this one is really important to me.

As you probably know, there's been a lot of hoopla about the new overdraft fee regulations. Banks are no longer allowed to automatically enroll customers in what they call, "overdraft protection." To us common folk, we generally call them "overdraft fees" or "allowing you to spend money you don't have so we can screw you out of more money we know you don't have."

I started thinking about my history with overdraft fees and some things I learned from Dave Ramsey. I wondered, who really pays all of these overdraft fees that account for 38 billion dollars per year in revenue?

This survey intends to find out. At the time of writing, I had 84 responses and I need hundreds more. Please, take my anonymous survey and ask all of your friends to do the same. Here is the Overdraft Fee Survey:Loading...

Array Function to Recode Data in Google Apps Scripts

2010-09-21T19:22:00.000-07:00

I went on my honeymoon with my beautiful wife last week and the week before. Having a little time off of work gave me the opportunity to get some work done :). I've been wanting for a while to develop a survey to find out what makes a good programmer.

I've been working with Google Docs and Google Forms to see what they're capable of. This spreadsheet posed a few difficulties. The primary problem was that I had a set of text values which needed to be recoded to numerical values from another range.

I wrote this array function for Google Apps Scripts in Google Spreadsheets to recode values based on an array of values.

Here's an example spreadsheet demonstrating the Array Data Block Recode Function.

Here's what the function looks like with a few tests:

function recode(data, values, valueColumnIndex)
{
  var  valueHash = {};
  
  // if the values are in an array, make a hash table
  if (values.constructor == Array)
    for (var i = 0; i < values.length; i++)
      valueHash[values[i][0]] = values[i][valueColumnIndex];

  else
    valueHash = values;
  
  var ret = [];
  
  // if the data are in an array, recursively recode them
  if (data.constructor == Array)
    for (var i = 0; i < data.length; i++)
      ret.push(recode(data[i], valueHash, valueColumnIndex));
          
  else
    ret = valueHash[data] != undefined ? valueHash[data] : data;

  return ret;
}

var values = [['a', '1', 'I'], ['b', '2', 'II']];

print(recode('a', values, 1));
print(recode(['a', 'b', 'c'], values, 1));
print(recode([['a', 'b'], ['b', 'c']], values, 1));
print(recode(['a', ['a', 'b'], [['a', 'b', 'c']]], values, 2));

/*
Results:
1
[1, 2, 'c']
[[1, 2], [2, 'c']]
['I', ['I', 'II'], [['I', 'II', 'c']]]
*/

/*
Google Apps Syntax:
=Recode(A1:B3, D1:F2, 1)
=Recode(A1:B3, D1:F2, 2)
*/

Regular Expression Search Bookmarklet

2010-09-21T14:14:00.000-07:00

I have an old website where I keep most of my bookmarklets. I'm planning on deprecating that site and just putting up some personal stuff (since I don't do what that site says I do anymore).

This bookmarklet is probably my most used bookmarklet. Basically, you enter a regular expression and each match in the page will be highlighted. It cycles through 16 color schemes to change the highlight color.

If you just want to install the bookmarklet, grab this link and drag it onto your bookmarklet toolbar in your browser.
Regex Search

If you want to use it on your mobile device, you can use my Mobile Bookmarklet Installer Bookmarklet (which, by the way, will install itself too).

I'm sure it doesn't work in IE, but I haven't tested it in a really long time. If you'd like to see what it would do in IE if IE didn't suck so badly, just click it.

If you're interested in the code, here it is!

// check to see if the variable searches has been defined.
// if not, create it.  this variable is to cycle through 
// highlight colors.
if (typeof(searches) == 'undefined')
{
  var searches = 0;
};

(
  function()
  {
    // just some variables
    var count = 0, text, regexp;

    // prompt for the regex to search for
    text = prompt('Search regexp:', '');

    // if no text entered, exit bookmarklet
    if (text == null || text.length == 0)
      return;

    // try to create the regex object.  if it fails
    // just exit the bookmarklet and explain why.
    try
    {
      regexp = new RegExp(text, 'i');
    }

    catch (er)
    {
      alert('Unable to create regular expression using text \'' + text + '\'.\n\n' + er);
      return;
    }

    // this is the function that does the searching.
    function searchWithinNode(node, re)
    {
      // more variables
      var pos, skip, acronym, middlebit, endbit, middleclone;
      skip = 0;

      // be sure the target node is a text node
      if (node.nodeType == 3)
      {
        // find the position of the first match
        pos = node.data.search(re);

        // if there's a match . . . 
        if (pos >= 0)
        {
          // create the acronym node.
          acronym = document.createElement('ACRONYM');
          acronym.title = 'Search ' + (searches + 1) + ': ' + re.toString();
          acronym.style.backgroundColor = backColor;
          acronym.style.borderTop = '1px solid ' + borderColor;
          acronym.style.borderBottom = '1px solid ' + borderColor;
          acronym.style.fontWeight = 'bold';
          acronym.style.color = borderColor;
    
    // get the last half of the node and cut the match
    // out.  then, clone the middle part and replace it with
    // the acronym
          middlebit = node.splitText(pos);
          endbit = middlebit.splitText(RegExp.lastMatch.length);
          middleclone = middlebit.cloneNode(true);
          acronym.appendChild(middleclone);
          middlebit.parentNode.replaceChild(acronym, middlebit);
          count++;
          skip = 1;
        }
      }

      // if the node is not a text node and is not
      // a script or a style tag then search the children
      else if (
        node.nodeType == 1
        && node.childNodes
        && node.tagName.toUpperCase() != 'SCRIPT'
        && node.tagName.toUpperCase != 'STYLE'
      )
        for (var child = 0; child < node.childNodes.length; ++child)
          child = child + searchWithinNode(node.childNodes[child], re);

      return skip;
    }

    // use the search count to get the colors.
    var borderColor = '#' 
      + (searches + 8).toString(2).substr(-3)
      .replace(/0/g, '3')
      .replace(/1/g, '6');
    
    var backColor = borderColor
      .replace(/3/g, 'c')
      .replace(/6/g, 'f');

    // for the last half of every 16 searhes, invert the
    // colors.  this just adds more variation between
    // searches.
    if (searches % 16 / 8 >= 1)
    {
      var tempColor = borderColor;
      borderColor = backColor;
      backColor = tempColor;
    }

    searchWithinNode(document.body, regexp);
    window.status = 'Found ' + count + ' match'
      + (count == 1 ? '' : 'es')
      + ' for ' + regexp + '.';

    // if we made any matches, increment the search count
    if (count > 0)
      searches++;
  }
)();

What Really Makes a Good Programmer?

2010-09-03T09:43:00.000-07:00

I've been working on a series of articles about mainframe migrations. My next post upcoming will be talking about source code translation tools. It got me to wondering, what is a good programmer?

I decided that the best way to figure out what makes a good programmer is to ask those who have a history of working with programmers.

I've been playing around with Google Docs and this seemed to be another solid opportunity to leverage some new Google Docs features, specifically Google Forms. I know this particular one is a little annoying but Google Forms didn't really give me many options for entering these data so I apologize for that.

I really appreciate your taking the time to read my blog and to help me with my research. So, without further ado, here is the What Makes a Good Programmer survey.

OK, maybe a little ado. Be sure to read and follow the instructions. They're pretty easy and, I think, relatively clever little recursive instructions to get a lot of responses. Thanks again for your help with my informal study.

If you've already taken the survey, I've published the results. Please don't look at the results of the "What Really Makes a Good Programmer" survey until you've completed it.

Loading the What Makes a Good Programmer survey...

Mainframe Migrations: Extracting Business Logic from Source Code

2010-06-25T14:39:00.000-07:00

I've been involved with several migration projects (some mainframe migrations and some migrating from an older platform to a newer one). At the beginning of any migration project, I like to ask, "Why do you want to migrate your application instead of just rewriting it?"

One of the most common responses I hear (among the ones I mentioned in my mainframe migration introduction post) is

We don't really have documentation; our source code is the documentation.

I've been in the software industry long enough to know that when schedules or money get tight (and they always do), documentation is the first thing to go, so I seldom expect adequate documentation and that's fine. Documentation is nice to have, but I'm pretty good and finding out what you need without it. After all, I get plenty of practice. By the way programmers talk, one would think there is almost never enough documentation and if there is, it's still not adequate.

I know that most projects lack documentation and that most code is legacy code (at least would be according to Working Effectively with Legacy Code by Michael Feathers). This is really no big deal and doesn't differ all that much from most projects. In fact, most of my projects start with a pretty clean slate. I get to go in, help the client document processes, recommend improvements, automate everything I can, and by the end of the project the client has a nice sleek system with clean documentation and good unit test coverage.

For some reason, that process frightens clients who already have a system in place. As a result, a lot of clients facing a mainframe migration elect to migrate rather than rewrite the application.

We just don't have the time or money to start from scratch. All of our documentation is in the source code.

I have two issues with this. First, the source code is not adequate documentation because you cannot derive business logic from code. Second, if you use the source code as your documentation, the review process will consume almost the same resources as would the discovery process on a brand new project.

You Cannot Derive Business Logic from Code

I know this sounds a little hard to swallow at first. To be sure, there's a lot of code out there that you can look at and come up with a very reasonable approximation of what was the original intent of the developer. If you assume the developer sufficiently understood the original intent of the business logic, your result will also be a reasonable approximation to the actual business logic.

But, no client has ever asked me for an application that is just a reasonable approximation of the requirements they have. What you'll find is that the best result you get from interpreting source code is a set of reasonable approximations to desired business logic. Every interpretation, no matter how reasonable, will need to be clarified or you'll risk having a dysfunctional application.

Here's an easy example. Go ask a programmer to "write a method that will verify that a person is old enough to drink." You'll probably get something like this:

function VerifyIsOfLegalDrinkingAge(age)
{
  return age >= 21;
}

Now, show that code to another programmer and ask what the original business rule was.

Chances are, you'll get, "ages 21 and older are of legal drinking age." While that's a close approximation, that's not the business rule you asked for. You said, "verify that a person is old enough to drink." You didn't say, "verify that a person is at least 21."

This is a really basic example and most problems in software engineering aren't as simple. In fact, even with simple problems, most code (especially legacy code) will not look like that. In your mainframe application, the above method would look more like this:

function CheckAge(age)
{
  // be sure that the person has appropriate identification
  // and that it is a valid picture id
  return age > MINIMUM_AGE;
}

You'll dig through thousands of lines of code just to find MINIMUM_AGE = 20 somewhere. Why is the minimum age 20 instead of 21? Because the code had a bug in it and it should've been age >= MINIMUM_AGE. Instead of fixing the method, they just decremented the age. Then, you'll have to find the code that's calling the method to figure out why you would verify that an age is greater than twenty. By the time you get this method implemented correctly, you'll have consumed far more time than it would have taken the client to say, "I can't serve beer to anyone under 21."

Now, consider the fact that developers who know their languages well know what the language will handle for them. For example, C# defaults integers to 0. If you were to ask a C# developer to write a method that returns true if someone is too young to drink, you might get something like this:

public static bool IsTooYoungForBooze(Person drinker){
    return drinker.Age < MINIMUM_AGE;
}

If you then ask someone to convert that to JavaScript, you'll probably get something like this:

function IsTooYoungForBooze(drinker)
{
  return drinker.Age < MINIMUM_AGE;
}

Seems like a reasonable approximation? In C#, if someone fails to set Person.Age, 0 will be less than MINIMUM_AGE; however, in JavaScript, undefined is not less than MINIMUM_AGE. This kind of bug tends to rear its head late in the game. Is this the kind of mistake you're willing to make because you had a "reasonable approximation?"

Reviewing Source Code Is Resource Intensive

I know it's tempting to expect a programmer to look at your source code on one screen and write brand new code on the other screen. It's just not going to work that way. If you don't believe me, pick one of your smallest methods and ask a programmer to document the business logic.Chances are, you'll get a dozen questions.

Why does this code never look at index 0 in this array? Where does this value get set? Is this always going to be 4 characters? What happens if this number is negative? What is this method trying to do? Who's calling it?

"Who's calling it?" is a very serious question. If you're prepared to list every place in the source code where that method is being called, be prepared for more questions about static or global variables, protection levels, database constraints, etc. If you finally get through that, be prepared for even more questions because now you're getting into the meat of the issue: Now that I know what it did then, what do you want it to do now?

Imagine how much improved your situation would be had you told your developer what the business logic was supposed to be instead of asking to have it inferred from a source that provides a best case scenario of a reasonable approximation.

What's the Solution?

One of my biggest pet peeves is when someone presents a problem but hasn't considered possible solutions. So, as not to break my own rules, I do have a solution to this particular problem. Rather than looking at your source code as the only source of business logic documentation, look at it as a guide and recognize that you simply don't have documentation.

I know it's rough to admit something like that, but if it makes you feel any better, nobody else has documentation either. Besides, admitting you have a problem is the first step to recovery, right? So now that we've acknowledged that we lack adequate documentation we can move on to bigger and better things. We can use the source code as a guide to get us (and keep us) moving in the right direction. We can write a new application in short iterations with open communication and good unit test coverage while we document business rules.

In my experience, a rewrite saves time and money in the short run. In the long run, as you consider feature changes and maintenance costs, it's no contest that a rewrite beats a migration every time. Keep in mind that your application is just an application. I know it seems like there's a lot going on in there, but it's just another application. Every developer I've ever talked to about mainframe migrations has said,

We shouldn't be migrating this. It would be so much easier just to rewrite from scratch and then it'd be a good program too. I mean, really . . . it's just a ____________.

Mainframe Migrations: Migrating Data and Data Structures

2010-06-22T13:30:00.000-07:00

It's hard to resist the urge to export data from a mainframe database, import them into the new database engine, and call the repository "finished." It is easy, on the other hand, to say "This has been working for 25 years; why would we change it now?"

Well, kind of a lot has changed in the last 25 years. For example, Adabas is a post-relational inverted list database. Sql Server on the other hand is a relational database with heaps and b-trees. I'm not saying that one is better than the other; to be sure, each has its merits. What I am saying is that they're different.

They store and retrieve data differently. They have different rules. They are optimized for different normal forms (again, with a background in both OLTP and OLAP, I'm not asserting that one is better than the other). With all of the ways that DBMSs differ, especially current vs. mainframe DBMSs, you just can't "lift and shift" and expect the same functionality or the same performance.

Continuing with the Adabas vs. Sql Server example (because that's what I have experience with), if you just pull the data right out of Adabas, you'll find that it is in need of serious refactoring. While it makes perfect sense to work with the Adabas data structure in Natural, it doesn't make sense using Sql Server and C#. The denormalized data will complicate your CRUD operations and clutter your database with empty results.

Then, you'll either have to filter your results or support CRUD operations that populate the empty rows so you'll get expected results back. In my experience, you can deal with a lot of these hassles by importing the data into a staging database and running an ETL process to get them into your application database. During the ETL process, you can eliminate empty rows, convert default values to NULLs, and normalize your data.

If you take the time to rebuild your database to support your application, it's really easy to keep your ETL process up to date to get the data into your database. On the other hand, if you build your application around the existing data structure, you'll spend countless hours pulling your hair out thinking to yourself, "who stores data this way?" You'll probably even find yourself cursing Adabas, Software AG, and mainframes in general. Meanwhile, it's your own fault!

You should have rebuilt the database and transformed your data to fit into it (which is easy), rather than leaving the database structure alone and forcing your programmers to compensate for the fact the the framework (and current software standards) won't support the intentional misuse of the new DBMS.

To put this another way, transforming data is repeatable and easy. It takes very little effort to extract data from a staging database and make them fit into a database designed to be efficient with your application. It's also not that hard to design the application database either. It will certainly take more time to design the database and the ETL process than it will to just dump the data into a database, but I assert that you will make up for that difference when your programmers start trying to fit a very large square peg into a very small round hole.

Just to make sure I'm not alone, I asked several of my colleagues who have work experience in mainframe migrations in a very informal survey. I asked them to consider a situation where they were migrating an application from Adabas to Sql Server. The source data are obviously the same and the result was to also be constant. That is to say that the entities presented to the BLL from the DAL should be clean, the code should be clean, and the data access should be equally perfomant. How much effort do you have to spend building the new database, writing the ETL process, and constructing the data access layer if you're "lifting and shifting" vs. rebuilding the database from scratch.

Here are the results:

To my colleagues, the choice seems clear. The fact is that you're going to have to transform your data at some point. You either have to do it in the ETL process and store the data in a cleaner format or you're going to have to do it in your DAL. If you do it in your DAL, it'll be harder to write the DAL because of the amount of transmogrification you'll have to do between your BLL and your repository.

And, let's just hope you never have to change anything. Storing your data in an unnatural format will make it very difficult to add features down the road, your DAL will be difficult to maintain (at best), and your entire application will most likely be considerably less performant than had you opted to save a little time and money and just refactor your database from the beginning.

Mainframe Migrations: Tips, Tricks, and How-To's

2010-06-22T07:52:00.000-07:00

Yesterday, I wrote a blog post about mainframe migrations. The top brass in my company took a look at it and told me that the world is just not ready for my thoughts on mainframe migrations yet.

I unpublished that post and will republish it at the end of this series. Leading up to my thesis I will post a series of articles detailing the things I've learned leading my last few mainframe migrations.

Migrating Data and Data Structures
The Documentation is in the Source Code
Source Code Translation Tools
The Original Authors are Unavailable
My Original Mainframe Migration Post

Be on the lookout for more updates. As I write these posts, each item will become a link to the article.

Custom jQuery Selector for External and Internal Links

2010-03-11T18:24:00.000-08:00

I was working on a website for my fiancee and her church group called The Diocese of Atlanta Young Adults. They were hosting an event they call the Young Adult Summit.

One of the requirements I had was to provide a warning to users before they left the page by after having clicked an external link. Using jQuery, I was able to bind to the click event with easy cross browser compatibility and I used jQueryUI to open a modal dialog box. Pretty basic stuff.

The one thing I regretted was that I had to use a class name to determine which links were external and which were internal. A few days ago, I discovered that jQuery supports custom selectors and I decided to write a pair of custom selectors for identifying internal and external links.

jQuery.extend(
  jQuery.expr[ ":" ],
  {
    /*
      /:\/\// is simply looking for a protocol definition.
      technically it would be better to check the domain
      name of the link, but i always use relative links
      for internal links.
    */

    external: function(obj, index, meta, stack)
    {
      return /:\/\//.test($(obj).attr("href"));
    },

    internal: function(obj, index, meta, stack)
    {
      return !/:\/\//.test($(obj).attr("href"));
    }
  }
);

I do have a few items of note. First, you'll notice that I'm not actually looking for the current domain name in the links. That's because I use relative links for all of my internal links these days. Thus, I know that if there's not a protocol definition that it's an internal link. If you need to identify internal links by domain as well, you can just pull that out of top.location.href and check for it too.

Second, you could technically use this selector on any DOM object. There are several ways to get around this. One way would be to verify that the object is an anchor tag. Another would be to verify that the href attribute exists. I just plan on using common sense.

Here's a quick usage example:

$("a:external").click(verifyNavigateAway);
$("a:internal").css("font-weight", "bold");

The Accidental Programmer

2010-03-11T16:59:00.000-08:00

I've been in the software field for some time now, and over the years I have worn many hats. I have been the sole developer in a psychopharmacological research lab; I have been a private contractor and security analyst; I have developed human resources software and data warehouses; I've been a programmer, a tech lead, a project manager, a VP, and a partner. I've hired and fired developers, laid off friends (and a fiancee), and interviewed at least a hundred candidates locally and overseas.

In all of my experience, I've discovered an unavoidable and intolerable fact: most programmers can't program. Just so we're clear, by "most programmers," I'm talking roughly 90 - 95 percent. Now, I know this isn't an original sentiment. Jeff Atwood talked about this in 2007 in an article which has since been quoted by dozens of people in the development community including folks like Phil Haack. So, if it's an already beaten dead horse, why am I writing about it again?

Well, frankly, I was wondering why there are so many bad programmers out there who seem evidently to be doing so well. Why are there so many crappy programmers getting work and making bank? Look at how many large companies are learning hard lessons from off-shoring experiences, yet everybody still wants to send work overseas! It's like there's a big chronic "WTF barrier" between business people thinking they could save a few bucks and programmers telling them how much it'll cost them in the long run.

Why doesn't anybody notice? Why doesn't everybody realize that these people don't know what they're doing? Well, it's because they produce programs that partially work. Jeff's right that most programmers can't even write a single line of code, so how is it that they manage to produce work that's functional enough to convince the world that they're capable developers? Well, I figured it out. They do it by accident; I call them accidental programmers.

By combining the forces of the internet, feature rich IDEs, code templating, and auto completion, an accidental programmer has all the tools he or she needs to accidentally write a semi-functional bad program without ever having to write a real line of code.

Now, I am sure my readers wouldn't let me get away with making such claims without providing empirical evidence, so here are some code snippets I believe you couldn't write on purpose:

tbPassword.Text.Trim().ToString().Trim();

protected void btnLogin_Click(object sender, EventArgs e)
{
    SqlClientUtilities sqlData = new SqlClientUtilities();
    SqlDataReader drPassinfo = null;

    string sLoginSQL = "select a.agentid,u.userName,a.FirstName,a.Lastname from dbo.person a, dbo.users u where  ";
    sLoginSQL +=  "  a.userid = u.userid and u.username='" + txtUserName.Text.Trim() + "'";
    sLoginSQL += " AND personid = '" + txtPassword.Text.Trim() + "'";

    drPassinfo = sqlData.SqlClientExecuteDataReader(sLoginSQL);

    if (drPassinfo.HasRows)
    {
        AppSupportUtils.WriteError("Records found");

        //valid login - redirect
        Session["userlogin"] = txtUserName.Text.Trim();
        Response.Redirect("Manage.aspx",false);
    }
}

/// <summary>
/// This class just returns an object which holds a date
/// </summary>
public class Date
{
    public int Year = 0;
    public int Month = 0;
    public int Day = 0;

    public Date()
    {
        Year = 1901;
        Month = 1;
        Day = 1;
    }

    public Date(System.DateTime dt)
        : this()
    {
        Year = dt.Year;
        Month = dt.Month;
        Day = dt.Day;
    }

    public Date(string dt)
        : this()
    {
        // en-US     M/d/yyyy
        CultureInfo MyCultureInfo = new CultureInfo("en-US");
        try
        {
            DateTime MyDateTime = DateTime.Parse(dt, MyCultureInfo);
            Year = MyDateTime.Year;
            Month = MyDateTime.Month;
            Day = MyDateTime.Day;
        }
        catch { ;}
    }

    public override string ToString()
    {
        return Month.ToString() + "/" + Day.ToString() + "/" + Year.ToString();
    }

    public override int GetHashCode()
    {
        return (int)(Year * 12) + (Month * 30) + Day;
    }
}

function invertBool(bool)
{
  if (bool == false) return true;
  return false;
}

function sendSecureVote(index)
{
  var checksum = Math.round(getFormattedDate() * 57 / 33 - 147 + 2009);
  startAjaxRequest("http://www.theserverhasbeenanonymized.com/voteCounter.php?checksum=" + checksum + "&voteFor=" + index);
}

function getFormattedDate()
{
  var date = new Date();
  return date.getMonth() + date.getDate() + date.getHours();
}

var ssnValidator = /[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]/;

My Mission Work in Central Tanganyika

2010-02-10T18:42:00.000-08:00

A few years ago, I started looking for ways that I can use my time and skills to contribute more to the world around me. One way I found that I could help my community was by volunteering for my local fire department as a volunteer member of the Alpharetta Fire Corps. I've really enjoyed my time volunteering with the Alpharetta Fire Department, and it has excited something in me that I can't really explain.

The first time I was called in to help on a structure fire, I realized what it means to give. It didn't really click at first because everything was so hectic. There were 4 fire engines, 6 police cars, and 2 ambulances. There was no time to realize what was really happening as I ran around performing my duties as a Fire Corps team member. After the house fire was suppressed, I had a moment to stop and reflect on what just happened.

A family not only lost their home, but everything in it. I started thinking about the things I own and how devastated I would be if I lost them. I mean, possessions can be replaced, but it's what they can't replace that is heartbreaking: the memories, the favorite blankets, the family photos.

That's when one of the firemen walked out the front door with a wet, singed, but intact photo album he found while extinguishing hot spots. I watched as the fireman handed the photo album to the homeowner. The homeowner tearfully gave thanks as the neighbors welcomed him and his family into their homes.

Feeling that selflessness made me want to find a way to do more. My fiancee works for the Episcopal Diocese of Atlanta, and the Diocese of Atlanta Young Adults are going on a mission trip to Dodoma, Tanzania. I told her I wanted to go and she eagerly arranged for me to meet with her and Bishop Alexander.

I couldn't hide my excitement as Bishop Alexander told me about the ways I could help the people in Tanzania. It turns out that the places we are going have pretty good technology, but have nobody to help them maintain it. They have many computers which are in disrepair.

Not only will I have an opportunity to fix their computers, but I'll be able to help teach them some basic computer repair. I'll also get to teach them how to use their computers and technologies better. I'm thinking about setting up a VPN endpoint down there so that volunteers here will be able to provide tech support without having to be on location.

I feel like I'll be able to do some good and I'm really excited about it. Lauren and I will be leaving for Central Tanganyika with the rest of the group on May 2nd and will be returning May 11th. The cost will be right around $3000 for each of us.

Lauren and I are hoping to defray our cost by asking our friends and family to help. I get about 85 readers on my blog per day so I thought I'd post this here too. If you have a few bucks to help us, we would really appreciate it.

Feel free to snail mail us:
Lauren Woody and Patrick Caldwell
295 Crab Orchard Way
Roswell, GA 30076

My email address is dpatrickcaldwell and I use gmail.

I'd really appreciate it if you included a note with your donation. I only have a few weeks in Tanzania, and I'd love your input on what I should focus on. Any ideas for maximizing their technological capability? Anything in particular you think I should teach them? Also, please let me know if it's okay for me to blog about you, your company, or your support.

I'd like to be able to thank our supporters on my blog, but if you would like to remain anonymous, that's no problem. Rest assured, we will be eternally grateful.

The No Fun at Work Rule

2010-01-28T20:35:00.000-08:00

I've got a pretty large government client at the moment. They feel better when they get to see me hangin' around the office (and can you blame them?). Problem is, hangin' around their office takes away some of the luxuries of my office. For example, my team and I like to go out to throw the football around from time to time.

Sometimes we get programmers block, or we need to talk about something, or we just need some sunshine so we mosey out to the parking lot for some ball. The other day, I asked one of my partners if it'd be okay if I brought a football to the client office. He looked at me like I was impaired.

Later that day, I was hanging out with another partner waiting to start a meeting. I said, "Hey Chuck, I'm gonna start smoking. I'm planning on smoking 6 cigarettes a day and I estimate that it'll take 10 minutes to smoke each one. Is that okay?"

Chuck said, "Well, Patrick, other than the fact that it's really odd that you are planning to start smoking, I don't really have a problem with it."

Then, I asked him, "Well, what if instead of smoking for an hour a day, Ryan and I spend 20 minutes a day throwing the football?"

Chuck leaned back in his chair and sighed. "Well, I'm going to have to think about that. I'm not sure that's going to look good to the client."

My point is, something is ass-backwards. If I want to spend an hour a day (on the clock) smoking cigarettes, nobody cares; but if I want to take a 20 minute break (off the clock) throwing the football, having a work related discussion, enjoying a brief reprieve from the stress of the work day, everybody thinks I'm going to look like I'm slacking off.

Don't get me wrong, Chuck knows that we enjoy our outdoor time back at the Emerald office. He knows there's benefit to it. Not only does it keep my team happy, but it helps keep the team cohesive and energetic. He's just concerned about how it will look to the client.

My point is, why is there this "no fun at work" rule? I've decided that I will, for the rest of my career, spend a portion of my time focussed on making sure that my employees are happy, comfortable, and healthy.

Your Field -- How to Land a Job as a Software Engineer

2010-01-11T21:40:00.000-08:00

Well. This is it. My last post in the How to Land a Job as a Software Engineer series. I spent a lot of time deciding whether this would be the first post of the series or the last.

I thought about making it the first post because it's really the first step to becoming a software engineer, but I saved it for last because I wanted to make sure I gave these thoughts plenty of time to ripen. If you've been following the series, you probably know how this series came about in the first place.

Emerald Software Group has been interviewing programmers in search of new development talent. I've used a lot of examples (both good and bad) from real life interviews I've conducted in the past few months. This particular story is one of the most important lessons I've learned in my career, so it's really important to me to tell it correctly.

I know that my style and sense of humor can be pretty sardonic and I know that many would be readers find that off-putting, but I will do my best to convey my sincerity in this post.

A few months ago, I interviewed a candidate who had a masters degree in computer science from a relatively prestigious technical university in Georgia. I expected great things from this developer and was excited about the interview.

I went through my standard series of programming questions and was disappointed by his lack of understanding of the technologies he had spent the last 6 years studying. After a while, I started feeling bad for the kid and decided to move on with the interview. After all, like I've said before, just because you don't know the information doesn't mean I'm going to assume you're incapable of learning it.

I decided to ask him a little about himself. I closed my notebook, leaned back in my chair, and asked, "what kinds of things do you program in your spare time?" He said, "I don't really program in my spare time."

I was pretty surprised, so I asked, "Do you read programming blogs or books? How do you learn about new technologies and techniques?"

"I don't really bother learning it until I need it."

I was fundamentally confused. I didn't understand how he had spent 6 years in school studying computer science and was looking to begin a career in the field, but that he neither knew anything about it nor had any interest in learning it. I thought for a minute and I asked him, "Why are you here?"

He said, "I'm looking for a job."

"No, I mean, why do you want to be a programmer?"

"Because it pays well."

"Do you like it?"

"Not really."

I was dumbfounded. I asked him a few more questions so that I didn't have to end the interview abruptly, I told him we'd be in touch, and I walked him to the door. If you want to write software for a living, you need to love it. It's not just a job; it's a craft. If you're not doing it for the love of the field, then you're committing an injustice not only against yourself but also against your employer, your teammates, and the customers.

To be sure, I don't just feel this way about software engineering, but about all careers. Whatever it is you do, do it for the love of your field. If you don't like what you do, let alone love it, you'll be miserable for most of your life and there is no amount of money that will make up for a lifetime of misery.

Now, I know that there are plenty of people out there who would be perfectly happy spending 5 days a week doing something they don't care about so that they can have more fun the other 2 days of the week. You cannot be great at what you do if you don't care enough to try and I will not tolerate mediocrity, especially intentional mediocrity.

So chose your field carefully. Don't get a degree in software engineering if you don't like it. Obviously, don't get two degrees. Think about the things that you love to do. What would you like spending your weekends doing? Take those things and find a way to make money doing them. If you make your living doing what you love, you'll always love what you do, you'll always be proud of your work, and you'll be in the top of your field.

Your Communication Skills -- How to Land a Job as a Software Engineer

2010-01-04T09:50:00.000-08:00

At long last (due to some intense deadlines), I'm getting around to the fourth installment of the How to Land a Job as a Software Engineer series. This concludes the job application sequence starting with your resume and ending with your interview.

This post is about maintaining good communication skills throughout the process. A very close friend of mine named Ryan LaFevre, who is a Ramblin' Wreck from Georgia Tech and a hell of an engineer, once mused to me, "I don't write good; I are a engineer." Obviously said in jest, I bring that up because I know a lot of programmers who feel like even a cursory grasp of language arts is completely unnecessary for anybody in a technical field. My former English teachers would probably be pleased to see me say that communication skills are of fundamental importance.

Do you ever watch those court room shows? If so, you've probably seen scenes where one attorney makes a damning argument or the witness breaks down in an emotional display, at which point the other attorney objects. The judge says, "sustained" and instructs the jury to ignore the preceding outburst and has the text stricken from the record. Now, imagine being on that jury. Could you possibly ignore that? Even if you don't make your verdict based on said event, could you possibly remain unbiased by it?

It's the same thing with your communication (both verbal and non-verbal). While I'd rather claim that I focus primarily on the skills you actually need to do your job, and while I can ignore your communication inefficiencies, I cannot ignore the feelings I had when I communicated with you.

For example, I'm not going to refuse to hire a candidate just because he's 25 years old and still can't discern the difference between they're, there, and their; however, there's a pretty good chance I will assume she's not all that bright. Thus, when I sit down with the resumes and my notes from my three favorite candidates to compare them side by side for selection, ceteris paribus, he'll get eliminated first because I won't feel like he has as much potential as the other candidates.

Here are some actual excerpts I've received from job hopefuls:

Hello Mr. Caldwell, I just wanted to touch base with you to see how your doing.

What about my doing? You leave my doing out of this and I won't say anything about your dumb.

The sample project I sent you is very simple on the front end, but it's back end is where all the magic happens.

I just read your email and its dumb is showing; it is back end is where you are not getting a job. Thanks.

I know what you mean about taking your work home with you. I've spent a lot of time with my computer in my underwear coding until the middle of the night.

First of all, gross. Second of all, your computer codes by itself? Third of all, what was your computer doing in your underwear?!?!?

Thank you very much for meeting with me today -- I really enjoyed talking with you -- If there's anything else that you need from me, please feel free to ask for it -- I'm available by phone most of the day and I'll get back to you as soon as possible

WTF? The period exists for a reason; please use it. If, by some chance, your period key is broken and you can't find a working keyboard with which to send your email, please hold alt and hit 0046 in lieu of the double hyphen.

i'll be available tuesday afternoon around 1:30 or so. is there anything i need to bring to the interview other than my resume? i'm really excited about the opportunity and i think emerald software group will be a great fit for me

i hate you (alt + 0046)

Also, sometimes it's not what you say, but how you say it (or, how often). Don't nag me please. I will get back to you; I promise. If you email me every day asking about the status of the job, on your third try (or second if I'm in a bad mood), I will simply let you know that you have been selected out of the applicant pool.

Written communication isn't the only place you can make silly mistakes. Oral communication is replete with its own difficulties. Sure, you less often misspell things orally than you do in written communication, but there are some things you're unlikely to get away with. For example, I really don't want to hear your stance on the adult entertainment industry, I don't want to hear a story about a time you were drunk, and I really don't want to know what crimes you've gotten away with. Keep it on a professional footing — please!

Here are some other hints for oral communication:

Don't swear a lot; I'll just think you've insufficient vocabulary.
Don't tell me a dirty joke you heard a few days ago. Let's get to know each other a little first
Don't opine on politics or religion. Chances are you won't offend me, but on the other hand, chances are I know someone at the office you would offend.
Don't tell me my school sucks. For that matter, don't tell me any school sucks. I don't care if you're just kidding and I don't care if you went to a rival school. If you say something disparaging about any school, I'm either going to think you weren't smart enough to get in or you're pompous. Either way, I don't want you on my team because I may later want to hire someone from that school.

Finally, it is worth mentioning that not all communication is verbal. There are plenty of ways you can non-verbally tell me you're not ready for the position in question. For example, don't be late. Don't smell bad. Don't wear your favorite pair of tattered jeans and raggedy flip-flops. Don't be over-confident. Don't be under-confident.

By all means, be yourself and try to have some fun. If you have fun, people are more likely to enjoy talking with you. If you have fun, the worst that can happen is you have a good time, meet some good folks, and learn some new skills.

Your Interview -- How to Land a Job as a Software Engineer

2009-11-20T09:28:00.000-08:00

This is the third installment of the How to Land a Job as a Software Engineer series. Now that you've submitted your resume and completed your prescreening, you've been called in to interview.

It's a pretty safe bet that you'll have several interviews to go through including talking to someone in HR, some management folks, and some technical people too. I do the technical interview around here and there are some things I've noticed interviewees do that consistently irk me. Here are things you shouldn't do on a job interview (in no particular order):

Don't tell me you don't need to answer my question. I've actually had several interviewees tell me that they don't need to answer my question because it's obvious that they know the answer. If I've asked you the question, it's because I want to know how you're going to answer it. Especially if it's obvious you have no idea WTF you're talking about.
Don't bother going if you embellished on your resume or cheated on your prescreener. If you tell me you're a 7 out of 10 with SQL and you can't tell me the difference between an inner join and a left join, I'm not going to hire you. If you tell me you're a 3 out of 10, then you have a fighting chance.
Don't ramble. I've only got so much time to talk with you. Most of the time, I enjoy interviewing and I enjoy the conversation, but I really do need to get back to work. Just answer the question I asked and move on.

Also, as a side note, when I was in graduate school, we talked a lot about interview techniques. The business oriented interviewers are likely to ask you questions like, "What's your biggest shortcoming?" or "Tell me about a time you had a conflict with a co-worker." These are cheesy questions and you're going to have a canned response, so you probably wonder why they even bother asking the question. Keep in mind that these folks have a lot of experience interviewing . . . much more than you do most likely. They ask the question because if you have a canned response, they can catch you off guard.

Here's the technique. I ask you a question, you give me the canned response, and I just look at you waiting for you to say more. In only seconds, the silence becomes deafening and you start talking again. Now, you're out of your comfort zone and you start telling the truth. The more I nod and keep my mouth shut, the more you confess. Moral of the story? Don't ramble. Just answer the question and then STFU.
Don't make things up. If you can't answer the question and you can't make an educated guess, don't just make something up assuming I won't know. I very seldom ask questions in an interview that I don't already know the answer to. If you make something up so that you don't have to say, "I don't know," I can almost guarantee that I'm going to know you're full of it.
Don't say you don't know the answer to a question unless you're absolutely sure you know nothing about the topic. Most of the time, I'm trying to figure out where you are in your professional journey. I don't expect you to know everything. Chances are, if you can give me a starting point, I'll prompt you until you get the answer. I just want to get a feel for how well you understand programming concepts. If you simply say, "I don't know" then I have nowhere to start and you don't get partial credit.

For example, if I ask you the difference between an interface and an abstract class, and you say, "I have no idea" then I have to operate under the assumption that you indeed have no idea. If you say, "well, I know they both define contracts for classes" then instantly you have partial credit. Then I'll ask you, "why would you use them?" You say something remotely intelligent about why you'd want to do this and you have even more partial credit. I'll keep asking you questions until I'm convinced that I've given you as much credit as you can possibly get.

By the way, this doesn't contradict the previous item. If you make something up and present it like it's fact, I'll assume that you're making things up. If you look at me and say, "you know, I'm not sure, but it sounds like . . . " and then you make your best inference drown from past experience . . . even if you're way out in left field, you still get partial credit.

The ThoughtWorks Agile Southeast Conference

2009-11-19T09:04:00.000-08:00

I'm taking a quick reprieve from my How to Land a Job as a Software Engineer series to write about a very informative and enjoyable event I attended this Tuesday. ThoughtWorks held its first professional conference in the Southeast and a friend of mine who is a thought worker invited me to go. It was a well organized event and several noteworthy thought workers presented.

We kicked off the day by listening to a keynote by Neal Ford called, "Why, Not How." Neal pointed out that there is solid evidence that agile works, but that few people investigate why it works. That is to say, the talk focused on why agile works rather than how to work agile. There was a great deal of merit to the things he said about the psychology and biology behind the effectiveness and efficiency of agile methods. This was one of my favorite talks, perhaps because it appealed to my background in behavioral neuroscience research at Duke, my experience getting my MBA, as well as my interest in becoming a better programmer. If only he had said something about airplanes it would've been perfect.

The breakout sessions started after Neal's keynote allowing us to chose between the management track and the technology track. I generally stuck with the management track. Just kidding. But, I wish I could've been in two places at once and I'll explain why shortly.

The first breakout session I went to discussed code metrics, tools for gathering metrics, and ways to interpret metrics. In fact, the significant interest in measurement may be one of my favorite aspects to the agile method. I suppose it appeals to my empirical nature; after all, if you can't observe it, it didn't happen. :)

Next, Anthony Pitluga and Ali Aghareza talked about cloud computing. They made an interesting distinction to me regarding Software as a Service (SaaS), Platform as a Services (PaaS), and Infrastructure as a Service (IaaS). They also pointed out a use for cloud computing that I can really get behind. Just so you know, I think there are very few instances where cloud computing is better than having your own infrastructure, but I believe these fellas were right on when they said it's perfect for testing. I mean, you can spin up (on demand mind you) a full test environment that very closely mimics your production environment on Amazon's EC2. Imagine that! Rather than doubling your capex to get two (or god forbid) three duplicate environments, you just build, test, and deploy your solution to EC2. You bang on it for as long as you need and the whole testing cycle costs a few hundred an hour. Very clever indeed.

I switched over to the management track for Saleem Siddiqui's talk about software complexity. He talked about essential complexity and incidental complexity in software, making a distinction between complexity that has to be in software and complexity that just sort of happens. Then, he talked about why it's better to focus on the latter rather than the former because, well, the former has to be there. He's a well spoken and interesting speaker to be sure.

After that, we took a much needed lunch break and had some really good food (although there was a vegimarian behind me in line who seemed to disagree, but since I'm a meatitarian I didn't notice). During lunch, Scott Conley gave his keynote about Platforms and innovation. As ThoughtWorks Chief Strategy Officer, Scott knows a lot about innovation and he's damn good at talking about it. Despite the incessant tinging of forks on plates, I got a lot out of Scott's talk.

After lunch, I went back to the technical track and enjoyed Graham Brooks's talk about testing web applications. He showed us a framework he's been working on that allows him to apply test driven development techniques to web applications.

Next, Dave Vollbracht talked about the perils of allowing everyone to have a unique development environment. It's important to note, though, that he didn't say anything about being able to try new tools. Basically, he said that if a team member finds a great tool and starts using it, that everybody should have it as well. That way, the machine you are working on may not be the same piece of hardware, but the environment will always be identical. That makes it much easier to image new development machines, to engage in paired programming, or to share team members with other teams. Dave also talked about several methods to achieve this uniformity.

Really, the biggest downside is that I couldn't be in both breakout sessions at the same time. I would really have enjoyed Ross Pettit's talk about Budgeting and the Financial Implications of Agile and I'm sure I would have loved to hear about Gregory Reiser's experience with offshore development. Can agile make offshore development actually work? I guess I won't know until next time.

Speaking of next time, I'll definitely be there and I'll be bringing friends. It was a good conference, and I enjoyed getting the ThoughtWorks perspective. I hope the next one will have a lot more attendance because I'd love to get other perspectives too.

Your Prescreening -- How to Land a Job as a Software Engineer

2009-11-10T12:55:00.000-08:00

My How to Land a Job as a Software Engineer series has me inspired and this is the third post I've written today. If you've ever applied for a technical position, you've probably met the prescreening questionnaire before. For software engineers, it generally comprises a set of questions about the language you'll be using and a few puzzles or programming tasks.

My company has something of a prescreening questionnaire and we're working on a more robust one. I've done them in the past and I have several friends who have done them for the companies they work for now. It's a common tool to assess your goodness of fit with the requirements of the position being filled.

So, you get the email from the HR department that says, "Thank you for your resume. You've been selected to move to the next stage of the process. Please complete this prescreening questionnaire and return it. You have one week from today." Sometimes the questionnaire will be online in quiz form and other times it'll just be a list of problems. Sometimes you're allowed to use the internet to help you and other times they ask you not to. In any case, my suggestion to you is to follow whatever rules they give you. Don't cheat! In my post about what hiring managers are looking for in a resume I kind of harped on the whole honesty thing.

Well, I'm about to do it again. Some people have their friends help them with the test or the coding problems or they Google when they were asked not to. Sure, this may help you get the job, but you are setting yourself up for failure. If you cheat on the test, they're going to find out when you go in for the technical interview. If they don't find out in the technical interview, they're going to find out that you're ill qualified once you start working. In any case, you're just gonna disappoint them and there's very little good that can come of it.

Work the problems and do it with integrity. If you don't get the interview or you don't get the job, it's for the best; that job was not for you and you're now better prepared to move on. After you submit your results, study the questionnaire. Learn everything you didn't already know. Then, learn more about what you know. Next time you have a prescreening task, you'll actually know the information you need. When you do get called into that interview, you're going to impress them with your knowledge and capability instead of disappointing them.

Your Resume -- How to Land a Job as a Software Engineer

2009-11-10T11:31:00.000-08:00

This is the first post of a series I call How to Land a Job as a Software Engineer. I chose this as the first post because it's really the first step in the process; however, I do not think it's the most important step because I believe that accolade belongs to selecting the right field in the first place. Also, from your perspective, finding a job to which to apply is the first step; however, I'm writing this series from the perspective of the person who makes the ultimate decision about who is and who is not worth hiring.

That being said, your resume has to be your foot in the door. Let's say that for a given job opening, we receive 100 resumes. At least half of those will be screened out before they even get to me. I'll eliminate at least half of the remaining resumes before we start inviting people to interview.

I've worked with resumes from every angle I can think of. I studied job applications and resumes when I was getting my MBA, I've been a job applicant, and I've reviewed dozens of resumes of job hopefuls. In graduate school, I read chapter after chapter about how a good resume is structured. As a job applicant, I perused every online resource I could find for hints and tricks. As a hiring manager, I can tell you what makes a difference to me.

But, before I get to my tips and tricks, I think it is worth pointing out what happens when a job opportunity opens up. First, the organization has identified a need. Second, the organization publishes that need. Third, many people apply. Fourth, one person is selected.

When you apply for the job, you feel like the organization is trying to hire the best candidate; well, they're not. Don't worry though because they think they're trying to hire the best candidate too; but they're not. In reality, they're trying to eliminate the worst candidates from each phase until there's only one left. I know it's something of a pedantic distinction, but it is important.

Why? Well, when I start looking at resumes, I'm not looking for reasons to hire someone. I assume that if you've sent me your resume, you feel like you'd be right for the job and that the job will be right for you. Therefore, when I look at your resume, I'm looking for reasons you're wrong. I'm trying to eliminate you because I don't want to waste your time and mine by having you in for an interview if I can tell it won't work out.

Sure, there have been times that something really stood out on a resume. Usually, it's a bad thing, but sometimes it's a really good thing. Just, keep in mind that things that stand out don't always stand out in a good way. So, in all forthrightness (and at the risk of seeming petty), here are some of the things that I find to eliminate you from the applicant pool:

I don't want to read your 5 page resume. In fact, I won't read your 5 page resume. I'll look at the first half of the first page and the last half of the last page. If you cured cancer in the middle of your career, chances are pretty good I'll never even know about it. If there's something you want me to know, put it in those sections. Even better, keep the whole thing succinct so that I don't think you're verbose.
Don't have grammatical or spelling errors in your resume. Ultimately, you're talking about a two page report about a topic you're the only living expert in -- you. If you're writing a doctoral dissertation about a topic you previously knew very little about and a few grammatical or spelling errors slip into the first draft, it's one thing. It's another thing altogether to write a two page paper about what you've been doing for the last 10 years and to make grammatical or spelling mistakes in your final copy. What does that say about your attention to detail?
Be factual; don't be boastful. I don't care if you think you're a great communicator or if you think you're an excellent problem solver with 12 years of experience leading successful teams. I just want to know how long you've been at it and what you've done. I'll figure out on my own if you're a great communicator or an excellent problem solver. Besides, how do I know that your definition of a "successful team" isn't one where no more than half of the team quit and 10% of your projects were on time and on budget? In fact, if you put this in your resume, I'm going to tailor your interview to prove you wrong.
Tell the truth! I mean, seriously . . . in the name of all things holy and good, don't lie on your resume. If you tell me you have 10 years of programming experience but you really only have 5, I'm probably going to figure it out. As soon as I figure it out, I'm just going to tell you to leave. I'm not kidding; end of the interview. If you tell me you have 5 years of experience and I need a 10 year programmer, you didn't want the job anyway because you're probably not ready yet; however, I'll think, "man, that person was honest and I'm gonna save that resume for the next junior opportunity."

Most of the time, I just need a programmer of any sort. So, if I'm just looking for a programmer and you say, "I have 5 years of programming experience," and I can tell that you're being honest, I may still hire you; I just won't pay you as much. So, just be honest. If you're honest, the worst that can happen is you don't get the job and everybody's happy (or at least at worst mildly disappointed). If you lie, the worst that can happen is you take a new job, everybody suffers, and you get the ax after your 90 day interim.

Now, here are some positive things I look for in a resume:

I actually do care what your hobbies are. I'm not going to hire you based on your hobbies, but if you're up against someone who is exactly equal to you in every way except that your hobbies are cooler, more relevant to the field, or are similar to my own then I'm more likely to hire you. In fact, I once invited someone in for an interview because I wanted to know more about his hobby. Came to find out that he was a very capable engineer, but was just a little too expensive for us. If I had the money, I'd have hired him on the spot.

It also makes you a real person. I look at a lot of resumes and none of them are real people, but if I know you're into underwater basket weaving, then I'm likely to remember you. In fact, I don't even look at the names on resumes. I just look at the resumes, chose the ones who seem capable, and hand them to my boss saying, "call these in please." If I see that you're into underwater basket weaving, I'll be like, "heh, what's this person's name?"

So, unless your hobby is rolling your poop into little balls, then go ahead and include a hobbies section on your resume. (Not that there's anything wrong with rolling your poop into little balls; I just don't want you touching company equipment.)
I want to see that you stick with jobs for a long time. While I know that contractors will have a lot of short term projects and few long term employments, I really like it if you consolidate your contract work into one item on your resume. Then, you can list the contracts you've worked on under that line item rather than listing each contract as though it was one job. It helps me rapidly review your work history. If nothing else, at least specify that the position was a contract position or I'm going to assume that you can't keep a job longer than a 9 months.
Apply for a job that's appropriate to your experience. If you've been programming for 10 years and still consider yourself a junior developer, I'm probably not going to call you back. I know that in most cases, one of two things will be true. Either you're not progressing at your craft and thus aren't worth the investment or you're better than the job you're applying for and you're just going to quit as soon as another opportunity comes up and then you're not worth the investment either. If you should be a senior developer, be a senior developer.

How to Land a Job as a Software Engineer

2009-11-10T09:43:00.000-08:00

My company is hiring again. It's a bittersweet process for me because I love meeting new people and getting new perspectives (and God knows we need the manpower -- er, person-power); however, it is a very frustrating process to endure parsing all of the resumes, selecting the first round for interviews, interviewing, rejecting, being left empty handed.

I started working with Emerald Software Group about three and a half years ago, and have been responsible for evaluating the technical capabilities of our job applicants for the last several years. Sitting on the other side of the table has given me some insight that I'd like to share with job seekers (especially, my fellow software engineers in need of gainful employment).

This is actually the first post in a series of posts about landing the perfect job for you. Nota bene, I didn't say "landing your dream job." I said, "landing the perfect job for you," because your dream job may not be the perfect job for you. For example, my dream job is to drive a Hog for the USAF, but alas the perfect job for me is as a software engineer. I just wanted to make this distinction now because the topic is going to come up again (and again).

Your Resume

I'll be starting with a post about resumes. After all, a majority of our applicants are eliminated before I even see their resumes. Then, I eliminate a majority of the remaining candidates before we invite anyone to interview. I know there are thousands of blog posts out there about resumes, but I want to be completely frank about it. Learn more about what a hiring manager looks for in a resume.

Your Prescreening

Next, I'll discuss the prescreening process. For those of you who don't work in a technical field, you'll probably find this a little foreign. A lot of technical industries have developed prescreening processes to establish your technical capability before you ever set foot in the building. For software engineers, these tend to be basic framework / language questions and some cute and clever programming tasks. I will give you hints to get the most out of your experience with the prescreening questionnaire.

Your Interview

Here's a great place to make an impression (or a great place to really screw it up). A good interview is a precipice. Keep in mind that your interviewer is not looking for a reason to include you in the next round of interviews; your interviewer is looking for reasons to exclude you. The ultimate goal is to eliminate all but one applicant and you want to make sure that it will be you if it should be. Here's how you make sure your job interview goes well.

Your Communication Skills

Communication skills are important even in technical fields. The way that you communicate says a lot about what kind of employee you'll be. People want to know that they'll be able to understand you, that they'll enjoy working with you, and that you'll bring something to the table. We can tell a lot about a candidate by the way they communicate with us. Learn how to communicate effectively and land your software engineering job.

Your Field

I thought long and hard about whether this should be the first post or the last post on the "How to Land a Job as a Software Engineer" topic. It's really the first step, but I've been saving it for last because I have a lot to say about this. I love writing software; if you don't, it'll show. In the post about selecting a field, I'll discuss what we look for when we interview job hopefuls. Selecting the right field for you is the only way to ensure that your perfect job is, in fact, your dream job.

Designing a Shampoo Bottle Takes an Engineer

2009-11-03T06:20:00.000-08:00

I was taking a shower this morning when I realized that it takes an engineer to design a shampoo bottle. My fiancee buys this Garnier Fructis stuff. I don't know if it's any good because I'm not smart enough to tell the difference between two shampoos, but she is so I assume it's not bad. In any event, it may have the dumbest design I've ever seen on a product container.

Take a look at the picture of the bottle. You see that little dark greed ball at the top? That's the mechanism for opening the bottle. Now, I obviously don't know in what order you execute your shower routine, but I'd hazard to guess that by the time you get to opening your shampoo bottle, your hands are already wet.

Now, let's say, just hypothetically, that over time a thin layer of soap or shampoo builds up on that stupid little ball. Now, your hands are wet and the bottle is damp and covered in soap and you have to try to pop the top of that bottle by what means? Grabbing? Prying? Squeezing? Try as you may, you can't get a grip on a semi-spherical wet soapy cap.

But, consider what happens over time. When you squeeze the shampoo out of the bottle and then close the cap, you've undoubtedly confined a small amount of shampoo between the cap and the shampoo dispensing opening. This shampoo dries while you're at work somehow morphing into the exact chemical composition of super glue! The next morning, you have to try to un-super glue your shampoo cap by using your wet fingers to apply friction to a soapy sphere.

WTF Garnier? Just, WTF?

IComparable.Between Extension Method

2009-10-21T08:17:00.000-07:00

T-SQL has a between statement to make it easier to determine if a test object occurs between two other objects. I needed similar functionality in C# but I didn't know what the type the objects would be; however, I did know they would always be IComparable. Obviously, there's no real need for an extension method here, but it did make things a bit more convenient.

Like T-SQL, the between extension method is inclusive and the start parameter doesn't have to be lower than the end parameter.

Here's what some test cases look like:

5.Between(1, 10) // true
5.Between(10, 1) // true
5.Between(10, 6) // false
5.Between(5, 5)) // true

Here's the method:

public static bool Between<T>(this T target, T start, T end)
    where T : IComparable
{
    if (start.CompareTo(end) == 1)
        return (target.CompareTo(end) >= 0) && (target.CompareTo(start) <= 0);

    return (target.CompareTo(start) >= 0) && (target.CompareTo(end) <= 0);
}

Strongly Typed GetValue Extension Method for SqlDataReader

2009-10-21T07:24:00.000-07:00

For a very long time, I've been bitching about the fact that the methods on the System.Data.SqlDataReader class which get a strongly typed result don't have overloads that take column names. I've seen a number of blogs that say, "the reason you can't do this is because you can't overload a method on return type," which is true but irrelevant. I don't see any reason the GetInt32(int i) method couldn't have a GetInt32(string name) overload that handles the GetOrdinal(string name) on your behalf (other than the fact that nobody wanted to write the two dozen overloads.

Well, finding this state of affairs unacceptable, I decided to rectify it. I expected it to be either difficult or annoying depending on the solution I accepted. First, I decided that it should be an extension method because I love extension methods and the fluency they provide. I also decided that it should be generic so you could have one method and specify the desired type. The tough decision came when I was trying to figure out how to handle all of the conversions between types.

Should I switch on the generic type and call the associated method on the SqlDataReader? That is to say, if you call GetValue should I call SqlDataReader.GetInt32 or should I just get the value as an object and handle all of the converting myself? Well, I decided that I'd rather deal with the difficulty of the type conversions than have a big ugly case statement in my method.

Then, I was looking around in the .net classes and found System.Convert.ChangeType and it made the whole process considerably easier. We also had a need to attempt to return the value as a specified type without throwing an exception if it failed (like int.TryParse) so I included a TryGetValue method as well.

public static T GetValue<T>(this SqlDataReader reader, string name)
{
    return (T)Convert.ChangeType(reader[name], typeof(T));
}

public static bool TryGetValue<T>(this SqlDataReader reader, string name, out T output)
{
    try
    {
        output = reader.GetValue<T>(name);
        return true;
    }

    catch (Exception ex)
    {
        if (ex is InvalidCastException || ex is FormatException || ex is OverflowException)
        {
            output = default(T);
            return false;
        }

        else
            throw;
    }
}

// usage examples

// string testString;
// bool result = reader.TryGetValue<string>("ColumnName", out testString);

// int testInt;
// bool result = reader.TryGetValue<int>("ColumnName", out testInt);

// int i = reader.GetValue<int>("ColumnName");
// string s = reader.GetValue<string>("ColumnName");
// DateTime dt = reader.GetValue<datetime>("ColumnName");

Agile Estimating and Planning

2009-09-14T05:05:00.000-07:00

A few months ago my company picked up a new project with a fairly major government organization. The mission is to migrate several applications from a mainframe (Natural / Adabas) to a modern environment (C# / SQL Server). We were told that the mainframe was going to be disabled in about 9 months . . . period.

I told the client very early in the process that a typical waterfall approach would be impossible to implement under the conditions of the project and that we should take an agile methodology. The client agreed; however, the lack of resources corralled a few of our team members back to the waterfall approach. After a few weeks in waterfall, we realized that there was no way a waterfall project could be successful in this case.

We officially switched back to an agile approach and the project has been swimming along nicely ever since. I owe a lot of this success to Agile Estimating and Planning by Mike Cohn.

Agile Estimating and Planning is part of the Robert C. Martin Series and is listed in every "Top 10 Agile Books" lists I've found. The book is full of well thought out logical explanations and detailed examples of agile software development practices and principles. It is a very easy read (I read it in 2 days) and the author did an excellent job of relating each chapter to other chapters in the book.

The book begins by pointing out the difference between planning an agile project and agile planning. Next, Mike describes several reasons planning fails in many projects (some of which I discuss in my blog post a few weeks ago called "Programming is a Gas").

The next several chapters discuss the benefits and techniques of abstracting estimations away from time and monetary units. This creates a set of relative estimations based on feature complexity that allows the project room to change over time (and promotes more open and honest communication with the client). Two techniques discussed specifically are story points and ideal days.

The following section is more planning oriented with a detailed discussion of themes, features, and prioritization. There is a discussion about what to do with epics (very large stories) and boundaries on which to split those stories. Conversely, there is also a discussion about combining several small but related user stories into one larger more meaningful one.

The scheduling section describes release and iteration planning, estimating velocity, and techniques for buffering projects (especially risky projects). the tracking and communicating section gives you a set of methods for monitoring the release and iteration plans and a tool set you can use to communicate this with the project stakeholders.

The next section describes how (and why) agile projects work so well in real life. High visibility into the project, frequent feature based planning, and small story size make agile projects preferred for many software development groups as well as many clients who have had the good fortune to have experienced a well run agile project.

The final chapter is a hypothetical case study which demonstrates the entire agile software development life cycle from beginning to end. In this chapter, you get to see how an agile project begins, how stakeholders become involved, how iterations are planned, and how easily problems can be solved. If you read carefully, you can also pick up a lot of insight Mike has to offer from his experience as an agile coach.

You can purchase Agile Estimating and Planning from Amazon.com at a deeply discounted rate.

Head First Design Patterns

2009-09-03T06:16:00.000-07:00

I read Head First Design Patterns about a year ago and it changed my life. Now, I know that a lot of non-programmers read this blog and I'm sure you must be thinking, "how can a book, especially one like that, possibly change your life?"

Well, it made my code much cleaner, much more flexible, and much easier to write. That's why I decided to make this the first book review on this blog. First of all, I'm a pretty big fan of the Head First style. I like most of the jokes and intentional thought provoking. It's a clever way to help people understand and remember what they're learning. Admittedly, sometimes it goes a little overboard, but I'd rather it go a little overboard than be a dry read. I read almost the whole book on a road trip from Atlanta to Orlando.

The book sets out to give the reader an understanding of what a design pattern is and what it's good for. A design pattern is a template for solving commonly occurring problems in software design. Head First Design Patterns discusses the details of about 15. You'll be surprised how many of these patterns you've seen before (and may have even written before). One of the big benefits you get out of this book is a common language to describe the patterns in your code. Next time you're doing a code review, you'll find yourself saying things like, "this is just a singleton" and your reviewer will understand what you're trying to accomplish without discussing the implementation details.

After reading this book, I've noticed more and more patterns:

The decorator pattern that you see throughout the .net framework. There are a lot of classes that take a stream and return a stream. Some read, some write, some encrypt — all of them are decorators.

The proxy pattern controls access to other objects. I wrote a blog post back in December of 2008 where I used the proxy pattern to write to multiple TextWriters.

The singleton pattern ensures that a class has only one instance and has a global point of access. It bears noting that there are a lot of programmers who have criticized the singleton to which the singleton replied in a public statement last year, "you can kiss my ass." It is true that the singleton can come back to bite you if you use it incorrectly; the singleton is the right tool for the job it has. Head First Design Patterns helps you understand what exactly that job is.

The template method pattern popped up the other day when I was looking at some PostSharp aspects. If you use the OnMethodInvocationAspect like I did with my memoizer, PostSharp will produce something very similar to a template method. Obviously, PostSharp isn't creating a base class and then overriding the template method, but it is allowing you to control the flow of the method without having knowledge of the method implementation.

The factory method and abstract factory patterns are also quite common and are well described in Head First Design Patterns. People use these patterns all the time. I've seen many data access layer implementations where a factory method loads a particular data access provider based on configuration options. You can achieve the same results with inversion of control containers like Ninject and autofac, but sometimes you just need a quick factory.

Almost every review I've seen of Head First Design Patterns has been positive; however, every once in a while you find a detractor like Jeff Atwood in his blog post from 2005. I think Jeff is probably a good guy and a good coder; he's just contrary and cynical. I know that being contrary and cynical doesn't negate your point, but it is a bit like crying wolf. I'm not going to address his argument in this post (because I could dedicate an entire post to the argument if I thought it'd do any good), but I think you should read it because he does have one some valid points.

Unneeded complexity is bad. I wouldn't say that design patterns should be eschewed altogether, but you should use the right pattern for the right problem. Reading this book should teach you to identify patterns yourself. What are some common problems you run into in your field? Can you think of a pattern for addressing these problems?

You have to learn to abstract your knowledge and apply it practically. Don't just decide that patterns are bad and start knocking books because the graphic on the cover has been reused. A pattern doesn't have to be necessary to be valuable. When the pattern provides maintainability, readability, testability, or simplicity then use the pattern; that's what it's there for. The pattern is just another tool in your tool belt.

I very seldom recommend you throw a specialized tool away just because you don't need it all the time. Buy (or borrow) Head First Design Patterns and focus on learning how to recognize and solve common problems. You won't regret it.

You can get Head First Design Patterns from Amazon.com for a great price.

Understanding Tax Brackets Takes an Engineer

2009-08-30T20:33:00.000-07:00

I've talked to dozens of people who have a dramatic misunderstanding of tax brackets (or tax rate schedules). In fact, I think I've run into more people who misunderstand how taxes work than who understand. So, I thought I'd demystify one small part of the American income tax system — brackets.

Many people believe that shifting from one tax bracket to another yields a major leap in income tax. This is incorrect. When you move from the 15% tax bracket to the 25% bracket for example, you're not suddenly paying an additional 10% in taxes. That is to say that if you make $33,950.00 per year (the highest salary for the 15% bracket) and you get a 1 cent raise bumping you up to the next tax bracket, that your change in income tax will actually be zero dollars.

Here's how that works. If you're in the 15% bracket, you pay 15% on income above $8,350.00 and you add $835 to that. The next bracket up is for people who earn more than $33,950.00 and less than $82,250.00. They pay 25% on income above $33,950.00 plus $4,675.00. Now, here's an interesting observation. If you take 15% of $33,950.00 - $8,350.00 (the maximum for the 15% bracket) and add $850, you get $4,675.00. That means that the maximum for the 15% bracket is the same as the minimum for the 25% bracket.

Thus, there's not a major leap between brackets. So, what are brackets for? Well, basically, the tax bracket just defines the rate of change between the minimum and the maximum incomes in the bracket. If you remember taking algebra back in the old days, you're just talking about slope. Basically, your tax bracket defines the slope of the line on which your tax is determined from your income.

Consider the slope-intercept form for a line: y = mx + b where b is the y intercept and m is the slope. If you look at the graph at the top of this blog (created with a free iPhone app called grafunc by Fabio Policarp), you'll see a blue and an orange line. The blue line represents the 15% bracket using the equation y = .15 (x - 8350) + 835 and the orange line represents the 25% bracket with the equation y = .25 (x - 33950) + 4675. Where those two lines intersect is where you switch brackets.

That's about all there is to understanding tax brackets. There's much more to understand about taxes, but that's another blog post altogether.

Programming is a Gas

2009-08-26T05:36:00.000-07:00

Programming is a gas. I don't mean programming is entertaining and exciting (even though sometimes it really is); I mean more along the lines of the physical state of matter. Programming is a gas. You see, a gas having perfect mobility and infinite expansion will take the size and shape of the container you put it in.

Parkinson's Law states that "work expands so as to fill the time available for its completion." I'm taking this one step forward and saying that "software development expands so as to take the size and shape of the constraints placed upon it."

I've worked with tens (if not hundreds) of software engineers and I can tell you one thing with certainty: ceteris paribus, the more the engineer charges for services, the better the result. Now, obviously, some engineers misrepresent their skill level intentionally or unintentionally, but . . . on average . . .

So, if this is the case, you can put several of these programmers together and derive that the more you pay a software consultancy, the better the result as well. I know to some, this probably just sounds obtuse. You might be saying to yourself, "well, some companies just have higher profit margins than others." 'tis true; some programmers have higher profit margins.

If you've been at the top of the game for 10 years, you deserve more than someone who just started programming (or someone who has stagnated for 10 years) because you can apply all of your experience to each new project. Similarly, a software consultancy with premium engineers (no matter how old the organization) can earn a higher margin because its engineers produce such valuable results.

Understanding this stipulation (whether you agree or not) is pivotal to the rest of this post, so I'll very briefly summarize what I'm saying: you get what you pay for!

So, why do so many projects turn sour? If you're a software engineer, think of the last 10 projects you worked on. If you're not an engineer, think of the last 10 projects your company had someone else work on. What's the first step of all of these projects? Almost uniformly, it's an RFP and if not an RFP per se, it's some kind of competitive bidding process.

That's where the situation starts going south — before the project even gets kicked off! Have you ever had a project where the vendor wasn't selected almost exclusively based on cost? When was the last time someone said, "whichever vendor comes back with the best feature ideas will get the contract" or "whichever vendor has the lowest turnover rate and the highest employee satisfaction will get the contract?" In fact, in a perfect world, I believe you could select a vendor based on its mission statement and core values (and perhaps a brief summary of its gestalt experience).

Almost every project I've been on has had two constraints . . . money and time. Before a pen ever touches the signature line on the contract, the project is doomed. The shape of the project is defined by money and time, not by desired features.

Thus, when a developer looks at the project, it is going to take the shape of money and time; however, to the customer, the project is expected to be shaped by the desired features. More often than not, the money x time shape fits quite neatly inside the feature shape.

So, what exactly am I proposing? I'm proposing that the team (whether it is internal or external) should be chosen based on values and trustworthiness. A team you can trust is much more valuable than a team which is cheap. Take the team you can trust and then define the shape of your project based on the features you want to see. When you ask the team for an estimate, it's not to compete for the project; rather, the estimate is so that you'll know if you have the budget for the project in the first place.

If you do it this way, you'll get an honest estimate. Many software companies will low-ball estimates because they know that once you're committed to the project, you'll dedicate whatever additional resources are necessary to complete it. Ultimately, it'll end up costing more in the long run because of the way each "phase" will build on the "phase" before it compared to a solution that was well architected from the get-go.

iPhone Bookmarklet Installer Source

2009-08-06T10:24:00.000-07:00

A few hours ago, I posted my iPhone Bookmarklet Installer Bookmarklet and several people have asked how it works.

Well, it's pretty basic. The effort is to loop through all the links on the page, identify the bookmarklets, and modify them so that the bookmarklet is appended to the URL somehow. The first time I tried, I used the '?' character making the bookmarklet look like a query string. This proved problematic though as a number of bookmarklets failed to convert cleanly. I didn't really investigate but I'd imagine it has something to do with the '&' character and/or '=' signs.

In any event, it's pretty easy to fix that by using the '#' character instead of ? because then it just looks like an anchor. It also keeps the page from having to reload. So, all you have to do is run the bookmarklet on your iPhone, click the bookmarklet you want to install, save the bookmark and remove everything up to and including the #.

Now, the actual bookmarklet code has been scrubbed a little to give it something of a smaller footprint like this:

(function(){
  var s,i,l=document.links;
  for(i=0;i<l.length;i++)
  {
    if(l[i].href.indexOf('javascript:')!=0)
      continue;

    l[i].href='#'+l[i].href;
  }
})();

But, that's a little hard to read, so here's the pre-scrubbed version. There's also an additional section I added later to make it easier to identify which links have been transformed (i.e., which are identified as bookmarklets). Here's the code:

// anonymous function
(function(){

  // put the link array somewhere
  var links = document.links;

  // loop through all the links
  for (var i = 0; i < links.length; i++)
  {

    // if the link does not start with javascript:
    // it isn't a bookmarklet and we can exit this
    // itteration and continue the loop
    if(links[i].href.indexOf('javascript:') != 0)
      continue;

    // update the href of the link prepending the
    // '#' character to make it look like an anchor
    links[i].href = '#' + links[i].href;

    // add some styling information to identify the
    // modified links
    var style = links[i].style;
    style.backgroundColor = '#eee';
    style.color = '#333';
    style.border = '1px solid #333';
    style.textDecoration = 'none';
    style.padding = '2px';
    style.fontWeight = 'normal';
  }
})();

iPhone Bookmarklet to Install iPhone Bookmarklets

2009-08-06T09:23:00.000-07:00

As a software engineer and a web developer, I have really grown to love the bookmarklet. I have a Regex Search Bookmarklet, my favorite javascript shell bookmarklet from Jesse's Bookmarklets, and of course, you've seen my blog post on Tabulate from the clever folks at Inventive Labs. However, installing bookmarklets on the iPhone without help from a computer is a royal PITA.

That's why I wrote the iPhone Bookmarklet Installer.

If you're interested in how it works, you can find details on my iPhone Installer Bookmarklet blog post. But, the short version is that I use Javascript to loop through all of the links on the page and make them look like anchor tags so you can just bookmark them on the iPhone and remove everything before the anchor tag character '#'.

With that out of the way, here's how you use it:

Run the bookmarklet on this page by clicking the iPhone Bookmarklet Installer. You should see all of the bookmarklets on this page change styles (They'll look something like this).
Click the bookmarklet you wish to install (n.b., the page won't reload).
Bookmark the page.
Edit the bookmark and remove everything up to and including the # character.
Name the bookmarklet appropriately.
Save and enjoy.

Leave the Poor Table Alone

2009-07-10T08:47:00.000-07:00

So, I realize that with some clever hacking and a butt-load of time, you can do just about anything with divs and CSS, but I think the anti-table thing has gone way too far! I've even seen people display tabular data without using tables. I mean, great. Look at you. You're fancy and all, but what have you accomplished? Tabular data is what tables were designed for in the first place.

So, why are people so steadfastly opposed to tables in HTML layout? Well, I've seen oodles of reasons including things like speed of data transfer, separation of content and layout, ease of redesign, no spacer gifs, lower html complexity, accessibility for persons with disabilities, etc.

Of course, there are sites out there that demonstrate all of the things that can be accomplished using nothing but divs and CSS and they're really great to look at and enjoy; I've learned a lot about CSS from them. But, just because you can doesn't mean you should. I mean, when was the last time you heard about someone doing something with "nothing but" and it wasn't a stupid human trick? Mac Gyver could take down a swarm of enemies with nothing but a Milk Bone, some shoe string, a little fertilizer, and duct tape; I, on the other hand, carry a Glock.

But, here's the real issue: whether you use tables or not, there're right and wrong ways to do anything. Just like there are wrong ways to use tables for layout, there are wrong ways not to. My point is that you can very cleanly and appropriately use tables for layout and still use CSS for almost all of your actual presentation.

As a result, I've yet to come up with a compelling reason to eliminate tables from layout altogether. For example, using a table for your overall layout can give you cross browser compatibility that is very difficult and effortful to achieve with CSS and divs. CSS and DIVs are rendered differently among the several browsers; however, CSS and tables are relatively consistent.

I've found that often times, I can use only one table and achieve exactly the layout I was looking for when a div design would require 3 or 4 levels of nesting. In this case, it is actually cheaper and less complex to use tables than it is to use divs and it took me a tenth of the time.

While I agree there should be a separation of concerns within the definition of websites to keep content, presentation, and behavior separated, using tables to position elements only is a very minor violation of this principle. In fact, using tables in this way is exactly the same as using divs this way. Afterall, any time you use an HTML tag your intent is to describe something about the way the content is displayed. Then, you modify that behavior with CSS. Really, the only thing you can't do with tables and CSS is fundamentally change the position of items.

I suppose that's where people are coming from when they mention redesign. Now, that may be the case and I don't really know that much about it. I'm not a web designer; I'm a software engineer. I spend a very small percentage of my time concerned with layout as there are usually much bigger fish to fry; however, I've never been on a redesign project that didn't require HTML changes.

As a result, it never made a difference whether the page was tables + CSS or divs + CSS; we always had to rewrite some HTML. Furthermore, it's generally in a template somewhere. In fact, many times, it has been easier for me to reposition elements on a page because I was using tables for layout rather than divs. The CSS required to move stuff around and make it cross-browser compatible is often a royal pain in the ass, but it's pretty easy for me to move a one line server side include or a user control in .net.

I never really understood why people used spacer gifs. I don't use them and never have so I don't know what that has to do with tables. I also don't know what tables have to do with accessibility. I've read some things about readers not being able to do anything with graphics because they can only interpret and present text. Divs, tables, plain text, whatever . . . it's all the same text so I don't really know what this has to do with tables for layout.

In my years of experience, I've gone through the dogmatic anti-table phase and I've gotten over it. I simply cannot come up with a good reason to eschew tabular layout. Most of the time, I can do what I want with divs and CSS and I do. When something is difficult with a div, I usually throw a table at it. As a result, I get the job done faster and I save clients money and they ultimately get the same quality software. I've also not had any complaints about accessibility, maintainability, or load times. To me, it's a win-all-around kind of thing, so until I can find solid arguments against tables, I say leave the poor table alone. It's another tool in the toolbox; use it if it makes your life easier.

Sign Placement Takes an Engineer

2009-07-06T09:51:00.001-07:00

I was driving into work this morning and I noticed a sign off to my right. It was a temporary sign which evidently was warning us drivers of some upcoming event. It was too small to contain the entire message so they broke it down into pages which displayed for about 3 seconds each. The first page I saw read:

July 12, 2009
From 6:00 AM
To 7:00 AM

Interested, I eagerly awaited the next page. About 3 seconds later the sign read:

Between
Exits 6 and 7

But, before I could find out what was happening during that block of time, I passed the sign. What in the world did that third page read? Should I plan on staying home? Should I leave home? What's happening?

Well, I started thinking about it and it occurred to me that it was pretty dumb to put that sign on the side of the road right after a curve. Why not put it after the curve in the ensuing half mile straight away?

So, I looked into it. If you take a look at the image above, you'll see the approximate location of the sign. I drew a line segment starting at the sign, passing the west most occluder, and ending at the median between the northbound and southbound lanes.

Then, I traced the median all the way up to the approximate angle where I could no longer see the sign because the visual angle was too narrow.

Next, I traced the fast lane (the longest route within the "polygon of visibility"). I measured it out in google maps and it turns out that the sign was visible for 448 feet.

The speed limit at this point on GA 400 is 65 miles per hour (approx. 95 ft / s). That means that the sign was visible for just shy of 5 seconds.

I'm sure by now, you've done the math and come to the same conclusion. If the sign pages rotate at a minimum of 3 seconds per page and the maximum visible time at or above the speed limit is 5 seconds, then it is only possible to read 2 pages of data. However, the message must be at least 3 pages in order to be complete.

There's no way to reasonably expect drivers to read the entire message and therefore most people passing that sign today will have no idea what the full message is. Evidently, when it comes to sign placement, it takes an engineer.

It Takes an Engineer

2009-07-06T09:30:00.000-07:00

I'm pretty excited about this post. Not only is it my first blog post in a while (I've been pretty swamped lately and haven't had a chance to write anything), but it's also the first new section I've started in a long time.

This entry marks the first post in a series of posts I call, "It Takes an Engineer." It's about all of those things we see in daily life that would be better if they consulted an engineer first.

I'm sure, if you're reading this blog, you probably know what I'm talking about. It's the stuff that makes you stop and say, "what the hell were you thinking when you did this?" Or, "you really thought this would be intuitive?" Or, "if someone thought about how this was going to be used, I wouldn't have to fix it all the time."

Well, this is going to be my venue for venting on this topic so I hope it provides enjoyment to all. If you have anymore good examples, let me know and I'll blog about them too.

iPhone MMS Fail

2009-06-16T10:59:00.000-07:00

So, I understand that Apple purports to have good reasons for not supporting MMS at the moment and that Apple plans to support MMS this summer for 3G and 3G S phones and AT&T seems to be proud of themselves for offering MMS for free on the iphone despite the fact that MMS is free with basic messaging with almost every other provider, but whatever. That's all well and good.

But, here's what pisses me off . . . aside the fact that MMS has been around a really long time and that the iPhone is one of the few phones that doesn't support it, I'm still really annoyed by the way AT&T . . . oh excuse me . . . at&t decided to hack together some form of MMS support.

If you don't have an iPhone, let me tell you how this goes. You get a text message. The text message links you to a simple at&t multimedia messaging page with text boxes for message id and password. Then, you enter your message id and password which are provided in the text message.

Now, if you don't have a computer handy and you want to view this message on your iPhone, you had better grab a pen, 'cause this message id looks something like this: esth9389uohh or 1ch309903hsuh or 47&75uck5r3411y3ffingb4d! How I'm expected to remember this, I don't know.

Then, the password is a little easier to remember. It's just two four letter words separated by a two digit number. So, why do I find this so frustrating? Well, simply because there is no reason under the sun to do this. There is absolutely no difference between my entering these codes manually and at&t tacking them onto the query string. In fact, if they did that, they could use longer keys and could make it harder to brute force.

Then, when I got an MMS message, I could just click the link and, lo and behold, there would be my message. I wouldn't have to try to write down the dumb message id and try to figure out if that l is a 1.

I've got a password for you at&t . . . epic69fail.

Soundex Extension Method for C#

2009-06-12T17:59:00.000-07:00

I've been pretty swamped at work lately so I've not been blogging much. Worse than that, I haven't really been able to do much creative programming and virtually nothing academic.

A few days ago, I was working on some basic name searching and recognized an opportunity to squeeze in some play time. It occurred to me that the users of our application will be looking people up by name and will often not know how to spell the name correctly.

I wanted to allow users to search for people by name without regard for different spellings. For example, if you're looking for Geoff McArther or Jeff MacArther, both names will be returned by the same search term.

To do this, I used a technology invented back in the 1920s for the US Census called soundex. The following is a set of extension methods that implement the soundex phonetic algorithm:

public static string Soundex(this string s)
{
    // by default, a soundex is the first letter
    // followed by three numbers
    return Soundex(s, 4);
}

public static string Soundex(this string s, int length)
{
    return FullSoundex(s)
        .PadRight(length, '0')  // soundex is no shorter than 
        .Substring(0, length);  // and no longer than length
}

public static string FullSoundex(this string s)
{
    // the encoding information
    const string chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
    const string codes = "0123012D02245501262301D202";

    // some helpful regexes
    Regex hwBeginString = new Regex("^D+");
    Regex simplify = new Regex(@"(\d)\1*D?\1+");
    Regex cleanup = new Regex("[D0]");
    
    // i need a capitalized string
    s = s.ToUpper();

    // i'm building the coded string using a string builder
    // because i think this is probably the fastest and least
    // intensive way
    StringBuilder coded = new StringBuilder();

    // do the encoding
    for (int i = 0; i < s.Length; i++)
    {
        int index = chars.IndexOf(s[i]);
        if (index >= 0)
            coded.Append(codes[index]);
    }

    // okay, so here's how this goes . . .
    // the first thing I do is assign the coded string
    // so that i can regex replace on it
    string result = coded.ToString();

    // then i remove repeating characters
    //result = repeating.Replace(result, "$1");
    result = simplify.Replace(result, "$1").Substring(1);

    // now i need to remove any characters coded as D  from
    // the front of the string because they're not really 
    // valid as the first code because they don't have an 
    // actual soundex code value
    result = hwBeginString.Replace(result, string.Empty);
    
    // i used the char D to indicate that an h or w existed
    // so that if to similar sounds were separated by an h or
    // a w that I could remove one of them.  if the h or w does
    // not separate two similar sounds, then i need to remove
    // it now
    result = cleanup.Replace(result, string.Empty);

    // return the first character followed by the coded
    // string
    return string.Format("{0}{1}", s[0], result);
}

Now, it may not yet be perfect as I had a pretty hard time finding the exact specs for the American soundex, but it should be pretty close. If you notice something wrong, please leave me a comment and I'll correct it.

Communication Clarity Fail

2009-06-12T10:26:00.000-07:00

When I'm not writing awesome software (grin), you can usually find me flying airplanes, dreaming about airplanes, thinking about airplanes, or blogging about airplanes.

Yesterday, I was getting ready to make my first solo in a Super Decathlon, and I spent an agonizing nerve-racked hour waiting for my instructor to get off of a conference call. I thought I'd try to work off some nervous energy reading.

Lately, I've been reading The Pragmatic Programmer. It's a really great book and will probably be the subject of several forthcoming posts reviewing the book, but that's not the topic here.

This post is about the coincidence of me sitting there at the airport reading about programming and finding this British Airways memorandum that was published in Pilot Magazine in 2006:

From British Airways Flight Operations Department notice:

There appears to be some confusion over the new pilot role titles.
This notice will hopefully clear up any misunderstandings.
The titles P1, P2, and Co-Pilot will now cease to have any meaning,
within the British Airways operational manuals. They are to be
replaced by

-Handling Pilot,
-Non-Handling Pilot,
-Handling Landing Pilot,
-Non-Handling Landing Pilot,
-Handling Non-Landing Pilot,
-Non-Handling Non-Landing Pilot.

The Landing Pilot is initially the Handling Pilot and will handle the
take-off and landing except in role reversal when he is the
Non-Handling Pilot for taxi until the Handling Non-Landing hands
the handling to the Landing Pilot at 80 knots.
The Non-Landing (Non-Handling, since the Landing Pilot is handling)
Pilot reads the checklist to the Handling Pilot until after Before
Descent Checklist completion, when the Handling Landing Pilot
hands the handling to the Non-Handling Non-Landing Pilot who
then becomes the Handling Non-Landing Pilot.
The Landing Pilot is the Non-Handling Pilot until the 'decision
altitude' call, when the Handling Non-Landing Pilot hands the
handling to the Non-Handling Landing Pilot, unless the latter call
'go-around', in which case the Handling Non-Landing Pilot
continues handling and the Non-Handling Landing Pilot continues
non-handling until the next call of 'land' or 'go-around' as appropriate.
In view of recent confusions over these rules, it was deemed
necessary to restate them clearly

Oh, thanks for clearin' that one up fellas!

Automatic SVN Updates for Offsite Backup

2009-05-19T07:10:00.001-07:00

My company has something of an offsite backup strategy. We have two external hard drives which are manually swapped out on a daily basis and someone takes the inactive one home.

We're a pretty small company and that works fine. Of course, while said person is in the office, both drives (the current drive and the offsite backup) are onsite. We also have a data center where we send our really important stuff too.

I've been doing my part to help in disaster recovery by keeping my own copy of the company SVN repository on an external drive at my house. The annoying thing was that I was always attaching the drive to my laptop and copying it over manually.

That process grew irritating so I installed TortoiseSVN on my home server and now I can just update the SVN folder and our other offsite backup is up to date. But then, I got a little tired of doing that too!

I decided to automate the entire process. First, I needed to connect to the office VPN (which is an OpenVPN server). Then, I needed to update the two repositories. Finally, I needed to disconnect from the office VPN.

The OpenVPN was really the hard part because if I opened a connection on the command line, I couldn't close the connection without sending an F4 keypress. I didn't want to write a windows app for this 'cause I should really be able to do this with a batch file.

Turns out, OpenVPN has a windows service and I can start those command line, so here's my script:

@echo off

echo net start "OpenVPN Service"       >> %esg%\tempGetESGSVN.bat
echo ipconfig /flushdns                >> %esg%\tempGetESGSVN.bat
echo svn up %esg%\Contractor           >> %esg%\tempGetESGSVN.bat
echo svn up %esg%\Internal             >> %esg%\tempGetESGSVN.bat
echo net stop "OpenVPN Service"        >> %esg%\tempGetESGSVN.bat
echo del %esg%\tempGetESGSVN.bat /F    >> %esg%\tempGetESGSVN.bat

%esg%\tempGetESGSVN.bat >> %esg%\GetESGSVN.log

@echo on

This batch file looks a little funny and that's because it is. Basically, it's creating another batch file which executes all of the steps and then deletes itself. I did this 'cause it made it a little easier for me to log. The batch file it creates looks like this:

net start "OpenVPN Service"
ipconfig /flushdns            
svn up %esg%\Contractor       
svn up %esg%\Internal         
net stop "OpenVPN Service"    
del %esg%\tempGetESGSVN.bat /F

Nota bene, if you don't have a command line SVN client installed, you can use Slik Subversion.

Conditional Check Constraints on SQL Server

2009-05-18T14:11:00.000-07:00

I was on Stack Overflow the other day (something I'm just getting into . . . not sure I care that much for it yet, but we'll see. I'm definitely enjoying it more now that I'm not restricted on my behaviors because I lack reputation points).

In any event, someone asked me a question about conditional check constraints. The consensus seemed to be that you couldn't do it so I figured I'd post the solution on my blog as well.

Basically, the thought was that you couldn't put the check constraint on a subquery or an aggregate so you had to use a trigger only. Well, that's not technically true since you can write a function to do the aggregating for you. I'm not sure I like the idea, but it is in fact possible and I'm looking forward to reading all of your comments about the topic.

So, here's my script that comprises an entire test including a test table, the function, the constraint, some inserts, the results, and cleaning up:

CREATE TABLE CheckConstraint
(
  Id TINYINT,
  Name VARCHAR(50),
  RecordStatus TINYINT
)
GO

CREATE FUNCTION CheckActiveCount(
  @Id INT
) RETURNS INT AS BEGIN
  
  DECLARE @ret INT;
  SELECT @ret = COUNT(*) FROM CheckConstraint WHERE Id = @Id AND RecordStatus = 1;
  RETURN @ret;

END;
GO

ALTER TABLE CheckConstraint
  ADD CONSTRAINT CheckActiveCountConstraint CHECK (dbo.CheckActiveCount(Id) <= 1 OR RecordStatus <> 1);

INSERT INTO CheckConstraint VALUES (1, 'No Problems', 2);
INSERT INTO CheckConstraint VALUES (1, 'No Problems', 2);
INSERT INTO CheckConstraint VALUES (1, 'No Problems', 2);
INSERT INTO CheckConstraint VALUES (1, 'No Problems', 1);

INSERT INTO CheckConstraint VALUES (2, 'Oh no!', 1);
INSERT INTO CheckConstraint VALUES (2, 'Oh no!', 2);
INSERT INTO CheckConstraint VALUES (2, 'Oh no!', 1);
-- Msg 547, Level 16, State 0, Line 14
-- The INSERT statement conflicted with the CHECK constraint "CheckActiveCountConstraint". The conflict occurred in database "TestSchema", table "dbo.CheckConstraint".
INSERT INTO CheckConstraint VALUES (2, 'Oh no!', 2);

SELECT * FROM CheckConstraint;
-- Id   Name         RecordStatus
-- ---- ------------ ------------
-- 1    No Problems  2
-- 1    No Problems  2
-- 1    No Problems  2
-- 1    No Problems  1
-- 2    Oh no!       1
-- 2    Oh no!       2

ALTER TABLE CheckConstraint
  DROP CONSTRAINT CheckActiveCountConstraint;

DROP FUNCTION CheckActiveCount;
DROP TABLE CheckConstraint;

Extension Method to Imitate Upto() in Ruby

2009-05-12T08:23:00.000-07:00

Last month, I wrote a blog post about using an extension method to imitate the ruby method times(). I recently found another cool method in ruby that I wanted to avail to the C# public.

Now, before I show you the implementation, I'll just say that it doesn't do anything the for loop wouldn't do on its own. I think the method has two benefits. One, I think it looks clean because it's basically a fluent way to create a for loop. Two, I like the fact that you can pass an action to the method.

In Ruby, the implementation looks like this:

5.upto(10){ |i| puts i; }

# output
# 5
# 6
# 7
# 8
# 9
# 10

I added the following extension method to one of my utility extension method libraries:

public static void UpTo(this int start, int upto, Action<int> action)
{
    for (int i = start; i <= upto; i++) action(i);
}

Here are the test cases and the results:

[TestCase(4, 12)]
[TestCase(-10, 10)]
[TestCase(10, 5)]
public void test_upto_iterator(int from, int upto)
{
    int test = 0;
    from.UpTo(upto, num =>
                        {
                            Console.WriteLine("Index: {0}", num);
                            test += num;
                        }
        );

    int expected = 0;
    for (int i = from; i <= upto; i++)
        expected += i;

    Assert.AreEqual(expected, test);
}

// ***** ESG.Utilities.System.Tests.IntegerExtensionTests.test_upto_iterator(-3,2)
// Index: -3
// Index: -2
// Index: -1
// Index: 0
// Index: 1
// Index: 2

// ***** ESG.Utilities.System.Tests.IntegerExtensionTests.test_upto_iterator(4,8)
// Index: 4
// Index: 5
// Index: 6
// Index: 7
// Index: 8

Stored Procedure to Unpivot a Table

2009-05-05T08:39:00.001-07:00

When I first started blogging, I wrote a post about this 3 liner (well, technically) stored procedure to unpivot data. Well, I was new to blogging and I didn't really have all of the great tools that I have now, so it was a pretty crappy post.

Well, I was just flipping through my archives looking at the lack of SQL Server posts that I have and thought, "I really should rewrite that one and use a good highlighter and post some sample usage."

So, I did, but before I get to the stored procedure, why don't I explain a little bit about what unpivoting is? Sometimes you don't know what kind of data you're going to be collecting so you can't really create a third normal schema to store these data. Instead, you use something called an Attribute Value Pair table (or AVP for short).

This table ends up being tall instead of wide, but you can store any data you please in there. Problem is, how do you get to a full and meaningful record? Well, the process to take tall data and make it wide is called pivoting. If, however, you have wide tables and you need to seed this AVP table, that process is called unpivoting.

I had to do this several times and I didn't really care to write a view every time I had to unpivot, so I wrote a sproc instead. Here's that stored procedure and it should work on all SQL Server 2008 and SQL Server 2005:

CREATE PROCEDURE UnpivotTable
(
  @tableName AS VARCHAR(512),
  @whereClause AS VARCHAR(2000) = NULL,
  @commonColumns AS VARCHAR(2000) = NULL
) AS
-- ==========================================================
-- Author:        Patrick Caldwell
-- Create date:   2007/12/19
-- Description:   this procedure unpivots data in a table
--                specified in the parameters with an optional
--                where clause.  every line has been commented.
--                the commonColumns variable stores a list of
--                columns that should be on every row.
-- ==========================================================


-- a variable in which to store the query
DECLARE @query VARCHAR(MAX);

-- build the query
SELECT            @query = ISNULL(@query + ' ', '') + QueryLine
FROM
(
    SELECT        QueryLine = 'SELECT ' +
                    ISNULL(@commonColumns + ', ', '') +
                    'ColumnName = ''' + COLUMN_NAME + ''', ' +
                    'ColumnValue = CAST([' + COLUMN_NAME + '] AS NVARCHAR(4000)) ' +
                    'FROM ' + @tableName +
                    ISNULL(' WHERE ' + @whereClause, '') +
                    ' UNION ALL'
    FROM          INFORMATION_SCHEMA.COLUMNS
    WHERE         TABLE_SCHEMA + '.' + TABLE_NAME = @tableName

    UNION ALL

    SELECT        'SELECT ' +
                    ISNULL(@commonColumns + ', ', '') +
                    'NULL, NULL ' +
                    'FROM ' + @tableName + ' ' +
                    'WHERE 1 = 0'
) AS lines;

-- execute the query
EXEC(@query);

For the sake of example, I'm going to need a test table. Here's the one I'm using:

CREATE TABLE Pals
(
  Id TINYINT IDENTITY(1, 1), -- so what if I only have 255 pals?
  FirstName VARCHAR(128),
  LastName VARCHAR(128),
  Gender CHAR(1)
);

INSERT INTO Pals VALUES ('Lauren', 'Woody', 'F'); -- no, really . . . F
INSERT INTO Pals VALUES ('James', 'Brechtel', '?');
INSERT INTO Pals VALUES ('Ryan', 'McGarty', 'M');

And finally, here are some example usages:

EXEC UnpivotTable @tableName = 'dbo.pals'
/*
ColumnName ColumnValue
---------- -----------
Id         1
Id         2
Id         3
FirstName  Lauren
FirstName  James
FirstName  Ryan
LastName   Woody
LastName   Brechtel
LastName   McGarty
Gender     F
Gender     ?
Gender     M
*/

EXEC UnpivotTable @tableName = 'dbo.pals', @commonColumns = 'Id'
/*
Id   ColumnName ColumnValue
---- ---------- -----------
1    Id         1
2    Id         2
3    Id         3
1    FirstName  Lauren
2    FirstName  James
3    FirstName  Ryan
1    LastName   Woody
2    LastName   Brechtel
3    LastName   McGarty
1    Gender     F
2    Gender     ?
3    Gender     M
*/

EXEC UnpivotTable @tableName = 'dbo.pals', 
                  @commonColumns = 'Id', 
                  @whereClause = 'Gender = ''F'''
/*
Id   ColumnName ColumnValue
---- ---------- -----------
1    Id         1
1    FirstName  Lauren
1    LastName   Woody
1    Gender     F
*/

Converting Hexadecimal or Binary to Decimal on SQL Server 2008 and 2005

2009-05-05T08:10:00.000-07:00

I just finished a blog post about Converting Decimal to Hexadecimal with T-SQL on SQL Server 2008 and 2005 and it got me thinking, "What if you have a hexadecimal string or a binary string and you want to convert it back to decimal?"

Well, in my previous article, I provided a SQL custom function that would allow you to convert from base 10 (decimal) to any other base you wanted. That way, you wouldn't need a Dec2Bin and a Dec2Hex function. Similarly, I don't think you should have both a Bin2Dec and a Hex2Dec to convert from binary or hexadecimal to decimal.

So, I wrote a custom function that converts from any base to base 10 and it looks a little like this:

CREATE FUNCTION ConvertFromBase
(
    @value AS VARCHAR(MAX),
    @base AS BIGINT
) RETURNS BIGINT AS BEGIN

    -- just some variables
    DECLARE @characters CHAR(36),
            @result BIGINT,
            @index SMALLINT;

    -- initialize our charater set, our result, and the index
    SELECT @characters = '0123456789abcdefghijklmnopqrstuvwxyz',
           @result = 0,
           @index = 0;

    -- make sure we can make the base conversion.  there can't
    -- be a base 1, but you could support greater than base 36
    -- if you add characters to the @charater string
    IF @base < 2 OR @base > 36 RETURN NULL;

    -- while we have characters to convert, convert them and
    -- prepend them to the result.  we start on the far right
    -- and move to the left until we run out of digits.  the
    -- conversion is the standard (base ^ index) * digit
 WHILE @index < LEN(@value)
        SELECT @result = @result + POWER(@base, @index) * 
                         (CHARINDEX
                            (SUBSTRING(@value, LEN(@value) - @index, 1)
                            , @characters) - 1
                         ),
               @index = @index + 1;

    -- return the result
    RETURN @result;

END

Here's a test query:

SELECT
   dbo.ConvertFromBase('110010110', 2) -- 406
  ,dbo.ConvertFromBase('120001', 3)    -- 406
  ,dbo.ConvertFromBase('626', 8)       -- 406
  ,dbo.ConvertFromBase('406', 10)      -- 406
  ,dbo.ConvertFromBase('196', 16)      -- 406
  ,dbo.ConvertFromBase('ba', 36);      -- 406

Converting Decimal to Hexadecimal with T-SQL on SQL Server 2008 and 2005

2009-05-05T07:33:00.001-07:00

Yesterday, I was doing a little maintenance on our internal bugtracker.net installation. Bug tracker uses the first column of a result set to determine the background color of the issue. I wanted to create a nice fade effect between green and red, but that requires that I convert a decimal number to hexadecimal in SQL Server.

I didn't really want to write a Dec2Hex conversion function for SQL Server because I could very well end up needing a Dec2Bin function as well (if I ever wanted to convert decimal to binary in SQL), so I just wrote a custom function to convert a decimal number to any base between 2 and 36. The same SQL custom function can convert decimal numbers to hexidecimal or decimal to binary (if you need to go the other way, look here to convert any base to decimal in SQL). Here's what that looks like:

CREATE FUNCTION ConvertToBase
(
    @value AS BIGINT,
    @base AS INT
) RETURNS VARCHAR(MAX) AS BEGIN

    -- some variables
    DECLARE @characters CHAR(36),
            @result VARCHAR(MAX);

    -- the encoding string and the default result
    SELECT @characters = '0123456789abcdefghijklmnopqrstuvwxyz',
           @result = '';

    -- make sure it's something we can encode.  you can't have
    -- base 1, but if we extended the length of our @character
    -- string, we could have greater than base 36
    IF @value < 0 OR @base < 2 OR @base > 36 RETURN NULL;

    -- until the value is completely converted, get the modulus
    -- of the value and prepend it to the result string.  then
    -- devide the value by the base and truncate the remainder
    WHILE @value > 0
        SELECT @result = SUBSTRING(@characters, @value % @base + 1, 1) + @result,
               @value = @value / @base;

    -- return our results
    RETURN @result;

END

Here is a test query:

SELECT
   dbo.ConvertToBase(406, 2)   -- 110010110
  ,dbo.ConvertToBase(406, 3)   -- 120001
  ,dbo.ConvertToBase(406, 8)   -- 626
  ,dbo.ConvertToBase(406, 10)  -- 406
  ,dbo.ConvertToBase(406, 16)  -- 196
  ,dbo.ConvertToBase(406, 36); -- ba

Extension Method to Imitate Times() in Ruby

2009-04-30T10:59:00.001-07:00

My friend and co-worker James is gettin' pretty darned good with Ruby. My experience with Ruby is severely limited. So far, the only thing I've done with it was to implement a "business rules engine" to host a DSL in Iron Ruby.

Today, he showed me a pretty cool method that's built into Ruby called int.times() and I was pretty much instantly jealousified! I decided I had to have one . . . and now I do:

public static void Times(this int times, Action action)
{
    for (int i = 0; i < times; i++) action();
}

I through together a few test cases to try it out. I had to make a few determinations though. If you call .Times on a negative number, it loops indefinitely (seemingly). I decided that you should never want to say "do something infinity times." If you do want to do that, you should just have your action call itself at the end of its execution. So, here's the test and the output:

[TestCase(4)]
[TestCase(-6)]
public void test_times_iterator(int times)
{
    int i = 0;
    times.Times(() =>
    {
        Console.WriteLine("Index: {0}", i);
        i++;
    });

    Assert.AreEqual(Math.Max(times, 0), i);
}

// Index: 0
// Index: 1
// Index: 2
// Index: 3

IComparer Extension Methods for Fluent Interface

2009-04-29T13:10:00.000-07:00

I've pretty much been a blogging fiend today! I love it when I find a topic like IComparer or Extension Methods that are entertaining enough to write about.

Today alone, I wrote articles about how IComparer works, using the proxy pattern to sort by multiple IComparers, and converting a Comparison delegate to an IComparer. Bragging aside though, I've found a fun way to combine my current favorite topics: IComparer and Extension Methods.

I wrote 3 extension methods to help me do some pretty neat things with IComparers and Comparisons. You'll see references to two of my prior articles in this post. The ComparerProxy sorts by multiple comparers and the ComparisonComparer converts a Comparison to an IComparer.

I'll show you the extension methods first, then I'll explain the reasoning behind them, and then I'll show you some usage.

public static class IComparerExtensions
{
    public static IComparer<T> Then<T>(this IComparer<T> priority, IComparer<T> then)
    {
        return new ComparerProxy<T>(priority, then);
    }

    public static IComparer<T> Invert<T>(this IComparer<T> comparer)
    {
        return new ComparisonComparer<T>((x, y) => comparer.Compare(x, y) * -1);
    }

    public static IComparer<T> ToComparer<T>(this Comparison<T> comparison)
    {
        return new ComparisonComparer<T>(comparison);
    }
}

So, lately I've been pretty big on fluent interfaces. I haven't written one yet, but I do use extension methods to make my implementations look a little more fluent. That's why, when I wrote the ComparerProxy, I thought, "Hey, wouldn't be cool if I could just take an IComparer and tack another comparer onto it? And then another? And another?" That's why I wrote the IComparer.Then() extension method.

Next, I realized that if you have a Comparison delegate, it'd be nice to be able to add that to a list of IComparers so I added the ToComparer method.

Finally, another thing that I often want to do is write an IComparer once but to be able to use it in reverse. Id est, I want to invert the logic. Obviously, I could List.Sort() and List.Reverse(), but that's much less efficient than having a CaseInsensitiveComparer and a ReverseCaseInsensitiveComparer.

The problem is, I don't really want to have to write both of those comparers. Instead, now I can call IComparer.Invert() and I get an IComparer that performs exactly like the original, but with inverted logic.

So, without further ado, here are some usage examples. I'm using the same test setup as I did with the ComparerProxy. I've also created several more IComparers to try out:

Comparison<Person> length = (x, y) => x.ToString().Length - y.ToString().Length;
IComparer<Person> lengthLast = new ComparerProxy<Person>(length.ToComparer(), last);

IComparer<Person> lastFirstComposed = last.Then(first);

I executed the following lines which produced the commented results:

_folks.Sort(length);
PrintList(_folks);

// Woody, Pete
// Woody, Lauren
// Caldwell, Cory
// Caldwell, Colin
// Brechtel, Jamus
// Woody, Royce Ann
// Brechtel, Ashley
// Caldwell, Jennifer


_folks.Sort(lengthLast);
PrintList(_folks);

// Woody, Pete
// Woody, Lauren
// Caldwell, Cory
// Brechtel, Jamus
// Caldwell, Colin
// Brechtel, Ashley
// Woody, Royce Ann
// Caldwell, Jennifer


_folks.Sort(lastFirstComposed);
PrintList(_folks);

// Brechtel, Ashley
// Brechtel, Jamus
// Caldwell, Colin
// Caldwell, Cory
// Caldwell, Jennifer
// Woody, Lauren
// Woody, Pete
// Woody, Royce Ann


_folks.Sort(last.Invert().Then(first));
PrintList(_folks);

// Woody, Lauren
// Woody, Pete
// Woody, Royce Ann
// Caldwell, Colin
// Caldwell, Cory
// Caldwell, Jennifer
// Brechtel, Ashley
// Brechtel, Jamus

Converting Comparison to IComparer

2009-04-29T11:25:00.000-07:00

Today, I was working with IComparers (mostly for my Demystifying IComparers blog post). I realized that there's a lot of use for IComparers when working with C# and Linq. I decided that I should dedicate an entire IComparer section to C# Icomparers.

I was working on fodder for several articles when I came across a little annoyance. There's not a Comparer class that takes a Comparison delegate as a constructor argument. Thus, I can call List.Sort() and pass a Comparison delegate or a lambda expression but I cannot create an IComparer that executes that delegate or lambda.

Dissatisfied with this state of affairs, I throw together this little class:

public class ComparisonComparer<T> : IComparer<T>
{
    private readonly Comparison<T> _comparison;

    public ComparisonComparer(Comparison<T> comparison)
    {
        _comparison = comparison;
    }

    public int Compare(T x, T y)
    {
        return _comparison(x, y);
    }
}

IComparer Proxy to Sort by Multiple Conditions

2009-04-29T11:04:00.000-07:00

I thought of another cool use for implementing IComparer.

If you've ever wanted to sort a list of objects using more than one comparer (i.e., you want to sort a list of people by last name and then by first name), then you probably had to compose your own IComparer object to do so. If you wanted to be able to dynamically change sort orders, then your task was more difficult.

To address this problem, I wrote a class called ComparerProxy which implements IComparer and proxies comparisons through a list of IComparers. First, I'll show you the class and then I'll explain a few nuances.

public class ComparerProxy<T> : IComparer<T>
{
    private readonly IComparer<T>[] _comparers;
    
    public ComparerProxy(params IComparer<T>[] comparers)
    {
        _comparers = comparers;
    }

    public int Compare(T x, T y)
    {
        int retVal = 0, i = 0;

        while (retVal == 0 && i < _comparers.Length)
            retVal = _comparers[i++].Compare(x, y);

        return retVal;
    }
}

So, what's happening here is that when ComparerProxy.Compare() is called, it loops through the collection of comparers until it either runs out of comparers and deems the two objects equal or it identifies a difference and returns the value of that difference.

That sounds confusing even to me and I wrote the class so I'll break it down. Basically, imagine you have a Person object (and soon we will). The person has a FirstName property and a LastName property. If you want to sort by LastName and then by FirstName, it would go something like this:

Sort the two objects by LastName
If the two objects have the same LastName, then sort by FirstName
If the two objects have the same FirstName, then the two objects are equal

If that doesn't make sense, please comment and I'll elaborate.

So, shall we see what it does? I created a Person class (very similar do the one described above) just to do some testing. I overrode the ToString method to print "Last, First" and I added two IComparers. Here's what that class looks like:

private class Person
{
    public string First, Last;

    public class FirstNameComparer : IComparer<Person>
    {
        public int Compare(Person x, Person y)
        {
            return x.First.CompareTo(y.First);
        }
    }

    public class LastNameComparer : IComparer<Person>
    {
        public int Compare(Person x, Person y)
        {
            return x.Last.CompareTo(y.Last);
        }
    }

    public override string ToString()
    {
        return string.Format("{0}, {1}", Last, First);
    }
}

I created a test list of Persons and a set of IComparers to test with including two complex IComparers using the ComparerProxy:

List<Person> _folks = new List<Person>();
_folks.Add(new Person { First = "Jennifer", Last = "Caldwell" });
_folks.Add(new Person { First = "Cory", Last = "Caldwell" });
_folks.Add(new Person { First = "Lauren", Last = "Woody" });
_folks.Add(new Person { First = "Colin", Last = "Caldwell" });
_folks.Add(new Person { First = "Jamus", Last = "Brechtel" });
_folks.Add(new Person { First = "Pete", Last = "Woody" });
_folks.Add(new Person { First = "Royce Ann", Last = "Woody" });
_folks.Add(new Person { First = "Ashley", Last = "Brechtel" });

IComparer<Person> first = new Person.FirstNameComparer();
IComparer<Person> last = new Person.LastNameComparer();

IComparer<Person> firstLast = new ComparerProxy<Person>(first, last);
IComparer<Person> lastFirst = new ComparerProxy<Person>(last, first);

I executed the following lines and, lo and behold, achieved the expected results:

_folks.Sort(last);
PrintList(_folks);

// Brechtel, Jamus
// Brechtel, Ashley
// Caldwell, Jennifer
// Caldwell, Colin
// Caldwell, Cory
// Woody, Lauren
// Woody, Royce Ann
// Woody, Pete


_folks.Sort(first);
PrintList(_folks);

// Brechtel, Ashley
// Caldwell, Colin
// Caldwell, Cory
// Brechtel, Jamus
// Caldwell, Jennifer
// Woody, Lauren
// Woody, Pete
// Woody, Royce Ann


_folks.Sort(firstLast);
PrintList(_folks);

// this one is the same as just sorting by first name
// in retrospect, I should've added some folks with the same first name
// c'est la vie
//
// Brechtel, Ashley
// Caldwell, Colin
// Caldwell, Cory
// Brechtel, Jamus
// Caldwell, Jennifer
// Woody, Lauren
// Woody, Pete
// Woody, Royce Ann


_folks.Sort(lastFirst);
PrintList(_folks);

// Brechtel, Ashley
// Brechtel, Jamus
// Caldwell, Colin
// Caldwell, Cory
// Caldwell, Jennifer
// Woody, Lauren
// Woody, Pete
// Woody, Royce Ann

Demystifying IComparer in C#

2009-04-29T08:03:00.000-07:00

I was looking through the analytics on my blog and I noticed a trend. I get a ton of searches for things related to the IComparer in C# article I wrote a few months back. I can only imagine that the reason people are searching for information on the IComparer so frequently is because the IComparer is a little mysterious to most people. It's pretty uncommon that you need to use one (let alone write it).

In truth, the IComparer Documentation isn't really all that helpful. Most examples of an IComparer that I found when I was working on my best fit select and my quickselect articles simply use x.CompareTo(y) or a built in IComparer.

Sure, that works, but I think people are hitting my blog trying to uncover the inner workings of the actual comparison.

Well, here it is. IComparer has one member method called Compare(object x, object y) (the generic version is strongly typed) which returns an integer. If the value of x is less than y, the result is less than 0. If the value of x is greater than y, the result is greater than zero. If the two values are equal, the result is 0.

Now, in most implementations I've seen, the results are nominal data (i.e., one trit valued -1, 0, or 1); however, I don't really see any reason you can't return ordinal, interval, or ratio data as well (although, returning ordinal or interval data would be relatively difficult as it requires knowledge of the rest of the collection). If that didn't make sense, take a look at levels of measurement. If you'd like me to elaborate on levels of measurement, leave a comment and I'll write an entire post about it.

Using my fictional class Person with the Age property, a common AgeComparer class would look something like this:

public class AgeComparer : IComparer<Person>
{
    public virtual int Compare(Person x, Person y)
    {
        if (x.Age == y.Age) return 0;
        return (x.Age > y.Age) ? 1 : -1;
    }
}

While that's useful for sorting, it doesn't really give you any information about the magnitude of the difference between the two objects and may as well just be called WhichOneIsGreater(object x, object y). If you were using the comparer to graph items in a list, this would be relatively useless. Also, it's an entire line of code more than you need, so I'm proposing a solution like the following:

public class AgeComparer : IComparer<Person>
{
    public virtual int Compare(Person x, Person y)
    {
        return x.Age - y.Age;
    }
}

One caveat is that some people use IComparers expecting 1, 0, or -1 (i.e., if (IComparer.Compare(x, y) == 1) {...}). Certainly, using a non-nominal result would break their code, but IMO had they read the documentation on IComparer, they'd have seen nothing about 1, 0, and -1 as the return value.

I challenge you to think of other comparisons you could make between objects and post them in the comments of this blog post. In the meantime, I'm going to start an entire blog section on IComparer and we'll see what we can come up with.

Required iPhone Knowledge for International Travel

2009-04-23T12:03:00.000-07:00

PDA Mode

You can disable the cellphone antenna on the iPhone by putting it in airplane mode. By default, airplane mode also disables the wifi antenna. You can re-enable the wifi antenna independently of the cellphone antenna by clicking Wi-Fi and flipping the switch to on. This will allow you to access the internet without incurring roaming charges.

Textual Intercourse

One thing you'll miss is text messaging. Do not fret Littlefoot; in the days between snail mail and text messages, people sent emails. iPhone works great with emails. Set up your gmail account on your iPhone and check it whenever you get internet access.

If you need more immediate response times, get a multi-protocol iPhone IM client. There are several great IM clients for the iPhone that you can get for free. My favorites are Fring and Palringo. I like the Palringo interface a lot for IMing, but I like Fring for IM and Skype support.

Phone Home

Calling home is a lot of fun when you're on an international adventure so you can make all of your friends and family jealous. I'm a big fan of the Skype unlimited U.S. land line calling plan for $2.95 so that's what I use. Also, Skype-to-Skype calls are free. There may be better VoIP services out there, but I haven't found one yet.

So, how do you Skype on the iPhone? Like I said before, Fring supports Skype and multiple IM clients and has a decent interface. As a note, when you use Fring to make a Skype call, you have to put +1 before the area code and phone number (i.e., +17705551212). Skype recently came out with its own iPhone application but it's spotty at best for me; you may have better luck.

Social Networking

Facebook has a great free iPhone application to keep you in touch with your friends and family back home. If you tweet, I really like TwitterFon. TwitterFon conveniently supports twitpic too.

Finances

When you're traveling internationally, it's nice to be able to check on your bank accounts back home; however, it'd be pretty tough to get to every page you're interested in within the 1 hour of internet access you get for 5 billion euro. If you use Mint.com (and you should), then you can get the free Mint.com iPhone application and check every balance at once and get a detailed transaction list.

Literature

Search the iTunes App Store for literature you may want. You can find travel guides, maps, and translators. Also, search the internet for images relevant to your trip. For example, find a large image of the subway system and break it into sections so you can view it on your iPhone.

Extension Method to Join Strings with a Delimiter

2009-04-07T14:38:00.000-07:00

I was talking with my coworker James Brechtel today about writing an extension method to join an array of strings together with a delimiter like string.Join() does. I'd prefer string[].Join() because it seems more fluent.

After I put the IList<T>.ToArray<T>() extension method together, implementing the fluent style join was a simple task.

Here's what that looks like:

public static string Join(this IList<string> list, string separator)
{
    return string.Join(separator, list.ToArray());
}

That's it! :)

ToArray Extension Method Converts IList to T[]

2009-04-07T14:17:00.000-07:00

I was talking with my coworker James Brechtel today about writing an extension method to join an array of strings together with a delimiter like string.Join() does. I'd prefer string[].Join() because it seems more fluent. I was throwing the IList<string>.Join() method together when I ran into a small snag.

I needed an array of strings for the string.Join() method. I put together a quick ToArray<T>() extension method which iterated through each item in the array and created an array T[]. The problem is, if I already have an array, that's a lot of extra work and some wasted memory.

So, I tried IList<T> is Array and it works great. I just cast the IList<T> as T[] and return it. Otherwise, I execute the iteration. Here's what it looks like:

public static T[] ToArray<T>(this IList<T> list)
{
    if (list is Array) return (T[]) list;

    T[] retval = new T[list.Count];
    for (int i = 0; i < retval.Length; i++) 
        retval[i] = list[i];

    return retval;
}

Pretty basic, but nice n' fast.

Why You'll Always Need a Programmer

2009-04-07T09:26:00.000-07:00

I've been writing software on various platforms for a little longer than a decade now and I have noticed a reoccurring trend. Customers are asking for and software vendors are trying to develop a way to allow business analysts to implement business logic in their applications.

The idea is that the business analyst knows the business inside and out. Therefore, if the business analyst had a way to implement the business logic, then there would be no need to have both the business analyst and the programmer (and the ensuing communication overhead).

I, however, am not concerned. In my experience, I have come to the conclusion that not only is the implementation of business a programming task, but it is a programming task that requires a strong development background.

What I mean is, you can write DSLs, query builders, and the like until you're blue in the face (presuming that you would eventually turn blue in the face writing DSLs and query builders), and the technically savvy business analyst will be able to learn to use these tools; however, without the foundation of good programming practices, this business analyst will produce vast amounts of technical debt which will, in the long run (if not the short run), cost the company much more than the programmer would have in the first place.

The Problem

Over the last several decades, programmers have developed practices and patterns that allow us to write applications that are easily upgradeable, scalable, maintainable, and even readable. We have built on this body of knowledge and we have tested it. Sure, there are some disagreements here and there, but for the most part, as we develop professionally we are forming a special way of looking at the world and solving problems which is relatively unique to programmers.

Similarly, business analysts have been doing the same thing. They look at the world and approach problems in their own special way. I believe they are two sides of the same coin. Both are responsible for understanding, documenting, and implementing business rules. One side communicates these rules to humans and the other to machines.

Enabling a business analyst to communicate directly to machines seems like a money saving flexibility that can't go wrong. It can go wrong — terribly wrong, and it can do so before you realize there's a problem. I believe that the business analyst would take as long to learn about the programming practices required to make solid architectural decisions as it would take the programmer to learn to be a business analyst.

If the business analyst is able to implement solutions, it is almost certain that there will be errors that a programmer would have caught. When the needs change, the analyst may even be able to address these changes, but this will lead not only to more errors, but also to an increased complexity that a programmer would have avoided.

Thus, the more work that goes into the solution over time, the more difficult it will be to maintain. Dealing with maintenance issues will lead to unintended consequences which will be corrected with more patches resulting in what programmers would liken to spaghetti code.

Eventually, the design entropy will lead to such a disaster that the vendor will need to be called in. Vendor engineers for large software companies can cost more than 300 dollars an hour.

Case Study - Business Process Automation

I have been working with a BPM engine for several years now. It's really great at what it does. It provides an asynchronous state engine, a visual workflow designer, and data persistence. The only real problem with this application is the way it is marketed to customers.

The company that produces the application claims that business processes can be automated without the need of a programming staff. They tout the visual designer as business analyst friendly. I agree that it is BA friendly . . . to read. Generating workflows is a vastly different story.

I have implemented this software for my own clients, for a very large government organization, and indeed for the software vendor itself. In every case, I have shown that even at a very low complexity, these implementations still required a knowledgeable programmer.

Case Study - Data Validation

I have a friend who works with a dynamic data collection application which implements data validation on the server side in SQL stored procedures and in javascript code snippets on the client side. The person who develops most of the data validation for one of his customers is an entry level programmer.

All of the validation works and everything is well and good until something changes. Even minor somethings can result in very intense development, debugging, and testing cycles. The people who developed these applications are very smart and clever people, but their lack of programming background has resulted in code which is almost impossible to maintain.

A more solid understanding of programming principles would have prevented probably 90% of the code repetition in these business rules. One small change wouldn't require a code change to every single business rule. Isolating bugs and errors could be accomplished through tests of small units of code rather than very large blocks of difficult to follow code.

The Solution

You will always need a programmer. If you are trying to develop a software based solution to save your company money, an architect with a solid programming foundation will produce a much more robust and more easily maintainable solution. The solution will last longer and will scale easier than anything your business analyst can come up with alone.

A good BA / software engineer team will produce benefits to your company that you, your BA, and your engineer could never have invented individually. The initial cost may be higher, but consider this.

Most of the time, your company will have to foot a hefty bill to train the BA. There will be evaluations, proof of concepts, and tests. More often than not, the determination will be that a BA just doesn't have the experience required. At that point, you will either have to hire a programmer and go through the same process or you will have to contract with the vendor or one of its partners.

The worse scenario is that your BA can implement a solution and that it works. When the application changes, you generate need for more application changes. These changes generate more changes. These changes compound and you're faced with a very costly overhaul of the entire system.

It is far better to start with a BA and a programmer. Send them both to the appropriate training. Have them both work together to produce adequate project documentation. Foster an environment where they are mutually productive. Not only will the whole be more than the sum of its parts, you'll save a great deal of expense and heartache.

More Yahoo Pipes

2009-04-06T06:26:00.001-07:00

Last week, I wrote a blog post about Yahoo Pipes. Shortly after I finished that post, I was doing some testing and my little brother Colin sent me an email complaint.

See, I am using my pipe to power an RSS feed in feed burner which I use both on my corporate profile and in my twitterfeed. I have facebook monitoring my tweets and updating my status.

The problem is, it looks pretty silly when I get a spam comment (or any comment really) and it pops up as my facebook status. So, when Colin sent me a message and said, "Pat, I have no idea what your status means or what that has to do with anything," I decided to update my pipe.

This is my new feed aggregation Yahoo pipe with context information.

Using Yahoo Pipes

2009-04-04T14:55:00.000-07:00

I was looking for a good way to combine a set of RSS feeds into one feed. I saw several web based services out there, but none was quite good enough for my needs. In my research, I happened across Yahoo Pipes. In about 3 minutes, I logged in, added 6 feeds, sorted them by publication date, and selected the top 25. Now, I have one RSS feed to aggregate everything including my two blogs (computing and aviation) and the comments from each one as well as my picasa albums and their comments.

The next thing you know, I have twitterfeed reading from that RSS feed and updating my twitter. The twitter facebook application then updates my statuses. Combining all of these services lets pretty much all of my friends and professional contacts know what I'm up to and what I'm publishing. Good stuff.

The only problem so far is that the twitter statuses look a bit confusing when I'm setting my facebook status to someone's comment on one of my blogs. I'm glad it's published, but I'm going to be looking into some of the transformations available in Yahoo Pipes to see if there's a way I can prefix posts to identify them as blog posts vs. comments.

PIN Entry Screens Considered Harmful

2009-04-04T14:14:00.000-07:00

I never wrote a "Considered Harmful" blog post before so I figured it was high time I did so. I often find myself at the grocery store, gas station, or fast-food restaurant keying my 4 digit PIN into some electronic entry system.

I look around and see someone watching me type in my code. Whether they're doing it intentionally or they're doing it incidentally, it still bothers me. I don't really want anyone seeing my PIN, particularly if that person wants to know it.

The other day, I was waiting behind some woman at the grocery store. In my frustration, I watched every move she made. She pulled her card out of her purse, handed it to the cashier, and started keying her PIN. From my angle, I couldn't see the actual numbers, but when she finished, I was completely shocked that I knew what her PIN was!

Evidently, without even noticing that I was doing it, I watched her key her PIN and I figured out what it was by the location of her presses alone. I had to be sure, so I asked, "ma'am, is your pin 4321?" Shocked and appalled she said, "why were you paying attention to that?" I replied, "it was an accident . . . I just happened to see where you were pressing."

That's when I had an idea. The digital keypads must be simply displaying the numbers on the screen. It seems that it would be completely reasonable to scramble the actual numbers so that nobody could decode your PIN simply by watching the location of your key presses.

More IList Extension Methods: Quickselect-ing from an Unsorted List

2009-03-16T07:38:00.000-07:00

Continuing my extension method kick and inspired by my extension method to select an item from an unsorted list that bets fits comparison criteria, I decided I wanted to try to implement an extension method to select from the list the kth best fit rather than just the best fit.

I left the best fit methods intact because they achieve worst case O(n) time, but the kth select worst case is O(n log n) because the worst case would quicksort the entire array. I later discovered that the method I implemented is called a quickselect. It's a divide an conquer algorithm based on the partitioning scheme of the quicksort.

Basically, with the quicksort, you pick an item (called the pivot) from the list and then put everything smaller than the pivot on the left and everything larger on the right. Then, you recursively quicksort each side of the pivot.

The quickselect uses the same technique, but rather than quicksorting both sides of the pivot, you quicksort only the side which contains the element you're looking for. Id est, if your pivot ends up at index 5 and you're looking for index 7, then you partition the right side. If the next pivot is index 8, then you partition the left side. You keep doing that until your pivot ends up being the index you're looking for.

Here's what the partition methods look like. There's really not a great reason to make them public, but there's also not a good reason to leave them private.

public static int Partition<T>
    (this IList<T> list, Comparison<T> comparison, int left, int right)
{
    int i = left;
    int j = right;

    // pick the pivot point and save it
    T pivot = list[left];

    // until the indices cross
    while (i < j)
    {
        // move the right pointer left until value < pivot
        while (comparison(list[j], pivot) > 0 && i < j) j--;

        // move the right value to the left position
        // increment left pointer
        if (i != j) list[i++] = list[j];

        // move the left pointer to the right until value > pivot
        while (comparison(list[i], pivot) < 0 && i < j) i++;

        // move the left value to the right position
        // decrement right pointer
        if (i != j) list[j--] = list[i];
    }

    // put the pivot holder in the left spot
    list[i] = pivot;

    // return pivot location
    return i;
}

public static int Partition<T>(this IList<T> list, Comparison<T> comparison)
{
    return list.Partition(comparison, 0, list.Count - 1);
}

public static int Partition<T>(this IList<T> list, IComparer<T> comparer)
{
    return list.Partition(new Comparison<T>((x, y) => comparer.Compare(x, y)));
}

public static int Partition<T>(this IList<T> list)
    where T : IComparable<T>
{
    return list.Partition(new Comparison<T>((x, y) => x.CompareTo(y)));
}

Isolating the partition logic makes it really easy to implement both the quicksort logic and the quickselect logic. In fact, in case you're interested, here's what the quicksort logic looks like:

public static void QuickSort<T>
    (this IList<T> list, Comparison<T> comparison, int left, int right)
{
    // pivot and get pivot location
    int pivot = list.Partition(comparison, left, right);

    // if the left index is less than the pivot, sort left side
    if (left < pivot) list.QuickSort(comparison, left, pivot - 1);

    // if right index is greated than pivot, sort right side
    if (right > pivot) list.QuickSort(comparison, pivot + 1, right);
}

The quickselect is basically the same thing, except that you don't quicksort both sides of pivot:

public static T QuickSelect<T>
    (this IList<T> list, int k, Comparison<T> comparison, int left, int right)
{
    // get pivot position
    int pivot = list.Partition(comparison, left, right);

    // if pivot is less that k, select from the right part
    if (pivot < k) return list.QuickSelect(k, comparison, pivot + 1, right);

    // if pivot is greater than k, select from the left side
    else if (pivot > k) return list.QuickSelect(k, comparison, left, pivot - 1);

    // if equal, return the value
    else return list[pivot];
}

public static T QuickSelect<T>(this IList<T> list, int k, Comparison<T> comparison)
{
    return list.QuickSelect(k, comparison, 0, list.Count - 1);
}

public static T QuickSelect<T>(this IList<T> list, int k, IComparer<T> comparer)
{
    return list.QuickSelect(k, new Comparison<T>((x, y) => comparer.Compare(x, y)));
}

public static T QuickSelect<T>(this IList<T> list, int k)
    where T : IComparable<T>
{
    return list.QuickSelect(k, new Comparison<T>((x, y) => x.CompareTo(y)));
}

Revised IList Extension Methods for Selecting Best Fitting Items from an Unsorted List

2009-03-15T17:33:00.000-07:00

Several days ago, I started writing articles about extension methods. One of the articles I wrote was about an IList extension method to select the best fitting item out of an unsorted list in O(n) (linear time). I was working on another article about implementing the quicksort and quickselect algorithms as extension methods.

I got pretty tired of having the same logic implemented in multiple places, but I hadn't yet figured out how to refactor it. Well, I did, so this is an update to those methods that keeps most of the work in one place.

I think it's a pretty cool implementation, it still works correctly, and it's still relatively fast. This implementation is also still side-effect free. That's one reason to use it over a sort/select method. If you need to leave your IList unsorted, then you can't sort and select. Also, if you are selecting from items with a custom IComparer, then even if you don't care about side-effects, you're still better off using this method.

On the other hand, if you don't care about the order of your list and your list type can be sorted with the native TrySZSort, then you're better off using Array.Sort() and selecting the last item in the list. However, if it can't be externed to TrySZSort then the .Net implementation of Array.Sort() uses quicksort which is O(n log n) and the selection is O(1). That's less efficient than the O(n) implementation below.

public static T GetBestFit<T>(this IList<T> list, Comparison<T> comparison)
{
    if (list == null || list.Count == 0)
        throw new ArgumentException("You cannot get the best fit from an empty list!");

    T currentFit = list[0];
    for (int i = 1; i < list.Count; i++)
        if (comparison(currentFit, list[i]) < 0)
            currentFit = list[i];

    return currentFit;
}

public static T GetBestFit<T>(this IList<T> list, IComparer<T> comparer)
{
    return list.GetBestFit(new Comparison<T>((x, y) => comparer.Compare(x, y)));
}

public static T GetBestFit<T>(this IList<T> list)
    where T : IComparable<T>
{
    return list.GetBestFit(new Comparison<T>((x, y) => x.CompareTo(y)));
}

public static T GetBestFit<T>(this IList<T> list, Func<T, T, bool> rule)
{
    Comparison<T> c = new Comparison<T>((x, y) => (rule(x, y)) ? 1 : -1);
    return list.GetBestFit(c);
}

Proxy Class for Making Any Object IComparable

2009-03-15T17:24:00.000-07:00

I've been continuing my thinking on the topic of extension methods and selecting items out of an unsorted list. I made some changes to the selection extension methods in order to make them more manageable.

It led me to another topic which is pretty much unrelated. I was thinking about the proxy pattern and trying to find another way to use it. Right now, I only use it in my TextWriter proxy but haven't used it anywhere else. But, as I was working on all of these extension methods, I realized how easy it was to sort and compare objects that were IComparable and it kept my method signatures nice and clean.

The problem is, not everybody uses IComparable and it's not always better to subclass the object just to get it. Instead, I wrote a proxy class that takes your object and gives you an IComparable. I don't really have any uses for it yet and it's not "cooked" enough to be production ready, but it does do the trick and it is an example of the proxy pattern at work.

public class ProxyIComparable<T> : IComparable<T>
{
    private T _object;

    public IComparer<T> Comparer { get; private set; }
    public Comparison<T> Comparison { get; private set; }

    public ProxyIComparable(T obj, Comparison<T> comparison)
    {
        _object = obj;
        Comparison = comparison;

        Comparer = new ProxyComparer(Comparison);
    }

    public ProxyIComparable(T obj, IComparer<T> comparer)
    {
        _object = obj;
        Comparer = comparer;

        Comparison = new Comparison<T>((x, y) => Comparer.Compare(x, y));
    }

    public int CompareTo(T other)
    {
        return Comparison(_object, other);
    }

    private class ProxyComparer : IComparer<T>
    {
        private Comparison<T> _comparison;

        public ProxyComparer(Comparison<T> comparison)
        {
            _comparison = comparison;
        }
        
        public int Compare(T x, T y)
        {
            return _comparison(x, y);
        }
    }
}