Size Matters: Improve Productivity with Cheap Screen Real Estate

It's hard to hire developers, and technical resources are expensive. Everyone is trying to figure out how to attract top talent, reduce turnover, and increase productivity of existing staff without reducing their work/life balance.

Early on, I decided that if you're going to spend 8 hours a day 5 days a week for almost 50 weeks a year doing something, everything about it should be as pleasant as it could be. I bought a new Macbook pro and two thunderbolt displays. Those investments very quickly paid off.

Until recently, the benefits of large technology investments were just intuition. Last week, I realized how much more productive I was working remotely than when I'm onsite with the client. One reason is that when I'm remote I get a lot more flow, but another reason is that when I'm remote, I have a lot more screen real estate so I can see a lot more information at once.

I decided it was time to be more productive onsite too. This time, I wanted to have more concrete justification.

Screen Selection

A friend of mine and I did a fair amount of research picking out monitors that gave us the best bang for our bucks. I'm really happy with my Thunderbolt Displays. They're IPS and they're quite high PPI. This time, I wanted to try something a little cheaper and a little larger and I wanted to try a VA panel so I ended up getting a BenQ BL3200PT. This monitor, I think, has the same panel as the Samsung S32D850T, which has a slightly smaller price tag.

Screen Real Estate

When I talk about screen real estate, I'm not talking about the diagonal size of the screen; I'm talking about the resolution of the screen. Having a larger screen doesn't do anything for you (at least not in the sense of a computer monitor) with regard to how much information you can fit on the screen at one time. The difference between an HD 1080 screen at 27" vs. 32" is just stretching. Resolution, however, is how many pixels the screen has. The more pixels there are, the more options the computer has to present data (provided you have the visual acuity to perceive the data without scaling).

These displays are 2560 x 1440 or WQHD and are the same resolution as my thunderbolt displays, albeit over a larger surface area. Thus the resolution is not quite as crisp as my thunderbolts, but still these are very high quality screens. If you're into lower pixel pitch and higher PPI, I don't blame you. Higher pixel density screens are much more pleasant to look at, but they do come at a cost.

Comparing Resolutions

There are two very common resolutions in offices these days. An older WSXVGA+ which is an 8:5 aspect display at 1680 x 1050 pixels. HD 1080 is also quiet popular which is the 16:9 display at 1920 x 1080. The HD 1080 has about 17.6% more pixels than the WSXVGA+. By comparison, the WQHD, also 16:9, has 2560 x 1440 pixels (sometimes called 1440P). It has about 77.8% more pixels than HD 1080 and about 110% more pixels than WSXGA+. That is to say that if you have 1 WQHD display, you have more pixels than a dual screen WSXGA+ setup.

Benefits of Screen Real Estate

The benefits of additional screen real estate, unfortunately, are easier to experience than they are to describe. Having additional real estate may mean that you will sense more data at the same time, but does not necessarily mean you will perceive these data. Similarly, having the ability see multiple windows does not mean that you won't just continue editing one file at a time in a full screen editor. On the other hand, not having the ability to optimize your workflow guarantees that you will not.

Anecdotes

I do have a few anecdotes that buttress my belief that having more screen real estate is inherently valuable. One time when I'm particularly keen to the benefits of higher resolutions is when I'm developing responsive designs. I'll often dock my browser's DOM inspector on the side and I'll slide the split pane to different responsive breakpoints watching the HTML/CSS changes flash in the inspector. The additional real estate means I can see meaningful data in the DOM inspector and the responsive design at the desktop breakpoint at the same time.

Another time I notice the benefits of having additional screen real estate is when working on native mobile applications. Native development often involves simulators and emulators. Scaling during emulation isn't always well supported. Thus, the additional pixels allows you to see the entire simulator at once.

Another example that I found particularly fascinating happened shortly after I took over a new project. I had been working on the project for a few weeks from the client site. One day, I was working remotely and I had my terminal window off to the left and I had the application in a browser to the right. This is a usual screen configuration for my workflow on my cinema displays. I had changed some code and the application reloaded. When it did, I noticed dozens of queries pop up in the log. None of the ORM queries had been optimized to prevent N + 1 queries. I spent a few hours fixing those, but may never have noticed them had I not had enough room on my screen to comfortably view the browser and the terminal at the same time.

Summary of Benefits

Typically on HD 1080 and especially on WSXGA+, I have only one window visible at a time. That's why I didn't notice the N + 1 queries until I was remote. There may be myriad other benefits of the additional screen real estate when considering looking at code in multiple panes, test driven development, debugging, guard and live reload, etc.

Costs

High quality screens can be expensive! The thunderbolt displays are almost $1,000 each. The BenQ was $550. The Samsung is $500. It's hard to imagine pulling out all of your existing displays and replacing them with displays that cost several hundred dollars. Heck, I spent hours justifying buying only one.

That's why it clicked. At some point I realized, it doesn't take much before the investment in display technology pays off. According to several job posting / salary research sites, the average developer in Atlanta makes about $120K per year. To keep the math easy, I'll call it $100K (also because I like lowballing estimates like this). Normally, overhead on a W-2 employee is roughly 25% - 50% of annual salary. If you have a very modest benefits package (and not a lot of PTO), then an average developer in Atlanta will cost about $125K per year.

Even if your organization buys these screens the same way I did (i.e., no sales tax and no bulk discount), then it'll cost about $550, or about 0.4% of the cost of an average developer. That means that if you bought 100 of them, it would cost less than half of the annual salary of your average developer for a year. Thus, if you have 100 developers and you get 100 screens and each developer is 0.5% more productive for the year, then you actually make money on the investment.

If the cost still seems too high, you don't have to get VA panels. In fact, 32 inch screens are pretty big (they take up a lot of desk real estate too). These 25" Dell U2515H IPS panels were only $350 at the time of writing. 100 of these would be less than 1/3 of the underestimated developer annual cost.

Recuperating Costs

The average American works 1,780 hours per year. To recuperate 0.28% of a developer's cost, the developer only needs to produce about 5 hours more per year. That could be working an additional 5 hours per year because work is more fun. It could be 5 hours saved by not having to tab between windows and the cost of brief context switches. It could be 5 hours not spent tracking down performance issues in production because some N + 1 queries sneaked through.

It could be 5 fewer hours spent recruiting great job candidates because the technology makes the company a better place to work. Developers with neat technology are more inclined to passively as well as actively recruit for you. This can bolster your recruiting efforts, also saving money. Similarly, developers with a satisfying work environment are less likely to leave their jobs reducing the high cost of turnover.

Studies on worker productivity show that American workers tend to have about 6 (and some studies show as few as 3) productive hours in an 8 hour workday. Better screens may improve ideal hours per day. That means if all of your developers show up Monday morning and find their brand new screens, and in their excitement they're productive for 8 hours on Monday and Tuesday (rather than 6) and 7 hours on Wednesday, then in the first 3 days, the screens have paid for themselves. I could be completely wrong about everything else and your net could still be more valuable somewhat liquid (they have good resale values) assets for free.

Remember, you only need 5 additional productive hours per year to cover the cost of 1 high resolution display … and that's if you buy new screens every year. If you plan on keeping the displays for 5 years, it's just 1 hour per year.

Incredulous?

Let's consider a more pessimistic case. Let's say your staff, on average is pretty junior (and you're using technologies that aren't particularly backend heavy. If that's the case, your developers may cost you an average of $80K salary and per year and $20K in benefits, taxes, recruiting efforts, etc. Let's say they also take no vacation or sick leave and thus work a full 2,080 hours per year. Further, your staff is productive 100% of working hours.

In this case, you're paying an average of $48 per productive work hour. If you buy very nice displays at say, $960 each, then you need 20 additional productive hours to cover the cost of one display. Even in this very unlikely case, that's less than 1% per developer. And they don't need to work 2,100 hours in a year per se. They can be 1% more efficient. That means if they're writing 100 lines of perfect code per day, then to cover the cost of a new display, they only need to write 100 lines of perfect code per day (and on the tenth day, perhaps they write 10 extra perfect line of code . . . or better yet, they delete a line or two).

More likely your average developer is closer to $125K per year, they work 1,780 hours per year, and they are productive fewer than 6 of every 8 hours. Thus, their cost is $94 per productive hour. Also, it's very reasonable to get smaller IPS WQHD panels for $350 each (especially if you have a large team and get a bulk or corporate discount). So, you need fewer than 4 hours per developer or 0.3% more productivity. An average developer contributes 10 lines per day to the codebase (Mythical Man Month) so to break even on your investment, your average developer only needs to net an additional 3 lines of code every quarter.

n.b., I don't support using lines of code as a measure of productivity. It's merely illustrative of the relative scale of the cost of a display to the cost of a developer. Try your own calculations using bugs sent to production, mean time to defect resolution, etc.

Recommendation

I recommend making an investment to replace low resolution displays for developers and pairing stations (and I would consider broadening the reach really to non-technical employees as well) with high resolution displays. I think that the cost of the hardware would be recovered very quickly by improved developer efficiency and performance. The investment is low risk because the cost is relatively low and the displays are a moderately liquid asset.

If you have 100 developers on your team, it may be hard to get approval for the capital to buy $35,000 worth of displays. In that case, you can always hire another developer (well, you can try at least) for $35,000 per quarter.

Display Options

Dell P2416D 24 Monitor with QHD 23.8-Inch Screen, Black
Dell UltraSharp U2515H 25-Inch Screen LED-Lit Monitor
Dell UltraSharp U2715H 27-Inch Screen LED-Lit Monitor
BenQ BL3200PT 32-inch WQHD 10-bit USBx4 HDMI Monitor built for Creative Class
Samsung 32" WQHD LED Monitor (S32D850T)

JavaScript for Photoshop - When Macros Aren't Enough

A few years ago, a fellow named Brian Dorn got in touch with me. He was working on his doctoral dissertation at Georgia Tech and he needed participants. I met him in Atlanta and found out that his study was about scripting with Photoshop. That was the first I'd ever heard of Adobe Photoshop Scripting.

A few weeks ago when I started working on a new word game This2That for Mobile Magic Developers, I needed to generate 38 tiles with letters, numbers, and punctuation. The task was tedious and the macros just couldn't make it any easier. Then I realized I had the embossing all wrong and I had to start over!

Frustrated, I tried to think of a better way. I remembered Brian Dorn and I started looking into this Adobe Photoshop JavaScript thing. I was very pleased when 15 minutes of script became an easily reusable tile generating utility. Now, I can take any PSD, open it up, select any text layer, and have the script generate a PNG for each letter I need

I thought this little known feature would make for interesting reading for both programmers and designers so here's my script:

// call the method that does all of the work
main();

// wrap the code in a method to make it easier to debug
function main() {

 // make sure you're working in a document and have a text layer selected
 if (!activeDocument || !activeDocument.activeLayer || activeDocument.activeLayer.kind != LayerKind.TEXT)
 {
   alert("Please select a document and a target text layer.");
   return;
 }
 
 // set up some information about the current file
 var textLayer = activeDocument.activeLayer;
 var path = activeDocument.path;
 var fileName = activeDocument.name;
 
 // remove the extension on the file name
 var extensionPosition;
 if (extensionPosition = fileName.lastIndexOf('.'))
  fileName = fileName.substr(0, extensionPosition);

 // get a good place to put the file
 var outputFolder = Folder.selectDialog("Select a target folder.", path);
 
 // set up the letters we want images for
 var characterMap = [
  ["question", "?"]
 ];
 
 // and add the lowercase alphabet and numbers
 characterMap = characterMap.concat(getAsciiRange(97, 26), getAsciiRange(48, 10));

 // for each character, update the selected text layer and save a file
 for (var i = 0; i < characterMap.length; i++)
 {
   var character = characterMap[i][1];
   var fileSuffix = characterMap[i][0];
   
   textLayer.textItem.contents = character;
   
   var file = new File(outputFolder + "/" + fileName + "_" + fileSuffix + ".png");
   var options = new PNGSaveOptions();
   options.interlaced = false;

   activeDocument.saveAs(file, options, true, Extension.LOWERCASE);
 }
};

// a little helper method to make a range of letters and their
// filename extensions
function getAsciiRange(from, count) {
 var result = [];

 for (var i = 0; i < count; i++)
 {
  var character = String.fromCharCode(i + from);
  result.push([character, character.toUpperCase()]);
 } 
 
 return result;
}
 

Mainframe Migrations: Source Code Translation Tools

I was working on a mainframe migration project a while back to move an application from Adabas and Natural to Sql Server and C#. Software AG developed Adabas and Natural back in the 70's so I was very pleased to have some Software AG developers and managers on the team.

Shortly into the project, one of the managers recommended a product Software AG has that translates Natural code into C#. Much to the surprise of my teammate, I recommended strongly against the software. In fact, if you've read any of my other mainframe migration articles, you know I was recommending against a "migration" strategy altogether (a lift-and-shift my teammate called it) vs. a guided redevelopment strategy.

In the process of making my argument, I spent a fair amount of time positing that a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap. The customer did decide not to use the translation tool, but in the process of making my argument, my curiosity was piqued (and I do have something of a strong research background) so I put together the What Really Makes a Good Programmer Survey.
For an in depth discussion of the results, see my Results of the "What Really Makes a Good Programmer" Survey post.

For this post, we'll be looking primarily at these data:
If you can't view this chart, I managed to keep a copy of the What Really Makes a Good Programmer Chart from the legacy charts API.

Because most of my respondents were programmers, I like to look at the group averages to eliminate selection bias. I group the "traits" into the following three categories:

• Traits that contribute to being a good programmer
  1. Has good problem solving skills
  2. Learns from experience
  3. Interested in learning
  4. Passionate
  5. Sees the big picture
  6. Recognizes patterns
  7. Communicates effectively
  8. Tries new approaches
  9. Detail oriented
  10. Seeks help when needed
  11. Interested in helping others
• Traits that are nice to have if you can get them
  1. Fast
  2. Co-located
• Traits with no effect on programmer quality
  1. Has a college degree
  2. Has a computer science degree
  3. Cheap
  4. Has certifications

So, back to my original stipulation that "a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap." While I still believe that statement to be true, when I conducted my survey, I asked the question in a more positive way so I can't actually claim that the tool is a bad programmer. I believe that it is analogous to a bad programmer; I just can't support that claim with these data. I'll explain why, but first, for the pedants, statements like "the community represented in my responses appears to believe that having good problem solving skills contributes to being a good programmer" will be abbreviated to "having good problem solving skills contributes to being a good programmer."

That being said, the data suggest that having good problem solving skills contributes to being a good programmer. If that stipulation is true, then the contrapositive must also be true that not being a good programmer means not having good problem solving skills. The inverse, however, is not necessarily true so I cannot claim that not having good problem solving skills contributes to not being a good programmer. I'll let you decide on the validity of that statement, but for the purposes of this post, I'll change my argument to, "using a source code translation tool is little more than hiring a programmer, who lacks the traits of a good programmer, simply because he's fast and cheap."

Think of a translation tool and go through the list of traits. The translation tool does not have problem solving skills at all, can't learn from experience and certainly isn't interested in doing so, has no passion, and doesn't even know what a big picture is let alone the big picture with your application in your environment. It may be able to recognize patterns technically, but it likely won't recognize patterns that are meaningful to your application. It's also unlikely that the tool will communicate effectively, create and try new approaches, or be interested in helping others.

I suppose you could say it's detail oriented (as long as the details are explicit and don't require problem solving) and it seeks help when needed (likely by way of errors). It's probably faster than a good programmer, though this may be negated if it's lack of the aforementioned traits produces less than useful results; that is to say as long as you don't need a programmer to fix the translation once it's finished the speed may be beneficial. Another good thing about the tool is that it's possibly cheaper than a good programmer (see previous caveat) and it's arguably well educated, but that doesn't seem to constitute a good programmer according to the survey.

Software development is an art almost as much as it is an engineering field. You hear many people talk of software craftsmanship. Allowing a craftsman to rewrite your application in the new framework from scratch using the old application as a guide will allow her to provide a unique expertise that translation tools currently don't have. She'll be able to apply a high level human analysis that the tool cannot.

For example, imagine there's a bug in the old code? The programmer can spot and fix the bug, but the tool will translate it into a prettier and newer bug. What if the old code is doing something unique to that language because the language doesn't have the features of the current language. A programmer can take advantage of these features and the tool cannot. A programmer will naturally analyze your business process as the application comes together and will make recommendations that may make your new application even better than the old one was in its heyday; the tool will not.

I can think of dozens of examples like this and I'm sure you can as well. Ultimately, I believe that allowing a tool to convert your legacy code into newer legacy code is a waste of time, money, and opportunity and if it doesn't cost more in the short run, it will cost more in the long run.

Switching on Enums: A Style Observation

I was digging through the framework today looking for another good dotPeek of the Week topic. I was perusing the Lazy<T> class and found an interesting snippet.

This post isn't quite like my previous dotPeek of the Weeks insofar as this is more of an example of what not to do. This is certainly merely my opinion, but one rule I try to follow when writing code is that more expressive is almost always better than less expressive. When I was looking at the Lazy class, I found a great example of this.

Here's the code (abridged for clarity … and also because the threading in this class will make for better discussion later):

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    default:
      // lock and set values
      break;
  }
}

Is there anything you notice about this code? Perhaps any unanswered questions as you read it and try to figure out what it does? Specifically, what exactly constitutes the default case?

As I read through this code, learning about some of the interesting thread safety techniques, I found myself pondering, "why would locking be the default behavior? In fact, what is the default case? Do the default values have something in common?"

I looked at the enum LazyThreadSafetyMode and found this:

public enum LazyThreadSafetyMode
{
  None,
  PublicationOnly,
  ExecutionAndPublication,
}

That's when I decided that, in most cases, when you're switching on a reasonably small set of values, it's best to express these values explicitly so that the people (including you) who have to maintain the code can better understand why the default case is the default case … even if there are no comments.

For example, the following code is functionally equivalent:

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    case LazyThreadSafetyMode.ExecutionAndPublication:
      // lock and set values
      break;
  }
}

Personally, I find the latter example much more expressive. It's obvious to me what the three cases are and I don't have to wonder what possible values can become the default case. In fact, I may even go so far as having the default case throw an exception in this class. I'd do this so that, for whatever reason, if someone were to change the LazyThreadSafetyMode enum and not implement that case for the new values in Lazy<T>.LazyInitValue(), they'd get an exception in testing instead of incorrectly using the default functionality.

7-bit Encoding with BinaryWriter in .Net

At work this week, I had the need to serialize objects and encrypt them while trying to keep the smallest data footprint I could. I figured the easiest thing to do would be to binary serialize them with the BinaryFormatter. It was, indeed, the easiest thing to do; however, the BinaryFormatter seems to come with a fair amount of overhead. By the time the BinaryFormatter was finished listing all of the necessary assembly-qualified type names, the 60 bytes of data I wanted to preserve were more than a kilobyte!

I needed another way so I extended BinaryWriter (mostly to get it to serialize the types I needed it to) and now my 60 bytes take 68 bytes to serialize. In the process of writing this class, I looked through the disassembled BinaryWriter using JetBrains's dotPeek and found this little gem (Write7BitEncodedInt(int)) and decided it'd make a great dotPeek of the Week:

protected void Write7BitEncodedInt(int value)
{
  uint num = (uint) value;

  while (num >= 128U)
  {
    this.Write((byte) (num | 128U));
    num >>= 7;
  }

  this.Write((byte) num);
}

This is how the BinaryWriter.Write(string) encodes the length of the string. Thus, if you have a string with fewer than 128 characters, it only takes one byte to encode the length. In my implementation, I used this to specify the length of collections too. In fact, when writing a 32 bit signed integer, you should be able to save space for all positive numbers less than 2 ^ 22 and break even (require four bytes) for 2 ^ 29. Encoding an Int32 that is greater than or equal to 2 ^29 will require five bytes. Thus, if your integers tend to be smaller than 536,870,912, you'll probably save space encoding this way. A similar function could be used for a long where all positive values less than 2 ^ 57 will result in at least breaking even.

Here's how it works:

1. Convert the number to an unsigned int so you can do arithmetic on positive numbers (after all, you're just writing bits)
2. While the converted value is greater than or equal to 128 (i.e., 8 bits)
   1. Write low 7 bits and put a 1 in the high bit (to indicate to the decoder more bytes are coming)
   2. Shift the 7 bits you just wrote off the number
3. When the loop finishes, there will be 7 or fewer bits to write so write them

I think this is really clever and very easy. Reading the data back is a smidgen more complicated (Read7BitEncodedInt()):

protected internal int Read7BitEncodedInt()
{
  // some names have been changed to protect the readability
  int returnValue = 0;
  int bitIndex = 0;

  while (bitIndex != 35)
  {
    byte currentByte = this.ReadByte();
    returnValue |= ((int) currentByte & (int) sbyte.MaxValue) << bitIndex;
    bitIndex += 7;

    if (((int) currentByte & 128) == 0)
      return returnValue;
  }

  throw new FormatException(Environment.GetResourceString("Format_Bad7BitInt32"));
}

Here's how this works:

1. Set up an int to accumulate your data as you read them
2. Set up a place to keep track of which 7-bit block you're reading
3. While your bit index is less than 35 (i.e., you've read no more than 5 bytes comprising 1 "more bytes" indicator and 7 data bits)
   1. Read a byte
   2. Take the byte you just read and logical conjuction (bitwise and) it with sbyte.MaxValue (127 or in binary 0111 1111)
   3. Left shift those 7 bits to the position in which they belong
   4. Use a logical disjunction (bitwise or) to write those seven bits into your accumulator int
   5. Add 7 to your bit index for the next 7 bits you read
   6. If the current byte you read does not have a 1 in its most significant bit (i.e, byte & 1000 0000 == 0)
      1. There are no more bytes to read so just return the current accumulator value
4. If you get to this point, you've read a 6th of 5 bytes and it's time to let the user know there was a formatting problem

.Net GetHashcode Functions

Last week, I wrote a post about a .Net method to combine hashcodes. In the process of designing that method, I looked into dozens of .Net's GetHashcode implementations.

If you'd like to know the principles of hashcodes, take a look at the aforementioned article. This one is part of the dotPeek of the Week series so I'll just be sharing the insight I got from the framework's implementations here.

First, some really basic ones:

// from Int32
public override int GetHashCode()
{
  return this;
}

// from Int16
public override int GetHashCode()
{
  return (int) (ushort) this | (int) this << 16;
}

// from Int64
public override int GetHashCode()
{
  return (int) this ^ (int) (this >> 32);
}

The Int32 implementation easily meets the hashcode requirements by returning itself. The Int16 copies itself to the top half of the Int32 and returns that. The Int64 takes the bottom half of itself and XORs it with the top half.

This XOR is the first valuable piece of information. If you look at the logical functions, the XOR produces the best bit twiddling for hashing. Basically, the XOR produces a relatively even distribution of bits as opposed to the OR and the AND operations which will be biased 3 to 1 in one direction or the other.

Some more complicated implementations:

// from String
public override unsafe int GetHashCode()
{
  fixed (char* chPtr = this)
  {
    int num1 = 352654597;
    int num2 = num1;
    int* numPtr = (int*) chPtr;
    int length = this.Length;
    while (length > 0)
    {
      num1 = (num1 << 5) + num1 + (num1 >> 27) ^ *numPtr;
      if (length > 2)
      {
        num2 = (num2 << 5) + num2 + (num2 >> 27) ^ numPtr[1];
        numPtr += 2;
        length -= 4;
      }
      else
        break;
    }
    return num1 + num2 * 1566083941;
  }
}

// from Tuple<>
internal static int CombineHashCodes(int h1, int h2)
{
  return (h1 << 5) + h1 ^ h2;
}

These two methods have some really good information in them. The String.GetHashcode implementation has what's called a rolling hash. It loops through the characters, does a barrel shift by 5, and then XORs with the next character.

While I liked this, I preferred the simplicity of the Bernstein Hash. The main component of the Bernstein Hash is the (i << 5) + i. i << 5 == i * 32 (except that bit shifting is much faster than multiplying).

Thus, i << 5 + i == 32i + i == 33i. The Bernstein Hash just takes the current hash value, multiplies it by 33, and XORs in the new value.

I didn't use 33 in my hash function because I feel like using prime numbers is healthy. Thus, instead of adding I subtract so my hashcode method is (hash << 5) - hash ^ value (or 31 * hash ^ value). This, by the way, is the way Java tends to do it.

Here are some interesting ones from System.Drawing:

// from Size
public override int GetHashCode()
{
  return this.width ^ this.height;
}

// from Rectangle
public override int GetHashCode()
{
  return this.X ^ (this.Y << 13 | (int) ((uint) this.Y >> 19)) ^ (this.Width << 26 | (int) ((uint) this.Width >> 6)) ^ (this.Height << 7 | (int) ((uint) this.Height >> 25));
}

// from Point
public override int GetHashCode()
{
  return this.x ^ this.y;
}

.Net Method to Combine Hash Codes

A while back I developed a helper method that I've been using to aid me in computing good hash values from sets of properties for an object. I was writing a dotPeek of the Week entry about .Net GetHashcode implementations and I decided to spruce up my implementation and post it on my blog.

I know It's pretty uncommon that you need to override the Equals method, but when you do, you have to take particular care in considering overriding the GetHashCode method as well. To make this a little easier, I developed a method I call CombineHashCodes().

Using CombineHashCodes, I can easily compute a well randomized composite hash code based on several parameters from an object. This curtails a lot of the work involved when dealing with objects that have complicated .Equals overrides.

Microsoft's documentation on Object.GetHashCode() lists the following rules:

A hash function must have the following properties:

If two objects compare as equal, the GetHashCode method for each object must return the same value. However, if two objects do not compare as equal, the GetHashCode methods for the two object do not have to return different values.

The GetHashCode method for an object must consistently return the same hash code as long as there is no modification to the object state that determines the return value of the object's Equals method. Note that this is true only for the current execution of an application, and that a different hash code can be returned if the application is run again.

For the best performance, a hash function must generate a random distribution for all input.

Thus, if you override the .Equals method such that it's no longer doing the default reference equality, you probably need to change your GetHashCode implementation. I've found this is usually a very simple task. I whipped up a quick example in LINQPad:

void Main()
{
 var lauren1 = new Person { FirstName = "Lauren" };
 var lauren2 = new Person { FirstName = "Lauren" };
 
 Console.WriteLine(lauren1.Equals(lauren2));
 Console.WriteLine("{0}.GetHashCode() = {1}", "lauren1", lauren1.GetHashCode());
 Console.WriteLine("{0}.GetHashCode() = {1}", "lauren2", lauren2.GetHashCode());
 
 if (lauren1.Equals(lauren2) && lauren1.GetHashCode() != lauren2.GetHashCode())
  Console.WriteLine("This is bad.  This is very bad.");
}

public class Person
{
   public string FirstName { get; set; }
}

/*
False
lauren1.GetHashCode() = 47858386
lauren2.GetHashCode() = 20006478
*/

As one would expect, lauren1 does not equal lauren2 and neither do the hashcodes; however, if we were to override the .Equals method on the Person object to compare just the first name, we'll find that lauren1 and lauren2 will be equal but will have different hashcodes! This violates the first rule of GetHashCode Club. Here's an example:

public class Person
{
   public string FirstName { get; set; }
  
   public override bool Equals (object obj)
   {
  var otherPerson = obj as Person;
  
  if (otherPerson == null)
   return false;
   
  return String.Compare(this.FirstName, otherPerson.FirstName, StringComparison.OrdinalIgnoreCase) == 0;
 }
}

/*
True
lauren1.GetHashCode() = 33193253
lauren2.GetHashCode() = 37386806
This is bad.  This is very bad.
*/

That's why we override GetHashCode as well and something like this will work:

public override int GetHashCode()
{
 return FirstName.GetHashCode();
}

/*
True
lauren1.GetHashCode() = 2962686
lauren2.GetHashCode() = 2962686
*/

That's all well and good (except the .Equals() does a case insensitive compare so "Lauren" and "lauren" will be equal but will result in different hash codes unless you do something like return FirstName.ToLower().GetHashCode(). Why'd I leave that error in the post just to parenthetically correct it . . . because I think it's a good demonstration of how easy it is to screw this up :)). Consider what happens when you end up with a more complex class that looks like this:

public class Person
{
   public string FirstName { get; set; }
 public string LastName { get; set; }
 public DateTime BirthDate { get; set; }
  
   public override bool Equals (object obj)
   {
  var otherPerson = obj as Person;
  
  if (otherPerson == null)
   return false;
   
  return String.Compare(FirstName, otherPerson.FirstName, StringComparison.OrdinalIgnoreCase) == 0
   && String.Compare(LastName, otherPerson.LastName, StringComparison.OrdinalIgnoreCase) == 0
   && BirthDate.Equals(otherPerson.BirthDate);
 }
}

You could leave the GetHashCode function the way it was previously, but then all Lauren's will be lumped in the same bucket together which violates the third rule about a random distribution. I've seen people combat this problem by concatenating the string values of the equals fields delimited by some extremely unlikely string and getting the hash code of that.

I felt like that was not really the best way to do it in case someone decides a colon is a good character in a first name or something worse.

Thus, I wrote something like this (which has recently been modified to what you see now based on my explorations for my dotPeek of the Week series). This is CombineHashCodes:

public static class HashCodeHelper
{
 public static int CombineHashCodes(params object[] args)
 {
  return CombineHashCodes(EqualityComparer<object>.Default, args);
 }

 public static int CombineHashCodes(IEqualityComparer comparer, params object[] args)
 {
  if (args == null) throw new ArgumentNullException("args");
  if (args.Length == 0) throw new ArgumentException("args");

  int hashcode = 0;

  unchecked
  {
   foreach (var arg in args)
    hashcode = (hashcode << 5) - hashcode ^ comparer.GetHashCode(arg);
  }

  return hashcode;
 }
}

With this method, you can get a nice pseudo-random distribution of hash codes without violating any of the GetHashCode rules as long as you include the values relevant to your Equals method in your call to CombineHashCodes:

public override int GetHashCode()
{
 return HashCodeHelper.CombineHashCodes(FirstName.ToLower(), LastName.ToLower(), BirthDate);
}

/*
True
lauren1.GetHashCode() = -891990792
lauren2.GetHashCode() = -891990792
*/

.Net Optimization for Int32

Another entry in the dotPeek of the Week series here. I was digging through some .GetHashcode() implementations (expect that as the next dotPeek of the week) and noticed some interesting implementations of overridden .Equals() methods.

I found this in Int16:

public override bool Equals(object obj)
{
  if (!(obj is short))
    return false;
  else
    return (int) this == (int) (short) obj;
}

public bool Equals(short obj)
{
  return (int) this == (int) obj;
}

So, I checked Byte and sure enough:

public override bool Equals(object obj)
{
  if (!(obj is byte))
    return false;
  else
    return (int) this == (int) (byte) obj;
}

public bool Equals(byte obj)
{
  return (int) this == (int) obj;
}

Can you guess how Char.Equals() is implemented?

public override bool Equals(object obj)
{
  if (!(obj is char))
    return false;
  else
    return (int) this == (int) (char) obj;
}

public bool Equals(char obj)
{
  return (int) this == (int) obj;
}

Even the unsigned int gets cast as a signed int in UInt.Equals(). I was pretty curious, so I looked at some other methods:

public int CompareTo(short value)
{
  return (int) this - (int) value;
}

public int CompareTo(byte value)
{
  return (int) this - (int) value;
}

public int CompareTo(char value)
{
  return (int) this - (int) value;
}

I took note and discussed it with a few friends. It seems to make sense that on a 32 bit operating system, it'd be optimal to work with 32 bit integers. It turns out, even on a 64 bit architecture, .net is optimized for the Int32. I found this gem online:

Best Practices: Optimizing performance with built-in types

The runtime optimizes the performance of 32-bit integer types (Int32 and UInt32), so use those types for counters and other frequently accessed integral variables.

For floating-point operations, Double is the most efficient type because those operations are optimized by hardware.

MCTS Self-Paced Training Kit (Exam 70-536): Microsoft® .NET Framework 2.0—Application Development Foundation

So storing integer values, no matter the ceiling of your expected value, is best done using Int32, particularly if they'll be operated on heavily like an index to a collection or as a counter in an iteration control structure.

Enumerable.Any vs. Enumerable.Count

This is the innagural entry into the section of my blog I'm calling "dotPeek of the Week." Workload permitting, it will be a weekly thing where I'll be using JetBrain's dotPeek - a free .NET decompiler to learn more about the .NET framework and share any interesting things I come up with.

In this post, I'll be sharing a little tip I picked up from my friend David Govek.

I can't .Count() the number of times I've seen a line of code like this one:

if (someEnumerable.Count() > 0) 
    doSomething();

I know I've done that myself a handful of times. I think it comes from the pre-.net 3.5 days when you were used to dealing with ICollections that had a .Count property that returned the value of a private field:

public virtual int Count { get { return this._size; } }

When 3.5 came out, the System.Linq.Enumerable class brought with it a Count extension method. I believe it was at that point that people started using .Count() everywhere, including on IEnumerables, which previously didn't have a method for getting the length of the enumerable.

Most of the time, there was very little pain because the engineers over at Microsoft were clever enough to help us out. The first thing they try to do in the .Count extension method is check to see if the IEnumerable is an ICollection. If it is, they just use the Count property which we already know is plenty fast; however, if it's not an ICollection, they have to iterate the Enumerable and count the elements.

Here's what that looks like:

public static int Count<TSource>(this IEnumerable<TSource> source)
{
  if (source == null)
    throw Error.ArgumentNull("source");

  ICollection<TSource> collection1 = source as ICollection<TSource>;
  if (collection1 != null)
    return collection1.Count;

  ICollection collection2 = source as ICollection;
  if (collection2 != null)
    return collection2.Count;

  int num = 0;
  using (IEnumerator enumerator = source.GetEnumerator())
  {
    while (enumerator.MoveNext())
      checked { ++num; }
  }
  return num;
}

If your source is indeed an Enumerable, this is still a fine way to find out how many items there are. The problem is, in the sample code where we're just simply checking to ensure that the Enumerable isn't empty, using .Count can prove costly. Checking that the count is greater than 0 has to enumerate the entire collection and count each item despite the fact that we know it's not empty as soon as we spot the first element.

Fortunately, the clever folks at Microsoft thought of this and also gave us .Any(). This extension method simply gets the Enumerator, calls .MoveNext(), and disposes the Enumerator. MoveNext tries to move to the next element in the collection and returns true until it passes the end of the collection.

Here's what the .Any() method looks like:

public static bool Any(this IEnumerable<TSource> source)
{
  if (source == null)
    throw Error.ArgumentNull("source");

  using (IEnumerator<TSource> enumerator = source.GetEnumerator())
  {
    if (enumerator.MoveNext())
      return true;
  }

  return false;
}

Thus, there's no need to Enumerate the entire Enumerable if you can use .Any() like this refactored code:

if (someEnumerable.Any()) 
    doSomething();

About dotPeek of the Week

Yesterday, I posted the first post of a series I call the dotPeek of the Week. Why am I using a JetBrains product name in my blog? Well, 'cause I like JetBrains. I love ReSharper for C#, IntelliJ for Android development, and of course, dotPeek for decompiling .Net code.

I also want to support JetBrains by encouraging people to use dotPeek so that they'll keep it free, unlike that other one that got bought by that one company and now is a paid app. Hopefully, I won't have to change the name of this section any time soon :).

The other (and primary) objective of this exercise is to learn more about the .Net framework and how it functions. If I find something interesting, I'll share it here on my blog.

Latest dotPeek of the Weeks

• All dotPeek of the Week posts

C# Yield Keyword, IEnumerable<T>, and Infinite Enumerations

Visual Studio 2005 came with a slew of .net and .net compiler features. One of those features that I particularly enjoy is the yield keyword. It was Microsoft's way of building IEnumerable (or IEnumerator) classes and generic classes around your iterator code block. I'm not going to discuss the yield keyword much in this post because the feature has been around a long time now and the internet is replete with discussions on the topic.

Despite the long life of the yield keyword, I still find it conspicuous when I see it in projects I work on. I suppose it's just rare that I find myself writing my own enumerable. As a result, when I see yield return, it tends to stand out. I started looking around to see how the rest of the programming world uses the yield keyword wondering if I was under-utilizing the flexibility provided.

Specifically, I wondered if I was missing out on the lazy nature of the iterator and the numerous linq extension methods optimized to take advantage of that aspect of iterators. What I mean by that is that an iterator doesn't need to store each sequential value in memory the way a collection would and thus you can use each value without necessarily increasing the memory overhead. Further, you can take advantage of calculations which tend to already be sequential in nature (like the Fibbonacci sequence for example).

The second thing I thought about was an infinite (well, sort of infinite) enumerable. I'm not sure if I feel like it's a bad idea or not so I wrote these examples to be unending? I may eventually find a use for such an iterator and then get burned when someone tries to call Fibbonacci.Min() and the application throws an overflow exception, but I suppose at that point I'll make it a method and take a sanity check variable.

In the meantime, here are a few examples of some iterators I thought were interesting and fun challenges:

static IEnumerable<ulong> Fibbonacci
{
    get
    {
        yield return 0;
        yield return 1;

        ulong previous = 0, current = 1;
        while (true)
        {
            ulong swap = checked(previous + current);
            previous = current;
            current = swap;
            yield return current;
        }
    }
}

static IEnumerable<long> EnumerateGeometricSeries(long @base)
{
    yield return 1;

    long accumulator = 1;
    while (true)
        yield return accumulator = checked(accumulator * @base);
}

static IEnumerable<ulong> PrimeNumbers
{
    get
    {
        var prime = 0UL;
        while (true)
            yield return prime = prime.GetNextPrime();
    }
}

static IEnumerable<List<uint>> PascalsTriangle
{
    get
    {
        var row = new List<uint> { 1 };
        yield return row;

        while (true)
        {
            var last = row[0];
            for (var i = 1; i < row.Count; i++)
            {
                var current = row[i];
                row[i] = current + last;
                last = current;
            }

            row.Add(1);
            yield return row;
        }
    }
}

Some of these methods use a few extension methods I adapted from places in the .net framework:

public static ulong GetNextPrime(this ulong from)
{
    for (var j = from + 1 | 1UL; j < ulong.MaxValue; j += 2)
        if (j.IsPrime())
            return j;

    return from;
}

public static bool IsPrime(this ulong value)
{
    if ((value & 1) != 0)
    {
        var squareRoot = (ulong)Math.Sqrt((double)value);
        for (ulong i = 3; i <= squareRoot; i += 2)
            if (value % i == 0)
                return false;

        return true;
    }
    return value == 2;
}

To keep the test console app clean, of course, I used my favorite IEnumerable.Each() extension method:

public static void Each<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    foreach (var element in enumerable)
        action(element);
}

Here's the sample code:

static void Main()
{
    Fibbonacci.Skip(10).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    EnumerateGeometricSeries(2).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    PrimeNumbers.Where(p => p > 600).Take(10).Each(Console.WriteLine);
    Console.WriteLine();

    foreach (var row in PascalsTriangle.Take(10))
    {
        row.Each(element => Console.Write("{0} ", element)); 
        Console.WriteLine();
    }

    Console.ReadLine();
}

// The second set of 10 elements in the Fibbonacci sequence
// 55 89 144 233 377 610 987 1597 2584 4181

// Base of 2 to the first 10 powers
// 1 2 4 8 16 32 64 128 256 512

// The first 10 prime numbers greater than 600
// 601 607 613 617 619 631 641 643 647 653

// The first 10 rows of Pascal's Triangle
// 1
// 1 1
// 1 2 1
// 1 3 3 1
// 1 4 6 4 1
// 1 5 10 10 5 1
// 1 6 15 20 15 6 1
// 1 7 21 35 35 21 7 1
// 1 8 28 56 70 56 28 8 1
// 1 9 36 84 126 126 84 36 9 1

.Net ObjectFormatter - Using Tokens in a Format String

If you've already read this article and you don't feel like scrolling through my sample formats, you can jump directly to ObjectFormatter on github to get the source.

At some point in almost every business application, it seems you eventually run into the ubiquitous email notifications requirement. Suddenly, in the middle of what was once a pleasant and enjoyable project come the dozens of email templates with «guillemets» marking the myriad fields which will need replacing with data values.

You concoct some handy way of storing these templates in a database, on the file system, or in resource files. You compose your many String.Format() statements with the dozens of variables required to format the email templates and you move on to the greener pastures of application development.

Now, you've got a dozen email templates like this one:

Dear {0},

{1} has created a {2} task for your approval. This task must be reviewed between {3} and {4} to be considered for final approval.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

No big deal, everything is going swimmingly, and the application goes into beta. Then, it turns out, the stakeholders don't want an email template that looks like that. That was more of a draft really. Besides, you should've already known what they wanted in the template to begin with. After all, it's like you have ESPN or something.

It's important to add information about the user for whom this action is taking place, so this is your new template:

Dear {0},

{1} has created a {2} task for your approval regarding {5}({6}). This task must be reviewed between {3} and {4} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {7}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

So far so good. You've updated the template, updated your String.Format() parameters, passed QA and gone into production. But, now that users are actually hitting the system, it turns out that you need a few more changes. Specifically, you need to add contact information for the supervisor, remove the originator of the task, and by the way, what kind of sense does it make to put a low end limit on a deadline? Here's your new template:

Dear {0},

A {2} task for {5}({6}) is awaiting your approval. This task must be reviewed by {4} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {7} at {1}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

Now you have an email template format with various numbers all over the place, a String.Format() call with more parameters than there are tokens, and you have to go through the QA - deployment cycle again.

I've gone through this process on almost every application throughout my career as a software engineer. Hence the ObjectFormatter. Now, my email template looks like this:

Dear {Employee.FullName},

A {Task.Description} task for {TargetUser.FullName}({TargetUser.UserId}) is awaiting your approval. This task must be reviewed by {DueDate} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {Supervisor.FullName} at {Supervisor.PhoneNumber}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

I find that the ObjectFormatter makes my templating much easier to maintain and much more flexible. It also usually makes my calling code a lot cleaner. Here's an example of the approaches you could take to populate the sample templates:

// plain string formatting
String.Format(template, Employee.FullName, Supervisor.PhoneNumber, Task.Description, String.Empty, DueDate, TargetUser.FullName, TargetUser.UserId);

// if you have a dto already built
ObjectFormatter.Format(template, myDto);

// if you don't have a dto built
ObjectFormatter.Format(template, new { Employee, Supervisor, Task, DueDate, TargetUser });

I've found that most of the time they ask for template changes, they want me to add some value that is already a property on an object that's already in my object graph because of the current email template. That way, when they come tell me they want he target user's name formatted differently, I don't even need to recompile (well, sometimes I do . . . I mean, I can't predict everything). I can implement a lot of changes by using objects I already know I'm passing into the ObjectFormatter.Format() method. Here's the new template with the changes and I didn't have to change a line of code to make it work:

Dear {Employee.FullName},

A {Task.Description} task for {TargetUser.LastName}, {TargetUser.FirstName}({TargetUser.UserId}) is awaiting your approval. This task must be reviewed by {DueDate} to be considered for final approval.

If you have questions, please contact your approvals management supervisor {Supervisor.FullName} at {Supervisor.PhoneNumber}.

This is an automagically generated email sent from an unmonitored email address. Please do not reply to this message. No, seriously . . . stop that. Nobody is going to read what you are typing right now. Don't you dare touch that send button. Stop right now. I hate you. I wish I could hate you to death.

Thank you,
The Task Approval Team

If you'd like to check out the source or use the ObjectFormatter in your own projects, look for ObjectFormatter on github. If you make any cool changes, please let me know and I'll try to figure out how to merge them into the repository.

Extension Method to Replace foreach With Lambda Expression

It's pretty often I find myself looping through some enumerable and performing an action on the elements. Sometimes it's just displaying results with Console.WriteLine. Other times I need to do something a little more complicated. In any case, every once in a while, I feel like the foreach statement and the for statement aren't really quite expressive enough.

That's why I have this little guy:

public static void Each<T>(this IEnumerable<T> enumerable, Action<T> action)
{
    foreach (var element in enumerable)
        action(element);
}

It's pretty basic but I like the way it looks and feels. I used it in my blog post about a C# UpTo Extension Method a la Ruby's int.upto method.

Here's a simple demonstration I wrote in LinqPad:

void Main()
{
    Enumerable.Range(1, 5).Each(Console.WriteLine);
}

static class Extensions
{
    public static void Each<T>(this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
            action(element);
    }
}

In writing this post (and perhaps because I've been spending way too much time with jQuery lately), it occurred to me that I may want to be able to chain my actions with another Each() or with other extensions from Linq perhaps:

void Main()
{
    Enumerable.Range(1, 5).Each(Console.WriteLine).Each(Console.WriteLine);
    // 1 2 3 4 5 1 2 3 4 5
}

static class Extensions
{
    public static IEnumerable<T> Each<T>
        (this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
            action(element);
   
        return source;
    }
}

The problem is though, that generally you don't want your action to enumerate your enumerable until something is to be done with the results. Instead, you often want it to be executed during the enumeration of your enumerable, so you'd write it like this:

void Main()
{
    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Where(i => i <= 5)
        .ToList();
    // 1 2 3 4 5 6 7 8 9 10
    
    Console.WriteLine();
    
    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Take(5)
        .ToList();
    // 1 2 3 4 5

    Console.WriteLine();

    Enumerable.Range(1, 10)
        .Each(Console.WriteLine)
        .Skip(5)
        .Take(5)
        .ToList();
    // 1 2 3 4 5 6 7 8 9 10
}

static class Extensions
{
    public static IEnumerable<T> Each<T>
        (this IEnumerable<T> source, Action<T> action)
    {
        foreach (var element in source)
        {
            action(element);
            yield return element;
        }
    }
}

GiantDropdown jQuery Plugin for Styling Large Select Lists

A few months ago, I was working on a performance management application for the Centers for Disease Control. They wanted a select box that contained a list of competencies from which they could compose a template. The problem was some of the items in the list were hundreds of characters long and, at the time of writing, most browsers would behave very badly with drop down boxes of that size.

I was sitting at home that night and decided I could probably come up with a jQuery plugin that would use the select list as the backing field and present a prettier display. I did and we ended up using it at the CDC. I decided I'd share in case anyone else runs into the same issue.

Here's an example of what one of the select boxes would look like:

What type of element would you like to see?
A short one
One that is a little longer
One that is a little bit longer than that
An element that is really quite long and takes up a lot of display room on the screen
An element that is so long and takes up so much room on the screen that you basically have no way of displaying the element without wrapping the text within it
Another one that isn't quite as long
One that is really very long but doesn't wrap around the screen
Short
Entheusiastic!
The last one


It's not very pretty and it behaves unpredictably with other DOM elements. With the GiantDropdown plugin, we can style it any way we choose and still preserve all of the original behavior of the select list (meaning, it even works with multi-selects and option groups).

Here are some samples of the giant-ified dropdowns:


If you'd like to get the giant dropdown jquery plugin, the can check out the Giant Dropdown jQuery Plugin Git Repository. Feel free to make improvements.

Improved UpTo Extension Method

Back in May of '09, I wrote a post about an extension method to mimic Ruby's upto.

Today, I was hanging out at Visual Studio Live and had another idea. In Ruby, the upto method takes an end value and executes a code block. In C#, interpreted this as an extension method that takes an end value and an action. They syntax ends up being pretty similar and it works fine.

Lately, I've been using an Each extension method for IEnumerables. I know an Each extension method seems like a waste, but I like the fluency of it for basic actions. So, given that I have the Each method on IEnumerable<T> anyway, I decided UpTo should really return an IEnumerable as well.

Here's my new UpTo extension method:

public static IEnumerable<int> UpTo(this int start, int end)
{
    while (start <= end)
        yield return start++;
}

Now, instead of passing your action to the UpTo method like Ruby's upto, you get your enumerable and pass your delegate to your Each extension method. Here's what it looks like:

static void Main()
{
    //for (int i = 5; i <= 10; i++)
    //{
    //    Console.WriteLine(i);
    //}

    5.UpTo(10).Each(Console.WriteLine);
    
    Console.ReadLine();

    // output
    // 5
    // 6
    // 7
    // 8
    // 9
    // 10
}

If you're using .Net 3.5 or newer, Linq shipped with an Enumerator class which has a Range static method so your code could look more like this:

public static class NumericExtensions
{
    public static IEnumerable<int> UpTo(this int start, int end)
    {
        return Enumerable.Range(start, end - start + 1);
    }
}

class Program
{
    static void Main()
    {
        //foreach (var i in Enumerable.Range(5, 6))
        //{
        //    Console.WriteLine(i);
        //}

        5.UpTo(10).Each(Console.WriteLine);
        
        Console.ReadLine();

        // output
        // 5
        // 6
        // 7
        // 8
        // 9
        // 10
    }
}

Another reason to make UpTo return an IEnumerable instead of making it a void action executer is because sometimes you don't really want to pass a large anonymous method to the Each method. You can use the same UpTo method like this:

static void Main()
{
    //foreach (var i in Enumerable.Range(5, 6))
    //{
    //    // Do a lot of stuff!!
    //}

    //5.UpTo(10).Each(i =>
    //    {
    //        // Do a lot of stuff!!
    //    });

    foreach (var i in 5.UpTo(10))
    {
        // Do a lot of stuff!!
    }
}

Results of the "What Really Makes a Good Programmer" Survey

Back in September I wrote a blog post called What Really Makes a Good Programmer? My goal was to ask various members of the development community what traits they thought contributed to the quality of a programmer. If you haven't taken the survey yet, I'd recommend you do that before reading the results. Wouldn't want to bias your opinions, right?

If you have taken the survey, but haven't told everyone you know to take it too, go ahead and do that; I'll wait.

Now that we've covered that, let's get on to the analysis. As you recall from taking the survey, there were 17 traits and you rated them on a scale of Very Important, Important, A little Important, Negligible, and Completely Unimportant. In order to do any analysis, I had to take these ratings and convert them to numbers.

I decided that A Little Important was the baseline and basically represented a lack of opinion on the topic. These traits are the ones most people feel are nice to have but aren't requirements. Basically, a good programmer often has these traits but not having them doesn't mean you're not a good programmer. I recoded these values with 0 points.

Very Important and Completely Unimportant are the ratings that people used when they felt strongly about the item. That means that, respectively, the trait is either absolutely necessary to being a good programmer or the trait has no bearing on the quality of programmer you are whatsoever (or perhaps a little bit of a negative indicator). These values were recoded as 10 and -10 respectively.

The central values of Important and Negligible are what I call the non-committal values. You feel that they make a difference but they're not quite important enough to bank on them. I gave these a 3 and -3 respectively.

For example, "Has good problem solving skills" was rated Very Important and "Cheap" was rated Completely Unimportant. One could say, "someone with poor problem solving skills would make a poor programmer." On the other hand, you cannot say, "someone who is not cheap is a poor programmer." Of course, the contrapositive could also be stated that, "someone who is a good programmer is also a good problem solver." The contrapositive "someone who is a good programmer is also cheap" is considered by the community to be a largely invalid assertion.

"Fast" and "Co-located" appear nearest to the baseline. This may be due to the fact that many of the respondents didn't know what I meant by co-located. In any case, one might say, "a good programmer is a good programmer whether or not she's in the same building," or that "just because you're fast doesn't mean you're good and just because you're slow doesn't mean you're bad."

"Communicates effectively" and "Interested in helping teammates" are moderately rated in favor of contributing to being a good programmer while the two "college degree" items are rated moderately against contributing to being a good programmer. For example, you might know a lot of programmers who are great developers but who lack the social skills or interest to become good communicators or team players. As a result, you might say that good programmers often communicate effectively and help teammates, but some good programmers cannot.

You might also say, "Many people without college degrees, let alone a computer science degree, are great programmers. Thus, while having a degree is helpful, you can learn to be a great programmer without it."

Now, I feel it is worth noting that "unimportant" does not mean "negative." Just because most of the development community feels that having a CS degree or other certifications is unimportant to being a good programmer, it doesn't mean that the degree itself is unimportant. It just means that having the degree won't make you a good programmer; you still have a lot of learning to do.

With notes on the analysis out of the way, what do you say we take a look at some data? Currently, the new Google charts won't let me force the display of the categories, so you'll have to hover over the dots you're interested in. Here's a legacy chart you can look at if the javascript version below is unsatisfactory. The legacy chart has current data; it's just not as fancy as this one:
I'm sure you noticed there are two series in this graph. All Responses represents the average responses for all respondents; however, as you noticed in the respondent histogram that I have a dramatically unbalanced sample with programmers outnumbering all other groups combined by almost 3 to 1. To get a more accurate representation of the "general feel" of the community, I included the Group Average measure. This is the average of the averages. It's sort of the electoral college of informal research.

This chart demonstrates the average responses for each group:

Improving Search Performance when using SQL Server 2008 Encrypted Columns

This is a story of courage, honor, and data encryption. In addition to being something of a tribute to one of my favorite games of all time, I do feel I need to preface this post with a bit of a disclaimer.

Generally, when I describe this problem to someone, we go through pretty much the same conversation.

Why didn't you try Sql Server 2008 Transparent Data Encryption? Why don't you just update your ACL? Why don't you try this? Or that?

Well, sometimes the environment, either technologically or politically, precludes some of your better options. Ultimately, you have to go with your best permissible solution to solve the problem at hand. That being said, let's say you've been charged with securing your organizations PII. Your only option is to encrypt the column with SQL Server's column encryption.

You start looking at the impact this is going to have on your current and future applications. If you're like me, one of the first concerns you're going to have is performance. For example, let's say you have the need to search on the encrypted field. A good example of an encrypted field that you'll likely have to search is the Social Security Number.

In my original benchmarks, I found the following results:Knowing that I was going to have to improve this performance, I started playing with some searching alternatives. First of all, it's obviously much faster to search an indexed column (especially one that is clustered), so my first goal was to generate an indexed column.

Ryan McGarty (my best friend and a damn fine programmer) and I discussed and quickly ruled out a basic hash. Sure, it'd be fast to search an indexed column containing the hash value, but that opens you up to a relatively simple plain text attack (especially with a known set of possible values). I decided that you could reduce the threat and still speed up the search by using hash buckets a la hashtables instead.

I concocted a hash function that produced a reasonable distribution within the buckets. My good friend and esteemed colleague David Govek pointed out that an MD5 hash would produce a pretty effective distribution between buckets (with high cardinality data like the social security number. David was right and I ended up with this hash function:

CREATE FUNCTION [dbo].[GroupData] 
(
 @String VARCHAR(MAX),
 @Divisor INT
) 
RETURNS INT
AS BEGIN

  DECLARE @Result INT;
  
  SET @Result = HASHBYTES('MD5', @String);
  
  IF (@Divisor > 0)
    SET @Result =  @Result % @Divisor;
  
  RETURN @Result;

END

I created the test data just as I had before, except that this time I also added the hash bucket. I created a clustered index on the bucket and I changed my select statement a little:

-- original select statement
SELECT *
FROM PersonData
WHERE 
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

-- hash key select statement
-- @Divisor is the divisor for the modulus operation in the hash function
SELECT *
FROM PersonData
WHERE 
  SocialSecurityNumberGroup = dbo.GroupData('457555462', @Divisor) AND
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

Here are my results:

So, what of the known plain text attack then?

So, I don't want to gloss over the plain text attack issue. Given the possible set of socials, the hashes would be very unlikely to have a collision. Thus, for most people, you'd be able to get their socials easily by hashing every possible social and joining to that table. By modulo dividing the hash value, I'm able to evenly distribute social security numbers among a known set of buckets. That means, I can control the approximate number of socials in each bucket given my set of values. I generally aim for about 1000 socials per bucket.

For example, an MD5 % 100 has a possible set of values from -100 to +100. That's 201 buckets so if you have 2010 rows of data to hash, you'll have about 10 rows per bucket. The benefit is that you now only have to decrypt 10 rows to find the exact row you're looking for. The detriment is that you've narrowed your possible result set. Within your own data set, you'd have narrowed it to 10 possible plaintext values; however, given that these values are unknown, you then have to look at the set of possible values.

Social security numbers have a set of possible values less than 1,000,000,000. It's hard to say exactly how many of them are the possible set of values, so let's say we're using only those socials currently in use by living Americans. The population of the United States at the time of writing was about 312,000,000. As I said, I usually aim for about 1,000 records per bucket. If I have 1,000,000 rows of data, I would modulo divide by 500 (1,001 buckets). If you knew my set of values, then you'd only have 1,000 possible values to reduce. Given that you know when and where I was born, you could probably narrow it to 20 or so possible socials.

But, you don't know my set of values, so you really have 312,000 values to chose from. Even if you did know the first 5 digits of my social security number (based on my state of issuance and my date of birth), your set of possible options would be so large, you'd probably be better off just pulling my credit report and getting my social that way.

Thus, while it seems to introduce a weakness to plain text attacks (and with low cardinality data it would be an issue), in the case of social security numbers, I don't believe it to be a reasonable attack.

Column Encryption in SQL Server 2008 with Symmetric Keys

Most of the time when I write blog posts, I do it to share ideas with my fellow developers. Sometimes I do it just so I can have a place to reference when I forget the syntax for something. This is one of those reference posts.

Recently I've been charged to column level encrypt some personally identifiable information. The present post is not intended to discuss the merits of column level encryption; rather, as I said it is to put a few code snippets up so that I can reference them later. If you should find yourself in a column level encryption predicament in a SQL Server 2008 environment, you may find these useful as well.

First thing's first. Get the database ready for column level encryption by creating a master key:

--if there is no master key create one
IF NOT EXISTS 
(
  SELECT * 
  FROM sys.symmetric_keys 
  WHERE symmetric_key_id = 101
)
CREATE MASTER KEY ENCRYPTION BY 
  PASSWORD = 'This is where you would put a really long key for creating a symmetric key.'
GO

Now, you'll need a certificate or a set of certificates with which you will encrypt your symmetric key or keys:

-- if the certificate doesn't, exist create it now
IF NOT EXISTS
(
  SELECT *
  FROM sys.certificates
  WHERE name = 'PrivateDataCertificate'
)
CREATE CERTIFICATE PrivateDataCertificate
   WITH SUBJECT = 'For encrypting private data';
GO

Once you have your certificates, you can create your key or keys:

-- if the key doesn't exist, create it too
IF NOT EXISTS
(
  SELECT *
  FROM sys.symmetric_keys
  WHERE name = 'PrivateDataKey'
)
CREATE SYMMETRIC KEY PrivateDataKey
  WITH ALGORITHM = AES_256
  ENCRYPTION BY CERTIFICATE PrivateDataCertificate;
GO

Before you can use your symmetric key, you have to open it. I recommend that you get in the habit of closing it when you're finished with it. The symmetric key remains open for the life of the session. Let's say that you have a stored procedure in which you open the symmetric key to decrypt some private data which your stored procedure uses internally. Someone who has access to the stored procedure can run it and then will have the key opened for use in decrypting private data. My point, close the key before you leave the procedure. Here's how you open and close keys.

-- open the symmetric key with which to encrypt the data.
OPEN SYMMETRIC KEY PrivateDataKey
   DECRYPTION BY CERTIFICATE PrivateDataCertificate;

-- close the symmetric key
CLOSE SYMMETRIC KEY PrivateDataKey;

Here's a little test script I wrote to demonstrate a few points. First, the syntax for encrypting and decrypting. Second, the fact that the the cipher text changes each time you do the encryption. This prevents a plain text attack.

-- open the symmetric key with which to encrypt the data.
OPEN SYMMETRIC KEY PrivateDataKey
   DECRYPTION BY CERTIFICATE PrivateDataCertificate;

-- somewhere to put the data
DECLARE @TestEncryption TABLE
(
  PlainText VARCHAR(100),
  Cipher1 VARBINARY(100),
  Cipher2 VARBINARY(100)
);

-- some test data
INSERT INTO @TestEncryption (PlainText)
SELECT 'Boogers'
UNION ALL
SELECT 'Foobar'
UNION ALL
SELECT '457-55-5462'; -- ignoranus

-- encrypt twice
UPDATE @TestEncryption
SET 
  Cipher1 = ENCRYPTBYKEY(KEY_GUID('PrivateDataKey'), PlainText),
  Cipher2 = ENCRYPTBYKEY(KEY_GUID('PrivateDataKey'), PlainText);

-- decrypt and display results  
SELECT
  *,
  CiphersDiffer = CASE WHEN Cipher1 <> Cipher2 THEN 'TRUE' ELSE 'FALSE' END,
  PlainText1 = CONVERT(VARCHAR, DECRYPTBYKEY(Cipher1)),
  PlainText2 = CONVERT(VARCHAR, DECRYPTBYKEY(Cipher2))
FROM @TestEncryption;

-- close the symmetric key
CLOSE SYMMETRIC KEY PrivateDataKey;