D. Patrick Caldwell on Software Engineering

Saturday, November 26, 2011

Mobile Magic Developers Co-founded by Autopilot Founder

Autopilot Consulting, LLC founder Patrick Caldwell has joined forces with long time friend and coworker James Brechtel to found Mobile Magic Developers, LLC.

James and I worked together for a few years several years ago and have been close friends ever since. It was an easy decision to get together and start developing mobile applications.

Mobile Magic Developers focuses on developing fun and exciting mobile products like educational mobile applications for kids, amusing home screen widgets, and mobile utilities. Autopilot Consulting focuses on … well consulting.

When your company is ready for your own custom mobile application, come talk to Autopilot about how we can help you wow your customers and outshine your competitors with a mobile application of your own. We'll apply our mobile experience as well as the talents of Mobile Magic to make sure you get the biggest mobile bang for your buck.

Mobile applications come in several varieties. Whether you're interested in native mobile applications or mobile skins for your web applications, we have the experience to get you there, save you money, and help you make the best decisions for your business, your customer, and your future.

I really appreciate comments so please feel free to comment on my posts. Whether you agree or disagree, I'd love to hear from you. Also, feel free to link back to your own blog in your comments. You can even subscribe to an RSS feed of the comments on this thread.

© 2008 — , D. Patrick Caldwell, President, Autopilot Consulting, LLC

Wednesday, November 16, 2011

Mainframe Migrations: Source Code Translation Tools

I was working on a mainframe migration project a while back to move an application from Adabas and Natural to Sql Server and C#. Software AG developed Adabas and Natural back in the 70's so I was very pleased to have some Software AG developers and managers on the team.

Shortly into the project, one of the managers recommended a product Software AG has that translates Natural code into C#. Much to the surprise of my teammate, I recommended strongly against the software. In fact, if you've read any of my other mainframe migration articles, you know I was recommending against a "migration" strategy altogether (a lift-and-shift my teammate called it) vs. a guided redevelopment strategy.

In the process of making my argument, I spent a fair amount of time positing that a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap. The customer did decide not to use the translation tool, but in the process of making my argument, my curiosity was piqued (and I do have something of a strong research background) so I put together the What Really Makes a Good Programmer Survey.
For an in depth discussion of the results, see my Results of the "What Really Makes a Good Programmer" Survey post.

For this post, we'll be looking primarily at these data:
If you can't view this chart, I managed to keep a copy of the What Really Makes a Good Programmer Chart from the legacy charts API.

Because most of my respondents were programmers, I like to look at the group averages to eliminate selection bias. I group the "traits" into the following three categories:

Traits that contribute to being a good programmer
1. Has good problem solving skills
2. Learns from experience
3. Interested in learning
4. Passionate
5. Sees the big picture
6. Recognizes patterns
7. Communicates effectively
8. Tries new approaches
9. Detail oriented
10. Seeks help when needed
11. Interested in helping others
Traits that are nice to have if you can get them
1. Fast
2. Co-located
Traits with no effect on programmer quality
1. Has a college degree
2. Has a computer science degree
3. Cheap
4. Has certifications

So, back to my original stipulation that "a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap." While I still believe that statement to be true, when I conducted my survey, I asked the question in a more positive way so I can't actually claim that the tool is a bad programmer. I believe that it is analogous to a bad programmer; I just can't support that claim with these data. I'll explain why, but first, for the pedants, statements like "the community represented in my responses appears to believe that having good problem solving skills contributes to being a good programmer" will be abbreviated to "having good problem solving skills contributes to being a good programmer."

That being said, the data suggest that having good problem solving skills contributes to being a good programmer. If that stipulation is true, then the contrapositive must also be true that not being a good programmer means not having good problem solving skills. The inverse, however, is not necessarily true so I cannot claim that not having good problem solving skills contributes to not being a good programmer. I'll let you decide on the validity of that statement, but for the purposes of this post, I'll change my argument to, "using a source code translation tool is little more than hiring a programmer, who lacks the traits of a good programmer, simply because he's fast and cheap."

Think of a translation tool and go through the list of traits. The translation tool does not have problem solving skills at all, can't learn from experience and certainly isn't interested in doing so, has no passion, and doesn't even know what a big picture is let alone the big picture with your application in your environment. It may be able to recognize patterns technically, but it likely won't recognize patterns that are meaningful to your application. It's also unlikely that the tool will communicate effectively, create and try new approaches, or be interested in helping others.

I suppose you could say it's detail oriented (as long as the details are explicit and don't require problem solving) and it seeks help when needed (likely by way of errors). It's probably faster than a good programmer, though this may be negated if it's lack of the aforementioned traits produces less than useful results; that is to say as long as you don't need a programmer to fix the translation once it's finished the speed may be beneficial. Another good thing about the tool is that it's possibly cheaper than a good programmer (see previous caveat) and it's arguably well educated, but that doesn't seem to constitute a good programmer according to the survey.

Software development is an art almost as much as it is an engineering field. You hear many people talk of software craftsmanship. Allowing a craftsman to rewrite your application in the new framework from scratch using the old application as a guide will allow her to provide a unique expertise that translation tools currently don't have. She'll be able to apply a high level human analysis that the tool cannot.

For example, imagine there's a bug in the old code? The programmer can spot and fix the bug, but the tool will translate it into a prettier and newer bug. What if the old code is doing something unique to that language because the language doesn't have the features of the current language. A programmer can take advantage of these features and the tool cannot. A programmer will naturally analyze your business process as the application comes together and will make recommendations that may make your new application even better than the old one was in its heyday; the tool will not.

I can think of dozens of examples like this and I'm sure you can as well. Ultimately, I believe that allowing a tool to convert your legacy code into newer legacy code is a waste of time, money, and opportunity and if it doesn't cost more in the short run, it will cost more in the long run.

Tuesday, November 15, 2011

Switching on Enums: A Style Observation

I was digging through the framework today looking for another good dotPeek of the Week topic. I was perusing the Lazy<T> class and found an interesting snippet.

This post isn't quite like my previous dotPeek of the Weeks insofar as this is more of an example of what not to do. This is certainly merely my opinion, but one rule I try to follow when writing code is that more expressive is almost always better than less expressive. When I was looking at the Lazy class, I found a great example of this.

Here's the code (abridged for clarity … and also because the threading in this class will make for better discussion later):

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    default:
      // lock and set values
      break;
  }
}

Is there anything you notice about this code? Perhaps any unanswered questions as you read it and try to figure out what it does? Specifically, what exactly constitutes the default case?

As I read through this code, learning about some of the interesting thread safety techniques, I found myself pondering, "why would locking be the default behavior? In fact, what is the default case? Do the default values have something in common?"

I looked at the enum LazyThreadSafetyMode and found this:

public enum LazyThreadSafetyMode
{
  None,
  PublicationOnly,
  ExecutionAndPublication,
}

That's when I decided that, in most cases, when you're switching on a reasonably small set of values, it's best to express these values explicitly so that the people (including you) who have to maintain the code can better understand why the default case is the default case … even if there are no comments.

For example, the following code is functionally equivalent:

private T LazyInitValue()
{
  switch (this.Mode)
  {
    case LazyThreadSafetyMode.None:
      // set the value
      break;

    case LazyThreadSafetyMode.PublicationOnly:
      // CompareExchange the value
      break;

    case LazyThreadSafetyMode.ExecutionAndPublication:
      // lock and set values
      break;
  }
}

Personally, I find the latter example much more expressive. It's obvious to me what the three cases are and I don't have to wonder what possible values can become the default case. In fact, I may even go so far as having the default case throw an exception in this class. I'd do this so that, for whatever reason, if someone were to change the LazyThreadSafetyMode enum and not implement that case for the new values in Lazy<T>.LazyInitValue(), they'd get an exception in testing instead of incorrectly using the default functionality.

Saturday, November 12, 2011

Elon University has #1 MBA Program

Elon University's part-time MBA program deserves a lot of credit for the success of Autopilot Consulting. Bloomberg Businessweek recently gave the Martha and Spencer Love School of Business (LSB) much deserved recognition.

In The Best Business Schools of 2011, Elon's MBA program was ranked first among competitors like UCLA, Carnegie Mellon, UC-Berkley, and Rice.

When I was looking for an MBA program to attend, I compared Elon's program with Duke University's and UNC's programs and was blown away by the comparative quality of the LSB program; the experience, friendliness, and helpfulness of the staff and faculty; and the extraordinarily low price tag.

Congratulations to the Elon LSB Faculty, Staff, Students, and Alum on a job well done.

Wednesday, September 21, 2011

Cow Says Moo for Android - Teach Your Kids Farm Animal Sounds

James, my partner in Mobile Magic Developers, and I have a handful of Android applications under our belts now. We loved working on Chinese Whispers, an idea my wife gave me for a new twist on The Telephone Game. James wrote an immensely handy application for getting your WiFi Passwords from your phone.

I wrote the Chuck Norris Fact of the Day Widget and the Jane Austen Fact of the Day Widget, and James wrote a really nice widget that shows you your last few installed applications called Latest Apps.

But… Then we Wrote Cow Says Moo

Cow Says Moo is definitely my favorite so far. Many of my friends and family are having babies now and my wife and I have one on the way. I noticed that kids are growing more and more interested in technology and are remarkably capable with it. My niece loved playing with my Motorola Xoom and I loved the way her face lights up when she touches the screen and something happens. That's where we got the idea for Cow Says Moo.

Cow Says Moo is an educational game to teach young kids and toddlers about farm animals and the sounds they make. Each picture is hand illustrated by Emily Upchurch Robinson and when pressed plays farm animal sounds. You can use the front facing camera on your device (or the rear camera if you prefer) to record video and sound while your child interacts with your mobile device or while you simply delight him or her with the sounds. You can use the built in media gallery to share your videos with friends and family. You'll be surprised how much fun you'll have playing Cow Says Moo with your kids and you'll get a lot of cute video footage like I did "User Acceptance Testing", i.e., playing Cow Says Moo with my godson AJ.

Check out Cow Says Moo, and as always, feel free to contact us with any comments or bugs.

Friday, September 2, 2011

7-bit Encoding with BinaryWriter in .Net

At work this week, I had the need to serialize objects and encrypt them while trying to keep the smallest data footprint I could. I figured the easiest thing to do would be to binary serialize them with the BinaryFormatter. It was, indeed, the easiest thing to do; however, the BinaryFormatter seems to come with a fair amount of overhead. By the time the BinaryFormatter was finished listing all of the necessary assembly-qualified type names, the 60 bytes of data I wanted to preserve were more than a kilobyte!

I needed another way so I extended BinaryWriter (mostly to get it to serialize the types I needed it to) and now my 60 bytes take 68 bytes to serialize. In the process of writing this class, I looked through the disassembled BinaryWriter using JetBrains's dotPeek and found this little gem (Write7BitEncodedInt(int)) and decided it'd make a great dotPeek of the Week:

protected void Write7BitEncodedInt(int value)
{
  uint num = (uint) value;

  while (num >= 128U)
  {
    this.Write((byte) (num | 128U));
    num >>= 7;
  }

  this.Write((byte) num);
}

This is how the BinaryWriter.Write(string) encodes the length of the string. Thus, if you have a string with fewer than 128 characters, it only takes one byte to encode the length. In my implementation, I used this to specify the length of collections too. In fact, when writing a 32 bit signed integer, you should be able to save space for all positive numbers less than 2 ^ 22 and break even (require four bytes) for 2 ^ 29. Encoding an Int32 that is greater than or equal to 2 ^29 will require five bytes. Thus, if your integers tend to be smaller than 536,870,912, you'll probably save space encoding this way. A similar function could be used for a long where all positive values less than 2 ^ 57 will result in at least breaking even.

Here's how it works:

Convert the number to an unsigned int so you can do arithmetic on positive numbers (after all, you're just writing bits)
While the converted value is greater than or equal to 128 (i.e., 8 bits)
1. Write low 7 bits and put a 1 in the high bit (to indicate to the decoder more bytes are coming)
2. Shift the 7 bits you just wrote off the number
When the loop finishes, there will be 7 or fewer bits to write so write them

I think this is really clever and very easy. Reading the data back is a smidgen more complicated (Read7BitEncodedInt()):

protected internal int Read7BitEncodedInt()
{
  // some names have been changed to protect the readability
  int returnValue = 0;
  int bitIndex = 0;

  while (bitIndex != 35)
  {
    byte currentByte = this.ReadByte();
    returnValue |= ((int) currentByte & (int) sbyte.MaxValue) << bitIndex;
    bitIndex += 7;

    if (((int) currentByte & 128) == 0)
      return returnValue;
  }

  throw new FormatException(Environment.GetResourceString("Format_Bad7BitInt32"));
}

Here's how this works:

Set up an int to accumulate your data as you read them
Set up a place to keep track of which 7-bit block you're reading
While your bit index is less than 35 (i.e., you've read no more than 5 bytes comprising 1 "more bytes" indicator and 7 data bits)
1. Read a byte
2. Take the byte you just read and logical conjuction (bitwise and) it with sbyte.MaxValue (127 or in binary 0111 1111)
3. Left shift those 7 bits to the position in which they belong
4. Use a logical disjunction (bitwise or) to write those seven bits into your accumulator int
5. Add 7 to your bit index for the next 7 bits you read
6. If the current byte you read does not have a 1 in its most significant bit (i.e, byte & 1000 0000 == 0)
  1. There are no more bytes to read so just return the current accumulator value
If you get to this point, you've read a 6th of 5 bytes and it's time to let the user know there was a formatting problem

Thursday, August 25, 2011

.Net GetHashcode Functions

Last week, I wrote a post about a .Net method to combine hashcodes. In the process of designing that method, I looked into dozens of .Net's GetHashcode implementations.

If you'd like to know the principles of hashcodes, take a look at the aforementioned article. This one is part of the dotPeek of the Week series so I'll just be sharing the insight I got from the framework's implementations here.

First, some really basic ones:

// from Int32
public override int GetHashCode()
{
  return this;
}

// from Int16
public override int GetHashCode()
{
  return (int) (ushort) this | (int) this << 16;
}

// from Int64
public override int GetHashCode()
{
  return (int) this ^ (int) (this >> 32);
}

The Int32 implementation easily meets the hashcode requirements by returning itself. The Int16 copies itself to the top half of the Int32 and returns that. The Int64 takes the bottom half of itself and XORs it with the top half.

This XOR is the first valuable piece of information. If you look at the logical functions, the XOR produces the best bit twiddling for hashing. Basically, the XOR produces a relatively even distribution of bits as opposed to the OR and the AND operations which will be biased 3 to 1 in one direction or the other.

Some more complicated implementations:

// from String
public override unsafe int GetHashCode()
{
  fixed (char* chPtr = this)
  {
    int num1 = 352654597;
    int num2 = num1;
    int* numPtr = (int*) chPtr;
    int length = this.Length;
    while (length > 0)
    {
      num1 = (num1 << 5) + num1 + (num1 >> 27) ^ *numPtr;
      if (length > 2)
      {
        num2 = (num2 << 5) + num2 + (num2 >> 27) ^ numPtr[1];
        numPtr += 2;
        length -= 4;
      }
      else
        break;
    }
    return num1 + num2 * 1566083941;
  }
}

// from Tuple<>
internal static int CombineHashCodes(int h1, int h2)
{
  return (h1 << 5) + h1 ^ h2;
}

These two methods have some really good information in them. The String.GetHashcode implementation has what's called a rolling hash. It loops through the characters, does a barrel shift by 5, and then XORs with the next character.

While I liked this, I preferred the simplicity of the Bernstein Hash. The main component of the Bernstein Hash is the (i << 5) + i. i << 5 == i * 32 (except that bit shifting is much faster than multiplying).

Thus, i << 5 + i == 32i + i == 33i. The Bernstein Hash just takes the current hash value, multiplies it by 33, and XORs in the new value.

I didn't use 33 in my hash function because I feel like using prime numbers is healthy. Thus, instead of adding I subtract so my hashcode method is (hash << 5) - hash ^ value (or 31 * hash ^ value). This, by the way, is the way Java tends to do it.

Here are some interesting ones from System.Drawing:

// from Size
public override int GetHashCode()
{
  return this.width ^ this.height;
}

// from Rectangle
public override int GetHashCode()
{
  return this.X ^ (this.Y << 13 | (int) ((uint) this.Y >> 19)) ^ (this.Width << 26 | (int) ((uint) this.Width >> 6)) ^ (this.Height << 7 | (int) ((uint) this.Height >> 25));
}

// from Point
public override int GetHashCode()
{
  return this.x ^ this.y;
}