Size Matters: Improve Productivity with Cheap Screen Real Estate

It's hard to hire developers, and technical resources are expensive. Everyone is trying to figure out how to attract top talent, reduce turnover, and increase productivity of existing staff without reducing their work/life balance.

Early on, I decided that if you're going to spend 8 hours a day 5 days a week for almost 50 weeks a year doing something, everything about it should be as pleasant as it could be. I bought a new Macbook pro and two thunderbolt displays. Those investments very quickly paid off.

Until recently, the benefits of large technology investments were just intuition. Last week, I realized how much more productive I was working remotely than when I'm onsite with the client. One reason is that when I'm remote I get a lot more flow, but another reason is that when I'm remote, I have a lot more screen real estate so I can see a lot more information at once.

I decided it was time to be more productive onsite too. This time, I wanted to have more concrete justification.

Screen Selection

A friend of mine and I did a fair amount of research picking out monitors that gave us the best bang for our bucks. I'm really happy with my Thunderbolt Displays. They're IPS and they're quite high PPI. This time, I wanted to try something a little cheaper and a little larger and I wanted to try a VA panel so I ended up getting a BenQ BL3200PT. This monitor, I think, has the same panel as the Samsung S32D850T, which has a slightly smaller price tag.

Screen Real Estate

When I talk about screen real estate, I'm not talking about the diagonal size of the screen; I'm talking about the resolution of the screen. Having a larger screen doesn't do anything for you (at least not in the sense of a computer monitor) with regard to how much information you can fit on the screen at one time. The difference between an HD 1080 screen at 27" vs. 32" is just stretching. Resolution, however, is how many pixels the screen has. The more pixels there are, the more options the computer has to present data (provided you have the visual acuity to perceive the data without scaling).

These displays are 2560 x 1440 or WQHD and are the same resolution as my thunderbolt displays, albeit over a larger surface area. Thus the resolution is not quite as crisp as my thunderbolts, but still these are very high quality screens. If you're into lower pixel pitch and higher PPI, I don't blame you. Higher pixel density screens are much more pleasant to look at, but they do come at a cost.

Comparing Resolutions

There are two very common resolutions in offices these days. An older WSXVGA+ which is an 8:5 aspect display at 1680 x 1050 pixels. HD 1080 is also quiet popular which is the 16:9 display at 1920 x 1080. The HD 1080 has about 17.6% more pixels than the WSXVGA+. By comparison, the WQHD, also 16:9, has 2560 x 1440 pixels (sometimes called 1440P). It has about 77.8% more pixels than HD 1080 and about 110% more pixels than WSXGA+. That is to say that if you have 1 WQHD display, you have more pixels than a dual screen WSXGA+ setup.

Benefits of Screen Real Estate

The benefits of additional screen real estate, unfortunately, are easier to experience than they are to describe. Having additional real estate may mean that you will sense more data at the same time, but does not necessarily mean you will perceive these data. Similarly, having the ability see multiple windows does not mean that you won't just continue editing one file at a time in a full screen editor. On the other hand, not having the ability to optimize your workflow guarantees that you will not.

Anecdotes

I do have a few anecdotes that buttress my belief that having more screen real estate is inherently valuable. One time when I'm particularly keen to the benefits of higher resolutions is when I'm developing responsive designs. I'll often dock my browser's DOM inspector on the side and I'll slide the split pane to different responsive breakpoints watching the HTML/CSS changes flash in the inspector. The additional real estate means I can see meaningful data in the DOM inspector and the responsive design at the desktop breakpoint at the same time.

Another time I notice the benefits of having additional screen real estate is when working on native mobile applications. Native development often involves simulators and emulators. Scaling during emulation isn't always well supported. Thus, the additional pixels allows you to see the entire simulator at once.

Another example that I found particularly fascinating happened shortly after I took over a new project. I had been working on the project for a few weeks from the client site. One day, I was working remotely and I had my terminal window off to the left and I had the application in a browser to the right. This is a usual screen configuration for my workflow on my cinema displays. I had changed some code and the application reloaded. When it did, I noticed dozens of queries pop up in the log. None of the ORM queries had been optimized to prevent N + 1 queries. I spent a few hours fixing those, but may never have noticed them had I not had enough room on my screen to comfortably view the browser and the terminal at the same time.

Summary of Benefits

Typically on HD 1080 and especially on WSXGA+, I have only one window visible at a time. That's why I didn't notice the N + 1 queries until I was remote. There may be myriad other benefits of the additional screen real estate when considering looking at code in multiple panes, test driven development, debugging, guard and live reload, etc.

Costs

High quality screens can be expensive! The thunderbolt displays are almost $1,000 each. The BenQ was $550. The Samsung is $500. It's hard to imagine pulling out all of your existing displays and replacing them with displays that cost several hundred dollars. Heck, I spent hours justifying buying only one.

That's why it clicked. At some point I realized, it doesn't take much before the investment in display technology pays off. According to several job posting / salary research sites, the average developer in Atlanta makes about $120K per year. To keep the math easy, I'll call it $100K (also because I like lowballing estimates like this). Normally, overhead on a W-2 employee is roughly 25% - 50% of annual salary. If you have a very modest benefits package (and not a lot of PTO), then an average developer in Atlanta will cost about $125K per year.

Even if your organization buys these screens the same way I did (i.e., no sales tax and no bulk discount), then it'll cost about $550, or about 0.4% of the cost of an average developer. That means that if you bought 100 of them, it would cost less than half of the annual salary of your average developer for a year. Thus, if you have 100 developers and you get 100 screens and each developer is 0.5% more productive for the year, then you actually make money on the investment.

If the cost still seems too high, you don't have to get VA panels. In fact, 32 inch screens are pretty big (they take up a lot of desk real estate too). These 25" Dell U2515H IPS panels were only $350 at the time of writing. 100 of these would be less than 1/3 of the underestimated developer annual cost.

Recuperating Costs

The average American works 1,780 hours per year. To recuperate 0.28% of a developer's cost, the developer only needs to produce about 5 hours more per year. That could be working an additional 5 hours per year because work is more fun. It could be 5 hours saved by not having to tab between windows and the cost of brief context switches. It could be 5 hours not spent tracking down performance issues in production because some N + 1 queries sneaked through.

It could be 5 fewer hours spent recruiting great job candidates because the technology makes the company a better place to work. Developers with neat technology are more inclined to passively as well as actively recruit for you. This can bolster your recruiting efforts, also saving money. Similarly, developers with a satisfying work environment are less likely to leave their jobs reducing the high cost of turnover.

Studies on worker productivity show that American workers tend to have about 6 (and some studies show as few as 3) productive hours in an 8 hour workday. Better screens may improve ideal hours per day. That means if all of your developers show up Monday morning and find their brand new screens, and in their excitement they're productive for 8 hours on Monday and Tuesday (rather than 6) and 7 hours on Wednesday, then in the first 3 days, the screens have paid for themselves. I could be completely wrong about everything else and your net could still be more valuable somewhat liquid (they have good resale values) assets for free.

Remember, you only need 5 additional productive hours per year to cover the cost of 1 high resolution display … and that's if you buy new screens every year. If you plan on keeping the displays for 5 years, it's just 1 hour per year.

Incredulous?

Let's consider a more pessimistic case. Let's say your staff, on average is pretty junior (and you're using technologies that aren't particularly backend heavy. If that's the case, your developers may cost you an average of $80K salary and per year and $20K in benefits, taxes, recruiting efforts, etc. Let's say they also take no vacation or sick leave and thus work a full 2,080 hours per year. Further, your staff is productive 100% of working hours.

In this case, you're paying an average of $48 per productive work hour. If you buy very nice displays at say, $960 each, then you need 20 additional productive hours to cover the cost of one display. Even in this very unlikely case, that's less than 1% per developer. And they don't need to work 2,100 hours in a year per se. They can be 1% more efficient. That means if they're writing 100 lines of perfect code per day, then to cover the cost of a new display, they only need to write 100 lines of perfect code per day (and on the tenth day, perhaps they write 10 extra perfect line of code . . . or better yet, they delete a line or two).

More likely your average developer is closer to $125K per year, they work 1,780 hours per year, and they are productive fewer than 6 of every 8 hours. Thus, their cost is $94 per productive hour. Also, it's very reasonable to get smaller IPS WQHD panels for $350 each (especially if you have a large team and get a bulk or corporate discount). So, you need fewer than 4 hours per developer or 0.3% more productivity. An average developer contributes 10 lines per day to the codebase (Mythical Man Month) so to break even on your investment, your average developer only needs to net an additional 3 lines of code every quarter.

n.b., I don't support using lines of code as a measure of productivity. It's merely illustrative of the relative scale of the cost of a display to the cost of a developer. Try your own calculations using bugs sent to production, mean time to defect resolution, etc.

Recommendation

I recommend making an investment to replace low resolution displays for developers and pairing stations (and I would consider broadening the reach really to non-technical employees as well) with high resolution displays. I think that the cost of the hardware would be recovered very quickly by improved developer efficiency and performance. The investment is low risk because the cost is relatively low and the displays are a moderately liquid asset.

If you have 100 developers on your team, it may be hard to get approval for the capital to buy $35,000 worth of displays. In that case, you can always hire another developer (well, you can try at least) for $35,000 per quarter.

Display Options

Dell P2416D 24 Monitor with QHD 23.8-Inch Screen, Black
Dell UltraSharp U2515H 25-Inch Screen LED-Lit Monitor
Dell UltraSharp U2715H 27-Inch Screen LED-Lit Monitor
BenQ BL3200PT 32-inch WQHD 10-bit USBx4 HDMI Monitor built for Creative Class
Samsung 32" WQHD LED Monitor (S32D850T)

Project Nostalgia: Warehouse Management

Early in my development career, I got a job at an entertainment company that rented party supplies like inflatable bounce houses, inflatable slides, human gyroscopes, etc. On my first day, I was introduced to my new boss, the owner of the company, on his way out the door headed to China for 2 weeks.

I introduced myself, thanked him for the opportunity, and asked what he wanted me to work on while he was gone. "You'll figure it out," he said, "make us some money." And out the door he headed.

Once I found my office and tracked down a computer, I sat down and thought, "what the heck am I going to do here?" Having just gotten my MBA, I decided I'd take a page from my business consulting class and just "see what my lantern shows me." I walked around asking my new coworkers what the company does, what they do, and what could make their lives easier.

There were the usual gripes, manual processes, duplicate paperwork, etc. One problem in particular, however, I found particularly intriguing. It turns out that the company had an uncomfortably high refund rate because they'd show up at parties and equipment would be missing. They also would only schedule a few parties a day, despite having plenty of equipment to spare, because it took so long to prepare for an event. Similarly, they required reservations be made a week in advance.

Shopping in Your Own Warehouse

If you're like me, when you make a grocery list, you write items down in the order you think of them. Then, when you go to the store, you walk up and down the items, grabbing things from the list and crossing them off as you go. Also, if you're like me, sometimes you pass an item and you have to go back. Or, you don't notice an item that you missed and you go home without it.

If you want to try an experiment (actual or thought), commit to going to the grocery store and getting everything on your list in the order it appears on your list. You'll probably find you forget less stuff, but you do a lot more walking around the store. It's not a particularly efficient way to organize a shopping trip. If you implement this in, say, a warehouse you may issue fewer refunds but you can't increase the number of events you can service in a day.

I'm sure by now, you're imagining myriad ways to organize a list of grocery items so that, if nothing else, at least you have some consistency. You could organize them alphabetically. That way, your items on your list will follow more or less the same pattern visit to visit. That may or may not be better than in the order you thought of them (assuming you remember grocery needs that are grouped together).

Imagine, however, that you can't sort your grocery list in the order you think of it. Imagine you can't even sort the list alphabetically. In fact, your only option is to sort the items on the list in the order the item first appeared in your grocer's inventory. Imagine how nonsensical it would be trying to track down where all of your groceries were. How much you'd forget (or perhaps even leave deliberately). That's what we had.

A Guided Tour

I printed our inventory list and I walked around the warehouse writing down the exact coordinates of every product and every component we had. I also estimated the volume, weight, and stack-ability (if you will) of the item.

Hoping to get a cheap win and to prove that it could make a difference (some of the guys in the warehouse weren't interested in their inventory lists being optimized), I got a few of the guys to agree to try it out for 1 day. I entered my data into the inventory table and sorted the inventory list by aisle, bin, then level. That was it. Only took a few minutes, was easy to revert if it failed, and I could switch it based on the user.

When I got back from lunch, one of my coworkers who sat outside my office said, "hey, warehouse really wants to talk to you … they've been in and out of here constantly since you left." I thought, "oh no . . . I probably broke something and they can't print inventory lists at all now."

I headed out to the warehouse preparing my apology. I had barely walked through the door when someone said, "hey Patrick, check this out?" There were several palates stacked with boxes and party equipment distributed along the loading dock. "What's up?" I asked.

Turns out my first sorted list was such a hit that they decided to start racing each other to see who could fulfill the inventory list faster. They had not only prepared the next days events, but several days after. I know it doesn't sound like it should take that long, but this warehouse typically had 4 or 5 people preparing 10 or 15 parties a day give or take.

A Deliberate Solution

Feeling I'd created some real value, I went back to my sorting. I updated the logic so that bins were in reverse order on odd number aisles (they started at the end of aisle 1, went to 2, the back up 3, and so forth). I sorted the items so that, given close proximity, the larger or heavier items would appear in the list first so they'd be on the bottom of the cart.

In the display, I grouped the items by aisle. Once I did this, I realized I should start at the end of the first aisle that had equipment on the list, then continue on the beginning of the next aisle with equipment (rather than just naively going up and down each aisle). With the help of the warehouse team and some statistics I pulled from the historical invoice data, we also started physically grouping items based on how frequently they were pulled on the same list.

Some of them were obvious. If you ordered an inflatable, you'd likely also get a blower. People who rent tents also rent chairs. Some were less obvious. People who rented dance floors typically didn't rent inflatables but did rent a bar and concessions. Once we established that not only could we control our own inventory but that we can control our own reports, it became a game to optimize the process.

In the end, it only really took 1 person to run the entire warehouse, though we staffed 2 because people liked working together. The rest of the warehouse team got promoted. It was actually something of a coveted job to get to deliver the equipment, spend some time outside, visit with the people having the party (partying people are usually pretty happy people).

We also started booking more parties. We even booked parties same day if we had the equipment and the people available. I'm sure there are COTS products that are really good at solving this very problem, but solving a real business problem with feedback from the people who were most impacted by it and seeing measurable gains in revenue was well worth the effort. It also help build some relationships and establish a collaboration they'd never had before (I was the first developer who worked onsite).

Remote Interview Pairing Challenge with Cloud9

My interview process typically involves just trying to get to know someone to see if we're a good match culturally. After all, if you have a good mind for development, I know you'll be great if we can put you in the right environment with the right culture. As a result, I tend to spend a lot more time talking about what kinds of problems a developer finds interesting, what sorts of roadblocks she finds frustrating, and what technologies we have in common.

There is, however, still the necessary evil of seeing how far along the candidate is on the programming journey. To that end, I usually try to do a very basic code challenge. Normally it's something simple enough that we can still have a healthy conversation while we solve it together as a pair, and it's not so academic as to be useless in real life. Mostly, I just want to pair together to solve a problem.

I give the candidate the opportunity to select the language of her choosing and typically I'll know it well enough that we can pair to solve the problem. I explain the challenge, we set up unit tests, write a failing test, and make it pass. Sometimes, if I don't feel like I know enough, I'll change it a little and we'll try to adapt to the changes together.

This has been such a positive experience for me (and for the candidates I've talked with after their interviews), that I've spent a fair amount of time practicing the process. As I recently discovered, it's such a fundamental part of my interview, that I was completely unprepared when a client asked me, at the last minute, to step into an interview to do a pairing challenge with a candidate . . . who was remote!

Remote Pairing Interview

We tried to come up with a few options for doing the code challenge. After all, our shop pairs pretty much all the time and we try to be supportive of remote work. I mean, we've solved this problem before; however, it's seldom without hiccups. I know that candidates are usually nervous in an interview and I feel a lot of responsibility to help set their minds at ease. To that end, I like to appear calm, collected, and organized.

We considered using tmate and vim. We talked about screen sharing with TeamViewer. We also thought about just letting this candidate work on a harder challenge, maybe something from Hacker Rank or Project Euler, and email us the solution.

Shortcomings

Each of these had a downside. Tmate would give the dev access to my machine and I wanted to let them do whatever they wanted without putting my machine in his hands. Alternatively, his machine could host but I wanted to be able to get to the solution later. Screen sharing with TeamViewer isn't as responsive as I like when pairing, especially if I want to be able to help out rapidly. Also, I type in Dvorak and sometimes that's an issue (turns out he does too . . . by the way). The async challenges, while good, don't let us know if we'll enjoy pairing together.

Enter Cloud9

I remembered, a while back, running across Cloud9 and playing with it a little bit. I thought, "hey, what if we just try this?" Cloud9 would give us a sandbox that we can work in with a full ubuntu virtual machine. The candidate can do whatever he wants to it, we can code, install tools, whatever. I said, "hey, give me a second . . . I'm going to try this Cloud9 thing . . . go ahead and get your account set up and I'll invite you to a project."

In about 5 minutes, the candidate, my co-worker, and I were pairing and running tests on our Cloud9 workspace. There were no security concerns, communication was simple, we still installed and used tmux and vim, we had a sandbox, I was able to save the code. It was a really great experience. Not only did we offer a position to the candidate, but the candidate has been happy on our team ever since. I consider it a sign of a good interview when a candidate takes a position and I think that Cloud9 had a lot to do with this one.

Next Steps

I've not used Cloud9 (or any web based IDEs for that matter) on day to day work. Unfortunately, I'm not really in a position to try it out on the free tier. At some point, I want to go ahead and get a subscription and see how well it works out for me. I certainly see the value of cloud based IDE services and I like the idea that the most expensive development hardware you need is a Chromebook.

Okay, so maybe that's a little hyperbolic, but not for long. We have pairing stations that we treat like cattle rather than pets. We can wipe it completely clean and have it up and running again ready to go in just a few minutes. The hardware isn't commodity, but it could be. In fact, it could just be a web interface to a virtualized instance and why couldn't that be accessed from a Chromebook, pairing station, laptop, tablet, or . . . maybe in a pinch . . . a phone? Personally, I like it and hopefully I'll get a chance to push Cloud9 beyond "hello world."

Our 2014 Charity Picks

We feel lucky every day. Our community has provided us with a great place to live and work. Our kids can grow and learn in safety and comfort. We're surrounded by opportunities personally and professionally.

For that, we are eternally grateful, and we like to give a little bit back to our community. We work hard to research the charities we choose to support and to make sure they're doing the greatest good with their contributions.

Here are the charities we selected for 2014

Episcopal Relief and Development

Episcopal Relief & Development works to save lives and transform communities worldwide. They rebuild after disasters and empower people to create lasting solutions that fight poverty, hunger, and disease. They working in 40 countries impacting the lives of nearly 3 million people around the world.

ERD has a high rating on Charity Navigator. While we typically seek a higher rating, it's very close to the highest it can be and we have close personal ties with the organization.

One thing we love about this program is you get to donate specific needs to the communities you're supporting. This is certainly the first time we've donated a cow and chickens. We also funded farming tools and community gardens, prenatal and postnatal maternity care, and 1 year of education for 5 girls.

Tiger Flight Foundation

A little closer to home, the Tiger Flight Foundation, base in Rome, Georgia inspires kids to be "Pilot in Command" of their own lives. All donations go toward the motivational and character development programs.

My relationship began with Tiger Flight as a volunteer pilot. I had a young man in my plane and as we taxied out, he said, "I've never been in a plane before." I asked him what was his favorite place to visit. He told me he'd never been out of Rome, GA. He was nervously excited as we started rolling down the runway.

We had barely left the ground when he could no longer contain his excitement and he looked at me almost teary-eyed and said, "I feel like I can see the whole world from here!" I thought, "from here kid, the whole world is in front of you . . . and I think you'll see it."

How to Market a Mobile App: Professional Design

The next phase in the How to Market a Mobile App series was to try a whole new look and feel. While I feel like the design of This to That was actually pretty good, it's obvious design is not really my strong suit. I hired a professional design company that reworked my icon for about $150.

I dropped the new icon into the game and had my wife play with the This to That theme generator for a while. She came up with what is now the current This to That default theme. We are pretty pleased with the way it turned out and it appears users are too.

Promotional Images

Another thing we changed were the promotional images that show up with the app on the app store. I had reviewed dozens of my favorite apps to see which ones I thought had the most compelling promo images. I noticed I was most enticed by promotional images that featured interesting design with the application screenshot embedded in the image (rather than the whole image itself).

I had tried my hand at this kind of design and these are what I had before:

In an effort to try something new (and to get a new release out faster), I decided to replace these with simple screenshots. You can see them on the This to That on iTunes page. I may have my designers throw together some new promotional images if I start to get enough data about new downloads that I'll be able to share any significant observations about their effectiveness.

Results

The new graphics don't appear to have had a huge impact on new users. It looks like about 10% - 20% more downloads than before. None of the current users have complained in comments or ratings so that's probably a good sign really. I think the most interesting statistic I've observed this time around is in session duration.

It appears whatever I added between 3.5 and 4.1 was displeasing to users. We went from an average of 4 minutes per session to about 2 minutes per session. 4.2 brought us back up and 5.0 and 5.1 have leveled us off at about 3 minutes per session, but almost all of our players now play every day.

A few other observations

• Very few people attempt to play multiplayer
• Many players open the drawer; very few swipe it
• Since we moved the help button, more people click help
• There are very few social sharing events
• Most common incorrect words: TOOS, BEAS, SEAP, BONS, BAIN

Next Steps

I have a release ready to go in hopes that we get more reviews on the app store (hopefully positive too). We have always had a feedback button in the app, but at some point the link broke so presently it takes you to the app store but not to the app's page. To fix this, we started using appirater. On a related note, I recommend using cocoapods for iOS app dependencies.

How to Market a Mobile App: Professional Review Sites

I've been working on the revealing (and humbling) How to Market a Mobile App series so I've found myself more open to some ideas that I usually am. A few months ago, I got an email from the owner of Mobi Apps Review. I've heard about sites like this before that establish a viewership on their website and on social networks. Some friends tried it out paying as much as $5,000 for one review.

Mobi purported to have 25,000 followers on twitter and the same on facebook. The offer was to write a review and publish it on all of their social networks for $25. I figured, it's worth 25 bucks just to know what a little exposure could do for me. Besides, it'll give me something to post about on the Mobile Magic Developers facebook page.

The Mobi Apps Review of This to That came out March 6th, 2014. It was nice to see that the review was generally positive. Not all of the reviews on Mobi are positive so I feel like the reviewer really did enjoy the game.

Results

I reviewed the Google Analytics for the 3 weeks before and the 3 weeks after the Mobi review was posted. I saw no discernible effect on new users.

Conclusion

I don't think the lack of results necessarily indicates that the review had no impact. To be sure, I could've spent more effort promoting the review. I also, unfortunately, have no insight into how many viewers actually saw the review. Of those viewers, I have no idea how many visited the This to That for iOS app store page. Even if I did, I have no way of knowing how many people who view the app store page actually download the application and whether or not the review had an impact on that.

All in all, I think it was worth 25 dollars, and I expect to get value out of it at some point. Now, I can mention the 5 star review in the profile description which needs to be revamped anyhow. I can also reference it in posts on our facebook page. The gestalt effect of having an additional third-party link out there is likely to help support my efforts in marketing This to That.

How to Market a Mobile App: Design Counts

As I discussed previously in the How to Market a Mobile App series, one of the first things I could do to improve user experience in This to That for iOS is not only to support the look and feel of iOS 7 but also to improve the color selections of the default theme. At the time I released the latest version of This to That, the iOS 6 SSL vulnerability hadn't been discovered yet, so I have a new default iOS 6.0 theme (that nobody will ever see again). :)

Results

There were no notable changes in new user acquisitions nor in returning user frequency. There may have been a slight reduction in session time and a slight increase in screen views per session. Reviews are still generally positive, but it appears changing the theme had very little impact on the performance of the application

Next Steps

I had a new icon designed professionally. I made the current icon and while I think it's pretty cool, it definitely doesn't look quite as engaging as the new one does. I need to re-release This to That with that icon in hopes that it increases new user acquisition.

Screenshots

Each set of screenshots shows the original theme on the left, the new default iOS 6 theme in the middle, and the new flat iOS 7 theme on the right. You can still select the "reflective" tile type in the theme builder to get the old look and feel, but the linen background is no longer available to iOS 7 users. Instead, it's just gray.

Basically, the theme is lighter, flatter, and sharper. The last few images demonstrate the difference in texts. Instead of inlaid icons for the letters, they're all just flat letters in a built in system font. Eventually, I think I'd like to switch the fonts throughout the application to use iOS 7's default font.

A Limerick Standup: AngularJS, PhantomJS, and Jasmine

In honor of St. Patty's Day, I decided a limerick standup would be appropriate:

I've made some good progress with Angular
Though nothing of mention particular
     I have some good tests
     That pass in PhantomJS
And if Jasmine were here, I would strangle 'er

How to Market a Mobile App: Knowledge is Power

If you're going to understand how changes you make to your application affect your users or how effective your marketing strategy is, you're going to need data. All of the app markets provide decent data about downloads, active users, ad impressions, etc., but these are no match for deliberate data collection and usage statistics.

In the early days of This to That for iOS, we didn't include any analytics. We relied heavily on iTunesConnect to provide the statistics we needed to determine the effects of our changes. Google Play provides much better analytics out of the box including active installs, device statistics, OS versions, etc., so we had to make a lot more inferences.

For example, we could see daily downloads but we had no idea how many users we were retaining. At that point, we were releasing often so we could compare our download statistics to our upgrade statistics to get a feel for that. We further refined our estimates when we added Game Center support. Comparing the new user statistics to the Game Center names I didn't recognize (seriously), I estimated about half of the users use Game Center. Thus, we could infer that for every player who reported a score to the leaderboard, there was a roughly equivalent user without a Game Center account.

We also planned to be ad supported out of the gate, so we were able to get some usage information from ad impressions. Our app has a roughly 98% fill rate and filled ads cycle about every 3 minutes. Combining that information with estimated users per day, we could guess how much time people spent playing our game.

Several versions back, we started using Google Analytics to collect anonymous (we're not the NSA after all) usage statistics. Now we get a much more detailed view of our users and how they play This to That. I recommend developers consider analytics to be an inextricable part of any minimum viable product. Without analytics, there's no way I could write the How to Market a Mobile App series as I'd have no way of knowing how these experiments impact our application's performance in the app market.

Starting Point as of Version 4.2

• 910 sessions lasting an average 3.5 minutes per month
• 108 of these sessions are new users (about half are iOS 7)
• About 80% of the sessions end on the first screen and last less than 10 seconds
• About half of the users play at least 10 times per month
• About 90% of users play This to That at least every other day

I think these statistics tell us a few meaningful things about This to That. The first and I think most obvious inference, which is the reason I'm writing this series (and is likely the reason you're reading it): people can't find and don't hear about This to That. To resolve this, we've added social features, game center, multiplayer gameplay, and have tried advertising. As of yet, to no avail.

Second, I think, is that many users never get past the first screen. I think that's likely because the look and feel of the default game theme may not be appealing (I'm not a designer after all). iOS 7 users are less likely to engage than iOS 6 users; it might be worth updating the default look and feel of the application for iOS 7 users.

I think there are some positive observations in these data too. Users who play the game tend to play it often (at least for a while). They tend to play several times a day for short periods. This gives me insight that the game should be quick to get into. Perhaps, for example, the game should start in the game screen and have a menu option for going to what's now the home screen.

Things to do in Version 5.0

• Create an iOS 7 default theme
• Clean up the default color scheme for all new users
• Make a better app description for iTunes
• Make more compelling graphics for the images in iTunes
• Clean up the icon

I'm hoping we'll see a larger number of new users by having more compelling marketing in our This to That iTunes profile. I'm also hoping that the improved icon and cleaner default theme will make new users more likely to complete their first game (most users who complete the first game complete many subsequent games). I'm hoping having more users and retaining a higher percentage of them will also result in a higher likelihood that users will share the game with friends.

There was a slight drop in retention after we released the multiplayer version of This to That. I think that users were frustrated when they tried to connect to a multiplayer game, but nobody else was playing. That frustration overpowered the interest in playing the original single player version. When we released that version, we were hoping it would encourage users to invite friends and family to play. Ideally, the new design and higher retention rate would also result in more available online users for random game center match creation.

What's it Going to Take?

Well, one thing I like to do as an independent developer is work on the cheap. I think that I've managed, on my own, to make a fun and entertaining application; however, I admit it lacks a certain je ne sais quoi. Perhaps it lacks the polish of the kinds of games people make when they have designers on staff. Perhaps it feels unprofessional. Of all of the negative feedback I've ever gotten, I don't think anyone has said, "the game isn't fun." So, I'm going to have some artwork done professionally. The game, in its current state (with the extensive theming) would be difficult to overhaul, so I'm going to focus on 3 components: the icon, the screenshots in iTunes, and the default theme. I'm hoping to get all of this done for about $500 - $750. I have a few quotes in and I think this is a reasonable expectation.

How to Market a Mobile App: Where to Start

Mobile Magic Developers has several applications on various mobile marketplaces now. My 2 year old daughter loves Cow Says Moo. My mother in law has been playing This to That for years. Our friends get goodnatured laughs with Said Obama. All in all, developing mobile applications has been a really positive experience.

In graduate school marketing classes, we spent a lot of energy discussing myriad ways large organization with massive marketing budgets could broaden their customer bases. I'm sure that information is still somehow useful but it surely doesn't help independent developers like Mobile Magic Developers who don't have a marketing budget, let alone enough to hire an MBA marketing consultant. In fact, our marketing campaigns like the rest of our expenses, are still being paid out of our pockets.

So, where do you start on the quest to release a popular mobile application?

I Have No Clue O.o

I sincerely have no idea. I've read a lot of good blog posts about marketing mobile applications. I've purchase some really interesting books on the topic. My brother has a BA in marketing. I did get some marketing exposure in college. Despite all of this, I still have some really great apps nobody uses.

That's why I'm starting the How to Market a Mobile App series (well, that and I haven't blogged in almost 2 years and it's high time I get back to it). I'm going to start by introducing, with complete candor, where we are with This to That for iOS today and how we spent the last few years getting to what is effectively a good starting point.

I have a new plan for spreading the word about This to That with a limited budget over the next few months. I'll blog about every strategy I try and how each idea fails and succeeds (including all of the nitty-gritty and sometimes embarrassing details). Every time I add a new post, I'll include it in the index below so bookmark this page for a convenient starting point.

Worst Case Scenario? :*(

Well, as I've shared, I don't really know much about successfully marketing a mobile application, but I have learned a handful of things not to do. That tells me that the worst case scenario of writing this series both for me the writer and you the reader (and subscriber hopefully) is that together we learn a few more things not to do. But, who knows? Perhaps we'll figure out the right formula for getting our wonderful apps into the hands of the target audience who can get the most value out of them.

How to Successfully Market a Mobile Application

• All posts in this series

First Small Business, Big Difference for the NBCF a Success!

If you followed our Small Business, Big Difference program, you know that our first Small Business, Big Difference to support the National Breast Cancer Foundation ended Februrary 18th, 2012.

Thus, there are two pressing topics:

First, I'd like to apologize for taking this long to get these results out there. I know it's been a while, but I've been a little busy. On February 17th, 2012 my wife delivered our first child and we've been enjoying some family time.

Second, when we set out to start the Small Business, Big Difference program, we knew it'd take a few tries for it to catch on. We figured our first attempt would produce modest results; however, we were so excited that we were able to generate such interest in the program that we rounded our donation up to $100.00.

We'd like to thank everyone who donated or participated in the program and we'd like to thank the NBCF for letting us raise funds for them with the Small Business, Big Difference program. We're looking forward to working with them again in a future promotion.

National Breast Cancer Foundation - Small Business, Big Difference

James and I recognize that we get a great deal of benefit out of writing mobile applications with Mobile Magic Developers. We enjoy the time together, the camaraderie, the learning experiences, and the opportunity to bring a little joy to a much larger audience. Along the way we've had a lot of support from friends, family, and the numerous customers who've downloaded our apps.

We wanted to find a way to give something back as an enthusiastic "Thank You" from the Mobile Magic Developers team to you. We've decided we're going to try something we call Small Business, Big Difference. Periodically, we're going to choose one of our applications and for a period of time, all proceeds from that application will be donated to a charitable organization. That includes sales and ad revenue.

For our inaugural Small Business, Big Difference attempt, we've decided that we'll donate all revenue generated by our Jane Austen Quote of the Day Widget available on the Android market to the National Breast Cancer Foundation. We've even added a new background for your widget just for this occasion!

Why the National Breast Cancer Foundation

We reviewed dozens of charitable organizations dedicated to promoting breast cancer research, helping breast cancer survivors, and comforting those fighting with breast cancer today. We considered each organization's rating on Charity Navigator and the NBCF scores on Charity Navigator were remarkable including a maximum score for accountability and transparency, 83.1% of revenue dedicated to program expenses, and a solid plan for growth.

We also felt strongly for the foundation's mission to save lives through early detection by providing education and mammography, particularly to women in need. The NBCF works with almost 100 clinics in all 50 states to provide free mammograms and diagnostic services to underserved women. Each sponsored facility is required to have the capacity to provide continued treatment to women after an abnormality is found or breast cancer is diagnosed.

How You Can Help

Your role is easy. Download the Jane Austen Widget from the Android market, use the "Share" buttons to share your quotes on Twitter and Facebook, encourage your friends to get the app. If you really like it, purchase the Jane Austen Widget Upgrade, the ads will go away and we'll donate the proceeds from your upgrade too.

The program will run for 1 week starting Sunday, February 12th, 2012 and ending Saturday, February 18th, 2012. Download the free application today and help us spread the word about the application and the promotion. The more people who have the free app, the more ad revenue we'll be able to donate. Don't buy the upgrade though until the week of Valentines Day so we can donate your upgrade too.

We hope to make this program a huge success because we're already looking into our next charity (hopefully promoting children's literacy with proceeds from our educational apps for kids). We're excited about this opportunity and with your help we can prove what a big difference a small business can make.

Mobile Magic Developers Co-founded by Autopilot Founder

Autopilot Consulting, LLC founder Patrick Caldwell has joined forces with long time friend and coworker James Brechtel to found Mobile Magic Developers, LLC.

James and I worked together for a few years several years ago and have been close friends ever since. It was an easy decision to get together and start developing mobile applications.

Mobile Magic Developers focuses on developing fun and exciting mobile products like educational mobile applications for kids, amusing home screen widgets, and mobile utilities. Autopilot Consulting focuses on … well consulting.

When your company is ready for your own custom mobile application, come talk to Autopilot about how we can help you wow your customers and outshine your competitors with a mobile application of your own. We'll apply our mobile experience as well as the talents of Mobile Magic to make sure you get the biggest mobile bang for your buck.

Mobile applications come in several varieties. Whether you're interested in native mobile applications or mobile skins for your web applications, we have the experience to get you there, save you money, and help you make the best decisions for your business, your customer, and your future.

Mainframe Migrations: Source Code Translation Tools

I was working on a mainframe migration project a while back to move an application from Adabas and Natural to Sql Server and C#. Software AG developed Adabas and Natural back in the 70's so I was very pleased to have some Software AG developers and managers on the team.

Shortly into the project, one of the managers recommended a product Software AG has that translates Natural code into C#. Much to the surprise of my teammate, I recommended strongly against the software. In fact, if you've read any of my other mainframe migration articles, you know I was recommending against a "migration" strategy altogether (a lift-and-shift my teammate called it) vs. a guided redevelopment strategy.

In the process of making my argument, I spent a fair amount of time positing that a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap. The customer did decide not to use the translation tool, but in the process of making my argument, my curiosity was piqued (and I do have something of a strong research background) so I put together the What Really Makes a Good Programmer Survey.
For an in depth discussion of the results, see my Results of the "What Really Makes a Good Programmer" Survey post.

For this post, we'll be looking primarily at these data:
If you can't view this chart, I managed to keep a copy of the What Really Makes a Good Programmer Chart from the legacy charts API.

Because most of my respondents were programmers, I like to look at the group averages to eliminate selection bias. I group the "traits" into the following three categories:

• Traits that contribute to being a good programmer
  1. Has good problem solving skills
  2. Learns from experience
  3. Interested in learning
  4. Passionate
  5. Sees the big picture
  6. Recognizes patterns
  7. Communicates effectively
  8. Tries new approaches
  9. Detail oriented
  10. Seeks help when needed
  11. Interested in helping others
• Traits that are nice to have if you can get them
  1. Fast
  2. Co-located
• Traits with no effect on programmer quality
  1. Has a college degree
  2. Has a computer science degree
  3. Cheap
  4. Has certifications

So, back to my original stipulation that "a source code translation tool is very little more than hiring a bad programmer because he's fast and cheap." While I still believe that statement to be true, when I conducted my survey, I asked the question in a more positive way so I can't actually claim that the tool is a bad programmer. I believe that it is analogous to a bad programmer; I just can't support that claim with these data. I'll explain why, but first, for the pedants, statements like "the community represented in my responses appears to believe that having good problem solving skills contributes to being a good programmer" will be abbreviated to "having good problem solving skills contributes to being a good programmer."

That being said, the data suggest that having good problem solving skills contributes to being a good programmer. If that stipulation is true, then the contrapositive must also be true that not being a good programmer means not having good problem solving skills. The inverse, however, is not necessarily true so I cannot claim that not having good problem solving skills contributes to not being a good programmer. I'll let you decide on the validity of that statement, but for the purposes of this post, I'll change my argument to, "using a source code translation tool is little more than hiring a programmer, who lacks the traits of a good programmer, simply because he's fast and cheap."

Think of a translation tool and go through the list of traits. The translation tool does not have problem solving skills at all, can't learn from experience and certainly isn't interested in doing so, has no passion, and doesn't even know what a big picture is let alone the big picture with your application in your environment. It may be able to recognize patterns technically, but it likely won't recognize patterns that are meaningful to your application. It's also unlikely that the tool will communicate effectively, create and try new approaches, or be interested in helping others.

I suppose you could say it's detail oriented (as long as the details are explicit and don't require problem solving) and it seeks help when needed (likely by way of errors). It's probably faster than a good programmer, though this may be negated if it's lack of the aforementioned traits produces less than useful results; that is to say as long as you don't need a programmer to fix the translation once it's finished the speed may be beneficial. Another good thing about the tool is that it's possibly cheaper than a good programmer (see previous caveat) and it's arguably well educated, but that doesn't seem to constitute a good programmer according to the survey.

Software development is an art almost as much as it is an engineering field. You hear many people talk of software craftsmanship. Allowing a craftsman to rewrite your application in the new framework from scratch using the old application as a guide will allow her to provide a unique expertise that translation tools currently don't have. She'll be able to apply a high level human analysis that the tool cannot.

For example, imagine there's a bug in the old code? The programmer can spot and fix the bug, but the tool will translate it into a prettier and newer bug. What if the old code is doing something unique to that language because the language doesn't have the features of the current language. A programmer can take advantage of these features and the tool cannot. A programmer will naturally analyze your business process as the application comes together and will make recommendations that may make your new application even better than the old one was in its heyday; the tool will not.

I can think of dozens of examples like this and I'm sure you can as well. Ultimately, I believe that allowing a tool to convert your legacy code into newer legacy code is a waste of time, money, and opportunity and if it doesn't cost more in the short run, it will cost more in the long run.

Elon University has #1 MBA Program

Elon University's part-time MBA program deserves a lot of credit for the success of Autopilot Consulting. Bloomberg Businessweek recently gave the Martha and Spencer Love School of Business (LSB) much deserved recognition.

In The Best Business Schools of 2011, Elon's MBA program was ranked first among competitors like UCLA, Carnegie Mellon, UC-Berkley, and Rice.

When I was looking for an MBA program to attend, I compared Elon's program with Duke University's and UNC's programs and was blown away by the comparative quality of the LSB program; the experience, friendliness, and helpfulness of the staff and faculty; and the extraordinarily low price tag.

Congratulations to the Elon LSB Faculty, Staff, Students, and Alum on a job well done.

Autopilot Consulting has a Logo!

At long last, Autopilot Consulting has an official logo! It's exciting to see this dream of mine coming together and now there's a face to the name.

I know it seems slow going. After all, I posted the announcement of Autopilot at the end of May and here we are in the middle of June and I'm just getting my logo together.

Well, it's true. I'm trying to get all of my ducks in a row in my spare time while most of my efforts are still spent trying to do a great job here at the Centers for Disease Control. We're not doing the same mainframe migration project anymore (but I still intend to finish that series someday). We've actually started doing some pretty interesting asp.net projects. I usually prefer mvc.net, but they focus on very rich user experience which is a good experience for a nuts-and-bolts kind of guy like me.

I noticed that not only are my blog posts becoming fewer and further between, they're also pretty non-technical. I promise to spend a little more time writing about the things I'm learning out there in the thick of it (and maybe I'll have a little advice from time to time for those who are thinking about going independent).

The Beginning of Autopilot Consulting, LLC

If you follow my blog, there are a few things I'd imagine you know about me:

1. I love writing software!
2. I love feeling like I'm improving the lives of my customers.
3. I love feeling like I'm doing my job well.
4. I love flying airplanes.

I Love Writing Software!

This is the very reason I got into the software industry in the first place. I get such a thrill out of a good days work programming. I love the challenge of solving problems as quickly, creatively, and efficiently as possible. A software engineer not only thinks about solving problems, but thinks about thinking about solving problems. Our craft is always changing and growing and it's great to be a part of this community.

Improving the Lives of my Customers

Whether I'm your employee, I'm a contractor for you, or you purchase my commercially available software, my goal is to find some way to improve your life. I've worked in paperless onboarding, business process automation, data warehousing, entertainment, e-commerce, insurance, pharmaceutical research, and academia.

There's one thing I've found in common no matter what kind of software I've been writing. That is that I find the most joy in my work when I can change peoples lives for the better. I like to think that some of the work I have done has bettered the human condition, and I am hopeful that my work will continue to do so.

Doing my Job Well

If I'm not doing my job well, I shouldn't be doing it. If I know I'm not doing a good job, I'm not satisfied with my work. If I'm not satisfied with my work, my customers can't be satisfied with my work. Sometimes you find yourself in a position where you can't meet your full potential. Other times, you don't know what your full potential is.

It is for this reason that I've decided to strike out on my own and form my own software company. I've been developing software for a long time working for various companies. I've decided that starting my company will allow me to touch the lives of more people.

I hope to continue consulting in the software industry while I build my library of commercial products including web based services and mobile applications.

Flying Airplanes

About the same time I got my first job with the word "programmer" in the title, I started flying airplanes. Since then I've flown tailwheel, multi-engine, upside down, through the clouds, and I can't get enough. One thing I've learned is that sometimes an airplane has so much going on, that you just can't do everything by yourself. That's why some planes require 2 pilots.

That's also why a lot of airplanes have autopilots. An autopilot is there to take care of going where you tell it to so you can focus on the important things. That's why Autopilot Consulting is here. We want you to focus on what's important &emdash; your business; we will handle the technology.

Together, we'll grow your business and improve the lives of your customers.

Do more. Work less. Put your business on autopilot.

Results of the "What Really Makes a Good Programmer" Survey

Back in September I wrote a blog post called What Really Makes a Good Programmer? My goal was to ask various members of the development community what traits they thought contributed to the quality of a programmer. If you haven't taken the survey yet, I'd recommend you do that before reading the results. Wouldn't want to bias your opinions, right?

If you have taken the survey, but haven't told everyone you know to take it too, go ahead and do that; I'll wait.

Now that we've covered that, let's get on to the analysis. As you recall from taking the survey, there were 17 traits and you rated them on a scale of Very Important, Important, A little Important, Negligible, and Completely Unimportant. In order to do any analysis, I had to take these ratings and convert them to numbers.

I decided that A Little Important was the baseline and basically represented a lack of opinion on the topic. These traits are the ones most people feel are nice to have but aren't requirements. Basically, a good programmer often has these traits but not having them doesn't mean you're not a good programmer. I recoded these values with 0 points.

Very Important and Completely Unimportant are the ratings that people used when they felt strongly about the item. That means that, respectively, the trait is either absolutely necessary to being a good programmer or the trait has no bearing on the quality of programmer you are whatsoever (or perhaps a little bit of a negative indicator). These values were recoded as 10 and -10 respectively.

The central values of Important and Negligible are what I call the non-committal values. You feel that they make a difference but they're not quite important enough to bank on them. I gave these a 3 and -3 respectively.

For example, "Has good problem solving skills" was rated Very Important and "Cheap" was rated Completely Unimportant. One could say, "someone with poor problem solving skills would make a poor programmer." On the other hand, you cannot say, "someone who is not cheap is a poor programmer." Of course, the contrapositive could also be stated that, "someone who is a good programmer is also a good problem solver." The contrapositive "someone who is a good programmer is also cheap" is considered by the community to be a largely invalid assertion.

"Fast" and "Co-located" appear nearest to the baseline. This may be due to the fact that many of the respondents didn't know what I meant by co-located. In any case, one might say, "a good programmer is a good programmer whether or not she's in the same building," or that "just because you're fast doesn't mean you're good and just because you're slow doesn't mean you're bad."

"Communicates effectively" and "Interested in helping teammates" are moderately rated in favor of contributing to being a good programmer while the two "college degree" items are rated moderately against contributing to being a good programmer. For example, you might know a lot of programmers who are great developers but who lack the social skills or interest to become good communicators or team players. As a result, you might say that good programmers often communicate effectively and help teammates, but some good programmers cannot.

You might also say, "Many people without college degrees, let alone a computer science degree, are great programmers. Thus, while having a degree is helpful, you can learn to be a great programmer without it."

Now, I feel it is worth noting that "unimportant" does not mean "negative." Just because most of the development community feels that having a CS degree or other certifications is unimportant to being a good programmer, it doesn't mean that the degree itself is unimportant. It just means that having the degree won't make you a good programmer; you still have a lot of learning to do.

With notes on the analysis out of the way, what do you say we take a look at some data? Currently, the new Google charts won't let me force the display of the categories, so you'll have to hover over the dots you're interested in. Here's a legacy chart you can look at if the javascript version below is unsatisfactory. The legacy chart has current data; it's just not as fancy as this one:
I'm sure you noticed there are two series in this graph. All Responses represents the average responses for all respondents; however, as you noticed in the respondent histogram that I have a dramatically unbalanced sample with programmers outnumbering all other groups combined by almost 3 to 1. To get a more accurate representation of the "general feel" of the community, I included the Group Average measure. This is the average of the averages. It's sort of the electoral college of informal research.

This chart demonstrates the average responses for each group:

Improving Search Performance when using SQL Server 2008 Encrypted Columns

This is a story of courage, honor, and data encryption. In addition to being something of a tribute to one of my favorite games of all time, I do feel I need to preface this post with a bit of a disclaimer.

Generally, when I describe this problem to someone, we go through pretty much the same conversation.

Why didn't you try Sql Server 2008 Transparent Data Encryption? Why don't you just update your ACL? Why don't you try this? Or that?

Well, sometimes the environment, either technologically or politically, precludes some of your better options. Ultimately, you have to go with your best permissible solution to solve the problem at hand. That being said, let's say you've been charged with securing your organizations PII. Your only option is to encrypt the column with SQL Server's column encryption.

You start looking at the impact this is going to have on your current and future applications. If you're like me, one of the first concerns you're going to have is performance. For example, let's say you have the need to search on the encrypted field. A good example of an encrypted field that you'll likely have to search is the Social Security Number.

In my original benchmarks, I found the following results:Knowing that I was going to have to improve this performance, I started playing with some searching alternatives. First of all, it's obviously much faster to search an indexed column (especially one that is clustered), so my first goal was to generate an indexed column.

Ryan McGarty (my best friend and a damn fine programmer) and I discussed and quickly ruled out a basic hash. Sure, it'd be fast to search an indexed column containing the hash value, but that opens you up to a relatively simple plain text attack (especially with a known set of possible values). I decided that you could reduce the threat and still speed up the search by using hash buckets a la hashtables instead.

I concocted a hash function that produced a reasonable distribution within the buckets. My good friend and esteemed colleague David Govek pointed out that an MD5 hash would produce a pretty effective distribution between buckets (with high cardinality data like the social security number. David was right and I ended up with this hash function:

CREATE FUNCTION [dbo].[GroupData] 
(
 @String VARCHAR(MAX),
 @Divisor INT
) 
RETURNS INT
AS BEGIN

  DECLARE @Result INT;
  
  SET @Result = HASHBYTES('MD5', @String);
  
  IF (@Divisor > 0)
    SET @Result =  @Result % @Divisor;
  
  RETURN @Result;

END

I created the test data just as I had before, except that this time I also added the hash bucket. I created a clustered index on the bucket and I changed my select statement a little:

-- original select statement
SELECT *
FROM PersonData
WHERE 
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

-- hash key select statement
-- @Divisor is the divisor for the modulus operation in the hash function
SELECT *
FROM PersonData
WHERE 
  SocialSecurityNumberGroup = dbo.GroupData('457555462', @Divisor) AND
  CONVERT(CHAR(9), DECRYPTBYKEY(SocialSecurityNumberEncrypted)) = '457555462'

Here are my results:

So, what of the known plain text attack then?

So, I don't want to gloss over the plain text attack issue. Given the possible set of socials, the hashes would be very unlikely to have a collision. Thus, for most people, you'd be able to get their socials easily by hashing every possible social and joining to that table. By modulo dividing the hash value, I'm able to evenly distribute social security numbers among a known set of buckets. That means, I can control the approximate number of socials in each bucket given my set of values. I generally aim for about 1000 socials per bucket.

For example, an MD5 % 100 has a possible set of values from -100 to +100. That's 201 buckets so if you have 2010 rows of data to hash, you'll have about 10 rows per bucket. The benefit is that you now only have to decrypt 10 rows to find the exact row you're looking for. The detriment is that you've narrowed your possible result set. Within your own data set, you'd have narrowed it to 10 possible plaintext values; however, given that these values are unknown, you then have to look at the set of possible values.

Social security numbers have a set of possible values less than 1,000,000,000. It's hard to say exactly how many of them are the possible set of values, so let's say we're using only those socials currently in use by living Americans. The population of the United States at the time of writing was about 312,000,000. As I said, I usually aim for about 1,000 records per bucket. If I have 1,000,000 rows of data, I would modulo divide by 500 (1,001 buckets). If you knew my set of values, then you'd only have 1,000 possible values to reduce. Given that you know when and where I was born, you could probably narrow it to 20 or so possible socials.

But, you don't know my set of values, so you really have 312,000 values to chose from. Even if you did know the first 5 digits of my social security number (based on my state of issuance and my date of birth), your set of possible options would be so large, you'd probably be better off just pulling my credit report and getting my social that way.

Thus, while it seems to introduce a weakness to plain text attacks (and with low cardinality data it would be an issue), in the case of social security numbers, I don't believe it to be a reasonable attack.

Overdraft Fee Survey

OK, I know I've been big on the surveys lately, but this one is really important to me.

As you probably know, there's been a lot of hoopla about the new overdraft fee regulations. Banks are no longer allowed to automatically enroll customers in what they call, "overdraft protection." To us common folk, we generally call them "overdraft fees" or "allowing you to spend money you don't have so we can screw you out of more money we know you don't have."

I started thinking about my history with overdraft fees and some things I learned from Dave Ramsey. I wondered, who really pays all of these overdraft fees that account for 38 billion dollars per year in revenue?

This survey intends to find out. At the time of writing, I had 84 responses and I need hundreds more. Please, take my anonymous survey and ask all of your friends to do the same. Here is the Overdraft Fee Survey: