Definition of Done

Marcello is considered a brilliant programmer by the team, willing to tackle any task is set upon him, he writes lots of very complicated code in little time. "It will be done in a couple of days", he likes to declare right before diving into his magic. Few days later you can find him typing effortlessly on the keyboard, immersed in his thoughts: "A couple more days, it's going to be very cool".
Some time later, he finally announces triumphantly: "It's done!". The new code is immediately committed to the main line ready to be tried out by designers, eager to cut their teeth with the new shiny piece of technology they were impatiently waiting for.

The new code crash the build.
This story is far too common in our world, but i wouldn’t blame the programmer, often very passionate about his work, so willing to please the hungry designers. The pressure to deliver features is often enormous, bugged and untested features are released, more time is spent fixing past features than developing new ones. And pressure piles up.
I blame the lack of a clear  Definition Of Done, that is able to define without ambiguities when a feature is ready to be deployed to the rest of team and go into production.
The final goal of the Definition of Done is to increase the rate at which working features are produced by the programming team in the long term.
The most common Definition Of Done that I've encountered in the Industry (and often used myself) is something similar to:
It's done when I wrote the code and it works on my machine.
This Definition Of Done is far from being unambiguous: in fact it means different things for each member of the team. To begin with, it's not clear what we mean here by "it works". Have I tried any possible condition in game? Or it works just in my testbed and I'm assuming it works in the game as well? Has someone from QA done a play through? Has he smoke tested the entire game or just a level? Has someone from the design team tried to use the feature? Does it actually do what the designers need? Do all automated tests pass? Is there any automated test at all?
And what about code quality: can code be changed easily if something doesn't work (and it will not work) or if something needs to be modified later (and it will need to be modified)?
Too many open questions without clear and unique answers to make this definition safe for the team. When a feature is "done", the content creators or other engineers can work on it with confidence, knowing that the chance of crashing or not doing what was asked for are minimized. Not zero. Just minimized.
A more useful Definition Of Done is a checklist of activities that needs to capture, more or less closely, the following concepts:
Uniqueness. A feature is "done" at the same stage of development regardless of who has been working on it.
Automation. Criteria can be checked automatically by tools, impact on actual production is minimal.
Quality. Code quality and implementation quality are guaranteed.
Agreeability. The entire team has to agree and stick to the Definition of Done, including designers and management.

The best defense is a good offense

 

My students might be moving their first careful steps in the dangerous lands of programming, but the problems I’m forcing them to face week after week are nothing short than “difficult”, even in code that is seemingly simple and straightforward. This week we were working on a sprite engine in C# (XNA), adding some code to select different sprite animations to play, where an animation is defined as an array of images. The sprite keeps track of the index of the current image being used in the current animation. Pretty straightforward…. crash.

Oh crashes happen, it’s part of the game, and we quickly found out the source of the problem:

Texture2D GetFrame(int index) 
{   
  return frames[index];    
}

An exception, IndexOutOfBound, was being thrown when a new animation is selected: the current index is being maintained by the Sprite class, and it’s not guaranteed that the new animation contains at least the same number of frames of the previous one. GetFrame() can, potentially, receive an index that is out of bound.

Solution: Make sure GetFrame() doesn’t fail if the index is out of bound.

Something like:

Texture2D GetFrame(int index) 
{   
  if (index > frames.Count)     
     index = 0;   
  return frames[index]; 
}

This is a very common approach to the problem that in my opinion is not actually solving it, merely hiding the dirt under carpet of an if guard. Surely, it “fixes” the bug, but at what cost: some added complexity (it’s an if, one more branch and one more unit test to write in an automated tested environment), that hides away the source of the bug; the system can still be in an inconsistent state. We are just trying to keep going in the presence of a blatant fault.

The more stable solution, I explained to the students, is to fix the source of the inconsistency, in this case either having one separate index per animation or, more simply, reset the frame index to zero when a new animation is selected: an animation is guaranteed to always have at least one frame, the code is thus safe, it doesn’t need a check in GetFrame() (one less branch) and it won’t crash… ever.

Unless we have an animation with no frames.

This is precisely the behavior I want: if there’s an animation with no frames, I want the application to crash and tell me if it’s in an inconsistent state that I can not ignore. If it crashes, I have to fix it.

Fail early, fail loud

This simple example shows my students a broader approach to software construction: offensive programming versus defensive programming. Instead of trying to prevent a crash by writing lots of defensive code that checks anything that can possibly go wrong (but obfuscates meaningful and useful code in a see of ifs), I prefer writing less code, but make it so execution stops as early as possible in the presence of inconsistent state, invalid preconditions, broken invariants.

In my experience this approach leads to less and simpler code, easier to modify and with less crashes.

An obvious drawback of offensive programming might be a live system, that is used by clients (content creators?) while it’s being developed: ring a bell? It’s exactly how we work in the Game Industry. Although I’m convinced that it produces less down time in the long run (by producing better and less crash-y code), it’s hard to convince an artist staring at a call-stack that this is a Good Thing™.

I’m again very open to suggestions here, how would you tackle the problem of writing lean and offensive code versus making the content creators experience pleasant in the presence of a (undoubtedly more rare) show stopper?

Some ideas:

• User friendly crash handler
  
• Automated unit, functional and smoke testing before release to minimize crashes
  
• Build staging

I played a lot in the past with unit and functional testing, especially in the form of asset testing to make sure that assets are sanitized before entering the system, with very promising results that led to fairly stable daily release cycles; something very much appreciated by content creators. Still that show stopper is hard to sell…

This post is also available on AltDevBlogADay.

A trip to Berlin

I’ve been teaching game programming at the University of Darmstadt for a couple years now: as they say, teaching something helps you clarifying it in your head. They also say that who can do something does it, who can’t teaches it

Last week I called two “volunteers” as usual to write some code at the main computer in front of the rest of the class, gave them some freedom to choose a solution for a little problem we were having in our XNA game. The solution they came up with was good and, then, they generalized it, writing some more code, and then some more code, and then a little bit more code.

I asked why.

We might need it later

I replied.

So next weekend you want to go to Berlin, spring is finally coming, time to enjoy a bit of sun, it won’t last long here in Germany. You evaluate all the available options, you may take the train (even if they are so expensive here), or go by plane which is not much more expensive than the train and takes less time, or rent a car which would give more freedom but costs a bit more. You discuss it among yourselves for a while, weighing pros and cons against the cost of each option; the train will be around 200 euro per person, same as the a plane ticket, while renting a car would cost around 100 euro per day plus petrol. You finally choose and you buy a car for 5.000 euro. Why? You might need it later. Maybe you will like Berlin so much that you want to travel there every weekend, or you will need the car to go to Amsterdam instead, or even Or maybe you will not do anything like that, and you will spend ten times more now to go to Berlin than if you just hopped on a train…

 

In our daily life we face this kind of tradeoffs and, with material things, we very often rationally choose to spend less now for something we need now, rather than spend more for something we might need later. I’m ignoring for this post the well proved fact that decisions are not always taken rationally (for a good book on the matter look here). In software we usually take the alternative route, we might need it later, we are willing to spend more now for something that more often than not we will not need later. We do it so often that we gave it a name: Over Engineering.

I believe that this peculiar behavior stems from the fact the cost associated to each line of code we write is not directly evident, it’s not money leaving our pockets. It’s also difficult to estimate precisely, which leads to the false perception that generalizing a solution will actually save time. It will not. Each line of code has a cost over time in terms of maintenance and added complexity, that is paid by the team day after day: if it doesn’t solve a problem that we have now, it’s a sunk cost. It requires a huge amount of experience to foresee that “we might need it later” is actually going to be true, and it also requires having gone through the same problem at least few times, which is not always the case in our Industry: we are on the bleeding edge of technology, uncharted territory, where things are complex by nature. Adding more complexity that is strictly needed is not going to help our cause.

Half jokingly with my students, I showed how they could solve the problem with half the number of lines of code, which will probably mean something like half the number of bugs at the end of the project if the idea is applied consistently.

You Aren’t Gonna Need It

Undeniable logic

One of the biggest mistakes I used to make in my career in the Industry was to think that a logic argument, well reasoned and with plenty of data to back it up was enough to convince reasonable and clever people, the kind of people I work with. An example is my favorite topic (no, it’s not commenting code), automated testing and unit testing: we all know that writing unit tests is a Good Thing™. We all do, right? There’s plenty of evidence and my argument for writing unit tests before even writing code is very solid. I was naturally expecting that after some resistance due to various reasons, the vast majority of my co-workers would eventually give up and embrace my reasoning, if not flat out try what I was advocating. But discussing automated testing here is not my goal now: my point is that very few people could actually see and understand my point. How frustrated I had been before realizing that it was all my fault.
It took years to understand that the undeniable logic in an argument is not enough to convince someone of its validity: it’s as important and most often even more important how the argument is expressed, how it’s told and explained.

John is a very clever programmer, extremely sure of himself, who likes to drill into the details of a problem; his method is tried and true, and he wants to give something else a go by himself before incorporating in his work. He likes to picture a problem and see a solution. I approach John and explain him that according to several studies writing Unit Testing  makes code more reliable and reduces cost of ownership for a code base, so he can focus on the general problem rather than fixing small bugs; doesn’t it sound correct? John thinks “bah” and moves on.
Trisha is a very clever programmer, who likes to keep herself informed by reading about how others in the Industry have solved similar problems in order to possibly produce something better for herself. She likes to think about the big picture rather than wasting time on irrelevant details. She wants to feel a problem before touching the right solution. I tell Trisha that if she tries Unit Testing for herself she will find that her code will be even more reliable, simpler and easy to change when she needs to change the details of the implementation, which is a result that she can immediately see after only few weeks. Trisha politely smiles at me thinking “bah” and moves on.
Things are not as simple as in my example, but we can immediately draw some interesting conclusions from it: in both cases what I told to John and Trisha is true and verifiable, but I failed to understand how they want information to be presented to them, which is radically different! John doesn’t care that others find Unit Testing useful, he wants to try for himself, while Trisha doesn’t want to try something if others haven’t already established it’s viable. John wants to see an argument not hear it, Trisha wants to feel it not see it.
Interestingly, addressing a group of people makes it even more difficult because different personalities are likely to be present, how about:

Writing unit tests before code is proved by different studies to be effective, which is something that you can see by yourself in just few weeks of trying it out, either on some small units or in a big problem that sounds difficult to tackle. You can easily feel the advantages or come to ask me for advice if you encounter problems.

How does it sound?
This realization makes leading a group of people even more challenging: now it’s important not only to have a solid and well thought-out vision, but also to understand the nature of the people we work with, how they want information to be presented to them, how they like to reason and respond to problems. Failing to communicate our vision is our mistake as leads.

Post also available on AltDevBlog.

No Comment

When I interview a new C++ programmer, my favourite question is not about const-correctness or virtual tables, but: "Do you comment your code?". I still ask about virtual tables, but asking about code comments always springs some interesting discussions and gives me a good insight about the level of care the aspiring programmer puts in crafting his code to not only make it correct and fast, but also to make it easy to work with. I witnessed puzzled faces wondering if that was a trick question (yes it is), and one unlucky candidate pushed his luck to a very bold "which good programmer doesn't comment his code!". Me.

More and more love to Stereo-3D in CryEngine [ITALIAN]

The interview is in italian, talking about Stereo-3D in CryEngine and my experience in the industry. The snowstorm in Frankfurt prevented me from flying to Milan, give up? No, I will have nothing of this nonsense! Skype solved the problem.

Straight from my living room… Enjoy.

VideoGamesParty.it

Also you can notice from the previous video that I’m getting slimmer and slimmer. Thanks to the guys at VideoGamesParty.it for their help.