Feb 22 2010

SCRUM-tastic Tuesday - How about some poker?

In my last post SCRUM-tastic Tuesday - Estimation guesswork, I proposed a simple approach to estimating work and managing expectations. After the post I got a few emails about the value of the Delphi Estimation method. Specifically, the writers mentioned the anonymous collection of estimates as the only real way to see how close or how far apart people see the size of a piece of work. Here is a basic summary of how it works:

  • An assigning manager approaches two or more developers separately and describes a single unit of work
  • Each developer estimates the work as they understand it and submits their estimates anonymously to the manager. Neither developer knows about the other and the manager does not know which estimate came from which developer
  • The manager reviews the estimates. If the estimates are more than 15-20% apart the manager calls a meeting to discuss the difference between the estimates
  • After the meeting, the developers re-estimate
  • This process repeats until the estimates are within threshold (15-20% in this case)

This sounds complicated but it does work. This forces people to explain why they saw things so differently, often leading to the discovery of assumptions and/or risks observed by one that was not considered by the other. This approach also addresses the issue of peer pressure. We have all seen it... Put the team in a room and ask them for an estimate and often you will only hear from a few people. Unless those few people are doing all the work, you may have estimates that don't reflect what the actual assignee can commit to. This kind of communication can go a long way to ensuring your estimates reflect real concerns. 

In a Scrum environment, the problem is time. The aforementioned process takes a lot of time and orchestration, which Scrum shops don't have. There is however another process that strives to capture the value of the Delphi method while prescribing a process that is fairly lean with respect to artifacts and time. Its called Planning Poker.

For me, this process gets to the heart of the matter, which is forcing full participation. The process is simple:

  • Bring all the devs together. Just like Delphi, explain each unit of work and ask the developers to estimate using numbers provided on a set of cards
  • Everyone shows their estimate card at the same time allowing everyone to see each others cards
  • The facilitator will be able to any see big differences immediately and ask each person to explain the assumptions and/or concerns reflected in their estimate
  • After a round of discussion, the team estimates again, repeating if needed

This process can go very quickly and provides all the same value as the Delphi method, although the artifacts created in this process often have a much lower fidelity. Strictly speaking, the cards are not critical... You could just as easily use yellow stickies, as long as the value choices are the same. Personally, I like using one of the many IPhone or Windows Mobile apps for this (Ex. Space Flight Orange).

While the previous example focused on dev shops this process is a great way for team managers and product owners to do rough estimation ahead of the planning sessions. This allows each party to get a sense of which pieces of work are larger or smaller and can guide selection of priorities. In this case, they are not trying to nail-down detailed estimates; instead; they are trying to define a rough approximation (often in days). This is also known as a Rough Order of Magnitude (ROM) estimate or a ball-park estimate.

Technically speaking, there is a little more to the process so feel free to read up on it. Whatever you decide to do, just remember, the most important part of getting reliable estimates from your team is getting people to communicate... chickens and pigs.


Feb 19 2010

What the heck is a torn-read?

If you've written any high performance asynchronous applications you may have found yourself putting up Memory Barriers to avoid those costly thread locks. More commonly known as Memory Fences, this technique can allow you to create lock-free read-modify-write operations which can give you that little extra bit of performance that some applications need. To be fair this is an advanced technique so if you are embarking on this sort of thing be very careful and do lots of testing.

The problem with using lock-free approaches is that you now have to deal with Atomicity on your own and this may not always be as straightforward as you might think. Lets look at a couple simple examples of what is and is not Atomic.

Single reads and writes are always atomic:

Int32 number = 123; /* single writes are atomic */
number.ToString(); /*single reads are atomic */

Unary operations are another story since they consist of two operations (one read and one write). As a result, using them in a lock-free environment can lead to unexpected results:

number += 2;

This means that another thread or process could alter the value of 'number' in between its read and write. When this happens you will certainly be left scratching your head wondering how 1+= 1 could possibly equal 1294.

64 Bit types present an even more subtle problem. Single reads or writes of 64 Bit types 'may' not be atomic the same way their 32 Bit counterparts are. Thats right, I said 'may' not... Here's the issue. To store a 64 Bit value in a 32 Bit environment the runtime needs two separate memory locations and consequently there are two separate instructions to read or write a value. This creates a vulnerability similar to unary operations called a torn-read. When this occurs one of the memory locations reflects a value from one thread or process while the other reflects a value from another thread or process. In the end, this will cause you no end of pain and is next to impossible to debug.

Thread or Process A writes 5,000,000,000 which is stored in two 32 bit registers as follows:

10101010000001011111001000000000 -- 10101010000001011111001000000000

Thread or Process B writes 10,000,000,000 which is stored in the same two 32 bit registers as follows:

11010100000010111110010000000000 -- 11010100000010111110010000000000

A read by either could result in the following combination:

10101010000001011111001000000000 -- 11010100000010111110010000000000

In 64 Bit environments this is not an issue because they only need one memory location and can read and write with only one instruction.

So how can you use a lock-free pattern and avoid this problem on 32-bit platforms? There are a couple ways to deal with this, but really only one worth mentioning... Interlocked.

Interlocked.Increment(ref number);

This class provides a number of methods to help ensure interaction with types such as these are done atomically.

So what is the moral of this story?  If you are going lock-free... Test, test, test...

Tags: ,

Feb 15 2010

SCRUM-tastic Tuesday - Estimation guesswork

Whether your using Scrum or some other Agile process, at some point you will be asked to Estimate your work. So what is that... as if you didn't already know: 

The ability to accurately estimate the time and/or cost taken for a project to come in to its successful conclusion.
- Wikipedia (http://en.wikipedia.org/wiki/Estimation_in_software_engineering)

The truth is; unless you are building templated software like toys on an assembly line; good software estimation is generally some combination of experienced guesswork, historical evidence, dumb luck, and black magic... If you are using processes that require higher fidelity artefact's like CMMI, SW-CMM, etc... then you may be familiar with estimation practices like COCOMO, Delphi or others. If you are working on very large projects, or you need to do long-range estimation, these techniques can make a lot of sense.

In Scrum, you have roughly one working day for the business to articulate its needs and priorities for the next iteration. During this same time the Team has to figure out what they will commit to delivering, how they might do it, who in the team will do it, and how long it might take. With so little time, traditional techniques just aren't practical so the estimation priorities need to be shaped accordingly. A key practice in any project is proper management and communication of expectations, risks, etc... So with that in mind, here are the tenets of our estimation process:

  • The artefact should be quick and easy to produce...  Keep it simple and reuse your tracking system if possible, letting the assigned task list represent your teams estimate (Ex. TFS, Microsoft Scrum, VersionOne, etc...). It could be as simple as a spreadsheet or even an email to all concerned
  • Communicate where the team has a lack of confidence in their estimates and why. Managers can use this to get a sense of how much contingency;if any; they need to add to an estimate to ensure customers are not too disappointed if things go the wrong way
  • Communicate what risks, resource shortfalls, or impediments the team is concerned about. This would include things that could place the teams ability to deliver commitments in jeopardy. Generally, these are items that need to be addressed by Product Owners or the Scrum Master
  • Communicate any assumptions the team is relying on as the basis of their estimates. This would include things that; if not true; could dramatically change the estimates

This may sound like a lot but this can be done very easily... Here's a sample of a spreadsheet that demonstrates each area:

As you can see, this would take very little effort, allowing the team to spend more time talking about more important details. On the other hand, this clearly communicates the points that shape expectations.

Even if you get the estimates correct, there is still one element that can cause you to estimate incorrectly and technically speaking this should be done before you estimate. What is it?.. Defining your teams Capacity. Essentially, we need to make sure we don't over-commit the team or any one member of the team. This process is pretty straightforward and does not require much more then a white-board discussion. Here are some things that should guide this calculation:

  • Will any member of the team be taking a vacation or any other planned time away from the team?
  • Are there any company holidays?
  • What is the team's productivity rate? This is the number of hours per day each team member is expected to be making progress towards the completion of committed tasks. This tries to achnowledge time spent on the phone, answering emails, attending meetings, training, lunch, etc... No estimate should extend beyond this number

Hopefully these tips will help make your estimates a little more successful.


Feb 11 2010

Hiding from the debugger

Category: Tips and TricksJoeGeeky @ 18:56

There is no doubt the modern debugger has made developers lives so much easier and without it... well... lets just not think about that. More specifically, the Microsoft Visual Studio debugger is really a beautiful piece of engineering kit. Having praised it, now lets talk about the pain it can cause us. Tools like this have become so advanced, providing so much detail that sometimes it can actually hamper our efficiency or in some cases give away more information than we are comfortable with. Consider the following examples:

  • When you have deep inheritance chains, auto-generated code for things like designers, or methods with high Cyclomatic Complexity, stepping through code can take considerably longer than you might like and if segments of the code are already trustworthy, you may wish to have the Debugger ignore them when stepping through code
  • When dealing with classes that have large numbers of private variables & properties that reflect the same values this can clutter up the variables windows makings your debugging experience less then optimal
  • When developing applications to be consumed by third-parties, you may wish to hide select members from the debugger. This could be a desired part of your obfuscation process or simply to make the debugging experience less confusing for consumers of your API

Lucky for us, these and other scenarios can be dealt with easily by using facilities provided within the System.Diagnostics namespace.

Lets start with hiding code from the Debugger:

public void MyHiddenMethod()
{ /* Do something */ }

By simply appling the attribute DebuggerHidden when stepping through code, any time this method is encountered by the debugger, it will skip over it. This will occur even if the method or a descendant has a breakpoint set.

Hiding select members can be just as easy. Lets say we have a simple variable that we want to hide. Before hiding the variable we can see that it is clearly visible in the debugger.

private string _internalMember;

To hide this member, all we need to do is apply the DebuggerBrowsable attribute:

private string _internalMember;

When debugging you will see that the member is now missing from variable windows such as Locals.

While writing this, I tried to think of some basic rules for when to apply this. The truth is, there aren't any "rules". This is something you will have to apply when it feels right for your situation. Keep in mind that applying techniques like this can lead to hiding important information, on the other hand it can make debugging experiences more productive. With that said, remember to never apply these techniques until you are absolutely sure the code being hidden is trusted. Take a look at the System.Diagnostics namespace for more tricks you can use to make you debugging experience better.

Tags: ,

Feb 8 2010

SCRUM-tastic Tuesday - Kick the Master from the circle

Category: Intellectual PursuitsJoeGeeky @ 19:00

In my post SCRUM-tastic Tuesday - Time to pay the piper, I talked a bit about the consequences of people missing meetings. In this article we will take a look at a situation where attendance; at least as part of the daily stand-up meeting; can keep the team from communicating effectively. I'm talking about the Scrum Master. In order to understand where the problem lies, lets first revisit what a Scrum Master is and is not.

What the Scrum Master is; or should be:

  • Someone who ensures all process stakeholders adhere to the Scrum process tenets
  • Someone who helps prevent the Team from losing momentum during a Sprint
    • Removes Impediments
    • Facilitates communications between the Team and resources outside the Team
    • Shielding the Team from external interference/distractions
  • Someone who schedules all meetings for major process milestones (e.g. Planning, Daily Scrum, Review, and Retrospective)
    • Communicates Agenda's
    • Ensures attendance of appropriate stakeholders
    • Facilitates meeting progress

What a Scrum Master is not; or should not be:

  • The team manager (operational, technical, or administrative)
  • Someone who makes delivery commitments with the Team

Ideally speaking, the stand-up would look something like the following image. Everyone standing in a circle with the Team communicating with one another and the Scrum Master taking note when someone has an impediment. 

But we don't all live in an ideal world and what often happens is the Scrum Master becomes a management figure. When this happens, the daily Scrum Meeting starts to look like the following image. Team members can stop talking to each other and start reporting status directly to the Scrum Master.

Once this pattern develops, communication within the Team become less effective. You have to remember that the Team has made a commitment to deliver at the end of the Sprint not the Scrum Master. To be successful, the Team needs to be able to communicate clearly to one another and self-manage.

The development of this pattern is understandable though. In part, the problem lies in the name Scrum Master. The word "Master" can create an implicit promotion for the person in that role, and this can disrupt the balance between the Teams role and the Scrum Masters role. Many organizations can make this problem worse by putting Team Managers in this position. While Scrum Masters should be empowered to make some decisions, this should not be confused with running the Team.

A better name might be something like Scrum Concierge, Expediter, or anything that re-enforces the role of someone focused on the continued momentum of the Team. Team Managers "can" fill this role, but in my experience it very rarely works. The Scrum Master has to be willing; eager even; to let the Team delegate work to him/her and the normal Manager/Managee roles just don't work that way.

If you find that your Scrum Master has received such a promotion, try having the Scrum Master stand outside the circle during the daily Scrum Meeting. They can still facilitate the meeting and interject when needed, but this will force the Team to start talking to each other again. This is a simple practice which you can implement on and off as needed.



Feb 5 2010

Find it faster amongst the collection

Category: Tips and TricksJoeGeeky @ 04:00

Here's a quick tip; pun intended. We have all had to find something in a Collection or List and in many cases it probably looked something like the following:

var tokens = new List<string>();
if (tokens.Contains("Foo")) { /* Do something */ }

Pretty simple, but if you are looking for multiple values you might have something like this:

if (tokens.Contains("Foo")) { /* Do something */ }
else if (tokens.Contains("Bar")) { /* Do something else */ }

or even this...

foreach (var searchToken in searchTokens)
    if (tokens.Contains(searchToken)) { /* Do something */ }

All these work fine, but under a large load or with very large collections you may find that these patterns are incredibly slow. There is however another approach that is substantially faster and most people I've spoke to never even knew it was an option. The alternative...

IEnumerable<string> values = tokens.Intersect(new[] { "Foo", "Bar" });

This is a nice simple call, even if you are only looking to match a single value. Elegance aside, the important bit is the math...  Is it really faster? Your results may vary somewhat but here are some numbers; in ticks; showing the difference between Contains and Intersect in a large collection. The size of the collection is not really important, as the point lies in the relative difference between each approach... Alright... it was a collection with 21.5 million entries, and each approach searched for the same five values.

  • Using .Contains five times: 1,015,560 ticks
  • Using one .Intersect call with five search values: 71,739 ticks

Keep in mind this was done a simple desktop to your results may vary... As it turns out a .BinarySearch(x) run five times returned the same values in 230 ticks, but this approach has limitations that you will have to read about on your own. Do your own testing, and you may find this a useful alternative and might make the difference between success and failure in a production scenario.


Feb 1 2010

SCRUM-tastic Tuesday - Time to pay the piper

Category: Intellectual PursuitsJoeGeeky @ 20:20

In my post SCRUM-tastic Tuesday - Your time is up, I talked a bit about meeting rules. In that article, I covered most of the stated rules, although one rule in particular was left un-addressed.

Attendance is enforced
- Me.

We all know those people who; for some reason; just can't arrive to any meeting on schedule. There are always "reasons", but at the end of the day it causes a ripple effect that inevitably prevents many meetings from being as effective as they often need to be. Think about it... We have all seen it time and time again.

  • The start of the meeting gets delayed
  • Agenda's have to be sped up or items removed to make up the lost time
  • Details have to be repeated for the benefit of those who were late
  • Questions asked by late comers that were already answered
  • Etc...

At some level this can be considered disrespectful, especially when you are dealing with someone who clearly does not believe in the meeting ever taking place. Scrum addresses this by limiting the number of prescribed meetings, balancing meeting time with the need to be productive outside a meeting environment. With that said, the process also requires the correct stakeholders be present, prepared, and engaged for those few meetings that are prescribed.

  • Sprint Planning (one every Sprint cycle)
  • Daily Scrum (daily 15 minute Stand-up Meeting)
  • Sprint Review/Demo (one every Sprint cycle)
  • Sprint Retrospective (one every Sprint cycle)

No more is this true than the Daily Scrum meeting. As it stands (pun intended), this meeting is only 15 minutes long and has a very lean 3 question Agenda. When people are late for this meeting it can really be disruptive and prevent the meeting from ever being successful. When dealing with the chronically late personalty all you can try and do is retrain or re-motivate them to arrive on time. There are lots of carrot and stick approaches, but only one has really proven to do the trick in my option... Fine them!.. Wait, wait, hear me out...

Without getting too Freudian on you, people need to have an indelible image of consequence left on their minds. People tend to pay attention when you threaten their pocket book. It doesn't take much, just one dollar [$] (or one Euro [€] for you Europeans) and they will definitely remember. There is something about that moment when you have to reach for your wallet or purse and hand over even a small amount of money. For some people this is an absolutely painful experience. Here is how you might structure this practice.

  • Make sure your daily stand-up is set for a time that is reasonable for everyone. Some people are early risers, while others are not
  • Pick a god clock... This clock is considered to be the "correct" time. (For example, we use the phone system time)
  • Get buy-in from the management team. Having their support will make the transition easier
  • Make the Fine small. 1 dollar for example
  • Make them pay publicly during the Stand-up... If they are going to cause a disruption by being late, make it a memorable one
  • The Scrum Master collects all fines and enforces the practice
  • At the end of the Sprint give the collected funds to a charity of the teams choosing or buy something fun for the team
  • Even the Scrum Master and chickens pay if he (or she) is late
  • Don't compromise... late is late, even if the moon fell on the motorway, or Klingon's stole their car battery

In the beginning, you will absolutely get resistance from someone. This will usually take care of itself and that person will often become a champion for the rules ensuring everyone else follows the rules to. Did you catch that? Read that sentence again. It is peer pressure.

Is all this too harsh? I don't think so. After all, time is money. Seems only fair you pay when you waste it. By the way, I have paid my fair share too Embarassed.