A better way to buy code. Use tests to buy code

by cashmonkey85 • 11 years ago

3 4 11 years ago

ColinWright2• 11 years ago

This is a lovely idea, but I'd like to share with readers an experience I had a while ago.

I was looking to outsource some coding, so I advertised and chose someone that seemed reasonable. I asked for some sample code doing similar things to what I wanted, they responded with some snippets that looked right, so we entered an agreement. I would provide some sample input and output, they would provide code that produced the right results. Once I verified that it did what was required, I would pay.

It was a problem domain that wasn't overly complex, and I thought it would be an ideal task to outsource. It was a lot more complex that the example I'll give here, but I want to give you a flavour of how it all went wrong. This is not the task, just something to explain how it went wrong.

I asked for code that would return "True" if the 2nd and 4th characters in a supplied string were the same, but otherwise all characters were different. So here are some examples of input and output:

    severity    True
    vagabonds   True
    unknowable  True
    unsnarled   True
    telegraphs  True
    robot       True

    faunas      False
    flimsily    False
    freighting  False
    rummaging   False
    mosaic      False
    shushes     False
    conflicting False

Remember, the actual task was described in full, was much more complex than this, and we were simply providing some example test cases.

What we got back was the equivalent of this:

    def test_string(x):

        if x == 'severity'    return True
        if x == 'vagabonds'   return True
        if x == 'unknowable'  return True
        if x == 'unsnarled'   return True
        if x == 'telegraphs'  return True
        if x == 'robot'       return True

        if x == 'faunas'      return False
        if x == 'flimsily'    return False
        if x == 'freighting'  return False
        if x == 'rummaging'   return False
        if x == 'mosaic'      return False
        if x == 'shushes'     return False
        if x == 'conflicting' return False

So to anyone who tries to specify their coding tasks by using tests - beware. There are coders who will strive to meet the exact letter of the requirements, and then demand payment.

For what it's worth, we never resolved this and eventually paid an agreed amount to make them go away. We took this as a reasonably expensive lesson about some of the potential pitfalls of out-sourcing.

cashmonkey85• 11 years ago

Hey, thanks for the example. This is definitely something we want to avoid with this platform and have thought of ways to mitigate it.

One way is we will allow the buyer to further test submissions as black boxes to see if the output matches their expectations.

The second way is with a rating system as extra information for the buyer and an incentive for the coders to write good code.

ColinWright2• 11 years ago

How do you avoid this scenario:

  * Customer creates job spec
  * Customer creates white examples
  * Customer creates black examples
  * Coder bids
  * Customer accepts
  * Customer sends spec
  * Customer sends white examples
  * Coder delivers code
  * Code fails black examples
  * Customer refuses to pay
  * Coder claims code meets spec

A better way to buy code. Use tests to buy code

Related Stories