Two Sides to Every Story...

Time and again I have end-users tell me "I want to score the reference checks to be sure we get someone that does good work"

My response ends up being "reference checks are not relevant (to score at least)".  I don't believe references are the ultimate test of whether someone does good (or bad) work.

Anyone who submits references is ONLY going to send in their best client references. Would you  'knowingly' submit a reference that was unsatisfied?  So for scoring purposes, you end up having references all getting high scores. 

As well, the people contacted for a reference check seldom provide 'critical information' as they want to avoid being held responsible for someone NOT getting work because of how their comments are 'scored'. 

Generally we use references for verification of what is stated in the response, and if anything is 'off' we use it as means to disqualify (as long as we've stated that purpose within our RFP!).

A alternative approach that a colleague uses is to state within the RFP "we reserve the right to contact client references NOT named within proposal for the purposes of..." that may help you get you a fuller picture, but again, there are always two sides to every story - I have a wealth of examples where people were dissatisfied with a contractor, but never gave feedback/managed the contractor during the course of the work.  So a 'bad' reference may have never given the contractor an opportunity to fix the issue.

Good or bad, the reference checks are only giving you a piece of the story...