In the aftermath of the Gomery report, many organizations have started using a "fairness" role with their large complex or politically sensitive procurements. The primary objective of involving fairness resources is to provide objective evidence that the process has been run in a fair, open and transparent manner. Having the role start with the process keeps the process ontrack and saves time later, versus the standard waiting for complaints and having someone audit the process 'after the fact'. I've done both roles a number of times, partly to deal with the perception of bias towards incumbents; alleviate union concerns of outsourcing, ongoing vendor relations issues, etc. Thanks to close contact with National Education Consulting Inc & the Legal Edge publication means keeping up-to-date with the latest/greatest court decisions.
Even in cases where RFPs are better developed (in terms of how evaluations would occur) with an incumbent in place, it might be useful to request a fairness advisor/monitor/auditor play a role. Generally, the procurement department itself doesn't hire the fairness role (as it is reporting on their conduct/process!) Personally, I believe a fairness role needs to be appointed by an assurance department or a higher authority. There are even 'little things' that should be considered in the course of 'fairness' such as having all the evaluators sign a disclosure agreement (much like how the vendors are to provide within their proposals). I sat in on an evaluation process where the chair did NOT request this, and the evaluators went through a dozen grant application proposals and it wasn't until they faced the last proposal, one evaluator disclosed his son worked for the firm and would recuse himself from that evaluation - which was good to do, HOWEVER, he was involved in the scoring/discussions on all the competitive proposals up until that point! That is still a conflict of interest, even if he wasn't going to play any role in the scoring of that last proposal, he influenced the other scores. The organization changed their practices for future evaluations, and dealt with the issue by awarding to 'all' grant applications received - eliminating the risk of breach of process. I've been in other processes where the evaluations had to be 'tossed' and a second evaluation had to occur.
Is this more bureaucracy? Red tape to an already onerous process? How do you think the fairness can be 'above reproach'?
Comments/discussion appreciated
Showing posts with label Evaluation. Show all posts
Showing posts with label Evaluation. Show all posts
Tuesday, May 3, 2011
Tuesday, June 8, 2010
Proposal Evaluations: Scoring
Image via Wikipedia
0 = No information provided
1 = Meets few requirements
2 = Meets most, but not all requirements
3 = Meets all requirements
4 = Meets and exceeds some requirements
5 = Meets and exceeds all requirements
You can find publicly available examples in the BC Ministry Guide to the RFP Process. There are similar guides in other public sector organizations (you just need to know where to look!)
The first step as an evaluator is to score each proposal on your own (individually). At this point it's just the individual stakeholder/end-user/subject matter expert reading/scoring without discussion with anyone else - I find it easier to read a proposal once, then score it on my second reading, but everyone has their own way of doing this. They can print the evaluation sheets and score by hand, or enter their scores into an evaluation spreadsheet, and save each as a separate file.
For the public sector, anything written down in the proposal or evaluation sheets IS FOI-able (it is deemed to be leading to a decision). Now on the other hand, I advise evaluators to not hold back from writing notes! I beg them to PLEASE, keep notes on why they scored things a certain way or if information is missing or unclear in the proposals. I also ask them to reference proposal page numbers in the comments so that our consensus discussion will move along more quickly, and we can easily find the information if ever asked to review the evaluations later.
As well, I advise evaluators that to keep things in context, when scoring, always consider how we will explain this score to the proponent in a debriefing? It's much easier to do debriefings if we have diligent notes in the scorebooks eg. "On pg 19, the vendor stated they did X but didn't demonstrate it" "On pg 24-27 they provide a case study of similar development work and the outcomes of the learning" (demonstration is done through action statements, and/or case studies).
As noted in a separate blog post, after the individuals finish scoring, we meet for consensus scoring.
Labels:
Evaluation,
Public sector,
Request for proposal,
scoring
Subscribe to:
Posts (Atom)