A Glimpse into Evaluating an RFP/RFQ

I've chaired many Request for Proposals evaluations. Many people new to the process (or who have not been involved in a structured evaluation process) have questions before committing to be an evaluator. Here's a brief overview (which also tends to be my opening remarks when debriefing vendors on their proposals).

1) The written word is law - only what is actually written in the proposal is evaluated. No outside knowledge nor past history nor links to websites are considered. So if a vendor answers a requirement by saying "you have worked with us before" or "please see our "About us" page on www.ourwebsite.... will get a score of "arguably meets requirement". To withstand legal scrutiny, the process can only evaluate what was submitted in the proposal by the closing date. Personal knowledge is deemed subjective; websites could be changed after the closing date, so neither would be fair for the process. (A note to those in other countries - Canadian Contract Law deems the RFP itself is a contract, and as such, there are some limitations to the process in Canada - we'll discuss this in another post).

2) Only the criteria included in the Request is evaluated - anything other than what was asked for would be considered a hidden preference - I could cite a few Canadian legal cases where municipalities have made the mistake of not citing a preference for a certain type of material or methodology over another.

3) Scoring is not like school - generally scoring is based upon a pre-defined matrix (eg. 0-3, 0-5 or 0-10) where the middle number "meets stated requirements and any perceived risk is deemed acceptable". Higher than 2, 3, or 5 in the above samples are for "clearly meets requirements with no added risk" or "exceeds expectations with acceptable or no added risk". Just meeting requirements won't win a contract, nor provide the public sector with any value-add/innovation. The public sector has to answer to its public, therefore perceived risks need to be accounted for. In many debriefings, I've advised vendors to face up to any weaknesses in their methodology/experience and advise how they will mitigate those risks (whether real or perceived) - ignore the weaknesses and risks to the public sector, and they will be scored accordingly.

4) Demonstrate Experience, don't just state it - This tends to be one of the most common errors made that result in a low score. Stating "We've provided these services for 20 years" tells the evaluators NOTHING - but demonstrating "For Client X, our firm was hired to provide Y. Our role was Z, we outlined A, B, C; completed D,E,F, and the resulting outcome was GHI." It doesn't need to be pages upon pages of excrutiating detail, but give action statements of relevant projects citing role, responsibility, tasks, outcomes to demonstrate the experience.

These are just a few tips on the inside view of evaluations - mostly what I've had to repeat numerous times. I'll provide more in future posts!

Rants vs Ideas for Change

I stumbled across a Why I hate RFPs blog yesterday - not that it's the first rant I've seen on these things, but it did make for interesting reading. Especially if you read the comments in support of this rant.

Now I could easily minimize some of the 'issues' raised, i.e. if it's a public sector RFP, they have to disclose who the incumbent is or it is deemed a hidden preference; the number of copies issue is based upon taxpayer perception that government money is being wasted on helping businesses get fat contracts (not that the perception is correct mind you!)

I'm thinking I could write an article on why I hate RFPs too, from a buyers view - but ranting doesn't solve the issues. It's similar to non-voters complaining about how the country is run...but no-one is offering "solutions".

An article in CA magazine in 2008 offered a 'solution' to the RFP. (Supposedly we, as professionals in the PMAC Agreed with him?). But again, it doesn't look at the whole picture. I read it as giving a recommendation to toss the baby with the bathwater, i.e. turn this RFP world upside down and do it differently by dealing with a few select firms and only having ONE write an actual proposal.

I can't possibly address all the issues (from the buyer's view) with this article. However, I can agree on some good points for improvement - clear specifications being the number one point. However, in reality, if government offices don't openly publicize a procurement, someone runs to their MLA, MP and complain that government money is being wasted; the process is unfair because not everyone had equal opportunity. That is what's behind the public competition - people complain the wrong company got the business; companies complain that they didn't get an equal chance because their competition has a personal relationship with the Ministry. Requests for Proposals were meant to create an arms-length, LEGAL process to evaluate fairly, objectively, and transparently. Not that the complaints about nepotism and unfairness don't still occur - but ranting why RFPs are bad isn't solving things.

Alot of what is being 'complained' about is due to policy and law. Experienced buyers (i.e. not consumer purchasing, but actual business procurement) know how to get what they need within the realm of objectivity, fairness and transparency. There are plenty of examples of multi-stage processes where only the pre-qualified, best value, experienced service provider is given the detailed specifications to propose a solution to. There are examples of draft RFPs where the buying organization is looking for vendor input to come up with a process to select the best solution, without it being deemed unfair, or led towards a single solution; there are examples of 'competitions' that suit the service being requested (IE black box "Iron Chef" competition for chef services at Government House will find link to archived article and edit this post).

So, instead of ranting, why don't we have a debriefing on our processes and learn from each other. We as buyers can learn about what we didn't 'ask for' or was unclear, and the vendor community can learn where law, legislation and lobbyists need to be addressed. After doing this for a number of years, so far, I am learning more about different industries and how to measure them in an objective way, and I have yet to have an unsuccessful service provider tell me to toss my competitive process.

end of rant

Protecting the Process - no talking please!

Public sector Request for Proposals (and other competitive processes) state that all enquiries are to be in writing. However, this instruction does tend to be 'ignored' by many vendors, who for various reasons, NEED to speak to a real person. Some people politely state they cannot discuss the process; others can be downright rude; and yet others mistakenly chat things up. Strangely enough, people don't realize the reasoning behind this instruction and risks associated with ignoring it:

1) Telephone game - remember as a child the secret game where you whispered something into the next person's ear and then it travelled down the line to end up something totally different from what it started as? Wikipedia notes that the educational factor of that game is "It shows how easily information can become corrupted by indirect communication"1 Most procurements involve a committee or procurement team, so that enquiry then needs to be communicated to the others and an answer generated. It would be good to know your question isn't mutated!

2) Perception vs. reality - there is an adage that when it comes to media and the public sector - perception becomes reality - whether there are secrets being disclosed to a person on the phone directly or not, if someone discusses/answers questions verbally to one proponent, many will believe there was 'more to it'. As well, tone of voice, exact words wouldn't necessarily be replicated even if it was written down afterwards for distribution.

3) What is lobbying? There are some terms popping up in RFPs now that will disqualify a vendor for 'lobbying'. The definition given in some of these solicitation documents is "must not attempt to communicate directly or indirectly with any employee, contractor...about the project...unless otherwise permitted". This may not always be clear, but "must" becomes a mandatory criteria, and if anyone has "attempted to communicate" outside of the allowable method (in writing) they have a non-compliant proposal and would be disqualified from the process! If they don't get disqualified, then the public sector organization *may* hear from other proponents about accepting a non-compliant proposal and end up paying out $$ for breach of their process.

4) Interesting turn of events - In a complete turnabout, an unsuccessful proponent (let's call him ABC) complained that the winner (DEF) was being allowed to do something that ABC had been told would not meet the requirements of the program. Nowhere in our solicitation document had we said this particular item would NOT be allowed (it wasn't even mentioned), so wondered where he got this information - he "admitted" that he had a meeting with Mr. S and discussed the program and this was something that he based his proposal upon. YIKES! Luckily for the project, Mr. S. wasn't on the evaluation committee, nor had he been involved in the RFP process. In fact the RFP stated explicitly that all enquiries were to be in writing, and would be posted publicly on the website plus "information obtained from any other source could not be relied upon". So ABC couldn't argue the process was flawed...I believe he was going to sue Mr.S. directly for misleading him.

Lessons? Questions and Answers in writing can become a part of the RFP and clarify requirements; everyone will have the same information (words); vendors won't be disqualified for starting a conversation about the process; and staff won't be in a position where they may end up named in a litigation for a misunderstanding/misinformation!




PSBlog by Katherine Caughran is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 2.5 Canada License.