Hands on Experience Counts...doesn't it?

I've heard of some scary 6 week, multi-stage hiring processes - resume (documenting education, experience, achievements) then a phone/pre-screening; an initial interview; a written test; and sometimes further interviews to test the ability/competency of the individual to ensure 'the best fit'. 

When seeking advice of consultants, advisors, experts, etc, are you putting them through the same scrutiny? (or even a fraction of that scrutiny?)

In the private sector, I saw far too many managers buy into glossy brochures & sales pitches to mean "these guys know their stuff", only to find out too late, "these guys have never worked in this area before".

People may complain, but for public sector projects (over a certain threshold), an RFP requires 'applicants' to document & submit their relevant Experience, Education & Capability for evaluation.  The RFP should outline minimum requirements to meet before considering approach/methodology and price.  In many cases, the end-users want someone who has 'done this before' so they aren't paying someone to 'learn their business' (taxpayers wouldn't be too impressed either).  The biggest issue I find, however, is too many RFPs state the requirement as "experience providing similar services", or "providing X services to organizations of 'similar size & nature', which unfortunately are too vague.

What would you define as "similar"?

• Would you hire a consultant who has experience in organizing a "cocktail & cause" party for 200 guests as having 'event management experience' for your charity's annual major fundraiser event?
• If you are in a research organization, would you consider someone who has written successful IT services proposals the same as someone who has written & received foundation grant funding?
• If you want a website for your organization, do you want the expertise of a branding firm, or the newly graduated student who knows all the 'latest & greatest IT Tools'? 

There are many individuals & businesses with great expertise in their industry, and time and again, I see many attempt to cross into other industries so they can expand their portfolios. Some have great success applying their knowledge to new, expanded areas...then there are the ones who "know what they know" but "don't know what they don't know" and are unwilling to consider the differences between industries/niches.  If you don't define your requirements for experience (let alone ask about hands-on experience)...will you be happy with the expertise you receive?

How are you scrutinizing the experience of your advisors?