Many CMS experts recommend that buyers create an elaborate scoring methodology when assessing proposals from vendors. This is supposed to be a way to carefully review each proposal while demonstrating to procurement that you have treated the bidders equally. I disagree with this approach . After a recent period of thorough research on best practices for selecting a CMS, I still wouldn’t recommend spending time scoring vendors. What if for some reason you did not want the bidder with the most points?
External consultants may not be the only ones talking about scoring. The pressure to develop scoring models may also come from your colleagues in legal or procurement; sometimes even from management. They may have used it during past projects in other industries and see this as the only transparent way to reach a decision about the best bidder. I still don’t recommend doing it; mainly as it undermines the possibility of freely deciding on the best bidder.
If you’ve created an elaborate criteria driven scoring system, it will inevitably be time-consuming to read each long proposal. Naturally, usability is critical, but how will you score it? And how about intangibles, such as having a local office. The answer to that is either a yes or a no, so how will that count?
Even if you’ve moved beyond the practical challenges involved in scoring, in the end you may easily end up with 8 bidders that score almost equally. This is not unusual and recently happened to one of our J. Boye group members. This resulted in procurement forcing them to have lengthy meetings with all 8 bidders, as they were not comfortable eliminating bidders with only a few points difference.
Based on our extensive work of advising CMS buyers over the years, we’ve never developed a scoring methodology. In fact, you can easily eliminate vendors without scoring, as in most cases many of the proposals will be weak on several of your requirements. Why then create the illusion of a scientific process? It is worth mentioning, that we also haven’t developed long lists of requirements in the RFP, but rather work with shorter scenarios in narrative.
We simply encourage buyers to develop a simple evaluation matrix rating. List vendors and your requirements in a table. Then briefly describe the response to each scenario by each bidder. For a multi-language scenario this could be as short as “OK” or “Good enough”, but could also be a bit longer, e.g. “OK, except for right-to-left text”. Doing it this way will save you quite some time, as you do not need to reach agreement on how many points a response merits. Also, by looking at the evaluation matrix you will easily be able to identify which vendors you should take a closer look at.
Do you agree or disagree? Either way before sending out the RFP, make sure you have internal agreement on how you will rate the proposals, so you know how to make your decision!
Thanks to Laurence Hart (@piewords) and Tony Byrne (@TonyByrne) for input.
Jeff Cram August 3rd, 2009 15:23
Could not agree more Janus. Great post. This line specifically hit it on the head for me:
“…In most cases many of the proposals will be weak on several of your requirements. Why then create the illusion of a scientific process?”
We use six “fit factors” to help align CMS selection to business requirements.
http://www.cmsmyth.com/blogs/cms_myth/archive/2009/05/30/does-your-cms-fit.aspx
There is a methodology that goes into how that is applied to a selection process, but i’d love to get your feedback on it.
Jeff Cram
ISITE Design
Seth Gottlieb August 3rd, 2009 15:23
I very much agree that elaborate scoring methodologies hinder rather than help the selection process. They tend to draw attention away from the solutions and give the illusion that one is making a quantitative mathematical analysis.
After reaching a short list of products, I recommend using a simple strategy of listing doubts of each of the alternatives and isolating aspects of the product that would constraining or difficult to live with. This post has more detail about the process: http://www.contenthere.net/2009/03/doubt.html
Mark Morrell August 3rd, 2009 15:23
Janus,
I strongly agree with you. I have not led but been involved in hours of agreeing, assessing and weighting complex scoring measures for a new search engine, document and content management systems. This was some years ago though!
I will make sure I avoid it next time and follow your advice. You’re right. Some proposals don’t need any scientific measures. You can clearly – and quickly – see do not meet your requirements.
Mark
Philippe Parker August 4th, 2009 15:23
Janus,
The biggest issue with your approach is that in my experience, procurement forces sponsors down this route. But there are reasons for this.
Firstly, the people doing the purchasing will at some point get audited. Auditors will demand to see evidence of how large capital expenditure was assessed. This isn’t just true in the public sector: boards want to know why a technology was selected.
Secondly, if the implementation goes wrong, or people struggle to adapt to new ways of working, the CMS inevitably gets the blame. In turn, the people who picked the CMS get the blame and stakeholders challenge the selection process. I’ve worked with more than one client where that has happened and it leads to 6 months of acrimony and obstructiveness. You have to prove to those not directly involved in the selection process that you have given the products rigorous consideration.
Thirdly, it may be the case that some consultancies are still coming up with feature-based matrices where one product scores 78% and another 83% (as Seth points out), but scoring doesn’t need to be that way. If you’ve already pre-qualified your suppliers properly and develop concrete scenarios as a basis for evaluation, scoring can provide a really useful framework for assessing tools. I don’t think it should be the final assessment method, but I do think it has value.
I hate to be standing alone in this company, but I don’t think that there’s a cut and dry answer to this. It is true that scoring gives the illusion of scientific process; but not scoring gives the illusion that there was no due consideration.
Philippe
Neil Lancia January 18th, 2010 15:23
As a vendor, I find that it is more important for the client to make a selection based on their own experience using the CMS rather than relying the responses a vendor has provided in an RFP or during the interview process. Vendors are hungry, and many are willing to say anything to get a deal closed. My recommendation is to:
1. Eliminate weak proposals quickly
2. Carefully qualify the remaining vendors using a table as you’ve suggested and select your top 2 or 3
3. Ignore product demos. Instead, take the time to evaluate the top performing CMS platforms yourself…its all about the user experience. Anyone can make a CMS look great in a demo, but you need to make sure that it performs under well for your unique business needs.
J. Boye | Intelligent use of spreadsheets in vendor selection May 18th, 2010 15:23
[...] the analysts at Real Story Group, I’ve previously advocated against the usage of scoring methodologies. My advice was instead to focus on scenarios and develop a simple evaluation matrix rating (in a [...]
Scoring Victim July 28th, 2010 15:23
I agree completely with Janus. There are too many intangibles that simply cannot be scored, and so they end up being ignored instead. I went through a process recently where one of the best vendors lost out by half a point (!) to the lowest cost vendor. Sure, they met the requirements, but the product is clumsy, difficult to use, and unlikely to grow with us. I wish our procurement folks were reluctant to rule out vendors based on a few points difference – they weren’t even reluctant about a half point difference!
Noz Urbina January 17th, 2011 15:23
Because I’m a consultant both buyers and sellers batter me with feedback and input about the selection process and our advised ways of going about it. I agree strongly that the illusion of science in the selection process is often a bit of a paper tiger specifically erected by those who want to protect themselves.
We also advise a simple matrix which is greatly standardised to save re-inventing the wheel, then extended on a customer by customer basis adding specific special requirements and then by adding a weighting multiplier to all items.
The team doesn’t have to collaborate on a ranking. Each stakeholder ranks according to their opinions (assuming each stakeholder is considered equal) and then they’re averaged. The result is indicative, not conclusive, and supplements the less tangible content in the evaluation documentation.