We also advise a simple matrix which is greatly standardised to save re-inventing the wheel, then extended on a customer by customer basis adding specific special requirements and then by adding a weighting multiplier to all items.

The team doesn’t have to collaborate on a ranking. Each stakeholder ranks according to their opinions (assuming each stakeholder is considered equal) and then they’re averaged. The result is indicative, not conclusive, and supplements the less tangible content in the evaluation documentation.

1. Eliminate weak proposals quickly
2. Carefully qualify the remaining vendors using a table as you’ve suggested and select your top 2 or 3
3. Ignore product demos. Instead, take the time to evaluate the top performing CMS platforms yourself…its all about the user experience. Anyone can make a CMS look great in a demo, but you need to make sure that it performs under well for your unique business needs.

The biggest issue with your approach is that in my experience, procurement forces sponsors down this route. But there are reasons for this.

Firstly, the people doing the purchasing will at some point get audited. Auditors will demand to see evidence of how large capital expenditure was assessed. This isn’t just true in the public sector: boards want to know why a technology was selected.

Secondly, if the implementation goes wrong, or people struggle to adapt to new ways of working, the CMS inevitably gets the blame. In turn, the people who picked the CMS get the blame and stakeholders challenge the selection process. I’ve worked with more than one client where that has happened and it leads to 6 months of acrimony and obstructiveness. You have to prove to those not directly involved in the selection process that you have given the products rigorous consideration.

Thirdly, it may be the case that some consultancies are still coming up with feature-based matrices where one product scores 78% and another 83% (as Seth points out), but scoring doesn’t need to be that way. If you’ve already pre-qualified your suppliers properly and develop concrete scenarios as a basis for evaluation, scoring can provide a really useful framework for assessing tools. I don’t think it should be the final assessment method, but I do think it has value.

I hate to be standing alone in this company, but I don’t think that there’s a cut and dry answer to this. It is true that scoring gives the illusion of scientific process; but not scoring gives the illusion that there was no due consideration.

Philippe

I strongly agree with you. I have not led but been involved in hours of agreeing, assessing and weighting complex scoring measures for a new search engine, document and content management systems. This was some years ago though!

I will make sure I avoid it next time and follow your advice. You’re right. Some proposals don’t need any scientific measures. You can clearly – and quickly – see do not meet your requirements.

Mark

After reaching a short list of products, I recommend using a simple strategy of listing doubts of each of the alternatives and isolating aspects of the product that would constraining or difficult to live with. This post has more detail about the process: http://www.contenthere.net/2009/03/doubt.html

“…In most cases many of the proposals will be weak on several of your requirements. Why then create the illusion of a scientific process?”

We use six “fit factors” to help align CMS selection to business requirements.
http://www.cmsmyth.com/blogs/cms_myth/archive/2009/05/30/does-your-cms-fit.aspx

There is a methodology that goes into how that is applied to a selection process, but i’d love to get your feedback on it.

Jeff Cram
ISITE Design