The vendor-neutral analyst firm Real Story Group (formerly known as CMS Watch) has recently published a few commentaries urging buyers involved in vendor selections to basically stop using spreadsheets. Complex scoring spreadsheets can certainly add to the complexity of a selection process, but are spreadsheets really always bad as an aid when making those important decisions?
Like the analysts at Real Story Group, I’ve previously advocated against the usage of scoring methodologies. My advice was instead to focus on scenarios and develop a simple evaluation matrix rating (in a spreadsheet). This can provide a visual overview which makes it easy to spot issues and key differences. In my experience this is a huge help in deciding which vendors to continue the conversation with.
In short, the problems with a detailed scoring methodology is that it comes with substantial risk; It can lead to the selection of an overly complex system and time invested in agreeing on scoring and updating spreadsheet, far from matches the value gained.
I caught seasoned CMS veteran Tom Wentworth from Ektron on camera a few weeks ago as he was filling out yet another spreadsheet as a part of a selection process. From the vendor perspective, this is a part of the daily routine for those involved in sales. Needless to say vendors have quite some experience filling out these and only rarely refuse to do it.
I’m a big fan of keeping things simple; yet vendor selection is a complex process with many potential pitfalls. It should be possible to make a good decision and complete the process in a month. If you do, however, decide to get lost in long and complicated scoring tables in the world of Excel, it could take much longer.
Jan May 25th, 2010 12:53
While I can understand the need for caution when using spreadsheets when selecting possible vendors as part of an RFI (request for information), I think the following should be noted:
- Yes, spreadsheets provided by vendors can be slanted to focus on questions that lean towards their products strengths… but
- Spreadsheets remain the most popular tool used and understood by many of us.
- Spreadsheets can be complex, but they can at least be used to measure and score quite accurately the functionality you desire in a content management system. All too often, other methods (from Magic Quadrants to detailed analyst reports) can be even more complex and unclear in how they draw their conclusions. Only exception in my mind is the ForresterWave which incidentally also uses spreadsheets so you can see exact scores they give and their weight
- The spreadsheets from the vendors are more intelligent that perhaps we give them credit for. The RFI spreadsheet from SDL Tridion for example gives clear advice on how to use the spreadsheet, also mentioning the pitfalls of using it. More importantly, it advises on focusing on your business goal so that the functionality you seek is aligned with your goal.
- Business goal focused spreadsheets seem awfully close to the scenarios you advise on.
So rather than advising to stop using spreadsheets, perhaps Real Story should consider evaluating the RFI spreadsheets that exist out there. There are nearly plenty of them about, and frankly I can’t see businesses refraining from using them. So how about it, Real Story? You evaluate the content management players, so why not their RFI spreadsheets?