Blog

What is a “Proof of Concept”?

Proof of ConceptWhen selecting a new CMS you definitely want to see the system in action before you sign the contract. A "Proof of Concept" (PoC) is often used as an opportunity to get a detailed demo, but there are many different opinions about what a PoC actually is. Let me clear up some of the confusion.

In our report, Best Practices for Selecting a CMS, we define a PoC in this way and argue that allowing 1 or 2 days should be enough:

A proof of concept is the stage in the process at which you invite a selected vendor to demonstrate the validity of the concept they have outlined in their proposal. Up to this point in the process, most has been on paper. Now it is time to see things in action. This is also an ideal opportunity to meet the vendor team and verify whether the chemistry is right.

Here are some other definitions of a web content management PoC to show that the term can be used in different ways:

  • CMS Watch (from the Web CMS Report): "(...) ask your list of vendor finalists to come in to your shop, install their packages on your hardware, and develop some sample templates and workflows that make sense to your team. (...) Note that, whether the proofs can be done concurrently, a proof-of-concept step could add a month or more to your selection cycle."
  • Jed Cawthorne (industry-expert and past J. Boye conference speaker in his feedback when 'beta testing' our report): "... neither a one or even a two day visit is what I would refer to as a "Proof of Concept". (...) I would consider a PoC to be something like what we got involved in at [Organisation]. We got [Vendor] to come in and build a portal environment for us, we went on some training courses and we evaluated it for 3 months (we paid for all this of course!) after which the [Organisation] decision was to not go down the portal route - that to me is a true Proof of Concept."
  • Seth Gottlieb (vendor-neutral analyst in the article POC, Prototype, or Pilot? When and Why): "A Proof of Concept is a small exercise to test a discrete design idea or assumption."

The PoC is an important step in the selection phase. Nevertheless, it is our opinion that you shouldn't spend too much time on it. You simply need to find out whether the system and the implementation team live up to your main requirements.

The process outlined by Jed resembles what we refer to as a scoping exercise - which is the next step. A scoping exercise usually takes 1-4 weeks and is carried out before the actual full-blown implementation with one vendor only. This vendor will have been carefully selected on the basis of their answers to the request for proposal and the (short) PoC. As a buyer you pay for the scoping exercise. One important advantage of going through this phase is that it is still possible to pull the plug if the vendor doesn't meet your expectations. This reduces the risks involved significantly.

This is a reminder that as a buyer, you have to work with - and stick to - a strict and transparent terminology when engaging vendors. They might not define the concepts in the same way you do, so in order to avoid misunderstandings, you should provide a short, concise glossary of the terms you use.

Thanks to @adambindslev, @cmswatch, @jedpc, @markmorrell, @sigurdmagnusson & sggottlieb for valuable input.

Peter Sejersen

Peter Sejersen worked at J. Boye from 2008 - 2011. He can be contacted via LinkedIn.

3 Responses to “What is a “Proof of Concept”?”

  1. Ahhh semantics ! What do we really mean when we say “PoC” ?

    We have just had an interesting discussion within my current organisation on what are the definitions of Proof of Concept, versus Prototype, versus ‘Pilot’ – our use of these labels may change depending on our organization and our context.

    However, I still say my example quoted above is truly a Proof of Concept, because after the work was undertaken, we went back to the CIO and the steering committee, who looked at the results and said “In our opinion this organization is not actually mature enough to make the most of the portal software” – so in fact our work could be said to have dis-proved the concept, and stopped us going further and wasting more money on it.

    Having said that, I agree with Peter’s definition of ‘scoping exercise’ above, its what you do once you have chosen your supplier and technology platform, so in my previous example we would have perhaps kept the PoC setup and worked with it and the vendor to fully scope the complete implementation (i.e. is connecting the ECMS to the portal in or out of scope ?)

    I guess the main thing is: caveat emptor, or buyer beware! Ensure you go through some process, however it is labeled to ensure the resulting deployed system actually meets your requirements.

  2. I think there are a number of elements to a PoC:

    1. During the selection process
    You should definitely have all vendors you’ve short-listed come in and show off their products, addressing not only the issues you’ve raised but also showing off their unique selling points, suggesting things that perhaps you haven’t thought of. I t think clients should positively encourage vendors to sell to them.

    2. Once you have a preferred solution
    This is the time to do a chargeable Proof of Concept. Get the vendor to come in and install their system and run it with a project group, ideally with stakeholders from across your organisation. The PoC must prove something that you think should be easy to do, perhaps that you do well already, and something you suspect will be difficult, such as single sign-on via your user directory. That way you will have proved that the riskier user-focussed and technical aspects of your implementation will be smoother to roll out.

    3. The proof is in someone else’s implementation too
    Finally, it’s a good idea to see someone else’s implementation before you sign the contract. Go and visit the vendor reference and ask them what they found tricky in their implementation. Put some caveats into your contract schedules around those issue.

  3. David Hibbs says:

    What Jed says in his first paragraph is an outstanding point. “What are the definitions of Proof of Concept, versus Prototype, versus ‘Pilot’ ? ”

    Prototypes and pilots can certainly stretch for months and sometimes even years. These almost always require payment as well due to the time commitment.

    A PoC of 1-2 days, on the other hand, seems more of a litmus test than a real chance to learn whether a solution is usable.Given the size and [potential] scope of a CMS solution, this may only be enough time to install the base CMS software!

    Also, consider whether you will learn anything in 2 days usage. Perhaps you will, but you must compare this to other software you use daily. After 2 days of using MS Word, could you use all of its features? Odds are high that even after many years of using MS Word, your knowledge is STILL not complete! Such is the nature of a CMS solution. You will have a few features you use a lot, and many more features that will require time to use and understand.

    If you want to keep your cycle short, you MUST have your team ready to experiment with the solution–and keep them focused–once it is set up. This means that considerations must be made for their normal workload! This marks the single largest obstacle most organizations face in a PoC, prototype, or pilot — personnel simply don’t have time to try out a solution and give a full assessment of whether it fits their needs (let alone their desires)!

    Plan at least a week or two to try out your new software. 2 Days is completely unrealistic.

Leave a Reply

Most popular posts from our blog
August 12, 2009 by Janus Boye
Selecting the right CMS is not an easy task with; there is in excess of 1,000 vendors in the very dynamic CMS…
February 16, 2010 by Janus Boye
All modern CMS vendors claim to be capable of delivering content to mobile devices. Some even offer additional modules to make the…
March 21, 2011 by Janus Boye
You may be impressed by some of the features during a CMS product presentation, but in reality many of the features that…
Recent comments
April 8, 2014 by Bertrand
I would include the Apache Sling website in this list ( http://sling.apache.org/ ) as it's what powers AEM. Understanding the…
April 8, 2014 by Scott Liewehr
Great resource, Janus. And thanks for the mention.
March 15, 2014 by John
Their site may not even be a CMS but more of a custom built web application. Also, how do you…