Project Management in Practice

Beyond the alphabet soup of PRINCE2, MSP, MoP, PMBOK, ITIL, Agile

Proof of Concept, Prototype, Pilot, Agile … confused?


Proof of Concept, Prototype, Pilot, Agile … confusedEvery industry sector has its own jargon that users or system developers use. IT is no different and plenty of words are used interchangeably that different stakeholders take to mean different things. This often makes for difficult conversations down the track. My colleague Nathan Heazlewood recently put in some effort in defining some of these. Thought he did an excellent job, well worth sharing. Hopefully it comes in handy in avoiding mismatches of expectation.

Proof of Concept (PoC): a small exercise to test a discrete design idea or assumption. An example of a POC is testing whether one technology talks to another.

Prototype: a system that tries to simulate the full system or at least a material part of it. May be completely discarded should a production version follow.

Pilot (or trial): A Pilot uses the full production system and tests it against a subset of the general intended audience. The reason for doing a pilot is to get a better understanding of how the product will be used in the field and to refine the product.

Agile development: An ‘agile’ application is one that is built to production quality in small incremental steps, some of which may be put into ‘production’ before the complete system is complete.

Production: A production system is the full system suitable for use by the complete target user group with the level of robustness and functional and non-functional agreed within the associated contractual documents.

Proof of Concept

A Proof of Concept (POC) is a small exercise to test a discrete design idea or assumption. An example of a POC is testing whether one technology talks to another. To be deliberate, a POC should clearly state what it is to be proven and to what degree. For example, it may not be enough that a message can be transferred from system A to system B if the complete solution calls a specific level of performance and reliability. You may need to test under certain conditions. POC’s often take the outward form of a simple “Hello World” but that may not be enough if the system needs to support an extended character set.

Characteristics In Scope Out of Scope
  • In the interests of minimising time and cost a Proof of Concept will relate to only small parts of a complete system.
  • A POC will utilise a development or test environment where possible to minimise change authorisation overheads of using a production environment.
  • Non-functional considerations will generally be ignored unless specifically requested.
  • Assessment of one or more important aspects of what might be involved with a production system. Only aspects specifically stated as being ‘in scope’ will be regarded as part of the ‘Proof of Concept’.
  • Functionality
  • Performance
  • User data for retention
  • Re-usable code
  • Scalability
  • Concurrency/capacity
  • Usability
  • Backup of data or code
  • Security
  • Backwards or forwards compatibility
  • Multi-browser or multi-OS support
  • Reliability
  • Availability
  • Documentation
  • Monitoring

Prototype

A prototype is a system that tries to simulate the full system or at least a material part of it. Prototypes are used to test the viability or usefulness of a system or subsystem. The only objective of a Prototype is to get a defined group of intended users of the projected system in front of the prototype and he/she/it (the user might be another system) should be able to visualize the experience of using the complete system. A prototype may provide some re-usable components that can be re-used in a pilot or production version, however it is also possible that it will be more efficient to re-do most or all of the system.

Characteristics In Scope Out of Scope
  • In the interests of minimising time and cost a Prototype will be assembled quickly with a level of quality intended for demonstration purposes only.
  • Functional aspects may be mimicked utilising ‘short-cuts’ or ‘work-arounds’ to minimise time spent on the prototype. These should be replaced with robust measures for a production system (e.g. ‘local copies’ of data may used rather than connections to data sources, components will not be fully documented or commented, testing may be minimised etc).
  • Non-functional considerations will generally be ignored unless specifically requested.
  • A prototype should only be tested for a defined period after which it will be ‘switched off’ while a production version is established.
  • Data used in a prototype should be ‘test’ data and will be disposed of at the end of the prototype demonstration period.
  • Components of a prototype may be completely discarded prior to building a pilot or production version.
  • Functionality
  • Usability
  • Backup of code
  • User testing
  • Limited user familiarisation
  • Performance
  • User data for retention
  • Re-usable code
  • Scalability
  • Concurrency/capacity
  • Backup of data
  • Security
  • Backwards or forwards compatibility
  • Multi-browser or multi-OS support
  • Reliability
  • Availability
  • Documentation
  • Monitoring
  • Full ‘factory’ or ‘unit’ testing
  • Training

Pilot (or Trial)

A Pilot uses the full production system and tests it against a subset of the general intended audience. The reason for doing a pilot is to get a better understanding of how the product will be used in the field and to refine the product. Maybe there are some open ended questions about scalability that only a live audience using the product can answer. You may consider prepping your pilot group to evaluate the full breadth of the application and give them channels to provide feedback.

Characteristics In Scope Out of Scope
  • A system as robust as stated within contractual documentation.

 

  • Functionality
  • User data for retention
  • Re-usable code
  • Usability
  • Backup of data or code
  • Security
  • Limited documentation
  • User testing
  • Training for pilot personnel
  • Other considerations As stated within contractual documentation (e.g. Statement of Work)

 

  • As stated within contractual documentation (e.g. Statement of Work)

 

Agile

An ‘agile’ application development is one that is built to production quality in small incremental steps.

Characteristics In Scope Out of Scope
  • Incremental releases of components of a system to a level of robustness as defined within the Agile documentation
  • Quality/robustness is intended from the outset, however only for a limited part of the overall system to begin with, with additional functions added over time.
  • As stated within contractual documentation (e.g. Statement of Work)
  • As stated within contractual documentation (e.g. Statement of Work)

Production

A production system is the full system suitable for use by the complete target user group with the level of robustness and functional and non-functional agreed within the associated contractual documents.

Characteristics In Scope Out of Scope
  • A system as robust as stated within contractual documentation.
  • As stated within contractual documentation (e.g. Statement of Work)
  • As stated within contractual documentation (e.g. Statement of Work)

Credits: Nathan Heazlewood, who adapted it from http://www.contenthere.net/2007/03/poc-prototype-or-pilot-when-and-why.html

Image Credit: austinstar.hubpages.com

6 responses to “Proof of Concept, Prototype, Pilot, Agile … confused?

  1. Brendon December 2, 2012 at 9:05 pm

    Very informative guide, very good reference. Thank you!

  2. Pingback: Considering a Proof of Concept or Proof of Technology? | the vendor chronicles

  3. Zia Ahmed November 30, 2013 at 12:17 am

    thanks Shoaib & Nathan. Informative.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: