Reliable data: crucial for benchmarking

apples-orangesAccording to Capers Jones article “sources of software benchmarks“, there are up to 24 sources for software benchmarks. Most of them have substantial amounts of data and are very useful. Unfortunately there are very few providers that allow their raw project data to be obtained by the industry. This implies that many benchmarks are not transparent because the underlying project data cannot be assessed. Using these sources for benchmarking might be comparing apples with oranges.

NESMA is gold member of the International Software Benchmarking Standards Group (ISBSG). This is an independent not-for-profit organization that collects industry data for projects (new developments, releases, enhancements) and maintenance (maintenance and support). The ISBSG data are ‘open’ and can be licensed by anyone in their raw (MS Excel) format. NESMA advises to use ISBSG data for benchmarking, as transparency is key.

The ISBSG defines some metrics like e.g. Project Delivery Rate (PDR) hours/FP, Cost/FP, Defects/FP, Size growth, schedule conformance, budget conformance, Speed of delivery (FP per calendar month).

This in order for the industry to be able to use the same definitions and to be able to benchmark to other industry data. The ISBSG collects data from the industry and grows, maintains and exploits two repositories: ‘New developments and Enhancements’ (over 6000 projects) and ‘Maintenance & Support’ (over 1200 applications). These repositories are an invaluable source for any organization that uses functional sizing, as it enables them to:

  • estimate projects in technologies of which they do not have their own data;
  • reality check existing estimates to industry averages;
  • benchmark completed projects to understand the competitiveness of the organization;
  • assess the reality value of RFP proposals in order to exclude the unrealistically cheap offers;
  • build estimation models;
  • analyze effects of specific cost drivers, such as development platform or team size;
  • etcetera.

See also the Benchmark tooling.apples-apples