UTip #3

IFPUG news announcement

 

But what is the accuracy of high level FPA compared to a detailed FPA?

Now that IFPUG published uTip #3, the users of the IFPUG method now officially got the green light to use size estimation methods that can be used  earlier in the software project lifecycle than the moment of a completed functional design. In uTip #3 the methods High Level FPA (In Nesma often called ‘global FPA’ or ‘estimated FPA’) and Indicative FPA are described. They are indeed very helpful methods to estimate the size of an application (or project) and many commercial organizations and governments worldwide have been using these methods for many years. In fact, in practice functional documentation is very rarely complete and detailed enough to carry out a detailed FPA as early in the software project lifecycle as it is needed to estimate the project. This is actually an argument made by many project managers for not using FPA, as they incorrectly assume that an FPA measurement can only be carried out on a complete functional design, which is too late in the project lifecycle to be useful for them for estimation purposes.

Especially with the industry moving away from the traditional development methodologies towards the more agile methodologies, functional documentation that is complete and detailed enough to carry out a detailed FPA becomes more and more rare, so one has no choice but to adopt an estimated method in order to be able to measure/estimate functional size.

Of course, with any estimation of anything … one should ask oneself what would be the accuracy of the estimation and what is the uncertainty that is introduced because of the estimate. The good news is that there have been studies published in which the accuracy of the High Level FPA method and the Indicative method compared to a detailed FPA were studied. In Sogeti Netherlands for example, the department Sizing, Estimating & Control (SEC) presented a paper on the Software Measurement European Forum in Rome (Italy), in June 2009, in which the accuracy loss and the effort savings on the use of High Level FPA and Indicative FPA of 42 projects was studied compared to detailed FPA measurements.

Although the Nesma method was used for this study, the results of that study should be comparable to the IFPUG method as well, as there are very little differences between the two methods in practice nowadays.

The paper, including the measurement data, is available on the Articles page. The main conclusions of the study are listed in the table below:

MethodAccuracyAccuracy lossEffortEffort gain
AverageAverage
Detailed100%100 norm hrs
High Level*98.5%1.5%67 norm hrs33%
Indicative83.5%16.5%18 norm hrs82%
The High Level FPA is also called ‘global FPA’ or ‘estimated FPA’ in the Nesma method

It turns out that using the High Level FPA method on average results in only 1,5% accuracy loss, which is usually not really a big deal early in the software project lifecycle. The indicative method is on average somewhat less accurate, but as this method can be used even earlier in the software project lifecycle (when only the number of logical files are known or can be determined), this method is very useful to create an early ‘rough order of magnitude (ROM)’ estimate of the expected functional size.

The effort hours column shows normalized hours instead of real hours. This means that the estimated method takes on average 67% of the time to measure compared to a detailed FPA measurement. The indicative FPA method takes on average 18% of the effort of a detailed FPA measurement.

 

About the authors

Harold van Heeringen, Edwin van Gorp and Theo Prins are software metrics consultants / software cost engineers for Sogeti Nederland B.V. They regularly publish on international conferences about practical aspects of software measurement. Most of their articles can be found on the Articles page.

A blog post represents the personal opinion of the authors
and may not necessarily coincide with official Nesma policies.
Share this post on:

6 Comments

Leave a Comment
  1. What percentage of your FP analyses is using the detailed method? Let us know on the forum.

  2. Harold and Theo: Thanks for posting this.
    I have used the light FPA method for the past ten years for a couple of reasons:
    – Early estimates require a high level view on the functional size. Detailed fp-analysis would lead to believe that there is accuracy where there can’t be.
    – Even in a baseline analysis, my clients often cannot give a detailed overview on the implementation of single attributes. Lack of documentation, lack of knowledge, lack of anything…. the reasons are manifold.
    – I usually offer fixed price estimates, thus to be cost-efficient, high-level analysis is the best choice.
    – I don’t want to bore the audience. Clients are quite satisfied when I finish a huge SAP application within 2 days (“We always thought it would take weeks, … how can it be that estimating only takes 2 days….”). And don’t we all want happy clients?
    Only at one occasion I almost began to sweat: I had trained people from a big telco company in light-FPA analysis. We had counted the volume of the organization’s 12-months development results. After reporting the fp-results, management did not dare to trust either me or their own staff, who knows? So they flew in another metrics company (they always fly them in from far away, hoping this makes it more effective) to redo the whole analysis using the detailed IFPUG method.
    You guess what happened then, don’t you? They found less than 1% deviation between our light analysis and their detailed analysis!

  3. I think the effort gain that is reported is rather conservative and may depend on your reporting standards. In her comparison (2 weeks vs 2 days) Eva shows an effort gain of 80% for a high level analysis. My experience is similar: a high level analysis is at least 4 times faster than a detailed analysis.

  4. At the IT confidence conference (Tokyo, October 2014), A presentation was given by Japanese researchers that credited the Nesma High Level FPA method to be responsible for increased interest in using functional sizing in the Japanese industry. I am trying to get the presentation, but there is already a short paper available, also through ISBSG slideshare: http://www.slideshare.net/ISBSG/matsumoto-towards-an-early-software-effort-estimation-based-on-the-nesma-method-estimated-fp

  5. I was very surprised to hear that IFPUG has announced FPA uTIP#3 ‘Early FPA’. Over more than 10 years NESMA’s ‘estimated analysis’ (which is identical to the High-Level analysis) is common practice in the Dutch metrics consultancy scene.
    The tables in the blog show unambiguously that there is only a slight difference in size between detailed analysis and estimated analysis. So it may be concluded that the accuracy of ‘high level’ analysis is nearly as good as detailed analysis, with a considerable gain in effort.
    In my opinion the real advantage is that ‘high level’ FPA can be applied to more coarse grained spec’s, thus earlier in the project. In my practice this means that a fairly accurate size estimate and project estimate can quickly be delivered at the end of Requirements Analysis (and before Detailed Design). I am very curious to the ‘Consistent Cost Estimating’ of IFPUG.

  6. SrideviDevathi says:

    Thanks for this post; its quite useful. We have used Quick FPA on many situations, and the accuracy of the estimate is always debated. This analysis would come in handy!

Leave a Reply