Rajeevan Velayudhan, Senior Director of Quality Assurance, MindtreePrint This Page
There are many business challenges that are often experienced in quality assurance (QA). Some include paying for capacity rather than actual output, spending too much time reviewing, resourcing and
planning, coping with QA spends that exceed estimation or receiving a list of value adds, but not experiencing any reduction in costs.
An output-based pricing model enables customers to define and measure improvements that provide benefit.
What is output-based QA and pricing?
This is a model where the partner charges the customer based on units of work.
The output / units of work are always measurable. By working with units, the partner is motivated to improve both efficiency and effectiveness in delivering the unit of work. The customer gets more units of work delivered leading to high productivity and reduced governance overhead. The resulting savings can then be applied to process and capability improvements, thus initiating a positive cycle.
An output-based model provides incentives to both parties to improve and deliver consistently better results.
Moving to an output / unit based model must be a systematic process. It requires common definitions, measurement criteria, governance methods and continuous improvement mechanisms. Different portfolios of an organization may be at varying maturity levels. Identifying the most mature portfolio to pilot this model and devising an adoption plan to evangelize this model with the other divisions is a key step in championing this model across the organization. Test Centers of Excellence if any should be leveraged. A corresponding partner engagement strategy should also be put in place.
Framework for output-based testing and pricing
Step 1: Define the unit of work
Defining the unit of work is a complex task. Though the definition will vary by the maturity of the enterprise, it must always be defined by looking beyond the basic test case and talk in the language of business
The foundations behind defining a unit of work are business significance of the unit, repeatability and availability of historical data. The intent is to make the details transparent to the customer by shifting focus to business significant SLAs.
Units will often be segregated into categories of tests. Some examples are user interface, validations, functional, and non- functional scenarios. A detailed well-categorized framework for defining units must be used.
An example is benefits payout testing in healthcare. An enterprise with thousands of customers may need to test a new benefit package. The type of testing undertaken for the customer is very similar, except for certain parameters such as coverage details. This is a classic case where health insurance enterprises and the testing partner both agree on a unit of work for testing which in this case, would be the number of claim scenarios complete healthcare packages tested.
The most important benefit of this model is clear separation of user acceptance testing and internal testing which often overlap in undesirable ways (sharing of test scenarios, sampling-based UAT testing and more). A detailed, quantifiable and mutually agreed framework for unit definition must be used.
Step 2: Agree on the associated SLAs
Once the unit of work is decided, applicable SLAs and permissible limits are built. This may include lead-time for work completion, impact of defect in production, defect density and defect distribution..
As change is the only constant, an additional input for defining SLAs will be the SLA framework. This framework will consider velocity of change, type of change and other evolution factors for the software in question.
It is also common to translate all SLAs to units of work to provide an inbuilt mechanism to feed the SLAs in billing. For example, a delay in code fix will be translated as additional x units of work; a delay in testing turnaround will be a reduction in x units of work.
The SLA framework will also define the amount of deviation that can be taken without impacting pricing and service quality. This framework must be established to both monitor performance as well as accommodate change.
Step 3: Agree on unit pricing
Once the unit and SLAs around it are agreed upon, the next step is to agree on pricing per unit. There are two common approaches:.
Top-down approach: Past work samples are looked at from a larger scale.
For example, unit level costs are arrived at by:
Looking into the past years’ QA spend
Examining what percentage of that QA will be taken by the partner
Bottom-up approach: Here, the effort needed is analyzed by testing and delivering the basic unit of work committed. This is done through past data, if we have granular data available, by inspecting trial runs, through subject matter expert comments and so on.
The unit pricing can be determined by either top-down or bottom-up approaches. Ideally they will match within reasonable limits.
Step 4: Agree on improvement goals
Once the initial targets and goals are set, the next step is to agree on quarterly or yearly improvement targets. These targets will look at improving the quality of testing from the current baseline, improving the turn-around time and reducing the effort required. It is important to agree on goals that are realistic and ones where the partner is able to provide a roadmap. After this the partner and customer mutually agree on translating these goals to redefined chargeback and SLAs. This enables both the customer and the partner to focus better on improving the output than spending time on operational items.
Agreeing on improvement goals is important to realize the benefits of a unit model, for both the customer and the partner.
The immediate benefits
The key benefits of a unit model are:
Know the pricing at the point of work allocation: Instead of going through detailed estimates at the point of allocating work, the QA effort expected and the associated SLAs are already known.
Skin in the game: Once the improvement goals are mutually discussed and agreed upon, they are translated to future SLAs and pricing. This helps to build vested interest to archive those goals, for both the customer and the partner.
Less time in non-core activities: Both the customer and the partner spend less time thinking through the value adds, future saving and resource details. This is because they have been translated as SLAs or price.
Where does a unit model not apply?
Among the four pillars of planning mentioned, the most important is unitizing the work and agreeing on the SLAs. There are testing services where we cannot unitize the work because of a wide variety in scope, insufficient past data, variation in amount of work, among others. Classic cases are testing for a new product and implementation testing in a new geography.
A partner with significant prior experience in similar areas will be able to accelerate the maturity to a unit-based model. Also, a unit-based model may not apply where the work is not well defined, but prior experience may accelerate reaching the desired state
A successful engagement is the one where the partner thinks about the client’s needs and develops appropriate solutions.
This is enabled by mutually aligning ideas and interests. Both parties should be able to reap the benefits of innovation. Moving away from a resource-based model to output-based model is one step toward this goal. The most important goals are unitizing the work and agreeing on the SLAs. There are testing services where work cannot be unitized because of wide variety in scope, insufficient past data, variation in amount of work among others. Classic cases are testing for a new product and implementation testing in new geography. This is where neither the customer nor the partner are comfortable in agreeing on SLA Details. In these cases, it is still best to assume pseudo units and work towards acquiring the data required for formal unit analysis.