Creating Sanity Amidst The Test Methodology Madness - Webinar Presentation by Mindtree Ltd.

Creating Sanity Amidst Test Methodology Madness Right-sizing your test methodologies

CONFIDENTIAL: For limited circulation only

ÂŠ 2011 MindTree Limited

Slide 1

Agenda

Rightsizing?

hen Identify the symptoms

hat Key aspects to right-size

ow MindTree Tools & Frameworks

Processes should work for you. You should not work for processes. CONFIDENTIAL: For limited circulation only

ÂŠ 2011 MindTree Limited

Slide 2

Poll Question #1

● Are your processes taking too much effort and time? ● Yes

● No ● Not able to measure

CONFIDENTIAL: For limited circulation only

Slide 3

When to Rightsize?

hen Identify the symptoms

• • • •

Test processes taking too much effort and time?

•

Difficulty in quantifying the quality in your test processes?

Metrics not helping you take decisions? Not able to justify the investments in testing? Test process not helping you identify the critical defects early?

Acknowledge the need. Don’t wait until you loose your sleep. CONFIDENTIAL: For limited circulation only

Slide 4

What to Rightsize?

hat Key aspects to right-size

• • • •

Quality goals / objectives

•

Tools and frameworks

Software life cycle methodologies

Engagement models Governance mechanism (metrics, reviews)

Tailor your processes for right fit. CONFIDENTIAL: For limited circulation only

Slide 5

Poll Question # 2

● Do you know what metrics should you be tracking for your testing needs?

● Yes ● No ● I do, but not sure if they are exhaustive

CONFIDENTIAL: For limited circulation only

Slide 6

Test Metrics Analysis and Decisions (TMAD) Model

Metrics Based Governance

MindTree Tools & Frameworks

• • •

• •

Business goals Quality goals Governance • Metrics, reviews, baselines Forecast • Risks, best practices Corrective & preventive actions Enablers

Statistical Models

CONFIDENTIAL: For limited circulation only

Historical Data

Org / Industry Baselines

Slide 7

Test Metrics Analysis and Decisions (TMAD) Model Metrics Analysis Proposed Metrics

Defect Containment Effectiveness

Goal

1. Improve testing quality 2. Prevent defect leakage

Associated Queries (RCA)

Undesired Behaviour (Risks)

1. Was the criteria to classify defect detection phase clear from a leakage perspective? 2. Are there clear entry and exit criteria between phases 3. Was the team's knowledge adequate to uncover leaked defects? 4. Was the scope of testing for each phase clear? 5. Do we have requirement traceability matrix?

1. Blame game 2. Manipulation of defect classification criteria

Metrics Decision Table Decision Support (Scope) 1. 1. Improve test case review effectiveness 2. Improve process compliance 3. Improve product and test design knowledge

CONFIDENTIAL: For limited circulation only

2. 3. 4. 5.

Corrective Actions (feedback) Clear entry and exit criteria between phases Process adherence Effective knowledge management Update test suite Get test strategy reviewed by customers

ÂŠ 2011 MindTree Limited

Raw Data Requirements

1. Defects detected in test phase 2. Leaked defects

Suggestions

Do RCA for every release and identify preventive actions

Slide 8

Test Quality Index (TQI) Model

● Bivalent questionnaire; Y or N

Identify areas of improvement

responses

● “Productivity per tester” vs.

“quality of deployed resources and resource deployment”

● Contextual metrics vs.

comprehensive score; across 30 process areas

Improve scores with each deliverable

Customize the scored checklist

● Scoring based on self assessment

Winner of the Testing Leadership Award at QAI Software Testing Conference CONFIDENTIAL: For limited circulation only

Slide 9

Poll Question # 3

● Do you have an integrated framework which helps you rightsize the test processes and methodologies?

● Yes ● No ● We have point solutions, but not an integrated solution

CONFIDENTIAL: For limited circulation only

Slide 10

MindTestâ&#x201E;˘ - Integrated Test Methodology Processes

Lifecycle Processes

Templates

Guidelines

Checklists

Best Practices

Type of Testing

Stage of Testing

Customization Tools

Technologies

Analysis

Continuous Processes

Domain

Audits

TQI

TMAD

Test Estimates

Feedback

Metrics & Measurement

ProjectSpace

ProcessNet

Test Dashboard

Radar

TestLink

Knowledge Management

Lifecycle Processes

Test Measurement and Metrics - ROI

Defect Management

Test Management

CONFIDENTIAL: For limited circulation only

ÂŠ 2011 MindTree Limited

Slide 11

Case Study The Customer

Business Impact • 90% coverage on test cases executed, a significant improvement over the earlier metric

The customer is the world's leading specialist in air transport communications and information technology (IT) solutions.

The Business Challenge • No testing guidelines • Inadequate process • Lack of measures for impact of testing • No single snapshot of engagement level governance

•

100% achievement on acceptable open defects for severity 1 and 2

• •

Customer overhead of test case reviews reduced by 60% Constant review of the test case execution helped us to evaluate and report how far/near is the application from the acceptance criteria at different phases of testing

•

Enabled development team to prioritize the defects that needs to be fixed for the immediate build

•

Customer is now able to ascertain the quality of the application

•

Reduced risks and enabled effective decision making through test dashboard

MindTree Solution Executed in-depth analysis to understand

•

Quality objectives and alignment between test process and application's business goals

•

Used Test Metrics Analysis and Decisions (TMAD) Model and Test Quality Index (TQI) to come up with appropriate metrics and processes

“MindTree's expertise in metrics is paramount in managing the engagement and seeing if process changes have made the hoped for improvement and if the application is ready for release. The metrics are also a great way to communicate progress across all the groups in this engagement.”

Collected key metrics as governance enablers to measure and drive decisions based on the health of the projects

CONFIDENTIAL: For limited circulation only

Senior Manager, Quality Integration

Slide 12

In Short…

You don’t have to search for clues in the labyrinth

Identify the need

CONFIDENTIAL: For limited circulation only

Evaluate the key aspects

Optimize the process

Measure and improve

Slide 13

MindTree Testing Services Overview

Testing Size and Focus

Testing Depth and Breadth

• Dedicated unit of 2200+ specialists focused on testing

• Tested over 2000 applications & products from chip to cloud

• Contributes 25% of the MindTree’s global business

• Services ranging from point solutions to long term CoEs

• Domain specific testing services and solutions

1500+

Testing Process and Delivery Excellence

Thought Leadership

• Testing pillars of excellence - MindTest™, TestLabs, TestingAcademy

• Part of ISO working group in defining new testing standard – ISO / IEC 29119

• Virtual practices and CoEs for effective delivery

• President of India’s leading testing forum STeP-IN

700+

20+

95%

delivering PREDICTABLE QUALITY through INDEPENDENT TESTING CONFIDENTIAL: For limited circulation only

Slide 14

MindTree Workshop

ARE YOU TRACKING THE RIGHT TEST METRICS FOR YOUR BUSINESS DECISIONS? Complimentary Workshop at your Doorstep REGISTER NOW www.mindtree.com/events/testingwebinar/index.html or email to Balaji_Thesma@mindtree.com

CONFIDENTIAL: For limited circulation only

ÂŠ 2011 MindTree Limited

Slide 15

Sylvia Veeraraghavan Sylvia_Veeraraghavan@mindtree.com www.mindtree.com CONFIDENTIAL: For limited circulation only

ÂŠ 2011 MindTree Limited

Slide 16