🎓 SEM III Exam Prep Syllabus Portal
🏠 Home
📖 2413MJET306B • Unit V • 4 Hrs

Unit V - Test Management, Automation, metrics and measurements

Comprehensive University Exam Preparation Notes, Model Question Answers & Comparison Matrices

🔍
📑 Quick Jump Navigation

📌 Syllabus Topics Covered

4 Hrs Weightage

📖 Comprehensive Theoretical Notes

Exam-Oriented Theory

5.1 & 5.2 Test Planning, Management & Test Reporting

Test Management: The managerial discipline of planning, estimating, scheduling, executing, tracking, and reporting software testing activities across the entire development lifecycle.

The Test Management Process:

  1. Test Planning & Estimation: Estimating testing effort using techniques like WBS (Work Breakdown Structure), Function Point Analysis, or 3-Point Estimation.
  2. Test Monitoring & Control: Continuously tracking test execution progress against planned milestones and applying corrective actions if delays occur.
  3. Test Reporting & Summary: Authoring the formal Test Summary Report containing pass/fail ratios, defect distribution by severity, open defect risks, and release readiness recommendations.

5.3 - 5.5 Test Automation: Architecture, Frameworks, and Tool Selection

Test Automation: The practice of using specialized software tools to execute pre-scripted test suites automatically, compare actual outcomes with predicted outcomes, and generate execution reports without human intervention.

Test Automation Framework Architectures:

  • Linear / Record-and-Playback: Simplest scripting; highly fragile to UI changes.
  • Modular Framework: Decomposes scripts into reusable independent functional modules.
  • Data-Driven Framework: Separates test logic from test data, reading multiple test vectors from external files (Excel, CSV, JSON).
  • Keyword-Driven Framework: Associates actions with keywords (e.g., click, typeText, verifyText) enabling non-programmers to write test sheets.
  • Hybrid Framework: Combines Data-Driven and Keyword-Driven approaches with Page Object Model (POM).
  • Behavior-Driven Development (BDD): Uses Gherkin syntax (Given-When-Then) with tools like Cucumber/SpecFlow.

Testing Tool Selection Criteria: Platform compatibility, programming language support, ease of maintenance, CI/CD integration, licensing cost, community support.

5.6 Test Metrics and Measurements (Mathematical Formulas)

Key Software Testing Metrics Formulas:

  • Defect Density: Quantifies software quality per module size: $$\text{Defect Density} = \frac{\text{Total Defects Found}}{\text{Module Size in KLOC (or Function Points)}}$$
  • Defect Removal Efficiency (DRE): Measures testing quality and filtering capability before release: $$\text{DRE} = \frac{\text{Defects found during testing (Internal)}}{\text{Internal Defects} + \text{Customer Defects found in production (External)}} \times 100\%$$
  • Defect Leakage (Defect Escape Rate): $$\text{Defect Leakage} = \frac{\text{Defects found by Customer in UAT/Production}}{\text{Total Defects Found}} \times 100\%$$
  • Test Case Execution Rate: $$\text{Execution Rate} = \frac{\text{Executed Test Cases}}{\text{Total Planned Test Cases}} \times 100\%$$
  • Test Case Pass Percentage: $\frac{\text{Passed Tests}}{\text{Total Executed Tests}} \times 100\%$.

🔑 Key Concepts & Examination Keywords

Quick Terminology
Test Automation Framework
An integrated set of guidelines, libraries, and coding standards providing a structured environment for building automated tests.
Defect Removal Efficiency (DRE)
A quality metric measuring the percentage of defects detected by testing before customer release.
Data-Driven Framework
A test automation framework that drives test execution by reading external data sheets containing input/expected values.
Page Object Model (POM)
A design pattern in UI automation where web pages are modeled as classes containing element locators and interaction methods.

🎯 High-Yield Important Examination Questions

8–10 Descriptive Points Each

Q1. Explain Test Automation Frameworks in detail: describe Linear, Modular, Data-Driven, Keyword-Driven, Hybrid, and BDD architectures.

10 MarksAutomation FrameworksCore
▼
📝 Detailed Examination Answer (10-Point Model):
  1. Definition and Need for Automation Frameworks: A framework provides a standardized set of guidelines, execution engines, reporting utilities, and object repositories for building scalable test suites.
  2. Linear Scripting Framework (Record & Playback): Testers record manual UI interactions; scripts run sequentially. Highly fragile because minor UI changes invalidate the entire script.
  3. Modular Testing Framework: Divides applications into distinct business modules (e.g., LoginModule, CartModule) creating reusable script functions, reducing code duplication.
  4. Data-Driven Testing Framework: Separates test logic from test input datasets; a single test script iterates over hundreds of data rows read from Excel/CSV/JSON files.
  5. Keyword-Driven Testing Framework: Separates test automation logic into tabular keywords (e.g., `openBrowser`, `clickButton`, `verifyText`) allowing manual testers to write automated tests in spreadsheets.
  6. Hybrid Automation Framework: The enterprise standard combining Data-Driven, Keyword-Driven, and Page Object Model patterns for maximum scalability and maintainability.
  7. Behavior-Driven Development (BDD / Cucumber): Uses human-readable Gherkin syntax (`Given [precondition]`, `When [action]`, `Then [expected result]`) bridging communication between developers and business analysts.
  8. Page Object Model (POM) Design Pattern: Creates separate Java/Python classes representing web pages, encapsulating UI locators (`By.id()`) away from test assertion scripts.
  9. Reporting & Continuous Integration (CI/CD): Generates rich HTML execution reports (Allure, ExtentReports) integrated into Jenkins/GitHub Actions pipelines for automated nightly builds.
  10. Return on Investment (ROI) of Automation: Automation requires high initial setup investment but saves thousands of manual testing hours during repetitive regression test cycles.

Q2. Explain the fundamental software testing metrics: Defect Density, Defect Removal Efficiency (DRE), Defect Leakage, and Test Execution metrics with mathematical formulas.

10 MarksTest Metrics & Measurements
▼
📝 Detailed Examination Answer (10-Point Model):
  1. Importance of Quantitative Metrics: Software metrics provide objective, data-driven visibility into software quality, testing efficiency, team productivity, and release readiness.
  2. 1. Defect Density Formula & Role: Measures the number of defects discovered relative to module size: $\text{Defect Density} = \frac{\text{Total Defects Found}}{\text{Size in KLOC (or Function Points)}}$.
  3. Interpreting Defect Density: Identifies high-risk, bug-prone modules in the codebase that require architectural refactoring or intensive code review.
  4. 2. Defect Removal Efficiency (DRE): Formula: $\text{DRE} = \frac{D_{\text{Internal}}}{D_{\text{Internal}} + D_{\text{External}}} \times 100\%$, where $D_{\text{Internal}}$ are defects found by QA and $D_{\text{External}}$ are bugs escaped to production.
  5. Benchmark Standards for DRE: A DRE $> 95\%$ represents high testing quality; a low DRE ($<85\%$) indicates ineffective test case coverage or inadequate test environments.
  6. 3. Defect Leakage (Escape Rate): Formula: $\text{Defect Leakage} = \frac{\text{Defects found in UAT / Production}}{\text{Total Defects Discovered across Project}} \times 100\%$.
  7. 4. Test Case Execution Rate: Tracks testing progress: $\text{Execution Rate} = \frac{\text{Executed Test Cases}}{\text{Total Planned Test Cases}} \times 100\%$.
  8. 5. Test Case Pass / Failure Ratio: Calculates $\text{Pass Percentage} = \frac{\text{Passed Tests}}{\text{Executed Tests}} \times 100\%$ and $\text{Fail Percentage} = \frac{\text{Failed Tests}}{\text{Executed Tests}} \times 100\%$.
  9. 6. Mean Time to Detect (MTTD) & Mean Time to Repair (MTTR): MTTD measures average time taken to identify bugs; MTTR measures average time taken by developers to diagnose, fix, and verify patches.
  10. Test Summary Report Integration: All metrics are consolidated into the final Test Closure Report to support executive release decisions.

Q3. Discuss the criteria for selecting test automation tools and compare Manual Testing and Automated Testing in terms of cost, speed, and suitability.

10 MarksTool Selection & Manual vs Auto
▼
📝 Detailed Examination Answer (10-Point Model):
  1. When to Automate (Automation Feasibility): Automate high-volume, repetitive regression suites, performance load tests, and cross-browser matrices; never automate one-time exploratory tests.
  2. Tool Selection Criterion 1: Application Compatibility: The tool must support target technologies (Web: Selenium/Playwright; Mobile: Appium; API: Postman/RestAssured; Desktop: WinAppDriver).
  3. Tool Selection Criterion 2: Language & Skillset: Tool should align with team technical proficiency (Java, Python, JavaScript, C#).
  4. Tool Selection Criterion 3: CI/CD & Cloud Integration: Must support headless execution in Docker containers and integrate with Jenkins, GitHub Actions, and cloud grids (BrowserStack, SauceLabs).
  5. Tool Selection Criterion 4: Licensing & Maintenance Cost: Evaluate open-source tools (Selenium) vs commercial licensed tools (QTP/UFT, TestComplete) including ongoing maintenance overhead.
  6. Tool Selection Criterion 5: Object Identification & Locators: Must provide robust object locator strategies (XPath, CSS, Accessibility IDs) with intelligent auto-wait mechanisms to avoid flakiness.
  7. Manual Testing Strengths: Essential for exploratory testing, usability evaluations, ad-hoc bug hunting, and UI aesthetics where human observation is mandatory.
  8. Automated Testing Strengths: Unmatched speed and repeatability for massive regression suites, eliminating human fatigue during repetitive execution.
  9. Cost Trajectory Comparison: Manual testing has low initial cost but scales linearly with time; Automation has high upfront scripting investment but near-zero marginal cost per subsequent execution.
  10. Balanced QA Testing Strategy: High-performing QA teams utilize a balanced pyramid: 70% Unit tests, 20% Automated API/Integration tests, 10% Automated UI & Manual Exploratory tests.

⚖️ Comprehensive Comparison & Difference Tables

8+ Comparison Criteria

📊 Manual Testing vs Automated Testing

Comparison ParameterManual TestingAutomated Testing
Execution AgentHuman QA tester manually interacts with UI and verifies results.Specialized automation tools and scripts execute test suites.
Execution SpeedSlow and time-consuming; limited by human reading and typing speed.Ultra-fast execution; executes hundreds of test cases per minute.
Repetitive Execution FatigueProne to human error, fatigue, and skipped steps during repetitive runs.100% consistent, reliable, and fatigue-free across repeated executions.
Initial Investment CostLow initial setup cost; requires no automation framework development.High initial setup cost; requires framework architecture and scripting.
Long-Term ROICost remains constant or increases linearly as project grows.High Return on Investment (ROI) over long-term regression cycles.
SuitabilityExploratory testing, UI usability, ad-hoc testing, visual layout checks.Regression testing, performance load testing, repetitive data entry, smoke tests.
Programming Skill RequiredDoes not strictly require programming or coding knowledge.Requires strong programming, framework architecture, and debugging skills.
CI/CD Pipeline IntegrationCannot be integrated into automated continuous integration pipelines.Seamlessly integrated into CI/CD pipelines for automated build verification.

📊 Data-Driven Framework vs Keyword-Driven Framework

Comparison ParameterData-Driven FrameworkKeyword-Driven Framework
Driving PrincipleDriven by external test DATASETS (inputs and expected values).Driven by action KEYWORDS (e.g., `click`, `type`, `verify`).
Target UserAutomation engineers with strong programming skills.Manual testers and business analysts without deep coding skills.
Data SeparationSeparates test script logic from external test data files (Excel/CSV).Separates test design steps into keyword tables and underlying code libraries.
Script ComplexitySingle script contains loop logic iterating over data rows.Requires building a keyword interpreter/engine that parses table keywords.
Initial Framework OverheadModerate initial framework development effort.High initial framework development effort to build keyword engine.
Test Case AuthoringTest cases are authored as programmatic test scripts.Test cases are authored as spreadsheets containing keyword step rows.
MaintenanceScript changes required when application workflows change.Easier maintenance; update keyword spreadsheets without touching code.
Ideal ApplicationForms with massive input validation combinations (e.g., Loan calculators).Large teams with many manual testers transitioning to automation.

⚡ Quick Pre-Exam Revision Summary

5-Minute Recap
💡 Core Takeaways & High-Yield Summary
  • Test management oversees planning, estimation (WBS/3-Point), execution, tracking, and reporting.
  • Automation frameworks: Linear (fragile), Modular (reusable), Data-Driven (Excel data), Keyword-Driven (keywords), Hybrid (POM + Data).
  • Defect Density = Total Defects / Size in KLOC; identifies high-risk modules.
  • Defect Removal Efficiency (DRE) = [Internal Defects / (Internal + External)] * 100%; measures test quality.
  • Defect Leakage = (Escaped Defects to Customer / Total Defects) * 100%.
  • Manual testing is ideal for exploratory/usability; Automation is ideal for repetitive regression/load testing.
← Previous: Unit IV
Back to Software Testing ↑