Definition

Email this page to a friend   

 Email to a friend

Test Documentation

(Alias: Test Plan, Test Design, Test Procedure, Test Case, Test Incident Report, Test Log, Test Summary Report)

If it's not written down it doesn't exist.
             - Anon

Test documentation is the complete suite of artifacts that describe test planning, test design, test execution, test results and conclusions drawn from the testing activity. As testing activities typically consume 30% to 50% of project effort, testing represents a project within a project. Testing activities must therefore be fully documented to support resource allocation, monitoring and control. This page identifies the types of documents you need to set up and run your test program and summarises their content.

The Test Documentation Tree

Tests must be planned and documented to ensure that test coverage is systematic and complete.

Test Documentation Tree

Test Plan

The test plan describes the testing process in terms of the features to be tested, pass/fail criteria and testing approach, resource requirements and schedules.

  1. Introduction
  2. Test items
  3. Features to be tested
  4. Testing approach
  5. Item pass/fail criteria
  6. Suspension and resumption
  7. Deliverables
  8. Tasks
  9. Environmental needs
  10. Responsibilities
  11. Staffing and training needs
  12. Costs and schedule
  13. Risks and contingencies.

Test Design Description

The Test Design Description refines the Test Plan's approach, identifying specific features to be tested and defining the test cases and test procedures that will be used.

  1. Features to be tested
    • Test items covered
    • Feature or feature combinations to be tested
    • References to software requirements specifications and software design descriptions
  2. Testing approach
    • Testing techniques to be used
    • Method of analysing test results (e.g. automated or manual)
    • Reasons for selection of various test cases
    • Environmental needs of test cases
  3. Test identification
    For each feature to be tested provide:
    • Test procedure identifiers and descriptions
    • Test case identifiers and descriptions
    • The pass/fail criteria.

Test Procedure Description

The Test Procedure specifies the steps for executing a set of test cases.

  1. Purpose
    • Test items to be tested
    • Expected outcomes
    • Test cases to be exercised
    • References to test item documentation
  2. Special requirements
    • Prerequisite procedures
    • Environmental requirements
    • Special skills
  3. Procedure steps
    • Set up. Preparation for the procedure
    • Start. Starting the procedure
    • Proceed. Procedure execution
    • Measurement. Test measurements
    • Shutdown. Suspending testing
    • Restart. Restarting the procedure
    • Stop. Bringing test execution to an orderly halt
    • Wrap up. Restoring the environment
    • Contingencies. Dealing with anomalous events
    • Logs. Logging test results.

Test Case Description

The Test Case Description describes a single instance of a test in terms of data inputs, expected outcome, required test environment and test procedures.

  1. Test items
  2. Input specifications
  3. Output specifications
  4. Environmental needs
  5. Special procedural requirements/rules
  6. Intercase dependencies.

Test Incident Report

The Test Incident Report documents any event that occurs during testing which requires investigation.

  1. Summary
    • Incident summary
    • Test item identification
    • Test procedure and test case identifier
    • Test log reference
  2. Incident description
    • Date and time
    • Expected results
    • Actual results
    • Anomalies
    • Procedure step
    • Environment
    • Attempts to repeat
    • Testers and observers present
  3. Impact
    • Impact on other test activities.

Test Log

The test log provides a chronological record of details about the execution of test procedures, records success or failure and describes anomalies experienced.

  1. Description
    • Test item identification
    • Test environment description
  2. Activity and event entries
    • Date and time
    • Author
    • Test procedure identifier
    • Staff present
    • Pass/fail
    • Error messages generated
    • Environmental information
    • Anomalous events
    • Test incident report identifiers.

Test Summary Report

The Test Summary Report summarises the results of testing and evaluates the test item based on these results.

  1. Summary
    For each test item provide:
    • Test item identifier
    • Summary test item evaluation
    • Test environment
    • References to test documentation
  2. Variances
    • From design specifications
    • From test plans, designs, procedures or test cases
  3. Comprehensive assessment
    • Comprehensiveness of the test
    • Features that were not sufficiently tested
  4. Summary of results
    • Tests incidents resolved
    • Tests incidents not resolved
  5. Evaluation
    • Limitations
    • Test performance (pass/fail)
    • Potential reliability and maintainability
  6. Summary of activities
    • Resource consumption
    • Performance against plan.

Case study: Two Angry Men

By Les Chambers

One day I was sitting at my desk minding my own business when I looked up and found I was surrounded by two angry men. A software component I had unit tested on the test floor failed when it was installed on-site. The failure was due to an obvious bug that should have been caught in testing.

Feeling the onset of a mild case of panic I reached for my test records. I had tested that unit about four months previously so my memory was hazy. The angry men were right, it was an obvious bug that should have been caught in testing. How could I have been so stupid?

I laid my hands on the appropriate documentation. What a relief! My records showed that I had run a unit test case for the function in question and it had passed. It looked like the problem was in integration testing. An interaction with another system element that I could not cover with a unit test had caused the failure.

I know, this did not help my assailants who's anger was entirely righteous. Corrective action was going to be expensive in terms of dollars, time and credibility with the customer. The failed software component would have to be fixed in the development shop, re-unit tested, re-integration tested, re-system tested, re-integrated on-site and go through acceptance testing again - while everyone including the customer sat around waiting.

The only positive was for me, at least the anger was focused somewhere else; which just goes to show, quite apart from being the right and professional thing to do, maintaining accurate, bullet-proof test documentation will one day, I am sure, help you avoid unfortunate and unnecessary unpleasantness.
Email to a friend