Realizing Test Cases, Test Scripts and Test Scenarios as Valued Testing Assets

One aspect of software testing that has endured over the many years of software development and testing is that of the need of test documentation. Like any practice, the way people perceive and apply test documentation varies depending on the context of projects and the lifecycles in use.

For example, early in software testing literature the implicit assumption was that all tests must be documented. Every textbook, every training course and every practice guide reinforced the idea of documented test cases and test scripts.

However, as time marched on, the idea of exploratory testing emerged and was embraced as a way to learn and test concurrently. Instead of starting a test with a pre-defined set of test cases and test scripts, some testers skip the pre-documentation and start testing directly, making note as they went. This had a major impact on test documentation because the perception can easily be, “Why spend all this time on creating test documentation when I can just get right to the test?”

The assumption is often that the tests will not need to be repeated, and that any effort expended in creating test cases or test scripts is a one-time cost with little or no payback.

Another major reason people may skip documenting test cases is that there is so little knowledge to draw from, there is little information to base the test upon. Or, perhaps the information is defined, but it changes too often. These are all valid concerns.

Then along came agile where performing the test took precedence over planning or designing the test. With other forms of documentation getting slim, such as user stories being favored over user requirements, test documentation followed that trend as well.

Today, even with exploratory testing and agile testing both very popular, people are coming back to the idea of more robust test documentation for several compelling reasons:

1. Test repeatability is essential for the ongoing testing of releases, especially for the purposes of regression testing and continuous integration (CI) testing.
2. The need to measure the status of testing for the purpose of determining readiness for release requires there be something to measure. While defect levels are important to measure, without test case status, they lack context.
3. Determining test coverage levels.
4. Using the test documentation as a basis for knowledge transfer.
5. Using test cases and test scripts as the catalyst for process improvement, especially the processes for testing, requirements analysis and documentation, and software development. By understanding where defects seem to be concentrated, more effort can be applied in preventing defects in those areas.
6. Test automation requires in many cases the ability to understand the test before automating it. Much of this understanding can be gained in the process of documenting a test in test cases and test scripts.

With the reasons listed above, it makes sense to see test documentation in the form of test cases, test scripts, test scenarios as a value-added investment as opposed to a cost that must be expended.

When any activity or item is considered an investment, it is also an asset that needs to be protected and cared for.

It is true that every test case and test script is one more item to maintain. Ignoring the maintenance commitment of test documentation is something that will cost much more later down the road in both time and money. In fact, inadequate maintenance of testware puts your test documentation effort at risk of being a total waste.

Just to level set, let’s define exactly what is meant by test cases, test scripts and test scenarios.

What Is A Test Case?

As we look at definitions, I must emphasize that each organization has their own way of expressing test cases. Some use the compact view that is less procedural, while others take a highly procedural view and resemble something more like a test script.

A useful definition of a test case is found in ISO 29119 Part 1, as a “set of test case preconditions, inputs (including actions, where applicable), and expected results, developed to drive the execution of a test item to meet test objectives, including correct implementation, error identification, checking quality, and other valued information”.

The ISTQB Glossary V3.2 definition of a test case is “A set of preconditions, inputs, actions (where applicable), expected results and post-conditions, developed based on test conditions.”

There is an important difference in these definitions. The ISTQB definition reads that test cases are based on test conditions. The ISO 29119 definition reads “to meet test objectives.” Test objectives are often much larger in scope than test conditions, which is why I prefer to think of test cases as a way to verify and/or validate specific test conditions.

Test cases are compact and focused. You might think of them as building blocks that can be assembled into suites and other collections.

An example of a test case would be testing the values entered in a search field, based on a variety of test conditions, such as full search, partial search, wildcard search, and so forth.

Test Case Example

ID: TC1
Function: Product Search
Test Condition(s): Full search, item in catalog
Input values: “Fur-lined hiking boot”
Expected Result: Matching products shown, along with suggestions based on the search.

As implemented in PractiTest:

How to Write a Test Case

Test cases are not difficult to write. In fact, one could create quite a number of test cases fairly quickly. Even more could be created faster with a test case generator. However, the hard part is writing test cases in a way that are maintainable, meaningful, and exercise the software beyond just positive confirmation or “happy path” tests.

The test design process can be best described as test analysis, test design and test implementation.

Step 1 – Test Analysis

The first thing is to identify the existence of a basis of testing, such as user requirements, user stories, use cases, and so forth.

The next step is to assess the adequacy of the test basis. Is it complete, clear, accurate, current and testable? This is also a good time to make sure the basis of testing has unique identifiers for traceability.

In test analysis, you can start to form the ideas for testing. These will often appear as test conditions.

Test conditions are very important in test case development, as they form the heart of test cases.

Step 2 – Test Design

In test design, the more complete view of test cases start to form. It is at this point you can apply techniques such as boundary value analysis to identify positive and negative tests that will add strength to tests.

In this step, you also define expected results. At first it may seem that researching expected results may seem to be a lot of work, but they actually save a lot of time. Without expected results in a test case, you must evaluate the test results by researching each outcome.

While it is true that some people may know the expected outcome based on experience, there is the possibility that future testers may not have that experience and knowledge.

Plus, expected results form a definite target to be hit in a test. Otherwise, it can be easy to see a test result and explain it away as “normal” or “correct” when it actually is not.

Step 3 – Test Implementation

In this step, you do two major things:

1. You translate the test cases into a test management tool.
2. You combine test cases to form test scripts.

Please note I make the assumption of using a tool. Some people use spreadsheets and other office-type documents, but I have found this to be a losing game. Test cases in spreadsheets simply are too difficult to scale and manage, especially with good tools available such as Practitest.

The use of an effective test management tool is a critical part of a test framework. A test framework is the key way in which test reuse is realized either in manual or automated testing. A test framework enables the creating of a library of test cases that can be organized in many ways, depending on the tests that are needed.

Step 4 – Refine and maintain as needed

It is very common to discover during the testing effort that new test conditions emerge. You may also learn that some tests are too complex or ineffective and need to be either deleted or de-prioritized.

What is a Test Script?

According to the ISTQB Glossary, a test script is “a sequence of instructions for the execution of a test.” ISO 29119 Part 1 defines a test script as “test procedure specification for manual or automated testing.”

With these definitions we see some ambiguity around the idea of a “test procedure.”

A test procedure can include all the steps required to set-up, perform and tear-down a test. That is a wide scope, but there is also the perspective of a test procedure as a collection of steps or test cases to test a particular task or process.

The latter is a what many people commonly mean when they refer to a “test script.” Just like a movie script, a test script describes a sequential flow of what is performed.

Keeping with the search field example, a test script might start with searching for an item, then continue on with selecting it for purchase, paying for the item, and so forth. In this example, a test script describes how to test a task or process.

Test Script Example

Objective: Purchase a product
Pre-requisite: Product in stock, customer has established account and is logged in.

Other Values of Testware

People who get the most value from test cases and test scripts have learned that there are many purposes of test documentation beyond testing. Here are some examples:

• Test cases and test scripts can greatly help end-users learn and test a new or upgraded system.
• As new users are hired, test scripts can help them learn the processes of how to use the software step-by-step.
• Many companies have compliance needs, such as the need to show they exercise due care over how their systems are controlled. One major aspect of showing control is quality control, which is software testing.
• Solid test documentation can also show due care of how a system was tested should legal issues arise. For this and other reasons, care should be taken to archive test documentation.

Two final thoughts…

First, don’t forget to review the test cases and test scripts! Test scripts and test cases, like any other project document can contain defects. So, it is a good idea to review them.

One good approach is to review the first few test cases and test scripts just to make sure the desired approach is being applied.

Then, as more test cases and test scripts are designed, other reviews are needed to make sure the testware are consistent and correct.

Second, look for defects as you create testware. It is interesting that many of the early testing authors considered test analysis and design as part of test execution, not test planning. The reason is that they learned that defects are often found by comparing the basis of testing (such as a requirement) to a test case being formed.

I hope this article gives you a good start in writing your own test cases and test scripts. Yes, they are still needed and add great value to your testing effort!