"Black box" testing

To warm up for the course, here are details of a task to do at home on your own computer (or on IDA's computers). The assignment is not graded. But it does give practice and understanding of the difficulty and importance of software testing, which is useful for future laboratory work in the course. The task is relatively large, so we expect that you collaborate (in whole class, if you want it). Feel free to discuss the task in larger groups and across class boundaries.

The specification of the programs you are to test in this task are given in the lab called "Tax Table" in your lab compendium. As you understand from the location of the specification, you will write your own version of this program later on. It means this task does not only give you insight in testing, but will also prepare you for that lab assignment.

Introduction

Writing program code is part of the wider term "programming". To ensure that the code you've written is working properly is an at least as important part. And that's what this task is all about. Below are a number of programs. Each are a possible student implementation of the "tax table" laboration. One of these programs is flawless, the other has one or more errors (there is no program that has the exact same set of flaws, all are unique in some aspect). Your task is to find good test cases that distinguishes the programs, and to find the program that are flawless.

NOTE! Several of these programs could be approved in class despite their flaws. Do not be afraid that we will require a great program when you do your own attempt later on.

NOTE! What you learn in STONE about commands, about redirecting program output, and about comparing files can perhaps come in handy during your testing efforts.

About testing

To be able to trust the programs you have written, it is important that you have tested them in as much detail as possible. An analogy here may be to test whether a calculator is working properly (which it hopefully does :).

A (simple) calculator has functions including addition, subtraction, multiplication and division. It also has positive values, negative values, and the value 0. To test the calculator we should test all arithmetic with all kinds of values.

Simple test cases can be 1+2, 0-5, 2.5*2 and 10/-2.5 where the calculator should give the correct answer if it works as intended. The mentioned cases give a very low confidence that the calculator works correct. To get a higher confidence that it works properly, we should test all combinations (for example, try multiplying by negative numbers). Soon you realize that the number of test cases can be very large, so it is important not to include same case twice, testing 3+4 is for example the same thing as testing 1+2 above, both use two positive numbers with addition (but testing 3+8 or 2+10 may both be interesting though, can you see why? *). Hopefully this gave an indication of what testing means.

If we do not test our programs, we can not be sure that they work the way we intended. You can not really say that a program works even after you tested it. All you can say is that it works for the cases tested, and that at least it is likely to contain fewer flaws than an untested program.

Instructions

Here is a brief description of what you have to do in this assignment.

1. Read the instructions for lab 1 (Tax Table).
2. First you need to download the set of flawed programs to your computer. You need a version matching your computer and operating system. Choose from the version that seems to be right for you in the table below. The "ZIP" file you download have 16 versions of the program. They are named:

         tax_table_0000
         tax_table_0001
         ...
         tax_table_1111
       

3. Now for the actual testing. What should you do to test programs and be able to distinguish the differences and see which program that is correct?

   Think of as many test cases as you can (a test case includes a first prize, a final price, a stride and a tax rate). Discuss this with your friends so that everyone test with unique test cases.

   Here are some other points that are important to consider:

   • Choose not only accurate test cases but also unreasonable test cases (eg, a negative price). A good program should handle invalid entries.
   • Use many test cases, so that each program is tested as exhaustive as possible. Try to try "different things". The same type of test cases usually gives no extra details.
   • Test ALL 16 programs with the same test cases and write down how each program reacted to each test case, and if that was the correct response or not. This is required so you later on can compare the programs.
   • Be thorough and write down your test cases.
   • Write down the result of each test case so that that the result can be reproduced.
   • Write down the errors that each program has.
   • Discuss and collaborate so that you do not test the same test cases. Ensure that one person keeps track of who does what in order to make the group effort progress faster.

Files to be downloaded

Here are a number of ZIP files with the executable programs. If you can not use any of these, it is good to hear from you so we may be able to use your computer to create new versions that may be good for others too.

If logged on to our computers you can run the programs directly by typing the full path to the program (no need to download!):

      /home/TDDD33/tax_table/tax_table_0000
      /home/TDDD33/tax_table/tax_table_0001
      ...
      /home/TDDD33/tax_table/tax_table_1111  

NOTE! If you click "Page in swedish" up to the right you get to the Swedish version ot the web page. It has more programs, but in Swedish. You can probably understand the Swedish version of the program easy enough, it is after all a very small program, and if you take a look at the English versions first I'm sure you can translate.

* Look at the number of digits in the operands and in the expected answer...

Page responsible: Torbjörn Jonsson
Last updated: 2014-08-31