What is model-based testing?
Utting, M., & Legeard, B. (2006) – Model-based testing is the automation of the design of black-box test. The main use of MBT is to generate functional tests. The difference from usual black-box testing is that rather than manually writing tests based on the requirements documentation, we instead create a model of the expected System under testing (SUT) behavior, which captures some requirements. Then the MBT tools are used to automatically generate tests from that model.
The models must be small but detailed, so we save cost to produce and accurately describe the characteristics we want to test. We can use a UML, but in order to be suitable for model-based testing, we need to add details about the behavior of the methods shown in the class diagram. We use notations for specifying behavior, for example OCL (Object Constraint Language) post conditions and state machine diagrams. Once we have a model we use a tool to automatically generate a set of abstract test cases, each test generated is a sequence of operations with the associated input values and the expected output value (the oracle). The next step is to transform these abstract tests into executable test scripts using a model-based testing tool. The resulting executable test may be produced directly is some programming language such as JUnit. We execute the test to try to detect failures in the SUT. The execution may be controlled and monitored by a test execution tool.
So the cost of using model-based testing is the time to write several pages of model details; the benefit is that we can then automatically obtain a comprehensive set of executable tests that cover all the different behaviors of each model and class.
Steps for model-based testing
- Model the SUT and /or its environment: write an abstract(smaller and simplier than the SUT itself) model of the system. We could use tools to check that the model is consistent and has the desired behavior.
- Generate abstract tests from the model: We can choose a test selection criteria to say which tests we want to generate from the model. The main output is a set of abstract tests (sequences of operations from the model) and also a requirements traceability matrix and some other reports.
- Concretize the abstract tests to make them executable: it may be done by a transformation tool or by writing some adaptor code. The goal is to to bridge the gap between the abstract tests and the concrete SUT.
- Execute the tests on the SUT and assign verdicts: the model-based testing tools will manage the test execution process and record the results.
- Analyze the tests results: This is similar to traditional tests analysis process.
The first two steps distinguish model-based testing from other kinds of testing.
Utting, M. & Legeard, B. (2007) – A reasonably mature test process is an important prerequisite for adopting model-based testing. So the adoption of MBT would correspond to at least level 7 of Test Process Improvement (TPI)
It is also useful to consider the modeling maturity level of a company that is adopting model-based testing.
Benefits of model-based testing
- SUT fault detection: in MBT fault detection power still depends on the skill and experience of the people who write the model and choose the test selection criteria.
- Reduced Testing Cost and Time: As well as costing less to design, MBT tests may also save time during the failure-analysis stage.
- Improved Test Quality: The use of an automated test case generator to choose the test cases from the model makes the design process systematic and repeatable. It becomes possible to measure the “quality” of the test suite by considering its coverage of the model.
- Requirements Defect Detection:It encourages early modeling, and this finds requirements and design errors because the model is precise enough to be analyzed by computer.
- Traceability: It can generate Reqs-Model, Model-Tests and reqs-Test
- Requirements Evolution: with manual test design, if requirements change, a large amount of effort is often required to update the test suite to reflect the new requirements. With MBT it suffices to update the model and then re -generate the tests.
Limitations of model-based testing
- It requires somewhat different skills from manual test design. This may require some training costs and initial learning curve.
- It is usually used only for functional testing.
- It requires a reasonably level of testing maturity.
- Some “pain” factors that can occur after one has started to use MBT can be:
- Outdated requirements that can build the wrong model.
- Inappropriate use of MBT: It takes some experience to know which aspects of the SUT should be modeled and which should be tested manually.
- Time to analyze failed tests: we must decide if the failure is caused by the SUT, the adaptor code or an error in the model. But the test sequences generated by MBT may be more complex and less intuitive than a manually designed test sequence.
- Useless Metrics: It becomes necessary to move toward other measurements of test progress because the number-of-tests metric is useless
- It cannot guarantee to find all the differences between the model and the implementation.
Some Model-Based Tools
- Case Maker
- Conformiq Test Generator
- LEIRIOS Test Generator – LTG/B or LTG/UML
- Zigma Test Tools
The proposed approach for “Model- Based Testing” helps companies to be more effective executing the quality process. Based on the fact that the tests are regenerated once the model is updated, thus avoiding manual rework.
In my opinion, the most difficult part for implementing Model-Based testing, besides the initial learning process, is not only modeling the system under test, but modeling a partial description of it in a detailed way. You need to have experience on how to do this to be successful. In that initial modeling process you can also detect failures in the requirements and the system itself, which is also an important advantage of model-based testing.
Despite the time consumed and the learning curve, the initial modeling process is in fact where the importance of model-based testing is; once you create an appropriate and valid model is just a matter of running the model-testing tools and generate a complete set of test cases that can be executed over the System Under Test (SUT) automatically.
You can always get feedback and improve the process and the effort of writing models; however its implementation could be hard and slowly and it’s based on experience and costs of modeling designing.
- Utting, M., & Legeard, B. (2006). Practical model-based testing a tools approach. San Francisco, CA: Morgan Kaufmann.