Method of Symbolic Test Scenarios Automated Concretization

When providing correctness checking for the models of software systems which include data types with wide range of values, a single symbolic behavioral scenario may cover a set of equivalent scenarios with concrete values. This feature is especially useful for systems with integer data types. However symbolic scenarios cannot be used as executable tests, they shall be concretized prior to execution. On the other hand, modern industrial software projects may contain many thousands of tests with nontrivial dependencies between their parameters. Manual selection and insertion of concrete values is impossible which requires full automation of the process. Besides, the actual experience in modern testing shows that efforts on bugs detection decrease significantly when directed method of selecting values is used instead of random selection of values. Concretization process shall follow a test plan prepared by tester. Such plans shall be flexible and generated based on standard templates or plans modified by user. Method of symbolic test scenarios automated concretization which resolves mentioned issues is described in the article. It allows to control coverage of boundary test parameters values which increases the quality of developed software. The developed method was successfully integrated into single complex technology of verification and testing which includes creation of a formal model based on initial requirements, automated symbolic verification, generation and concretization of symbolic behavioral scenarios, generation of test sets based on concretized scenarios and analysis of tests execution verdict. Results of method application within integrated technology are also shown.


Introduction
In the scope of software lifecycle the cost of software defects increases dramatically in accordance with development stage [1]. Avoiding defects on the stage of requirements gathering and detecting them on early stages of project lifecycle reduces the amount of corrections in the software and overall cost of development. This makes usage of methods and tools for model-based verification and testing extremely valuable [2,3]. However in the toolsets which mainly resolve problems of model-based approach (automation of requirements formalization, creation of behavioral models, verification of generated model-based scenarios, requirements coverage analysis [4][5][6][7]) arises the combinatorial explosion problem of possible behavioral scenarios which shall be tested [8][9][10][11]. Methods of symbolic verification are very effective to reduce the behavioral space. It is possible to specify ranges of possible parameters values in symbolic scenario. Each symbolic scenario represents a set of concrete scenarios with equivalent behavior (with same sequence of events). This means that to provide required coverage of complete model behavior it is enough to select several specific scenarios from each group of behavioral equivalence instead of having to check all possible parameters values. This allows to significantly reduce the number of scenarios covering the functionality of application in the scope of selected coverage criteria. However for code generation of executable tests only scenarios with concrete values of parameters are needed. Given that modern industrial software requires many thousands of tests with complex dependencies of parameters values it is impossible to manually count and insert appropriate concrete values based on ranges in symbolic scenario. The concretization process shall be completely automated. This paper describes the automated concretization process for symbolic test scenarios in the scope of VRS/TAT toolset [12] providing automated generation of test scenarios based on requirements specifications formalized with basic protocol notation [13], which is a representation of Hoare triple [14].

Overall Scheme of Concretization
VRS includes symbolic trace generator STG [15] which observes the formal model behavioral space and creates traceslinear sequences of events in the model. Model states are also saved in traces. The mail tool for concretization is called Trace Concretization Tool (Fig. 1). It consists of three modules -Concretizer, ValueCalculator and Concretization View which interact between each other.

Steps of Concretization Algorithm
Concretization process is iterative, on each step a single parameter is concretized. The process terminates after concretization of the last parameter in the trace. Below some definitions are introduced. Transition in the formal model in VRS terms is a basic protocol representing parameterized transition from one model's state into another. Basic protocol B(x) is represented by the following expression: a formula of basic logic language, which are called precondition and postcondition respectively; P(x)a process of basic protocol (in current casea sequence of parameterized signals in MSC format). Trace parameters are parameters of its signals. Formula of basic language may contain variables and constants, arrays of elements of simple types, functional types. Variables which may change their values during system execution are represented by attributes and attribute expressions. Trace is a sequence of the following type: where S are model's states, Bbasic protocols, xlists of their parameters. The following steps of concretization algorithm can be specified:  restore of initial symbolic trace  obtain ranges of allowed values for basic protocol's parameters  interactive concretization of trace parameters  save concretized trace. All steps except interactive concretization are executed automatically by internal means of VRS and hidden from outside. The most interesting for the user are implemented tools of the concretization which provide the control of concretization process and make the technology flexible enough for testing all modes of software functionality.

ValueCalculator Tool
This tool implements automatic calculation of concrete values for symbolic parameters within test scenarios. One or several rules can be used for calculation: left value of the range, middle value or right value. Examples of values calculated based on ranges and selected rule are shown in the Possible values for each parameter on each step of behavioral trace are calculated automatically by the means of VRS. Selection of the rule for value calculation is provided by corresponding set of options (Fig. 2):

Fig. 2. Options for selecting concretization rule
Based on calculated values of symbolic parameters the STG creates traces with concrete values which can be executed on the model. When two or three rules are selected there will be two or three concretized traces generated for each symbolic scenario.
An example of tool execution is shown below. Test scenario contains a signal which turns on a radio station on the car radio. Radio station number is the signal's parameter (Fig. 3):

Fig. 3. A part of symbolic test scenario
If overall number of radio stations is 9, ValueCalculator will calculate the following values for the channel_number parameter depending on selected concretization rule: "1" (for the Left rule), "5" (for the Middle rule) and "9" (Right rule). If all three options are selected (Fig. 2), there will be three concretized traces generated with different values of channel_select parameter. A part of concretized trace with Right rule value selection is shown below (Fig. 4):

Fig. 4. A part of concretized trace with right value selected
The user can select default concretization rules and repeat generation of concretized traces with corresponding values or use ConcretizationView tool to create own test plan.

ConcretizationView Tool
This tool provides the ability to specify any concrete values from the possible range for one, several or all parameters in test scenario. The tool is implemented as a View element in Eclipse IDE. It allows to display the contents of concretization table and specify desired values of symbolic parameters. This is performed by adding "C" symbol on the row with required parameter in the "Rule" column and desired value in the "Value" column. Continue with the example of turning on a radio station of the car radio. If the range of parameter's possible values varies between 1 and 9, then for example value 7 is neither left, nor middle, nor right value of the range. The only possible way to concretize a trace with this value is to explicitly specify it using ConcretizationView tool (Fig. 5):

Fig. 5. ConcretizationView user interface
As a result the concretized trace with value 7 will be generated (Fig. 6):

Fig. 6. A part of concretized trace with user-defined value
Applying ValueCalculator and ConcretizationView tools together the user can obtain all tests required to satisfy specific test criteria. For example, a set of tests covering all possible values of one parameter and only boundary values of another parameter. The concretization process terminates when the complete set of tests required for execution is obtained.

Results
Created tools were applied for preparing tests in telecom software projects. Symbolic scenarios of possible systems behaviors contained up to several hundred of basic protocols. For testing process all symbolic parameters in generated scenarios shall be concretized which is extremely time consuming without tools of automation. For example, using described approach to concretization in a small project with 11 basic protocols allowed to concretize all traces in 2 minutes. For a project with 151 basic protocols the concretization took about 20 minutes. While manual concretization of such project takes about 3 working days. Clear that in projects with several thousand of basic protocols it is impossible to concretize symbolic scenarios without automation toolset. The table below shows the comparison between manual and automated approaches to concretization: