In cases where the predicted parameter cannot be estimated in a quantitative scale, it is evaluated in a qualitative scale. The qualitative estimates obtained in this case are the results of the ranking of objects, i.e. their location according to the degree of increase or decrease of the estimated parameter. The ranks assigned to objects indicate the degree of severity of the measured property, but do not say how much this property is stronger or weaker in objects with different ranks. Ranks can be assigned by increasing (the more pronounced the property, the higher the rank) or by decreasing (the stronger the property, the lower the rank).
In the case when the expert believes that all objects differ from each other in the property being evaluated, he assigns each object its own, different from others, rank, so the number of ranks is equal to the number of evaluated objects or in other words, the assessment is made on a scale with the number of divisions equal to the number of objects. If, in the opinion of the expert, among the evaluated objects there are those in which the measured property is expressed in the same way, then he assigns them the same ranks. Each repetition of ranks leads to a decrease in the length of the scale on which the ranking is made by one. As a result, if there are duplicate ranks in an expert’s assessments, his estimates can be compared with the estimates of another expert only if the number of recurring ranks coincides. Otherwise, it is not possible to compare these estimates with each other without special treatment, since these estimates are actually obtained by measuring using scales of different lengths.
The procedure for bringing the rank scores of experts to a comparable form is called rank standardization. Its content is easiest to understand with a specific example. We have rank assessments by three experts of the contribution of five objects (executors, departments, etc.) to the achievement of the common. The ranking is carried out in descending order (the first rank is the most pronounced property). The results of their work are presented in Table 1.
Table 1. Initial rank assessments of experts.
Spanish 1 | Spain 2 | Spain 3 | Spanish: 4 | Spanish 5 | Altogether | |
Expert 1 | 2 | 2 | 1 | 4 | 3 | 12 |
Expert 2 | 3 | 2 | 1 | 3 | 2 | 11 |
Expert 3 | 1 | 3 | 2 | 5 | 4 | 15 |
Total | 6 | 7 | 4 | 12 | 9 |
As can be seen from these data, the first expert assessed the contribution of the two performers in the same way, the second expert twice recognized the contribution of two pairs of performers as the same, and only the third expert considered that the contribution of each performer is different. As a result, estimates of the contribution of performers in scales of different lengths (from 3 to 5) were obtained. Confirmation that the estimates are not comparable with each other is the different amount of assessments for each expert. If they used the same scale, that amount would be the same.
In the process of standardizing ranks, instead of ranks, all dimensions of one expert are assigned places according to the following rule. All his grades are consistently viewed and there are those that have the lowest rank. The first object with the lowest rank is assigned the first place, all subsequent objects having the same rank are assigned places increasing by one. The seat number is also increased by one when moving to objects that have the next rank, and the process of assigning places is repeated, i.e. each subsequent repetition of the new rank is assigned a place one more. As a result of this procedure, we get the number of places for assessments of one expert equal to the number of evaluated objects (Table 2).
Table 2. Places for expert assessments.
Spanish 1 | Spain 2 | Spain 3 | Spanish: 4 | Spanish 5 | Altogether | |
Expert 1 | 2 | 3 | 1 | 5 | 4 | 15 |
Expert 2 | 4 | 2 | 1 | 5 | 3 | 15 |
Expert 3 | 1 | 3 | 2 | 5 | 4 | 15 |
Next, for each group of objects that have the same repeating rank (in Table 2, these are cells of the same hatching), the average value of the places is calculated. The found average values of places for objects with repeated ranks, as well as places (initial ranks) for other objects are standardized ranks (Table 3). The obtained standardized ranks, in contrast to the initial ranks, are comparable to each other since they are the result of measurement in the same scale for all examiners. The sum of standardized ranks for each object reflects the collective opinion of experts and is the final collective assessment (last line of Table 3).
Table 3. Standardized ranks.
Spanish 1 | Spain 2 | Spain 3 | Spanish: 4 | Spanish 5 | Altogether | |
Expert 1 | 2.5 | 2.5 | 1 | 5 | 4 | 15 |
Expert 2 | 4.5 | 2.5 | 1 | 4.5 | 2.5 | 15 |
Expert 3 | 1 | 3 | 2 | 5 | 4 | 15 |
Sum of standardized ranks | 8 | 8 | 4 | 14.5 | 10.5 | |
Results of the evaluation of performers by experts | 2 | 2 | 1 | 4 | 3 |
It should be noted that the standardization of ranks is not a procedure for simply uniformly stretching shorter scales to the required length. Lengthening scales in standardization is done by stretching them only in those places where there are repetitions.
The degree of consistency of the opinion of experts is assessed using the concordance coefficient . To do this, you first need to calculate the average amount of ranks 
according to the formula:
where: 
– number of experts;
– the number of evaluated performers.
For this example 
, next, for each object, you need to find 
– the sum of the squares of deviations of the sum of standardized ranks from the average sum of ranks 
:
where: 
is the sum of the standardized ranks for the i-th object.
The results of the calculations 
are summarized in Table 4.
Table 4. Calculation 
.
Spanish 1 | Spanish 2 | Spanish 3 | Spanish 4 | Spanish 5 | Altogether | |
The sum of standardized ranks (Table. 3.) | 8 | 8 | 4 | 14.5 | 10.5 | |
Average rank amount | 9 | 9 | 9 | 9 | 9 | |
Difference | -1 | -1 | -5 | 5.5 | 1.5 | |
Square difference | 1 | 1 | 25 | 30.25 | 2.5 | 59.5 |
If all the experts were unanimous, i.e. their estimates coincided, and they evaluated all the performers, since this turned out to be standardized ranks, then 
it would be the maximum. Let’s denote it with 
, it can be calculated by the formula:
The coefficient of consent is the relation 
to 
, if its value exceeds 0.4, then the opinion of experts is considered to be consistent.
For the example 
given = 90, 
/
=0.66>0.4 and the expert opinion can be considered consistent.