Practice guide for sensory panel training Part 3: Measurement of the performance of sensory assessors and panels

The requirements in accordance with DIN EN ISO 8586:2014-05 and training plans for sensory assessor and panel qualification were presented in Parts 1 and 2 of the practice guide for sensory panel training. Part 3 will now focus on measuring the performance of individual sensory assessors and entire sensory panels with different methodological approaches. The intention of the guide is to draw attention to the essential aspects to be observed when measuring the performance of sensory assessors and panels. The backgrounds and criteria for determining performance will first be described, and selected methods for measuring and optimising it will be presented; these can be carried out even without an advanced knowledge of statistics. Case examples will be provided for a practical illustration of various applications so that specialists and managers working in the field of sensory analysis and evaluation in the food and beverage sector can obtain valuable suggestions for practical implementation. However, these must always be adapted to the respective issue in the specific company and to the sensory evaluation projects undertaken there.

Besides the availability of the technical infrastructure that is crucial for sensory analyses and adherence to ‘good sensory analysis and evaluation practice’, one of the most essential prerequisites for reliable sensory analysis and panel results as well as for valid data in human sensory analysis is the qualification of the sensory staff. By no means least, the high professional importance of a high-performance sensory panel that also ideally supports product development and quality assurance in the sense of food fraud and food safely (see IFS, BRC standard, etc.) in the food industry is also made clear by the speedy amortisation of capital invested in food sensory analysis (see DLG Trendmonitor 2019).

Proof of professionally qualified sensory assessors and panels whose performance is monitored additionally concerns all organisations that operate as test institutions or test laboratories for third parties, use sensory methods in their food analysis and quality monitoring and have been accredited in accordance with ISO/IEC 17025:2018-03.

Figure 1 shows the most common areas in which sensory assessors and panels are used. As shown there, a distinction is generally made between the two method groups of discrimination tests and descriptive tests in terms of the respective problem definition in the case of sensory evaluation methods. There are also methods that are assigned to the discrimination tests, but use elements from the area of the descriptive methods in the form of variants:

•    Discrimination tests 

• Product comparison to determine (slight) differences that can be recognised by means of sensory analysis (e.g. Triangle Test, Duo-Trio Test, Paired Comparison Test, Ranking Test)
• Product comparison, particularly in the context of sensory quality control (e.g. IN/OUT Test, Difference from Control Test with the involvement of reference samples) While these are primarily discrimination tests in methodological terms, they can be extended with descriptive elements from the descriptive test methods in the respective descriptive variant (descriptive IN/OUT Test or Descriptive Difference from Control Test), however.

•    Descriptive tests
Determination and description of sensory product characteristics; the intensity attribute can additionally be tested in individual methods.

Sensory analysis methods that are focussed on objectively ascertaining sensory product differences are discrimination tests. In this process, slight sensory differences between products usually have to be recognised and named.

There are also the descriptive tests, which are used to describe sensory product characteristics (attributes, descriptors) in detail and to measure their intensity. Such sensory product profiles (often shown in the form of spider web charts) are required in order, e.g. within the context of product management, to compare competitors’ products with own products, to define sensory specifications for a new product and therefore also the basis (e.g. reference samples, also referred to as ‘gold standards’) for quality control or to identify sensory product changes on use of alternative raw materials, in the event of modified ingredients, after long periods of product storage or as a result of interactions with the packaging.

Sensory methods that are also used, in particular, in sensory quality assurance in order to ensure the consistent quality level of the manufactured products include IN/OUT Tests (Inside/Outside Tests) or Difference from Control Tests, for example. These are primarily assigned to the discrimination tests. However, descriptive variants also exist for both of these test methods. The objective of these sensory tests is to analyse presented products in comparison with a defined reference sample (e.g. product specification, ‘gold standard’) and to identify possible deviations. Based on the results achieved in this process, a product rating is then also carried out in the sense of product release, rework or product blocking.

Pursuant to the DIN 10973 standard, as distinction is made between three forms of IN/OUT Test, the general [categorical], scaled and descriptive test, each of which is used to test whether the presented sample lies inside or outside of the specifications in comparison with a reference. In the descriptive variant of the IN/OUT Test, the method can ‘be extended by simple profiling of the key attributes/descriptors in addition to the inside/outside assessment’. The descriptive part then follows on immediately after the discrimination part of the test. In terms of the discrimination and descriptive test elements, the same also applies to the Descriptive Difference from Control Test (DIN 10976).

While sensory assessors and panel members are trained specifically in terms of their sensory expressiveness and powers of description for their work in descriptive sensory panels, i.e. they have to internalise the identification and designation of characteristic sensory attributes/descriptors and their quantification on the basis of defined scales, focus in the qualification of discriminatory sensory panels is placed on the selective perception of what are usually slight sensory differences. High acuity of the senses and knowledge of the relevant product quality parameters of specific product categories are the essential goals of qualification in this case. Irrespective of the area of food sensory analysis and evaluation for which the sensory assessors and panels are qualified and used, regularly checking and monitoring their performance are crucial to the quality and meaningfulness of the sensory evaluation results to be delivered.
 

Various aspects have to be taken into consideration when getting the practical implementation of a performance review off to a start. Many companies almost certainly have company-specific requirements, e.g. with regard to data protection on the part of the works council or concerning the use of software tools on the part of the IT department, which can lead to restrictions in data evaluation. If the sensory assessors’ individual test results therefore can not be used for purposes of performance evaluation and monitoring or also for motivating the sensory assessors (i.e. to illustrate their own sensory skills) due to data protection reasons, not even in anonymised form, and the conclusion of confidentiality agreements or similar forms of ‘secure data usage’ is not possible either, compressed data in the form of the evaluation of group results provides indications for identifying performance-related weaknesses and improvement potentials, albeit with limited meaningfulness. Equally, this can be used as the basis for planning training measures and implementing training courses to further improve and optimise the sensory assessor groups’ sensory performance. The respective panel leader must determine which options are appropriate and permissible in this regard based on the company-specific circumstances, and must then ultimately evaluate and decide on them on an individual basis.

From a general professional point of view, a distinction can generally be made between various approaches to the practical implementation of a performance review and assessment for sensory assessors and panels (Rogers, 2018); these must be taken into consideration in the respective planning in advance:

1.    Analysis object:
It must be defined whether the monitoring of an individual sensory assessor’s performance is to be examined or the review of a sensory panel’s ‘bundled’ performance is to be analysed instead. These individual results can also be placed into relation with one another, meaning that a sensory assessor can be compared with a panel or a sensory panel can be compared with panels from other companies or company locations.

2.    Procedure:

a)    Ongoing/new projects:
Sensory assessor and panel data from ongoing projects can be used for further evaluations pertaining to performance or results from new projects developed specifically for monitoring can be used for assessment.

b)    Existing/modified product samples:
The product samples to be analysed can be either market samples or products from internal production. In addition, products can also be specifically modified (‘spiked’) for practice purposes, e.g. by reinforcing the impression of vanilla in desserts through the addition of a higher quantity of vanilla aroma or similar. The advantage of this is a detailed knowledge of the product, and it clearly ensures that there are differences in specific attributes. These are then used as key measurement variables for categorising the performance that has to be achieved by the sensory assessor or panel.

3.    Time period:
As shown in Table 1, the performance of individual sensory assessors or panels can be reviewed either selectively (at the end of a training unit) or also over a longer period of time within the framework of monitoring.

Sensory performance has to be trained on a continuous basis. Ideally, sensory assessors are deployed regularly and only with short interruptions. In such a case, the results from ongoing projects can be used for the performance review. Developing a corresponding concept and appropriate documentation that shows the separate evaluation of data from the actual sensory analysis project is then also recommended in this regard. In addition to evaluations from ongoing projects, determining performance with additional tests, possibly within the framework of regular training courses, or also with short tests prior to the actual project sessions is also recommended.

While the performance review in the area of descriptive tests is also extensively described and standardised in DIN EN ISO 11132:2017-10, with the result that company-specific options that are adapted to the respective product category can be derived on this basis for practical implementation, the design of measures in the area of ‘discrimination tests’ is the responsibility of the respective panel leader, who has to compile corresponding information from various literature sources if necessary. Regardless of whether the sensory assessors are used in the context of discrimination tests or descriptive tests, the three criteria of discrimination ability, repeatability/reproducibility and homogeneity, as described above, always apply as the basis for reviewing performance.

If the ability to discriminate is to be reviewed, the sensory assessors must be provided with several similar samples to test. When selecting the samples, it must always be ensured that the differences are neither too small nor too large, as the risk of subsequently interpreting the result as (excessively) good or (excessively) poor discrimination ability is otherwise run. A sensory assessor’s or the panel’s discrimination is poor if a) no sensory difference is ever found or b) what are actually different products are evaluated as very similar in terms of their sensory attributes and/or intensities. Training courses that train both the acuity of the senses (i.e. Ranking Tests, Triangle Tests) and also the use of scales (i.e. the representation of the extent of the difference) help to improve the ability to discriminate.

The quality of the repeatability is clearly indicated by comparing the test results of both individual sensory assessors with themselves and also those of the sensory panel as a whole with itself. In this process, identical samples should also be rated identically when they are assessed again, i.e. identical or similar results and scale values should be delivered. ‘Concealed double samples’ that are integrated into a single session and presented to the sensory assessors as an equivalent product (same batch) (i.e. same sensory assessors/panels, same product, same place, same time) are suitable for reviewing this capability, for instance. ‘Repetition sessions’ are an additional option in this regard. In this process, the sensory assessors are required to complete several similar sessions, i.e. the defined sensory tests with the same samples (same batch, otherwise comparable samples) are carried out again at certain intervals over a long period of time (i.e. same sensory assessors/panels, same product, same place, different time).

Good homogeneity is ensured if the degree to which the sensory assessments of the assessors match compared to those of other assessors and the result achieved by the sensory panel as a whole is high. This can be clearly determined by means of a data evaluation.
 

There is also the option of data comparison with other sensory panels in this context. Such round robin tests (proficiency tests) are conducted to review the respectively used analysis method and to compare the results of one’s own sensory panels with those of other panels. Round robin tests, which have already been established in chemical analytics for some time now in the context of good laboratory practice (GLP), can be organised on a product-specific and intracompany (e.g. comparison of sensory panels at various company locations) or industry-specific (e.g. cross-company comparison of sensory panels) basis in food sensory analysis. To do this, the organising body sends uniform samples for all sensory assessors; these have to be analysed according to the same previously defined test methods within a specified period of time by the participating sensory panels (i.e. different sensory assessors/panels, same products, different places, different times). The submitted results can be evaluated according to the previously described criteria to measure the performance of a sensory panel.