Less-than-ideally-competent assessors in TS

Daniel Gile

Peer reviewing, screening of papers for conferences and assessments for professional promotion of academics are all-pervasive forms of research evaluation. Ideally, they should be conducted by evaluators well versed in the relevant fields, topics and research methods. In TS, the small size of the TS community, the diversity of research issues and paradigms, the absence of mature methodological traditions, language barriers etc. generate a situation somewhat remote from ideal conditions, and much assessment of research is done by evaluators who do not have the required thematic and/or methodological expertise. I often find myself in this situation, having to referee a paper or assess doctoral work in an area and/or using a methodology with which I am not familiar enough because another evaluator with proper qualifications who knows the relevant language(s) is not available.

LICAs (Less-than-Ideally-Competent Assessors) may fail to identify significant added value – and/or to detect methodological flaws in a work. The former does not do justice to the author, and the latter can be damageable to many more people, especially in the case of a PhD dissertation: once a researcher has been awarded a doctoral degree, s/he gains official academic ‘respectability’ and may be put in charge of younger scholars. If the degree awarded does not reflect genuine competence, deleterious consequences may extend over several generations of researchers with a significant effect on the relevant community.

Poor presentation and overall weaknesses in an author’s rationale, including sampling procedures and general inferencing from data, can be detected with a sharp mind without specific expertise in the relevant area – though some training in critical reading can hone relevant skills for scholars without training in empirical research methods. LICAs can offer useful contributions to both authors of the work assessed and editors/scientific committees, but are likely to fail to detect omissions and misrepresentations of theories and other contributions as well as incorrect application of specific research methods.

It is therefore important that whenever possible, in every team of assessors, at least one expert be involved. If assessment is done by LICAs only, risks of an inaccurate outcome of the procedure cannot be avoided. Such less-than-ideal-assessment situations are still frequent in TS. For historical reasons, they are part of our present environment, and should be acknowledged as one of our weaknesses. The situation is improving steadily as the mass of TS work done at advanced levels is expanding, and more works are assessed by TS experts well versed in the relevant areas.

But we LICAs who are still called upon to help need to be aware of our limitations and associated risks - and to be humble and conscientious in our work. As to colleagues being assessed, they should be encouraged to present their work and its added value as explicitly as necessary for potential LICAs and to remember that assessments may well be the best that colleagues can offer them at this point and will probably be a source of useful input - but are still fallible.