|
[back to Resources]
The editorial process through the looking glass
Daniel Gile - Université Lumière Lyon 2
Gyde Hansen - Copenhagen Business School
Published in Gyde, Hansen, Kirsten Malkmjær and
Daniel Gile (eds). 2004. Claims, Changes and Challenges in
Translation Studies. Amsterdam/Philadelphia: John Benjamins. 297-306.
Abstract
Editors of scholarly publications are gatekeepers
who decide whether to publish manuscripts or not, but their decisions
are constrained by institutional factors, sociological factors, and
the availability of referees for peer-reviewing. The quality of the
input of such referees depends not only on their thematic and methodological
expertise, but also on their motivation, on their personal bias and
on sociological factors. Although this suggests that editorial decisions
are less objective and reliable than their idealized representation,
the referees’ comments are often convergent to an encouraging degree.
The main comments made by referees on 42 manuscripts submitted to these
proceedings are listed. They share a large common denominator, which
suggests that the fundamental norms they are based on are the same.
1. Introduction
Conference proceedings are a well-established tradition,
and are often awaited impatiently not only as a reflection of the meeting,
but also as a publication vector for participants, and frequently as
a justification to their university for their presence and participation
in the event. However, publishers do not necessarily allow sufficient
space for all papers to be published, and editors must choose. How they
do this is not always clear to authors who submit their manuscripts.
Basically, the process is very similar to the editorial
process in other types of collective volumes and in journals in that
it is based on the peer review system. This paper discusses the concept,
as well as recurrent issues, in particular as they crystallise in the
field of Translation Studies; it then discusses issues in refereeing,
and briefly mentions follow-up possibilities after comments are received
from the reviewers. This paper is based on its authors’ experience as
editors in the field, and also includes, by way of illustration, a case
study with an analysis of the most common types of weaknesses reported
in papers submitted for publication in proceedings in the field of TS.
2. The editorial process
2.1 The editor's role as a gatekeeper
In the scientific community, where a scholar’s publications
are his/her visiting cards, passports and stepping stones towards professional
promotion, editors of journals and collective volumes have an important
three-pronged institutional role to play. Firstly, they create and offer
publication space to individual scholars. Secondly, they act as gatekeepers
and guarantors of the scientific quality of texts they publish (see
Judd et al. 1991:17f). Thirdly, through the peer-review system, they
offer guidance and text-improvement opportunities.
As gatekeepers who decide to publish or not to publish,
editors seem to be in a position of power. And yet, their action is
constrained by several factors, including constraints from the publishers
and pressure from the research community, difficulties in finding adequate
referees for proper implementation of the peer review system, and “political”
and diplomatic considerations that inevitably influence their decision-making
process.
2.2 Constraints and policies
Thus, a system which is theoretically designed to
provide reliable guidance and assessment to editors in their role as
gate-keepers in the publication process has in-built limitations which
can be difficult to overcome.
To these limitations, one should add the expectations
of the publisher in some cases, and of the researchers' communities
in other cases (for instance, when editing the proceedings of a conference),
let alone the expectations and reactions of individual researchers,
who may need to publish for institutional or personal reasons and who
may find it difficult to accept the editors' decision to reject their
paper, or even the critical comments from the referees.
Several types of policy are observed in the field.
The easiest one to implement consists in accepting virtually all papers,
and merely performing some cosmetic surgery on the wording, correcting
errors in the lists of references, re-formatting papers to applicable
standards. While this may not be the most efficient policy in terms
of scholarly quality and promotion of solid scientific norms, it can
be considered legitimate when the purpose is only to reflect the state
of the art in a field, or to provide the written record of a scholarly
meeting.
At the other extreme, editors may decide to implement
a strict quality policy, accepting only manuscripts which pass severe
scrutiny. Such a strategy is possible for editors of prestigious journals
in established disciplines where there is much research production and
much competition. In TS, due to the small size of the community and
the general state of research, it is doubtful whether any editorial
board can afford to act in such a strict manner.
Most often, the editorial policy with respect to quality
is somewhere in-between, and quality thresholds for acceptance vary
as a function of supply and demand: if many good papers are offered
and fill the required number of pages, there is no need to accept mediocre
submissions. When there is more publication space than material, acceptance
standards fall.
In such a situation, it is difficult to set standards
in advance. For each issue of a journal or collective volume, submissions
may have to be refereed first, and the decision to publish or not to
publish is taken when the total picture is clear.
2.3 Referees in the peer review system
As its name indicates, the peer review system is based
on the work of scholars with a similar background to the authors', whose
role is to provide useful input, which takes the form of both assessment
for the benefit of the editor, and guidance to the authors. It is significant,
but not often noticed, that in its very name, the system refers to “peers”
rather than to “experts” or some other authority-carrying agent: the
idea, which is corroborated by experience very regularly, is that even
without a substantial difference in training and experience between
referees and authors, the very fact that it is easier for anyone to
detect another person's weaknesses than one’s own, makes it possible
for the former to help the latter (“…well-trained scientists can make
mistakes and sometimes overlook important problems of design” – McBurney
1990:16).
So much for theory. In practice, the system is a bit
more complicated. Firstly, "peers" are not always peers. Sometimes,
acknowledged authorities are asked to do the reviewing, and sometimes,
referees are much less experienced and qualified than the authors themselves.
In the 'guidance' part, i.e. specific comments on errors or weaknesses
and specific suggestions for improvement, this is generally not too
problematic. In fact, excellent criticism is often received from conscientious
beginners, who may react with high sensitivity to all segments in a
text which contain unclear, ambiguous writing or faulty logic, whereas
more experienced readers already familiar with the field, who may also
be busier, may focus on more specific points and overlook such segments.
With respect to overall assessment of manuscripts by referees as opposed
to specific comments, experienced, more highly qualified scholars are
clearly in a better position than beginners to provide useful input.
In established disciplines with a large community of
scholars, referees may not be too difficult to enlist. In other cases,
and in particular in the rather small, but diversified TS community,
finding such reviewers may be a real problem. Firstly, in some sub-areas
of a discipline, there may simply not be more than a very small number
of scholars competent enough for refereeing purposes. The problem in
international events (as opposed to national ones) is compounded in
papers written in languages other than English, which has more or less
become the international language of science, in the field of TS as
in other fields. More generally, refereeing effectiveness can be discussed
under several components:
2.3.1 The language issue
Though not a scientific or intellectual criterion, the language barrier
is formidable, even in a multilingual society such as the TS community
(or perhaps especially in this community, where authors may insist
on their right to write in their language and count on their fellow-researchers’
comprehension). Not only are language skills essential when assessing
and/or correcting the style of a text, but they are also important
in terms of comprehension, especially in theory-oriented papers. How
many TS scholars can review German texts on court interpreting, Spanish
texts on research methodology, French texts on contrastive linguistics
and translation? The problem was a very concrete one in this volume
as well, when scholars well versed in certain fields turned out not
to have the language in which the relevant papers were written. In
several instances, the less than ideal choice was between refereeing
a paper written in a language one knew well, but in a field not familiar
enough, and refereeing a paper addressing a field one knew well, but
written in a language in which one did not trust one’s comprehension
ability fully.
2.3.2 ‘Thematic’ expertise
The requirement for sufficient relevant knowledge on the specific
theme in the field is obvious, but in practice, it is not always easy
to meet, because editors do not always have access to referees with
the required qualifications. When they lack ‘thematic’ knowledge,
readers may miss errors in the manuscript, especially in the literature
review and references, but also in the interpretation of quotations
and in the use of concepts. On epistemological and methodological
issues, they may also fail to see the relevance of certain points
made by authors, or miss gross errors.
When the referee’s lack of familiarity with a field
leads to unjustified criticism, the refereeing process can become
counter-productive. The most conscientious and careful referees faced
with this dilemma tend to focus on the structure and rationale of
the paper and to formulate questions rather than assertive criticism
when in doubt, thus alerting the authors and editors to the existence
of potential problems, without making any damaging claims.
2.3.3 Methodological expertise
In a field as heterogeneous as TS, research paradigms are diverse
and quite different from each other. Some scholars may adhere to the
methods and norms of cognitive psychology, others to those of literary
studies, of philosophical discourse, of sociology, of formal linguistics,
etc. These are not necessarily mutually compatible, and if referees
adhere to one paradigm and the author of a refereed text to another,
this may lead to negative assessments predicated on an opposition
between schools of thought, not on actual weaknesses in the paper
within the paradigm chosen by the author. Challenging a paradigm is
legitimate, and a potentially powerful driver of scientific progress,
but criticising an author’s method using the criteria of a paradigm
other than the one s/he has chosen to follow can be counter-productive
(see Kjørup 1996: 109, 113, 116 f.).
2.3.4 Motivation
However ‘objective’ and impersonal the peer review system is supposed
to be within the idealistic view of science, personal factors may
come in strongly in the refereeing process. Critical reading is serious
work, as it involves reading a text carefully, making sure one understands
it, assessing it, reporting on the findings, and making suggestions
for improvement. If there are serious design flaws, and if weaknesses
are numerous, it may take much time and effort to list them comprehensively
and to suggest alternative formulations, inferences or actions to
help an author understand the problem and do the appropriate remedial
work. In such cases, the willingness of referees to devote considerable
work to the process can make much difference, and it is not rare to
see one reviewer only send back one or a few general paragraphs as
input to the editors, and another offer several pages of detailed
comments on the same paper.
2.3.5 Personal bias and sociological factors
One final component, which is not in line with the ideal of objectivity
in science, but which is nevertheless endemic in human society, is
personal bias. Scientific society is a competitive one, both in terms
of theories and schools of thought and in terms of personal rivalry
(see Plastow and Igarashi 1989). Actually, as analysed by Kuhn and
many others (see Kuhn 1970, Chalmers 1990), such sociological forces
may reasonably be considered useful in that they drive researchers
into accumulating evidence to attack existing paradigms (or defend
them, depending on which side they are on), thus contributing to scientific
development. Nevertheless, in an activity having such a strong assessment
component as refereeing, positive or negative attitudinal bias, whether
conscious or subconscious, can result in unbalanced judgement. Scholars
may be eager to find fault with their opponents' or rivals' work,
or be reluctant to criticise their allies' and friends' writings.
The problem is compounded by status issues: when
reading a well-known, experienced scholar's work, referees may be
reluctant to criticise, not only out of respect for the author, but
also out of fear of ill feelings or 'retaliation' if recognised. Indeed,
in a community as small as that of Translation scholars, it is often
easy to recognise an 'anonymous' author and an 'anonymous' referee
through their distinct style, ideas and references.
3. Common problems in submitted papers: a case study
3.1 Introduction
After the EST Congress in Copenhagen, more than 50
papers were submitted for inclusion in the proceedings. Following the
tradition and conventions of scientific editing and publishing, they
were submitted to the peer-review process. All papers were reviewed
by at least 2 referees, and most of them by three. In a few cases, only
general assessments were received (see above), but specific comments
were made on 42 of them. Referees mentioned both positive aspects, such
as the presence of innovative components, interesting information, interesting
ideas, etc., and problematic areas. In line with its focus as explained
in the introduction, this paper analyses the latter, where constructive
work could be done, but should not be misinterpreted as a pessimistic
statement or diagnosis.
The following types of perceived problems were reported
by referees. Since this is a single case study, they are given not as
a demonstration of any general proposition, but by way of an illustration
of what we consider a not atypical pattern.
Also note that for reasons that readers will understand
easily, no attempt will be made to present the full set of data, with
all the comments made by each referee on each paper. This information
will remain confidential, and only a synopsis will be given here, in
the hope that it will be sufficient to provide interesting and useful
input.
3.2 Frequently mentioned weaknesses in papers for
this volume
-
The most numerous comments from referees report
missing data: in order to gain better understanding of the author’s
claims or conclusions, reviewers felt that they needed data which
was not provided in the relevant papers, on experimental conditions,
on actual results, on the nature of the texts involved, etc. (also
see Strauss/Corbin 1998: 268ff, Judd et al. 460ff.) This problem
was reported in 18 out of the 42 papers (43%) for which detailed
comments were received from referees.
-
In a related but not identical category, in 11
papers (26%), reviewers reported problems with the bibliographical
references, which were missing, incorrect, and/or used inappropriately
from secondary sources (also see Gibaldi 1999).
-
The second largest number of comments pertained
to weaknesses in the rationale of papers. Reviewers reported that
authors made unsubstantiated or excessive claims, speculated, mixed
categories, over-generalised, and more generally made inferencing
errors in 17 out of the 42 papers (40%) (see Judd et al. 1991: 27ff,
258 f. on various types of validity and bias).
-
This problem in the rationale is linked with problems
in the design of empirical studies. Out of 20 papers reporting such
studies, 12 (60% - the highest proportion) were reported as having
a design that did not correspond to the stated aim of the study,
or as comparing groups that were not comparable, or using samples
that were too small, or not representative, or as failing to control
relevant variables (also see Judd et al. 1991: 75ff, 215ff.).
-
The third largest set of comments considered that
papers (13 out of 42 - 31%) presented no innovation or added value.
-
Other frequent comments had to do with specialised
concepts, which were not defined at all or not defined clearly enough,
or used in a way different from their established use in other disciplines
or in paradigms from which they were taken (9 papers out of 42 -
21%).
-
Referees also commented on the excessive length
of papers or segments thereof, in particular the introduction (8
papers out of 42 - 19%).
-
Still other comments reported clumsy or pompous
style, repetitions in the structure of the paper, missing units
in graphs, abbreviations and acronyms which were not spelled out,
lack of indications as to the objectives of the papers or studies
(see Judd et al. 1991: 472 about procedure and style).
3.3 Assessing the truthfulness of the referees’
comments
Before analysing these data, a few points must be
made: Firstly, no strong claim is made as to the degree of representativeness
of the sample of papers, of referees and/or of comments. The numbers
are given here as a rough indication. Nevertheless, the following could
be pointed out:
-
The set of more than 42 papers reviewed for these
proceedings can be considered of reasonable size for a sample of
this nature, and we (the authors of this paper and editors of this
volume) did not feel it differed markedly in respect of problem
areas from other sets of papers received and reviewed for other
volumes in the field in recent years.
-
An important point is that while there were a
few disagreements (less than a handful) between the referees and/or
editors on the general value of papers, no disagreement was expressed
on any specific comment. Thus, while it is not claimed that referees
are holders of the truth (they are as fallible as any scholar -
see the discussion in Section 2.3), in this case, there is good
reason to consider that their judgement on specific weaknesses has
some reasonable inter-subjective value
-
Referees and editors come from different countries
and have different backgrounds. These factors suggest that the picture
which emerges is not likely to be a highly biased reflection of
the general situation in TS. The only potentially marked difference
between this case and other, similar cases, is a possibly stronger
will on the part of the editors (there was at least one editor among
the referees of each paper) to stress quality in the editorial process,
in spite of the other factors mentioned in Section 2.1.
3.4 Analysing the data
Looking at the data, it appears that referees have
reacted on the basis of a number of norms, which are made explicit in
research textbooks (see for instance, McBurney 1983, Shipman 1988, Babbie
1992, Judd et al. 1991, Gibaldi 1999, Strauss/Corbin 1998), in particular
the requirements that science be explicit, in providing all the information
required for readers to be able to understand and assess reports on
scientific studies (“Great care is taken to specify the exact conditions
under which observations are made so that other scientists can repeat
the observations if they desire and can try to obtain the same results
as the first scientists”- McBurney 1990: 7; also see Strauss/Corbin
1991: 266f about replicability), that it be logical in its inferences,
and that scientific endeavours contribute at least some innovation,
rather than just review existing data and information (in the words
of the editor of the Journal of Bone and Joint Surgery
who gave a plenary at a medical conference in Paris in the early nineties,
“the reviewer will ask three questions: Is it true? Is it new? Does
it matter?” - also see Beaud 1988).
A strikingly large proportion of papers received, more
than one in three, did not observe these principles. This suggests a
lack of specific awareness of the importance of these norms in research
which, in turn, could be ascribed to the lack of appropriate training
in research principles and their concrete implementation.
A third interesting observation is the relatively high
number of problems arising from trans-disciplinary ventures. As they
explore new avenues with theories and concepts from related disciplines,
scholars with a solid track record in TS seem to become susceptible
to misperceptions. Beyond the case study of this volume of proceedings,
problems have been observed when TS scholars without the corresponding
background ventured into statistics, philosophy of science, linguistics,
and cognitive psychology. Such weaknesses could probably be avoided
by co-operating with researchers from the relevant disciplines, and,
of course, by appropriate training in such disciplines.
4. Referees’ comments and beyond
The next and obvious questions are, firstly, what authors
can/should do after they receive the referees’ comments. Most authors
only view the peer review process as an assessment, and focus on its
short-term effects in terms of acceptance or rejection of their manuscript
and of requirements for rewriting. However, it could also be an excellent
opportunity to improve research and writing skills. Comments can be
classified into three levels of
-
Some comments are easy to deal with. Following minor
suggestions for grammatical correction and stylistic improvement,
correcting factual errors, correcting errors in the bibliographical
references, improving the graphical presentation of data are ‘mechanical’
and should cause no problems to authors.
-
At a second level of difficulty, the referees’ suggestions
challenge the authors’ ‘local’ assessments (on specific points): does
a previous finding really mean what the paper says it means? Does
another author really say what the paper reports it is saying? Is
a quoted piece of work good enough to be used as a reference? Should
another study not be cited to strengthen the case? Is the newly coined
terminology appropriate, or should it be replaced by better terms?
Should a particular assertion based on the findings not be toned down?
-
At the highest level of difficulty are suggestions
that challenge fundamental choices and inferences made by the author.
In some cases, if the design of an empirical study is seriously flawed,
if a theoretical discourse on a concept, model or theory is predicated
on a serious misunderstanding of its nature, the paper cannot be salvaged.
In other cases, it is possible to improve it without starting again
from scratch. For instance, the data may be re-processed with a different
method, and findings can be re-interpreted, which may lead to a new
conclusion, which may not be in line with the ideas that authors set
out to defend. Again, this may be associated with considerable psychological
difficulty for them.
5. Conclusion
The peer-review process and the subsequent decisions
and actions discussed here are not the only components of editorial work.
Upstream, there are contacts with a or several publisher(s), definition
of editorial policy, sometimes calls for papers with specific guidelines
as to the length and format of texts. Downstream, when revised papers
have been accepted, there is still much reading, checking, writing, copy-editing,
etc. However, the most stimulating and constructive part certainly lies
in dealing with the issues highlighted in this paper.
The preparation of proceedings involves a chain of actions,
decisions and reactions from authors, referees and editors. However clear
and straightforward the principles may be, their implementation is therefore
lengthy, complex, and sometimes a bit painful. However, if referees and
editors have enough time and good environmental conditions for serious
work on the manuscripts, and if authors accept their comments as potentially
useful input, the editorial process offers excellent skill honing opportunities
for all parties in the process.
References
Babbie, Earl. 1992. The practice of social
research. (Sixth Edition). Belmont, California: Wadsworth.
Beaud, Michel.1988. L’art de la thèse.
Paris: Editions la découverte.
Chalmers, Alan. 1982. What is this thing
called Science? (Second Edition). Milton Keynes: Open University
Press.
Gibaldi, Joseph. 1999. MLA Handbook for
Writers of Research Papers. (Fifth Edition). New York: The Modern
Language Association of America.
Judd, Charles M. & Eliot R. Smith & Louise H. Kidder.
1991. Research Methods in Social Relations. (Sixth Edition).
Orlando: Harcourt Brace Jovanovich College Publishers.
Kjørup, Søren. 1996. Menneskevidenskaberne.
Roskilde: Roskilde Universitetsforlag.
Kuhn, Thomas. 1970. The structure of scientific
revolutions. (Second Edition, enlarged). Chicago: The University
of Chicago Press.
McBurney, Donald. 1990. Experimental Psychology.
Belmont, California: Wadsworth.
Plastow, Michael & Yoshihide Igarashi. 1989.
The Mind of Science. Tokyo: Kyoritsushuppan.
Shipman, Martin. 1988. The limitations
of social research. (Third Edition). London and New York: Longman.
Strauss, Anselm & Juliet Corbin. 1998. Basics
of Qualitative Research. (Second Edition). California: Sage.
[back to Resources]
|