The peer review system is usually used to evaluate research proposals and scientific articles that are submitted for funding and publication respectively. As jobs and careers of researchers are at stake by a result of a peer review evaluation, it is not surprising that the effectiveness and fairness of the system are matters of great concern. Despite the apparent importance of peer review in influencing funding decisions, and despite growing criticism of alleged deficiencies and biases in peer review systems, not very many empirical studies have been done. Moreover funding institutions would rarely question the validity of their procedures and even when they do they do not publish the results.
Scientifically small countries, for example Korea, have adopted the peer review system from the developed countries without probing it to find out whether it is appropriate for their situation, although it is argued that the traditional peer review system is inadequate and probably dangerous for such countries because of the domination of personal relations and politics. Thus there is great need to examine its workings and unintended side effects of peer review in scientifically small countries. This study focuses on the peer reviewer processes of research proposals for funding.
The main objectives of this study are as follows:
(1) to identify biases and the factors influencing the evaluation outcome of peer reviewers in assessing research proposals.
(2) to measure the degree of the evaluation errors (leniency tendency and halo effects) and to find out their affecting factors.
(3) to inquire into the factors having influence on the interreviewer agreement and relability.
(4) to measure applicants' reaction to the peer review system and to examine its influencing factors.
(5) to assess the potential effectiveness of blind review and to inquire into the factors influencing author recognition of reviewers in blinded research proposals.
Eighty-four hypotheses were generated and tested using two kinds of data, one collected from a questionnaire survey of 1,098 reviewers and 476 professors who applied for research funding from the Core Research Support Program(CRSP) of the Korea Science and Engineering Foundation(KOSEF) in the areas of mathematics, physics, biology, computer science, and electrical engineering during fiscal year 1996, and the other acquired from the data base of evaluation scores of their research proposals. Correlation analyses, path analyses, moderated regression analyses, t-tests, and multiple discriminant analyses were employed to test the hypotheses.
The major findings of this study are as follows:
(1) There were high correlations between reviewer ratings and the following characteristics of ① the applicants: prestige rank of current university, prestige rank of undergraduate school graduated, nation of Ph.D. training, professional age, academic recognition, KOSEF funding history, and experience in helping KOSEF, ② the research proposals: research stage. innovativeness, and mainstream/anti- mainstream, ③ the reviewer-applicant relationship: difference of prestige ranks of current university between a reviewer and an applicant, degree of familiarity, perceived similarity, and liking, and ④ the reviewer-proposal relationship: similarity of research areas. Also, reviewer ratings and the previous scientific performance of the applicants had significant influence on funding decisions.
(2) The following characteristics of the reviewers had significant correlations with leniency tendency: accountability to the review, need for affiliation, satisfaction with KOSEF funding history, perception of the intrinsic rewards of review, professional age, and age. And the following characteristics had significant correlations with halo effects: innovativeness of research, mainstream/anti-mainstream of research, number of papers published in Korean journals by reviewers, and liking.
(3) Interreviewer agreement was affected by an proposal characteristic of mainstream/anti-mainstream of research, and interreviewer reliability by innovativeness of research and variance of rating experiances of reviewers.
(4) Funding decision had high correlation with dissatisfaction with review structure and process, perception of the effectiveness of feedback, and applicants' reaction to review. Dissatisfaction with review structure and process, and perception of the effectiveness of feedback mediated the relationship between funding decision and applicants' reaction to review. And, satisfaction with KOSEF funding history of applicants had moderating effects between funding decision and review satisfaction and between funding decision and review system satisfaction.
(5) Blind review was found to be effective on the whole at keeping an author's identity secret from an experienced reviewer. Only 39.4% of the blind reviewers correctly recognized blinded applicants. There, however, were significant correlations between author recognition and the following characteristics of ① the applicants: age and academic recognition, and ② the reviewers: age and review experience.
The implications of the findings for the peer review system, suggestions for improvement, and future research directions were discussed.
