1. Introduction to PCI and PCI Psychology
2. Submission requirements, review policy and workflow
3. Recommender Role Overview
    3.1 Main differences between recommenders and journal editors
    3.2 General advice
    3.3 What you should NOT do
    3.4 What you don't NEED to do (but you can if you want)
    3.5 What you SHOULD do
4. Your Role as the Recommender
    4.1 Triage assessment (performed by Managing Board)
    4.2 Pre-handling assessment (performed by potential recommender)
    4.3 In-depth review (performed by assigned recommender)
    4.4 Desk-rejecting after recommender assignment
    4.5 Assigning reviewers
    4.6 Review outcomes
    4.7 Exercising judgment
    4.8 Editing or discarding reviews
    4.9 Managing appeals
5. Common challenges

---

1. Introduction to PCI and PCI Psychology

 

PCI Psychology reviews and recommends preprints related to psychological research and scholarship that are posted to any institutional or organizational repositories and have not been previously published in a peer-reviewed scholarly journal. PCI Psychology is not a journal, but foremost a preprint recommendation service that supports subsequent publication of manuscripts in academic journals, as well as other dissemination options. At this time, PCI Psychology does not review or recommend postprints, i.e., articles that are published elsewhere. 

You can read more about PCI and PCI Psychology here.

2. Submission requirements, review policy, and workflow

See How It Works for more information. 

3. Recommender Role Overview

PCI Psychology recommenders manage the evaluation of articles in the same way as action editors do at journals, with a few differences (see below). PCI Psychology only accepts (“recommends”) preprints of high scientific quality that are methodologically and ethically sound.

3.1 Main differences between recommenders and journal editors

Recommenders handle a manuscript evaluation only if they are willing to do so; as a recommender, you should not feel obligated to take charge of an evaluation. You have to be positively motivated to do so because you are scientifically interested in the preprint and have the relevant expertise to serve as a recommender. You also have to be available to take charge of the evaluation throughout the whole process, from submission through feedback to the authors to recommendation, which may involve multiple rounds of revision.

As there is no Editor-in-Chief, you are solely responsible for editorial decisions. If you have co-recommenders, you share this responsibility with them. The Managing Board oversees these decisions and can be consulted when necessary.

The evaluation history (including reviews, your decisions, and author responses) is published by PCI Psychology at the end of the evaluation process only if you elect to recommend the preprint.

If you decide to accept a manuscript for recommendation, you will write a recommendation text. This text describes the qualities of the preprint, including the subjective reasons for your interest in it. You can find a template of this text in the Template folder of the recommenders’ google drive.

There is no acceptance/rejection rate to be respected. All articles of high scientific quality that are methodologically and ethically sound should be recommended. Articles that do not achieve these standards should not be recommended.

3.2 General advice

• Follow the PCI Psych Code of conduct.
• Be constructive, kind, and respectful in your decisions.
• Be precise and clear.
• Remain open to new approaches. 
• When inviting reviewers to evaluate a preprint, bear in mind that a broad mix of scientists of different genders, career stages, and geographic origins is desirable.
• Only agree to handle manuscripts when you foresee being able to do so in a timely manner. PCI Psychology aims to return first decisions on manuscripts to authors within 60 days.

3.3 What you should NOT do

Do not agree to handle the evaluation of a preprint if you have not taken the time to read it, if you are not familiar with the subject, and/or if you will not have time to handle it expeditiously from submission through recommendation.

You should not have any conflict of interest with the authors or with the content of the article (see the PCI Psych Code of conduct​). If you do, you must decline the invitation. 

You should not invite reviewers who have a conflict of interest with the authors or with the content of the article (see the PCI Psych Code of conduct​)​.

Do not discredit negative or null results. A major goal for PCI Psychology is to shift the evaluation criteria from being centered on the findings to being centered on the conceptualization and design. Thus, a poorly designed study, regardless of its findings, is likely to be rejected. In contrast, well-designed studies will have a higher likelihood of a successful recommendation following review, even if they produce null findings, negative findings, inconclusive findings, or findings that contradict widely accepted results.

You should avoid unconstructive, ambiguous, and unsupported comments.

3.4 What you don't NEED to do (but you can if you want)

You do not need to determine whether the article falls within the scope of the PCI. Once a submission has been validated by the managing board, its scope is considered suitable for that PCI.

You do not need to check whether scripts, code, and/or data are available to the reader (e.g. repository link/DOI or appendix); this is a prerequisite for submission to PCI and is checked by the managing board.

You do not need to check for plagiarism. If an article has been validated, this has been checked by the managing board. However, if you have ethical concerns the MB may have missed, please contact the MB.

You do not need to copy and paste the comments of the reviewers in your decision (as the reviews are automatically attached).

You do not need to submit an “acceptance” decision when you are ready to recommend the article. You should directly post your recommendation.

3.5 What you SHOULD do

Reply to the invitation to handle an article as quickly as possible, by accepting or declining the invitation on the PCI Psych website. Once you agree to handle a manuscript, you are expected to do so until final rejection or recommendation.

Start inviting reviewers as soon as you can ascertain that the manuscript should go out for in-depth review (see details on the role of the recommender below), or as soon as you receive the revised version of an article that you intend to evaluate again. Send invitations to 5-10 potential reviewers within 48 hours, and then send new invitations until you find at least two reviewers willing to review the preprint. This process of finding reviewers should, ideally, take no more than one week.

Try to personalize your reviewer invitation letters as much as possible, to maximize response rates. PCI Psychology has reviewer templates that you will access when inviting a reviewer and can modify. 

Search for additional reviewers if the reviews received are too light and insufficient to enable you to come to a decision.

Make a decision, informed by the reviews. This is your duty and responsibility, not that of the reviewers. The expertise of the reviewers should be helpful, but you are not bound to agree with their assessments.

Post your editorial decision or write your recommendation text within 10 days of receiving the reviews or the revised version of the preprint.

Your editorial decision should summarize the most relevant comments of the reviews and your own points. 

State whether you disagree with some of the points raised by the reviewers.

Feel free to reject an article at any step of the evaluation process (including before sending it out for peer review, or after multiple rounds of review) if there are scientifically sound reasons justifying this rejection.

Pay attention to errors in the text of the article (typography, grammar, etc.). A recommended article must be of the highest quality in terms of its form, because it is a completed, final article. If language in the manuscript is unclear enough that it hinders comprehension of the study, the recommender should (politely) request that the authors improve the writing of the article.

If you decide to recommend the preprint, you must write a recommendation text. This text is a short article (between 300 and 1500 words). It has its own title, and articulates the basis for the recommendation, describes the strengths of the preprint, and the nature of the review process.This text also contains references (at a minimum, the reference to the article being recommended).

Inform the managing board if there is likely to be any delay or if you are unexpectedly unable to carry out your recommender duties. 

Inform the managing board if you suspect scientific misconduct.

 

4. Your Role as the Recommender

PCI Psychology recommenders oversee the peer review process, deciding which preprint submissions will ultimately receive a PCI Psychology recommendation. This section provides guidance on the roles and responsibilities of a recommender and provides advice for dealing with common challenges. To ensure a minimum understanding of the role and responsibilities, prospective PCI Psychology recommenders are required to pass a multiple-choice test and receive certification as a PCI Psychology recommender before handling their first assignment. Recommenders must also complete a declaration of interests indicating any other editorships or professional roles that could be perceived to conflict with their role at PCI Psychology.

A PCI Psychology recommender is analogous to an action editor at a traditional peer reviewed journal. The recommender invites and assigns expert reviewers, assesses submissions and the reviews against the assessment criteria, and issues recommendations. Recommenders typically handle submissions either within or close to their scientific or scholarly expertise.

We ask our recommenders to reflect on their potential biases to help make academia more equitable. Please read the Bias Reflection Guide (Foster et al. 2021) before deciding whether to accept an invitation to be a recommender for a manuscript, before writing a response to the authors, and before writing the recommendation text. 

4.1 Triage assessment (performed by Managing Board)

Triage assessment refers to the phase of evaluation, conducted by the Managing Board, prior to in-depth peer review. When a submission is received by PCI Psychology, it is assigned to the Managing Board, which includes researchers with significant journal editorial experience. At least two Managing Board members then review the submission and perform a pre-review triage assessment. There are four possible outcomes of this triage process:

• Outright desk reject. A submission that falls short of the review criteria may be rejected outright by the Managing Board with no opportunity for the authors to submit a revised version.

• Desk rejection with the invitation to submit a revision. Where a submission falls significantly short of the review criteria but is deemed to be sufficiently promising, it will be desk rejected (without in-depth review) but with the invitation to submit a revised version that addresses specific points raised by the Managing Board triaging members. This may also happen if the submission fails to meet the minimum criteria for submission (see below).

• Validation. Where a submission meets the minimum criteria for submission and comes sufficiently close to meeting the review criteria, the submission will be validated and relevant recommenders will be invited to handle it. This may result in two different outcomes:

• A recommender initiates peer review. A qualified recommender decides to handle the available manuscript. The MB approves the recommender to handle the preprint, and the peer review process starts. In some cases, the manuscript may still be desk-rejected at this stage without going out for peer review.

• No available recommenders to handle the preprint. In some cases, PCI Psychology may decline or postpone the review of preprints due to capacity issues, because of a higher number of submissions than can be handled at a particular time, a lack of expertise among the current pool of recommenders, or an inability to secure recommenders or reviewers for the manuscript within two months. In any such case, authors may choose to resubmit their manuscript at a later date.

4.2 Pre-handling assessment (performed by potential recommender) 

Submissions judged promising are validated by the Managing Board and available to potential recommenders. Before deciding to handle a preprint, confirm that:

• You have read the preprint 
• You feel equipped to handle it and are competent in the subject, including being at least somewhat familiar with its subject, content, and methodology
• The preprint’s subject is interesting to you
• You will be able to handle the preprint in a timely manner
• You have checked that you have no conflict of interest with the preprint, including its content and authors (see Code of Conduct)

After confirming these, you may accept the invitation to handle the preprint. Please note that handling assignments happen on a first-come-first-served basis, and multiple recommenders may receive invitations at the same time, so another recommender might accept the assignment before you. 

4.3 In-depth review (performed by assigned recommender)

Start the in-depth review process as soon as possible.  

Required criteria

Please see the Required Criteria in the “How does it work?” page for a list of requirements and expectations for submitted preprints.

When evaluating non-empirical papers, recommenders should consider the clarity and originality of the central argument or framework, as well as its grounding in the existing literature. The manuscript should provide a comprehensive, balanced, and critical synthesis of relevant research, avoiding undue emphasis on selected findings. Recommenders should consider whether the paper identifies clear gaps, unresolved issues, or new directions for future research and practice. The logic and coherence of the proposed ideas or models should be rigorously evaluated, along with their potential to advance understanding or generate novel hypotheses. Additionally, the paper should be accessible to a broad audience, with technical terms clearly defined and concepts well-articulated.

Guidelines for Empirical Research

1. Research Design and Ethical Considerations

• Clarity and Justification of Research Questions/Hypotheses: Are the research questions or hypotheses clearly stated and grounded in existing literature or theory? Do they address a significant gap or problem in the field?

• Appropriateness of Study Design: Is the chosen study design (e.g., experimental, correlational, qualitative) suitable for addressing the research questions or hypotheses? Does it incorporate necessary controls, randomization, and blinding where applicable?

• Protection of Human/Animal Subjects: Does the study adhere to ethical standards for research involving human or animal participants, such as obtaining informed consent and ensuring confidentiality? Are potential risks minimized, and are there appropriate safeguards in place? Has the research received approval from an appropriate ethics review board? 

2. Data Collection

• Sampling Methods and Sample Size Justification: Is the sampling strategy clearly described and appropriate for the study's objectives? Does the sample adequately represent the population of interest and is the sample size adequately justified through formal power or sensitivity analyses?

• Measurement Validity and Reliability: Are the instruments and procedures used for data collection valid and reliable? Is there evidence supporting their use in the current context?

3. Data Analysis

• Analytical Appropriateness: Are the statistical or qualitative analysis methods suitable for the data and research questions? Have the authors included adequate statistical controls and reported zero-order associations in uncontrolled models? Are assumptions underlying statistical tests checked and reported?

• Transparency and Reproducibility: Are the data analysis procedures described in sufficient detail to allow for computational reproducibility and future replication? If applicable, are data and code shared in accordance with open science practices? If preregistered, is there an accessible link to a time-stamped document?
  

4. Interpretation of Results

• Alignment with Evidence: Do the conclusions logically follow from the results presented? Are alternative explanations considered and discussed? 

• Claims and Limitations: Are the limitations of the available evidence appropriately considered? Are the claims well-calibrated to the evidence presented?

• Contribution to the Field: Does the study advance knowledge in psychological science? Are the implications for theory, practice, or future research clearly articulated?

5. Reporting Quality

• Clarity and Organization: Is the manuscript well organized and clearly written? Does it adhere to relevant reporting guidelines?

• Disclosure of Conflicts of Interest: Have the authors disclosed any potential conflicts of interest? Is there transparency regarding funding sources and affiliations?

• Contributorship: Is there a contributorship statement (e.g., using the CRediT Role Taxonomy)

4.4 Desk-rejecting after recommender assignment

It is possible that fatal flaws within a preprint only become clear after a deeper examination. If, after careful consideration, you come to the conclusion that the preprint is so flawed as to not be worth sending out for full peer review, it may be “desk-rejected” at this point. The assigned recommender (you) will write a detailed decision letter explaining the reasoning behind the decision.

4.5 Assigning reviewers

As a recommender, you are responsible for inviting reviewers. Aim to assign 3 field-specialist reviewers (and never less than 2) to assess the first submission of a new manuscript. Assigning 3 field-specialist reviewers will generally require the recommender to invite 5-10 reviewers to achieve a sufficient number of agreements. Where possible, recommenders should avoid inviting reviewers with a clear conflict of interest such as a close relationship to the authors that includes recent co-publication, joint funding, or other obvious collaborations; or being at the same institution (see the PCI Psych Code of Conduct for more information on what is considered a conflict of interest). As part of the submission process, authors can recommend up to five independent expert reviewers. Where a recommender intends to invite a reviewer suggested by the authors, it is the recommender’s responsibility to ensure that the suggested reviewer is independent and does not have a conflict of interest.

Where you are also a field-specialist in the topic of the submission, and have no conflict of interest, you can also serve as one of the 3 specialist reviewers, in addition to adjudicating on the submission as recommender. However, your review can never replace one of the two necessary field-specialist reviews to meet the minimum number of reviews before a recommendation can be issued. 

Authors can also list up to five opposed reviewers, and need not provide any reason for opposing their involvement in the review process. It is the policy of PCI Psychology that no specifically-named opposed reviewer will be invited to assess a submission. Note that you are not obliged to adhere to more sweeping requests from authors, such as to exclude all reviewers from a particular university department, field, or institute.

In addition to the recommended 3 field-specialist reviewers (minimum of 2), you can invite reviewers to assess any specific part of a submission, such as statistical analysis, methodology or computer code, even when the reviewer lacks expertise in the specific field or discipline of the investigation.

Reviewers are requested to complete their reviews within 4 weeks, with automatic extension to 6 weeks. Longer extensions are negotiable and recommenders should exercise judgment in enabling a longer review period. 

You are free to consult with the Managing Board for advice on specific cases as they arise.

If you are struggling to find reviewers, try using Jane to find researchers conducting similar research and who do not have conflicts of interest with the authors of the submission you are handling. If you are still having difficulties, you can always email the authors to ask for a list of potential reviewers with whom they do not have any conflicts of interest, and we can also help on the Managing Board (we recommend asking us for help before contacting authors). If you communicate with the authors at all, please only use email and make sure to cc contact@psych.peercommunityin.org so the Managing Board is aware of the interactions. We made a Reviewer Invitation Cheatsheet to help you keep track of the guidelines and organize your search. Feel free to download it and fill it as you handle a submission.

Reviewers are welcome to co-review with a junior colleague. To credit the co-reviewer, invite the junior colleague as a separate reviewer. After they have accepted the invitation and the joint review has been written, they should paste the complete review at the junior academic's review space. In the senior academic's review space, they should state "Joint review: see review by X for details." If the reviews are anonymous, then state "Joint review: see review below for details" and note in the junior academic's review text that this is the joint review.

4.6 Review outcomes

Following peer review, you are responsible for conveying the reviews to authors and issuing a range of judgments on submissions. There are three possible outcomes following each completed cycle of peer review:

• Reject. Where expert reviews identify major flaws in a submission that are uncorrectable without designing a completely new program of work, then the submission should be rejected outright. Effective triage assessment helps keep the rate of outright rejection following review relatively low. 

• Invitation to revise. Where the reviews identify shortcomings that are, in principle, correctable to satisfy the review criteria, the authors should be invited to submit a revised manuscript, accompanied by a comprehensive response to reviewers.

• Recommend. Where a submission is judged to meet the review criteria, it will be recommended and a recommendation text will be issued.

The Managing Board (MB) is responsible for overseeing and confirming the decisions of the recommender. The MB is ultimately responsible for all recommendations and, where necessary, can overrule a recommender's decision.

4.7 Exercising judgment

Reviews provide advice to you, the recommender, not binding judgments. While it can be desirable for reviewers to reach a consensus in the assessment of submissions, a consensus is not required to reach a recommendation, and you should always be guided by your own judgment and that of the Managing Board. Moreover, in cases where only minor revisions are required, you should, where possible, avoid overburdening reviewers with further review requests. While there will be reasonable exceptions to this rule, you should assess minor revisions before issuing a recommendation and without additional input from reviewers.

In cases where reviewers offer competing evaluations, it is your responsibility to help steer the authors toward a resolution. It is therefore vital that you read all the reviews and the submission before rendering a recommendation, which is standard practice for recommenders on all submissions, not only those where the reviewers disagree.

4.8 Editing or discarding reviews

Reviewers and recommenders are expected to adhere to the PCI Psych code of conduct, avoiding the use of abusive or discriminatory language in reviews and decision letters. Reviews that violate the code of conduct may be returned to the reviewer for editing before becoming part of the open review record. In cases where a review contains no information of value, you may discard the entire review. Depending on severity, you may also feel it is appropriate to leave the review unedited but, if so, you should flag specific content in the review as unnecessary for the authors to address in revision. You should also feel free to contact reviewers and ask them to provide a higher quality review before submitting their decision. It is not permissible for recommenders to alter the content of a review without explicit permission from the reviewer. 

Open review policy

PCI Psychology publishes all reviews of recommended manuscripts, with reviewers retaining the choice to sign their reviews or remain anonymous. (The exception to this is when one of the authors is a PCI Psychology Managing Board member, in which case all reviews must be signed. Managing Board members have access to confidential information about all submissions, which poses an ethical problem if a reviewer wishes to remain anonymous from the authors). Reviews and recommender decision letters are published on the PCI Psychology platform no later than the point of acceptance. Reviews of rejected submissions are not published.

4.9 Managing appeals

Authors of rejected manuscripts may appeal the PCI Psychology recommendation by emailing the PCI Psychology Managing Board and the recommender who handled the manuscript. Appeals must be received within 30 days of PCI Psychology issuing a decision. The email must clearly state the case for why the decision should be changed. Appeals will only be considered if the authors a) identify factual errors made by the reviewers or recommender that had a major impact on the decision, or b) can provide a substantiated claim of unfair treatment and/or bias in the review process. Appeals for any other reason will be denied without further consideration. Appeals that meet the identified criteria will be discussed among a panel that includes the recommender and two members of the Managing Board. The panel will review the appeal and vote to uphold or reverse the original decision. Final decisions will be based on majority vote (i.e., decisions need not be unanimous). Decisions following appeal are final.

5. Common challenges

This section offers advice on how to handle a range of common challenges experienced by recommenders at PCI Psychology.

Dealing with late or non-responding reviewers

Occasionally, complications in the review process do may delay submissions. The most extreme delays arise where a reviewer agrees to assess a submission but then never submits the review and fails to respond to reminder messages about the review. Such cases typically require the recommender to find an alternative reviewer, effectively restarting the process and delaying the decision by several weeks. As insurance against this eventuality, recommenders are advised to assign three to four reviewers for each submission—that way, a decision could be issued if one or even two of the reviews is not forthcoming. Recommenders should also be flexible about allowing short extensions to review deadlines. Where possible, recommenders should invite the same reviewers back for a resubmission that requires additional review. 

Seeking specialized reviewers

Submissions can sometimes benefit from specialized review when the reviewers who are evaluating a manuscript are competent in the main substantive topic but may lack the required knowledge to evaluate the validity of a specific method or analysis that is central to the study and/or its results. In this case, consider seeking an extra reviewer who does not need to necessarily be competent in the substantive area of the study. This specialized review is particularly useful where you yourself lack specialist expertise. Specialized reviewers should be an addition to the two required reviewers for any recommendation, i.e., they do not “count” towards the minimum number of required reviewers. 

Availability of data, code, or materials

Authors will sometimes contact recommenders with questions about the requirements concerning making their data publicly available. In response, you can direct authors to the PCI Psychology Transparency and Openness Promotion (TOP) guidelines policy. Broadly, this policy requires that authors make all study data, digital materials and code publicly available unless there are specific ethical or legal restrictions that prevent it. In such cases, authors must instead specify in the manuscript the process (if any) by which requesters can obtain the data, materials and code, including any specific conditions they must meet. You should also contact the Managing Board with any queries about specific cases.