Workflow - Goals of Peer Review

Should peer reviewers evaluate a submitted paper for the pursuitworthiness or acceptability of the content of the paper?

Ideally, a workflow needs to clearly articulate the goals of its peer review process. In the traditional workflow, the answer to this question is far from obvious. While some reviewers review submissions for pursuitworthiness, others review for acceptability, yet others do both.1 As a result, it is often unclear whether the content of published articles is to be taken as accepted by the respective community or as or merely as considered pursuitworthy by the editors and reviewers? As noted by Shaw and Barseghyan,

Many of our practices suggest that it might be the former. For example, in Naomi Oreskes’ widely cited study on the consensus on climate change, she uses the content found in publications as a measure of acceptance.2 Moreover, many reviews reject papers due to their purported flaws suggesting that they should not be published because they are not acceptable.3 Yet, at other times, the fact that something was published is only taken to mean that it was considered to be worthy of further attention. Philosophers, for example, will be acutely familiar with this view: no one reads the newest paper on realism in Philosophy of Science to see what the community believes about realism. They merely search for stimulating and interesting ideas.4p. 3

Thus, it is vital to clearly state the goals of the peer review process.

In the scientonomic context, this question was first formulated by Hakob Barseghyan and Jamie Shaw in 2019. The question is currently accepted as a legitimate topic for discussion by Scientonomy community.

In Scientonomy, the accepted answers to the question can be summarized as follows:

• The goal of peer reviews in the scientonomic workflow is evaluation for pursuitworthiness rather than acceptability.
• Scientonomic knowledge is best advanced by:

1. documenting the body of accepted communal knowledge knowledge in an online encyclopedia;
2. scrutinizing this accepted knowledge, identifying its flaws, and formulating open questions at seminars, conferences, publications, and other in-person or online formats;
3. publishing journal articles that propose modifications to our current knowledge and documenting these suggestions;
4. evaluating the suggested modifications with the goal of reaching a communal consensus and changing the respective encyclopedia pages when a verdict is reached.

Scientonomic History

Acceptance Record

Here is the complete acceptance record of this question (it includes all the instances when the question was accepted as a legitimate topic for discussion by a community):

Community	Accepted From	Acceptance Indicators	Still Accepted	Accepted Until	Rejection Indicators
Scientonomy	22 December 2019	The publication of Shaw and Barseghyan (2019) is and indication of the acceptance of the question.	Yes

All Theories

The following theories have attempted to answer this question:

Theory	Formulation	Formulated In
Goals of Peer Review - Pursuitworthiness (Shaw-Barseghyan-2019)	The goal of peer reviews in the scientonomic workflow is evaluation for pursuitworthiness rather than acceptability.	2019

If an answer to this question is missing, please click here to add it.

Accepted Theories

The following theories have been accepted as answers to this question:

Community	Theory	Accepted From	Accepted Until
Scientonomy	Goals of Peer Review - Pursuitworthiness (Shaw-Barseghyan-2019)	25 February 2023

Suggested Modifications

Here is a list of modifications concerning this topic:

Modification	Community	Date Suggested	Summary	Verdict	Verdict Rationale	Date Assessed
Sciento-2019-0001	Scientonomy	22 December 2019	Accept that the goal of peer-reviews in the scientonomic workflow is evaluation for pursuitworthiness rather than acceptability.	Accepted	The decision was made during the 2023 scientonomy workshop. The modification was summarized by Paul Patton as essentially a ratification of current scientonomic practice. Jamie Shaw raised some concerns about how we don’t have adequately defined norms that must be satisfied for pursuitworthiness, which may make this modification trivial. Discussion about how peer-reviewers’ notions of pursuitworthiness may veer close to acceptability ensued. Nevertheless, the modification passed with 83% of the votes to accept (10/12).	25 February 2023

Current View

In Scientonomy, the accepted answers to the question are Goals of Peer Review - Pursuitworthiness (Shaw-Barseghyan-2019) and Scientonomic Workflow (Barseghyan et al.-2016).

Goals of Peer Review - Pursuitworthiness (Shaw-Barseghyan-2019) states: "The goal of peer reviews in the scientonomic workflow is evaluation for pursuitworthiness rather than acceptability."

In the scientonomic workflow, the goals of peer review are to assesses a paper for pursuitworthiness of the modifications suggested in the paper. Thus, peer reviewers should not evaluate submissions for acceptability, but only for pursuitworthiness.

Scientonomic Workflow

Scientonomic Workflow (Barseghyan et al.-2016) states: "Scientonomic knowledge is best advanced by:

1. documenting the body of accepted communal knowledge knowledge in an online encyclopedia;
2. scrutinizing this accepted knowledge, identifying its flaws, and formulating open questions at seminars, conferences, publications, and other in-person or online formats;
3. publishing journal articles that propose modifications to our current knowledge and documenting these suggestions;
4. evaluating the suggested modifications with the goal of reaching a communal consensus and changing the respective encyclopedia pages when a verdict is reached."
   
   The key stages of the workflow are:

• Pose Questions: The goal of this stage is to scrutinize the current state of the scientonomic theory and our knowledge of scientific change and identify as many open questions as possible. The annual seminar on scientonomy hosted by the University of Toronto's Institute for the History and Philosophy of Science and Technology is currently the main venue facilitating this stage of the workflow.
• Suggest Modifications: The goal of this stage is to advance our knowledge of scientific change by proposing modifications to our current body of knowledge. These suggested modifications are published and properly documented. These modifications are currently published in the Journal of Scientonomy but, in principle, they can be published in any journal which makes use of the scientonomic mechanism of modifications. Once a modification is published, this encyclopedia documents that suggestions and invites experts to review it.
• Evaluate Modifications: The goal of this stage is to assess the suggested modifications and decide which of them are acceptable and which are not. This is done by the community of scientonomists on the respective discussion pages of this encyclopedia. If a consensus emerges, the fate of the modification is documented. If a modification causes disagreement among scientonomists, it becomes a topic of discussion during scientonomic workshops, which aim at bridging the gaps between opposing parties and arriving at consensus.
• Document Changes: The goal of this stage is to document all the changes in our communal body of knowledge. If a modification is considered acceptable by the community, then the respective articles of this encyclopedia are modified to reflect that change. If a modification is considered unacceptable, then the respective verdict is documented for that modification.

The primary role of this encyclopedia in the scientonomic workflow is to document the current state of scientonomic knowledge, trace all suggested modifications, and list open questions.

Here is an outline of the main stages of the scientonomic workflow:

This workflow gives researchers a simple way of knowing where the community stands on different topics, i.e. what theories it currently accepts, what open questions it tries to answer, what modifications have been proposed and how they have been assessed. It ensures that our communal knowledge is advanced in a piecemeal and transparent fashion:

• Piecemeal: modifications to the communal mosaic are suggested one by one, which allows for a sober critical evaluation of these suggestions by the community.
• Transparent : suggested modifications and their evaluations are properly documented, so that there is no mystery as to whether, when, or why a certain modification was or wasn't accepted.

The workflow is scalable, as it can - in principle - be implemented in other fields of digital humanities and beyond.

Related Topics

This question is a subquestion of Scientonomic Workflow.