How PaperScore Works

PaperScore accepts articles from all disciplines, particularly from interdisciplinary and multidisciplinary studies. It does not categorize articles, but rather tags them with keywords.

Instead of an editorial board, PaperScore uses a collective intelligence algorithm that is implemented in a computer code and controls the peer-review process with minimal human intervention.

After the peer-review process, the authors see the reviewers’ comments and median scores, and then can decide whether to publish their manuscript or not. If published, it will be indexed in several databases such as Scopus, PubMed and Google Scholar, and will be accessible by all universities and academic institutions around the world. It can also become an NFT (ERC 1155) on the Ethereum blockchain.

Whether the authors decide to have their manuscript published or not, they can receive valuable feedback and comments from the experts who reviewed it. This helps them to improve their manuscript and submit it to other journals if they want to.

Go to Guidelines page

Entrenched Incumbents

The current research publication industry is dominated by a three-sided network effect between authors, institutions, and journals. Institutions and authors want to publish in well-known journals, while journals want to publish works of well-known authors (and institutions) who have published in well-known journals. This can lead to self-fulfilling prophecies. PaperScore aims to break this outdated ecosystem by completely decentralizing the peer-review process.

Moreover, the current ecosystem gives editors the authority to select reviewers or reject a submission without any peer-review. They also make the final decision on each submission. But like all people, editors are also susceptible to biases. As a result, journals are subject to a number of editorial biases. Such biases are reinforced if the editors are always succeeded by those who have published in the journal before.

Publication Bias occurs when studies that find no significant effects are not published, even though the results were obtained rigorously. Discipline Bias occurs when multidisciplinary and interdisciplinary studies are rejected for not fitting in the particular study areas of the major journals. PaperScore provides a home for such homeless studies.

PaperScore Review ProcessReview Process

Once an author submits a manuscript to PaperScore, the backend code randomly selects up to five potential reviewers based on the manuscript’s keywords and citations. Each potential reviewer receives a message and decides whether or not to evaluate the manuscript. Reviewers also have the option to refer manuscripts to other potential reviewers they deem to be experts in the field. The next reviewer will also have the same option.Once an author submits a manuscript to PaperScore, the backend code randomly selects up to five potential reviewers based on the manuscript’s keywords and citations. Each potential reviewer receives a message and decides whether or not to evaluate the manuscript. Reviewers also have the option to refer manuscripts to other potential reviewers they deem to be experts in the field. The next reviewer will also have the same option. Based on the small-world phenomenon, each chain will get to a suitable reviewer in fewer than six referrals on average. Most likely, it will be fewer than three referrals because of the purposeful randomization that incorporates relevant information such as keywords.

Based on the small-world phenomenon, each chain will get to a suitable reviewer in fewer than six referrals on average. Most likely, it will be fewer then three referrals because of the purposeful randomization that incorporates relevant information such as keywords.Currently, the PaperScore review process is double-blind and is controlled by the scaffolder (the Editor In Chief); the scaffolder will try to find the best reviewers for each manuscript and calibrates the rating scores to improve their accuracy and reliability. In the future, however, the scaffolder will be out of the loop and the process will be fully automated and triple-blind, moving towards a neutral meritocracy.

Currently, the PaperScore review process is double-blind and is controlled by the scaffolder (the Editor In Chief); the scaffolder will try to find the best reviewers for each manuscript and calibrates the rating scores to improve their accuracy and reliability. In the future, however, the scaffolder will be out of the loop and the process will be fully automated and triple-blind, moving towards a neutral meritocracy.

Rating Scores

Each reviewer can anonymously rate a manuscript on four dimensions:

Writing & Composition
Rigor & Validity
Novelty & Contribution
Importance & Value

These dimensions are defined in theScore Guidelines. Reviewers can also leave anonymous comments to support their rating scores. Once a sufficient number of reviewers submit their rating scores, the median score for each dimension is calculated and reported to the author(s). Author(s) can only see the anonymous reviewer comments and the aggregated median scores, but not the individual scores, or the names of the reviewers.

Instead of accepting or rejecting, PaperScore amends each manuscript with comments and scores. This provides valuable information for authors and potential readers.

Publication Decision

Upon seeing the anonymous comments and median scores, the author(s) can decide to release, revise, or withdraw the manuscript. If they decide to release the manuscript, it will be published with its anonymous comments and median scores. However, if they are not happy with the review results, they can choose to revise and resubmit the manuscript; the manuscript will then undergo the same review process obtaining a new set of rating scores and comments.

Therefore, every submitted manuscript that follows the essential formatting requirements, such as anonymity, will be reviewed and then published if, and only if, the corresponding author decides to release it. Since PaperScore is fully electronic, it is not limited to the traditional space limitations of the paper-based journals and can include thousands of papers in its database. To find the most relevant and high-quality studies, the readers can search, filter, and sort papers based on keywords, authors, dates, and, most importantly, median scores.

Reviewer Points

Most of the traditional journals rely on anonymous reviewers whose performance is only visible to the editors of that journal. Therefore, the reviewers spend time evaluating the manuscripts based on the hope that the editor remembers their contributions. However, this provides little incentive for meticulous and thorough reviews in time. PaperScore solves this problem by defining reviewer points.

The PaperScore code automatically accumulates and calculates reviewer points for reviewers and referrers. For each submission, the median reviewer(s) for each dimension will earn reviewer points. The median reviewers are those whose rating scores are closest to the median score for each dimension. Also, the referrers who referred the manuscript to those median reviewers receive partial points for their accurate referrals. The relative amount of reviewer points for accurate rating and accurate referral can be fine-tuned through an RSM algorithm.

While the reviewers and their referrers are kept anonymous for every manuscript, the aggregated reviewer points for reviewers are publicly visible on their profiles. Also, the top reviewer points will be listed on the PaperScore homepage to publicize the best reviewers. The reviewer points will give the reviewers credit for their contributions and incentivize for better reviews and referrals.

The Vision

PaperScore envisions a future where scientific publishing is governed by intelligence—both artificial and collective—rather than editorial gatekeeping. By replacing traditional editorial boards with an AI-driven evaluation system, PaperScore ensures that every manuscript is assessed purely on its merits. Leveraging fine-tuned domain-specific LLMs, retrieval-augmented generation, and multi-task learning architectures, PaperScore provides precise, consistent, and impartial manuscript evaluations across all four critical dimensions: Writing & Composition, Rigor & Validity, Novelty & Contribution, and Importance & Value. This approach minimizes human biases, democratizes the research publication process, and fosters a transparent, meritocratic ecosystem for research dissemination.

At its core, PaperScore has a hybrid human-AI review pipeline, progressively transitioning to full automation through reinforcement learning from human feedback (RLHF). The platform employs SciBERT embeddings, Longformer architectures, and domain-adaptive transformer models to tackle the most challenging aspects of manuscript evaluation—areas where current NLP capabilities remain limited. Unlike general academic tools that merely assist with writing or citation management, PaperScore's specialized evaluative AI delivers comprehensive manuscript assessment with unprecedented consistency. Following a three-phase implementation roadmap, PaperScore will launch the Writing & Composition assessment in Q2 2025, add the Rigor & Validity assessment by Q4 2025, and complete full automation across all evaluation dimensions by mid-2026.

For mathematical validation and Automated Theorem Proving, PaperScore integrates GPT-4, PaLM 2, and WolframAlpha, enabling Formal Verification of scientific claims with precision surpassing human reviewers. By utilizing Axiomatic Reasoning and Neuro-Symbolic Knowledge Graphs, PaperScore sets new benchmarks for rigor in research evaluation, targeting: >80% consistency with expert reviewers, and >95% precision in methodological error detection. All evaluation processes will maintain complete transparency through detailed reasoning paths and evidence chains, allowing authors to understand precisely how scores were derived, thereby establishing trust throughout the academic community.

PaperScore is more than an alternative to traditional publishing—it is the future of knowledge validation and dissemination. Every manuscript is indexed in major academic databases and can be minted as an ERC-1155 token (NFT) on the Ethereum blockchain, securing its provenance and accessibility. Our pilot partnerships with three leading academic institutions will demonstrate how PaperScore can complement existing publication channels while dramatically improving efficiency and fairness.

By removing artificial publication barriers while enhancing quality standards through quantifiable, multi-dimensional assessment, PaperScore accelerates scientific progress without compromising rigor. This platform fosters an impartial, interdisciplinary, and transparent research ecosystem, pioneering an open-access paradigm where scientific contributions are judged solely on their intrinsic merit. Hopefully, PaperScore will increase cross-disciplinary citations in all research areas and will improve publication opportunities for researchers from underrepresented institutions. Beyond research publishing, PaperScore's specialized evaluative AI extends to patent evaluation, grant review, and content moderation, transforming scholarly communication across industries.

* This is an attempt to explain the general purpose of PaperScore and is subject to change. Parts of the above mentioned processes and the methods used in this software are included in the US patent application 63151798 (pending patent): "Method and Apparatus for Decentralized Evaluation of Papers, Articles, Artifacts, and Designs" (Inventor: Hamed Khaledi)