This is stifling Machine Learning Research
Most of my content revolves around machine learning research/ideas. There is a reason for this. While coding-heavy tutorial articles/videos are more popular and valuable, technologies change over time. The ideas and nuances of the techniques however don’t. My goal is to take these complex machine learning concepts that are often found in papers/research and introduce them to you so that you can choose what tool you need to use in your specific context. This article is different. In it, I will take a step back and talk about the peer review system.
There are so many papers being published that if one were stack them on each other as they are published the top of the stack would be moving faster than the speed of light. This raises a paradox because nothing can move faster than the speed of light.The paradox is resolved when we realize the top of the stack of articles is carrying no information.
Old physics joke about peer review and the flood of papers being published
Specifically, I will talk briefly about what is, and then what benefits it provides. I will also talk about why we need an overhaul and provide some suggestions on how it could be achieved. The goal of this article is to generate bring attention to the situation and generate some discussion on the system and how it can be improved.
An Introduction
Let’s first understand the basic mechanics of the Peer Review system. Generally, there are 3 parties involved: the author, the editor, and the (in most cases anonymous) reviewers. The author submits a paper which is then passed around to the reviewers. These reviewers comment on multiple aspects of the paper, suggesting improvements etc. The editor then looks at all these reviews. If there is a largely negative review, editor might reject the paper. Otherwise, they will pass the feedback back to the author, who will then enact these changes. This process happens till the paper is published in a journal/publication.
Benefits
Why is this system so spread throughout science? Such an expert provides certain benefits:
- It allows for feedback from multiple sources. Different reviewers are going to bring different experiences and look for different things. This can help in catching errors compared to having just one person/group looking at things.
- Enables experts to verify the work. This is why replicable research is so important.
- If the author is angling for a particular journal, feedback from experienced reviewers will give them pointers on presentation, content, and tone.
Weaknesses
To say the Peer Review system is bad would be inaccurate. It is responsible for many of the great progress of the modern world. However, it is outdated and needs to be updated. There are a few areas where the Peer Review System is failing Machine Learning Research.
- Inconsistency. This is a huge problem with the Peer Review system. The video above explains this in pretty great detail. Identical papers might be accepted or rejected by different editors/reviewers.
- Can kill innovation. In one of the most ironic twists, this system can destroy innovation. Imagine this scenario. You’re a researcher/Ph.D. student. Being published in the prestigious journal ABC will boost your career. You’re going to do research closely aligned with this publication. Getting too creative will actively hurt your chances of getting published here. Similar story for getting funding to pursue your research.
- Misaligned incentives. Many researchers are just incentivized to publish things. Imagine you’re a researcher at Google. You get bonuses per accepted publication. You decide to go for publication ABC. You go through previous papers in ABC, take a paper, and make some changes to its findings. Then you present it. Your paper is accepted in and you make a nice bonus. This is the reason why so many papers/architectures are just tweaks of what is already done. Also, the reason why you see so many Google/Facebook/Big Company papers on arXiv. Why is this a problem? This floods the space with many lower-quality papers. That makes it harder to sift through to find interesting/innovative papers. It also diverts resources away from good research.
How to Fix This
Far be it from me to criticize without giving solutions/improvements. Here are some ways that we can work towards fixing the problems of this system.
Compensate Reviewers (well)
This might come as a surprise to you. What good would that do? By compensating reviewers we can demand higher quality work from them. The review becomes a part of their work. This will let them dedicate more time and effort to the cause. This will allow for a greater quality of feedback and discourse around the paper/topics. More time spent on review will also naturally reduce the inconsistency in the process. Such a vital aspect of knowledge growth should not be left unpaid.
Standardization
Many prestigious Machine Learning journals have some kind of a checklist. However, there is a lack of standardization/SOPs with which ML research can be evaluated. Such a move would allow people to have a clearer direction and increase the quality of submissions/drafts. Here is a part of the Machine Learning Reproducibility Checklist being pushed for by many prominent researchers in the field.
Defining quality/innovation standards will also allow for more consistent reviewing across the board
Pivot from Submissions as a Primary Metric
Things such as citations/number of publications can be useful ways to evaluate the researcher/their work. However, far too often these end up being the only aspects considered. Such evaluation turns research into a popularity contest. And in such cases, the works that win are often the ones that conform to the standards instead of pushing the norms.
Penalize Low-Quality Submissions
There is no penalty for big organizations spamming papers. These aren’t really intended to improve the discourse and are often meant to act as advertisements for the company. And/Or just bonuses for the researcher. We need to have some level of penalty to stop flooding. One possible way is to have an upper limit on the number of submissions that an organization can submit. After this, additional payments could be charged. This would make sure that people/organizations only submit the best quality papers (and provide resources for better reviewer compensation).
Obviously, this is not the be-all-end-all to this. Bringing change requires a lot of careful thought. There are likely other ways we could improve the process. Or there might be problems with what I’ve described. The point of this article was to shed some light on the topic and generate some discussion around it. Share your thoughts through either the comments or by reaching out to me personally. I’d love to hear them.
If you liked this article, check out my other content. I post regularly on Medium, YouTube, Twitter, and Substack. I focus on Artificial Intelligence, Machine Learning, Technology, and Software Development. If you’re preparing for coding interviews check out: Coding Interviews Made Simple.
Reach out to me
If that article got you interested in reaching out to me, then this section is for you. You can reach out to me on any of the platforms, or check out any of my other content. If you’d like to discuss tutoring, text me on LinkedIn, IG, or Twitter. If you’d like to support my work, use my free Robinhood referral link. We both get a free stock, and there is no risk to you. So not using it is just losing free money.
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
If you’re preparing for coding/technical interviews: https://codinginterviewsmadesimple.substack.com/
Get a free stock on Robinhood: https://join.robinhood.com/fnud75
