What do scholars want in a journal?
Members of the European Society for Cognitive Psychology (ESCoP) have just learned that their executive committee unanimously took the courageous step of abandoning the society’s journal to its publisher. Instead, we are focusing our efforts on launching a new journal, one that is designed to appeal to and benefit scholars. You can find Sebastiaan Mathot’s summary of why we took this decision here. In short, it was frustrating for us to imagine that we could spend the coming years working hard to make our journal better, but that ESCoP would not necessarily derive any additional benefits from our extra effort. If scholars are going to invest so much, we should make sure that scholars are the principle beneficiaries of our work.
We now have the opportunity to create a cognitive psychology journal that serves scholars, that puts our priorities first. How do we want journals to function? This is what I want:
- Rigor: I want editors and peer reviewers to hold me to high standard. When I submit a paper, I hope that reviewers suggest ways I could improve it. If my research design has a fatal flaw that limits the conclusions that I can draw too much, I want them to catch that and stop me from publishing it. I do not want the editors to reject my otherwise excellent paper merely because they do not imagine that it will incite a lot of publicity.
- Transparency: I want to have access to data and materials of the papers I read. Sometimes this is the only way to get the details necessary to evaluate the methods. This may not be crucial for a casual reader, but it is absolutely crucial for the scientist thinking of planning an extension of the published work, or the scholar trying to meta-analyze some aspect of the data that the original authors did not thoroughly describe. Much time is wasted on the misunderstandings that arise from relying merely on narrative text for conveying every detail that was important for running or analyzing a study. If we leave this to the authors’ discretion, it is all too likely that the data and materials will not be preserved and curated properly. I want scholars to encourage each other to pre-emptively curate their work.
- Accessibility: I want my work to be accessible to everyone. I believe that accessibility will increase its impact. I don’t want to pay a huge fee to a publisher to compensate for the loss it takes when making my work accessible.
- Promotion: I want my work to spark discussion and new ideas. I want the journal I publish in to match the efforts I take to let interested readers know about my work, and to take steps that encourage interested readers to discuss the work.
A scholarly society is in the ideal position to create a journal that prioritizes scholarship first. ESCoP’s motives are aligned with the motives of the scholars seeking to publish with us. We don’t want to hide your work behind paywalls to anyone who hasn’t paid for access. We don’t need to own your work and generate profits from it; though we haven’t settled on a fee yet, the margin we assume beyond the necessary fees for the labor involved in processing an article will be thin. If any profit is realized, it will be spent by ESCoP on things valuable to scholars, like affording opportunities for summer schools and conference attendance to PhD students. Most importantly, ESCoP wants to take pride in the quality of the work it publishes. We do not need to fill a specific number of pages, nor do we need to limit ourselves to some arbitrary number of papers. We can publish and promote as much high-quality cognitive psychology as scholars want to entrust us with.
ESCoP has entrusted me and my editorial board to design an open-access journal that serves the scholar. We’re excited about this. Are you? What policies would you appreciate in such a journal? Please tweet them to me @CandiceMorey.
11 thoughts on “What do scholars want in a journal?”
Fantastic initiative. I would really like to encourage you to have at least the tenured reviewers sign their reviews by default, unless they can provide a convincing reason to be anonymous. For (some of) my motivation, see http://osc.centerforopenscience.org/2014/05/15/anonymous-peer-review/
I really do not think that paying to publish our own paper will allow us to gain in transparency and in fairness during the review process. Moreover, the suggestion to commit the reviewers to give their name is the best way to subvert the system even more. We are not living in Alice in Wonderland and if one if strongly positive about one of our papers, it will be almost impossible to avoid a positive bias when it will be time to judge his or her paper.
Well, no one has determined yet what it would cost. OA journals range a lot in cost, starting from free. ESCoP will be trying to reach a low rate with our journal. Unlike many others, our primary motive isn’t making a profit.
> I really do not think that paying to publish our own paper will allow us to gain in transparency and in fairness during the review process.
This is a problem of *some* forms of open access—but not all. Take the best bad example, Frontiers: They publish to make money, and reviewing is (probably) light as a result. But ESCoP doesn’t publish to make money; they publish to get good science out there, and authors pay a slight fee to keep the system running. *ESCoP is us*, and they/ we are as (un)corrupted as we/ they are. The new journal will *never* pressure reviewers to accept a paper that they don’t consider suitable for publication.
> Moreover, the suggestion to commit the reviewers to give their name is the best way to subvert the system even more. We are not living in Alice in Wonderland and if one if strongly positive about one of our papers, it will be almost impossible to avoid a positive bias when it will be time to judge his or her paper.
This is true—no question. The question is whether the other bias is not worse: Currently, many scientist feel that the review process is unfriendly and unconstructive because reviewers are never held accountable for their reviews. Open peer review is an experiment that may succeed or not. And whether the new journal will participate in this experiment is to be decided by the community. (Opinions seem divided on this.)
In general, and without meaning to dismiss your opinion, it’s important not to get stuck in “status-quo anchoring”, that is, not seeing problems with the current situation because it is so familiar. Every new development (such as open access) has downsides—but despite those downsides it may be still preferable to the status quo. Right now, the status quo is extremely bad: Normal (non-academic) people don’t have access to the science they paid for.
Before I write my thoughts on this, there is one question I am wondering whether someone can answer for me. What has to happen for an open access journal to become a highly regarded, high-quality journal?
I see no reason in principle why an OA journal cannot become highly regarded. OA or not, the esteem of a journal reflects its contents. It needs to have excellent content, and must promote its excellent content so that everyone becomes aware of it.
Another factor I think is important is the quality of the editorial process. We all have experience of lazy editors who treat peer review like a vote. They send form letters announcing that the peer reviews are enclosed, and seem to make no more of a decision than counting the number of reviewers who chose accept and reject. I don’t think this is the way to find excellent content for a journal. You need editors who are willing to invest more in the process, to weigh and judge the criticisms of the reviewers, to try to make the best use of everyone’s time by directing revisions. I think that if a journal became known for consistently offering constructive, critical editorial judgment, this could contribute to its reputation.
I did an informal study of this, and you are right in the sense that an example exists: The Journal of Machine Learning Research has outperformed the journal it replaced (Machine Learning). Another one is TACL instead of Computational Linguistics.
However, the failures are also very prominent: Frontiers, PLoS ONE (at least in my eyes). One important point may be that my two examples above are probably not for profit, but the last two are. Could be a major factor, as you point out elsewhere.
How do JMLR and TACL fund themselves? I don’t think they charge authors. How can they do this? This is a question I would get an answer to before deciding to create a journal. I don’t think it’s a given that the author has to pay to publish.
This is a great effort. It is also a lot of work and I’d encourage academics to support such efforts and provide constructive feedback. The current exploitative publishing system only exists because we academics are too busy and driven by our own work and the idea of spending time on taking ownership of publshing seems like another task for which there is no time. So we keep sawing with a blunt saw because we don’t have time to sharpen or replace the saw. Though I am a social psychologist, I may borrow from and model on your ideas to develop a similar journal for my discipline. My best wishes.
Here are my suggestions. I should mention that I am an outsider to cognitive psychology. I’m primarily a linguist, but my area is psycholinguistics, and that interfaces with psychology. Some of my best friends are psychologists; but I am very clearly an outsider in this community.
Here is my wish list, in no special order (apologies if I repeat other points already made above):
1. Short papers as the default: I think that one big problem in psych* is the amazing verbosity of the papers. How is it that computational linguists can get their papers into 8-16 pages but psych*s need 50? One can stick the details into an online supplement. See https://www.transacl.org/ojs/index.php/tacl. TACL is an interesting journal: it was created as a response to the amazingly slow review cycle of Computational Linguistics, the leading (MIT Press) journal in the field. If there really is such an important paper that needs 10 pages of discussion, then maybe designate that paper as a Discussion paper, so that other researchers can be invited to comment on it (a bit like the Bayesian Analysis model).
2. Registered reports: This should be provided as an option.
3. Obligatory data + code release with submission: All code and data must be released as part of the review process.
4. Costs: The journal must be free, perhaps with an option to pay if the author wants to contribute to the journal. That way, if the PI has money, they can help the journal. How many ESCoP members are there? Could they pay 5 Euros more every year to finance the journal running costs so it can be free for authors to publish there? If ESCoP has 10,000 members, that’s an annual budget of 50k. Journals like Language, the flagship journal of the Linguistic Society of America, demands that at least one author of a submission be an LSA member. This is a reasonable demand for ESCoP as well. Another option is to talk to the European Research Council. They hand out millions of Euros to researchers. Could they subsidize a journal? I bet they could.
5. Reviews and post-pub comments: Reviews (anonymous or not) should be published as online supplements of a published paper. A comment section should be allowed after publication (it’s OK and perhaps even desirable if only ESCoP members can comment).
6. Print archiving: Microtome publishing provides print archiving, which I think is important for a journal. See http://www.mtome.com/
I suggest looking at the Journal of Machine Learning Research as an inspiration for the upcoming ESCoP journal. It has a good reputation and it seems to be doing everything right:
You can also read the resignation letter of the editorial board members of Machine Learning to Kluwer here (below).
I just saw that ESCoP only has 650 members! So, perhaps the best way is to get external funding, e.g., from the ERC.
This is a great development!!! Well done ESCoP 🙂
We recently started a new journal for the same reasons in the European Health Psychology Society. We approached Ubiquity Press, a not-for-profit publisher, to handle the publication side. We implement some of the suggestions of Shravan (by the way, “some of my best friends are psychologist”, brilliant :-)), such as a strong Full Disclosure policy. We have blind reviews that become unblinded and citable upon acceptance (I personally think they should become unblinded upon *decision*, so also for rejections), and we want to work to enable post-publication reviews, as well. And we aim to publish all data – in other words, also of studies with flawed methodology.
Maybe we can learn from each other – so Candice or Sebastiaan, feel free to drop me a line if you want. We have the current policies at http://healthpsychologybulletin.com, and some FAQ at http://ehps.net/content/hpb-faq. The editorial explaining stuff in detail is currently being typeset – I can send it to you if you’d want to see it already.
In any case, great initiative!!! 🙂