Ethan Mollick, prominent AI scholar and enthusiast, had a good, if leftfield, benchmark for LLM capabilities - its ability to draw an otter on a plane. Over the years, he asked successive generations of models to produce an image of just that. And boy did they improve!
Well, I’ve got my own benchmark - the suffragist peace theory. About three years ago now, I tasked a model with bottoming out this interesting claim from political theory: that countries with more female emancipation are less likely to go to war.
Back in 2023, this was tough going — I had to do lots of legwork, and the data was, if I’m honest, a bit sketchy. My conclusion:
Overall, there’s no doubt that transformer AIs will change social science research, in a more profound way than generating fluent, plausible prose. And I would expect better data tools to arrive imminently. The code transformer wasn’t really designed for what I’m using it for, and it’s still in beta anyway. Interesting times.
Well, here we are, October 2026. The models have advanced at an astonishing rate. Every day brings news of new breakthroughs - just today, for example, three new proofs from OpenAI that, if substantiated, will have large implications for maths, physics, and computer science.
Here in the social sciences, change is coming too - though we’re some way down OpenAI’s list of things to set its state-of-the-art internal model to work on. When Kissinger opined that academic disputes were so bitter because the stakes were so low, I bet he was thinking of social science, not pure maths.
Anyway. Here’s where we are today. Over lunch, Claude wrote me a paper on the suffragist peace. My instructions were minimal (forgive my typos, I was hungry):
many months ago on openAI, I set about using llms to find data and do analysis to explore whether the suffragist peace theory was right. It\s a nice benchmark. You've come on loads since then. Let's go again. I want you to do the lit review, hypothesis generation, data gathering and analysis. And I want you to write the draft paper, with all that - including figures and tables. I can help insofar as you have problems downloading data sets you find. But I can't help intellectually, except in seeking clarification of anything that's unclear. Ack? Let's do this!
It did need some logistical help too - academic paywalls being what they are. But I was content to be the lowly research assistant and download the pdfs it asked for. Towards the end, its proof read it caught a fun error - it had used data from a preprint for one of the key articles, and found a new dataset from the final, published version. Good catch, Claude.
So, here’s the abstract, which links to the final paper (posted below the line, in the footnotes).
Abstract
An influential recent argument holds that giving women the vote made democracies more peaceful, and that women’s suffrage, rather than democracy alone, explains why democracies rarely fight each other. The claim is hard to test because almost every democracy enfranchised women within a few decades of 1918. A comparison of democracies with and without women’s suffrage is therefore mostly a comparison of nineteenth-century democracies with twentieth-century ones. Using public data on every pair of states from 1816 to 2014, I re-test the claim with methods designed to separate the effect of suffrage from the effect of the era in which it arrived.
The central comparison in the original study — pairs of democracies where women could vote against pairs where they could not — does not replicate: the difference is much smaller than reported and statistically insignificant, and it rests on 16 disputes, all before 1919. After 1918, pairs of democracies that both excluded women had no disputes at all. A second, broader measure does replicate, including after 1918: democracies that enfranchised women were more peaceful with each other. But moving every country’s suffrage date up to 40 years earlier, to years when women could not vote, leaves that association as strong or stronger. What matters is which countries enfranchised women, not when they did it; those countries were already becoming more peaceful before women voted. Comparing countries with themselves over two centuries does show a link, but it depends on nineteenth-century great powers, and in the years after women gained the vote countries did not start fewer disputes than comparable countries that had not yet enfranchised women. Women’s suffrage marks the democracies that became peaceful, but the global record of interstate conflict cannot show that it made them so.
It’s a cracker! But is it right? I think you’d have to ask an expert, and I might. We really ought to investigate further, since Claude’s analysis challenges the received wisdom in important ways. You know, a good next step is to give it to two more LLMs to perform peer review. Last week saw a preprint weighing the performance of AIs versus humans as scholarly peer reviewers. Guess which was better? Yes - not the one insisting the that authors cite more from Kenneth Payne, or rewarming stale old personal disputes from decades of intellectual sparring.
I’d send this to a leading journal. But they almost all stipulate no AI writing. Which begs the question, ‘Why?’ What are we doing here - performative ludditism?
