How to fact-check an AI-drafted book
List every claim a reader could check, then trace each one to a source you have opened yourself. Start with numbers, quotations, names and citations, where invented details do the most harm. If you can’t confirm a claim, cut it or rewrite it as your opinion. An AI can help you find what to check, but its say-so isn’t a source.
Why AI drafts need this
A language model writes what is likely, which isn’t the same as what is true. It can produce a plausible statistic or a citation to a paper that doesn’t exist, in the same even tone as everything else.
One study measured this directly. Walters and Wilder had ChatGPT write short literature reviews on 42 topics and checked the 636 citations. In their words, “55% of the GPT-3.5 citations but just 18% of the GPT-4 citations are fabricated”, and “43% of the real (non-fabricated) GPT-3.5 citations but just 24% of the real GPT-4 citations include substantive citation errors”. Those were early-2023 models, and newer ones will differ. Take away the shape of the problem instead of the number: a citation can be invented outright, and a real one can still be wrong in its details.
What to look for, and how to check it
| Claim | What goes wrong | How to check |
|---|---|---|
| Statistics | The number is invented, or real but old, or from a different group | Find the original study or dataset. Note its year and who it measured |
| Quotations | The words are close but not what was said, or the wrong person said them | Find the exact sentence in the original text |
| Citations | The paper doesn’t exist, or the authors, year or title are wrong | Look up the title and open the paper. Read what it actually claims |
| People and companies | Wrong job, wrong date, an invented event | Use the person’s or company’s own site, or a reliable news source |
| Laws, rules and prices | True once, changed since | Use the official page, and put the date you checked in the book |
| “Studies show” | No study behind it | Name the study or cut the phrase |
Examples and anecdotes need the same care. If the draft describes “a client” or “a coach in Denver”, either replace it with something that really happened, or say plainly that the example is made up.
A method for what you can’t place
Mike Caulfield’s SIFT is built for checking things found online, and it works well here. His four moves are: Stop; Investigate the source; Find better coverage; and Trace claims, quotes, and media back to the original context. The last is the one AI drafts need most. A good claim usually has a traceable origin, and if every search leads back to the same unsourced line, treat that as your answer.
Your name is on it
KDP’s guidelines are direct: “You are responsible for verifying that all AI-generated and/or AI-assisted content adheres to all content guidelines, including by complying with all applicable intellectual property rights.” That sentence is about content rules and rights, but readers apply the same standard to facts. A nonfiction book that gets one checkable thing wrong makes readers doubt the rest of it.
Where Chapbook fits
Chapbook can narrow the job, not do it. You can add up to 10 sources to a book’s notes, and the drafting step may cite only those by their keys; a citation to anything else is stripped out. Web research overviews read part of a long source and say so, so check the original before relying on a quotation.
The Polish read-through then looks over the whole book and flags claims of fact or figures with no supporting source, citations that don’t match the source they name, and contradictions between chapters. It quotes the sentence, and where one small edit would fix it, offers a change for you to accept or reject. It is an AI reading your book, so treat its list as a place to start. It only knows your sources, and it can miss things.
The sample on this page was drafted with Chapbook and left unedited. Its brief asks for a method, so most of what it says is advice to weigh and not fact to check. Try the steps above on it anyway.
What to do
-
Mark every checkable claim
Go through each chapter and highlight numbers, dates, names, quotations, citations, and any sentence that says what research or experts found. Don’t check yet. Just list them.
-
Check the riskiest first
Start with anything about health, money or the law, then figures and quotes, then named people and companies. If time runs short, you have covered what could hurt a reader.
-
Open the original
Find the study, report or book itself and read the passage. A blog post that repeats the number isn’t the source, and neither is the model that wrote the sentence.
-
Keep, fix or cut
If the source agrees, keep the claim and note the link. If it says something else, fix it. If you can’t find it, cut it or rewrite it as your own view.
-
Have a person read it
Give the checked draft to someone who knows the subject, and ask what looks wrong. In a regulated field, that reader should be a qualified professional.
