Information gain and slop: how we measure them

Information gain

A reader comes to a text with questions: what exactly has to happen, where, when, how much, under which conditions and why. Information gain is the share of these questions that the text answers. At 100% the reader has nothing left to ask; a form with empty fields is close to 0%. The model in your browser estimates this share as a rating from 0 to 10 and shows it in steps of 5%.

The idea goes back to Claude Shannon, who measured information in bits: one bit is the answer to one yes-or-no question (Shannon, 1948). A bug report that says “large exports fail” leaves the developer asking “12,300 rows? 11,000?”, one yes-or-no question after another; “exports over 10,000 rows fail” answers them all at once. Warren Weaver described three levels of communication: whether the symbols arrive, whether they carry the intended meaning and whether they change what the receiver does (Weaver, 1949). Slopcheck works on the second and the third level.

Information gain in SEO

In SEO the term means what a page adds beyond the pages its reader has already seen. Google holds a patent on such a score, “Contextual estimation of link information gain” (US 11,354,342 B2, granted on 7 June 2022). A patent does not show that a score is used in ranking; Google’s public guidance asks whether content provides “original information, reporting, research, or analysis” (Google Search Central). Slopcheck measures the part that 1 text shows on its own, without comparing it to other pages: the share of its reader’s questions it answers, from 0 to 100%.

What a statement rules out

Rudolf Carnap and Yehoshua Bar-Hillel described the meaning side of information: a statement carries more information the more possible states of the world it rules out (Carnap and Bar-Hillel, 1952). “The export fails for orders over 500 lines in Chrome 128” rules out far more than “the export sometimes fails”. A named threshold is the strongest form: “3.1.4 works, 3.2.0 fails” tells the developer which release to open.

Whether a statement is true is a separate question (Stanford Encyclopedia of Philosophy). Slopcheck does not check facts: “orders over 500 lines” scores the same whether the real threshold is 500 or 5,000.

What matters to the reader

Ronald Howard showed in 1966 that information is worth what it changes in a decision (Howard, 1966). For a developer, “Chrome 128” decides which of 3 browsers to test; “the export is important” decides nothing. On a product page, “4 EUR per user a month, first 30 days free” changes the decision to buy; “a seamless experience” does not. In a bug report, the version, the steps and the threshold are what change the developer’s next step.

Slop

Slop is the share of the text, counted in words, that answers no question of the reader. We see 4 kinds: empty sentences that would fit any text, obvious ones that rule out nothing the reader considered, irrelevant ones that change no decision, and unresolvable ones such as “as discussed” or “same as there”. Repeats of what the text has already said count too. In the result, every slop sentence is highlighted.

In terms of conversation, slop breaks 2 of the maxims described by Paul Grice: be as informative as required and no more, and say what is relevant (Grice, 1975). Repeats are what Shannon called redundancy: the part of a message the reader could predict from the rest.

Style is not slop: “we fixed the CSV export” and “the CSV export issue has been resolved” count the same.

How the two numbers relate

Slop carries no information, so the more of a text is slop, the lower the ceiling on its gain: a text that is 60% slop scores at most about 50%, and one that is 95% slop or more scores at most 10%. A full 100% is shown only when no sentence is slop; otherwise the gain stops at 95%. Length is not the goal: a bug report of 50 words can score 90%.

Bug reports, tasks and product pages

We wrote down 27 questions a reader brings to a text: 7 for any text, 8 for bug reports, 7 for tasks and 5 for product pages. A bug report answers which version, which steps, what was expected and what happened, always or sometimes, and since which release. A task answers what, where, why, how to check that it is done and what not to touch; good acceptance criteria read like “34 settings, 4 tabs, no text changes”. A product page answers for whom, what it does, what proves it, the price and the next step. The model learned these differences from 2,424 examples rated by a language model, so you do not need to choose a genre before you paste a text.

5 versions in 4 days

Version 1, on 19 October 2026, counted function words, fillers and concrete details per 100 words. Version 2 added the rarity of words, measured against frequency lists we counted from about 600,000 words of Wikipedia text per language; on 55 web pages rated by two language models it was noise (AUROC 0.45 to 0.49), so we dropped it. Version 3 judged by concreteness per 100 words; on 37 new pages set aside for testing, its “too little substance” verdict was right in 1 case of 9, so it became a hint.

Version 4 used a catalogue of about 1,000 patterns in 5 languages and separated padded pages from dense ones on the same 37 pages (AUROC 0.75; we had seen those results during development, so the figure is optimistic). On real bug reports it was at chance (0.47 to 0.52). That failure moved us from counting words to counting the reader’s open questions, and to version 5 (s2) on 22 October; on 24 October we retrained it for 12 languages (s3).

What bug trackers taught us

We took 1,355 Mozilla Bugzilla reports from 2023 to 2025. Reports closed as incomplete or “works for me”, or whose author was asked for more information, were longer than reports fixed without such a request (a median of 165 words against 124) and more concrete: outsiders fill in the full template, which appears in 38% of “works for me” reports and 14% of fixed ones, while engineers write short notes that rely on shared context. The outcome measured who wrote the report, so from then on we compared reports by the same author.

On Apache Jira we collected 11,129 bug reports from 165 projects filed since 2022 and compared fixed reports with those closed as “Cannot Reproduce” or “Incomplete”. One finding shaped the method: a language model read 240 of these reports, 120 pairs by the same author, blind to the outcome and the author. Asked whether a version or environment was given, it found one more often in the poorly received reports (63% against 50%). Asked what a developer would still need, it named the version or environment more often for the same reports (48% against 33%). Mentioning a detail is not the same as answering the question.

Rules from a manual review

We rated 35 texts by hand, most of them on a scale from 0 to 10: 0 is a form with nothing in it, 5 to 6 can be done after a couple of questions, 9 leaves one open question, 10 leaves nothing to add. On 28 of these texts our first question catalogue raised 86 questions; we marked 35 of them, 26 as fair and 9 as excessive. The result is a written guide of 37 rules and 33 test cases.

Examples of rules: a statement without a limit means always and everywhere, so a missing version costs nothing unless the text says “sometimes” or “after the update”; a prohibition counts only if it closes a path the task implies; contradicting requirements cost the most. In one test the same facts, rewritten with an explicit link and a named object, went from 1 to 9–10 out of 10. In another, a report rated 10 by hand had been closed as “Cannot Reproduce”: a tracker outcome is a signal, not a verdict.

Language models as raters and as a teacher

We turned the 37 rules into a rating guide for large language models and compared 5 models on the 27 texts with numeric hand ratings. The rules came from these texts, so this shows how closely each model follows the rules, not how it does on new texts: the best two landed within 1 point of the manual rating in 22 of 27 texts (rank correlation up to 0.94); the weakest, in 12 of 27.

One of the two then rated 2,424 texts sentence by sentence: 600 public Apache Jira tasks and bug reports, 384 texts with the same facts that language models wrote in 4 styles, 600 texts that the same model wrote for training in German, Polish, Czech, Croatian and Russian, and 840 texts, 120 in each of Slovak, Slovenian, Bulgarian, French, Italian, Spanish and Dutch, that language models wrote without seeing the rules. That is 18,420 labelled sentences, 21.4% of them slop. The small model in your browser learned from these labels. In cross-validation it agrees with the teacher at a rank correlation of 0.85, and on texts it did not learn from it finds the teacher’s slop sentences with an AUROC of 0.92. On 25 hand-rated texts it agrees with the manual ratings at a rank correlation of 0.81, but the rules on top of it come from the same texts, so the held-out tests below are the fairer measure.

No language model reads your text when you check it. A model call would send the text to a third party, could give a different answer to the same text and would cost money per check; the small model runs in about 350 KB of fixed weights.

Predictions written down first

For 5 experiments we wrote our predictions to a file before running the test; for the pattern test on 20 October 2026 the file’s MD5 checksum begins with ff01ce1a. We predicted that the plain version of a text would beat the assistant-style one in 85% to 95% of pairs; it did in 79%. We predicted that rejected Bugzilla reports would be shorter; they were longer, with a median of 165 words against 124. On same-author Jira pairs 3 of 4 predictions missed. After the Bugzilla miss, every later comparison used reports by the same author.

What did not work

Word rarity (AUROC 0.45 to 0.49). The share of function words and fillers, which on held-out pages reached an AUROC of only 0.31, worse than chance. A “too little substance” verdict that was right 1 time in 9. Counting units of meaning or ticking a 5-point completeness checklist, which agreed with a reading of the whole text at a rank correlation of only 0.18 to 0.26. 8 indicators modelled on search-engine signals, rejected at the design stage. Comparing a page with the top 3 to 10 search results, which needs a paid search API. Broad versions of the explicit rules, which fired on 33 Jira reports; every one we checked was a false alarm. A language model at check time: it came closest to the manual ratings, but it would send your text to a third party.

How we test the method

We hold out 24 briefs in English and German, each written in 4 versions by language models. The plain version scores higher than the padded one in all 24 pairs and higher than the assistant-style one in 83%. For each of the 7 languages added on 24 October we hold out 6 more briefs in 4 versions: the plain version beats the padded one in every pair, and the assistant-style one in 100% of pairs in Slovak and Slovenian, 92% in Italian and Dutch, 83% in Bulgarian and French and 58% in Spanish.

On 120 held-out pairs of Jira reports by the same author, the fixed report scores higher in about 60% of pairs. A language model that reads the same 240 reports blind, without the outcome or the author, reaches 64%. The English and German briefs and the Jira pairs were left out of training, but we used them to choose between variants of the explicit rules, so on new texts the results may be slightly lower. Whether a report ends up fixed or closed as “Cannot Reproduce” also depends on the bug itself: it may already be fixed in a newer version, occur in only one environment or come from a configuration error, so 100% is not possible on this test.

How the estimate is computed

For the whole text the model reads 182 signals, such as numbers, versions, conditions, thresholds, names, quoted messages, vague words, unsupported claims, repeats and empty template headings, plus words and word pairs hashed into 16,384 cells; a second model with 68 signals and the same hashed words decides for each sentence whether it is slop. The text model is a linear regression on these signals and the sentence model a logistic regression, both fitted on the 2,424 rated texts. 3 rules from the manual review apply on top of the model. A start time after the deadline costs 25 points. A task that opens with “split into 4 tabs” and never says what to split gets at most 20%. A template with empty headings gets at most 10%. The weights take about 350 KB and load with the page, and a text of 3,000 characters is checked in under 100 milliseconds.

Sentences and languages

We split the text into sentences at full stops, question and exclamation marks, ellipses and line breaks, but not inside “v8.1”, “1.200” or a web address, and not after a date such as “7. Juni”. 12 languages are supported: English, Polish, Czech, Slovak, Croatian, Slovenian, Bulgarian, German, French, Italian, Spanish and Dutch. The language is detected automatically: 99.6% correct on our earlier test sets and 97.1% on short texts in the new languages. We compiled the lists of slop phrases for the 11 languages other than English ourselves; for German, Polish, Czech and Croatian we found no open ones. Free checks take 80 to 3,000 characters, which covers 97.5% of the Jira bug reports we collected, not counting code blocks. Known bias: the language alone shifts the estimate by a few points. With the same signals, Spanish, French, Czech and Italian texts score 5 to 8 points lower than English ones, German and Dutch about 4 points higher, so compare scores within one language.

What we do not measure

Slopcheck does not measure who wrote a text, whether it is true or how it will rank in Google or Bing. On a sample of the public RAID benchmark in English, rule-based text features caught at most 28% of AI-written texts at a 5% false-alarm rate. That is why we do not offer authorship detection. A 2-line note can be exactly right for a team that shares its context: the estimate sees only the text.

Questions

How do I check a text for AI slop?

Paste it into the field on the home page: 80 to 3,000 characters, up to 5 texts a day for free. You get information gain, the slop share and the slop sentences marked in the text.

Is my text stored or sent anywhere?

No. The check runs in your browser: the text is not sent to us and is not stored. The result link keeps the text in the part of the address after the # sign, which browsers do not send to servers.

Which texts can I check?

Articles, product pages, posts, tasks, user stories and bug reports in 12 languages: English, Polish, Czech, Slovak, Croatian, Slovenian, Bulgarian, German, French, Italian, Spanish and Dutch, from 80 to 3,000 characters.

How do I write a good bug report?

State how to reproduce the problem: which steps, which version, which device and at which threshold it starts to fail. In a study of 180 reproducible Android bug reports, 92.2% missed at least 1 step (Johnson et al., 2022). “The export fails for orders over 500 lines in Chrome 128” rules out far more than “the export sometimes fails”. Paste the report into the check to see which sentences answer none of these questions.

Is Slopcheck an AI detector?

No. Slopcheck estimates how much a text tells its reader. It does not say who or what wrote it: a generated text with real facts can score high, and a hand-written one with none scores low.

How is it different from checkers that look for AI tells?

Tools that count em dashes, “delve” or “tapestry” look at the surface of a text. Slopcheck looks at what the reader gets: a text can avoid every tell and still answer none of the reader’s questions.

What is a text passport?

A record of one exact text: its hash, information gain, slop, the method version and the date of the check, on a page with its own link. Whoever has the text can paste it there and see whether it matches: if even 1 character differs, the hash differs. Each edit is saved as a new version with its own scores.

Do I need a plugin or an SDK to use the API?

No. On the Team plan the check is one HTTP request with the text. It fits a CI step, a pre-publish hook in a CMS or a field in your task tracker.

Sources

Check your text

Paste an article, a product page, a task or a bug report and see its information gain and slop.

Check a text for free

How we count