Everything tried, what failed, what worked, and what to patch.

pangram-redteam

Adversarial evaluation of Pangram 4 (text.external-api.pangram.com) on a HARO-style reply email. Key in ~/.claude/pangram.env. score.py file... scores files (cached in cache/, log in results.tsv). gen.py MODEL OUT SYSFILE USERFILE generates with OpenAI.

Result

original.txt = 0.805 AI (High). FINAL.txt = 0.286 Human (High), all facts retained, 2026-10-11. Pangram is deterministic (same text -> same score). Label boundary: Human <= ~0.35, AI >= ~0.42, grey zone between.

What moved the score (ai_assistance_score, single window)

What did NOT work (all 0.94-1.0, humanizer head fires on the degraded ones)

• "Sound more human" rewrites: loose/digressive, choppy, messy email register
• Typos / lowercase / dropped punctuation (humanizer 0.40), clunky grammar (humanizer 0.81), fragments
• Dense specifics (names, suburbs, dates, invoice numbers)
• Fresh generations: GPT-5.5, GPT-4.1, Claude, plain or heavily-constrained persona prompts
• Finnish round-trip translation (0.94, humanizer 0.47)
• Zero-width spaces: normalised away (0.816 = no change)
• Genre shift: SMS thread 1.0, interview transcript 0.998
• Quoted email thread + signature: creates a second window scoring 0.04 Human -> verdict "Mixed", but longer quote merged into one window and scored 0.93

Method that worked

1. Score the original. 2. Ablate one sentence at a time (1 credit each) to find the 2-4 sentences carrying the signal.
2. Delete them; confirm Human. 4. Re-insert their facts as sub-clauses/parentheticals inside sentences that already score low.
3. Add a sign-off. Never rewrite whole paragraphs: every fresh LLM rewrite scored higher than the source. Carriers here were evidential/framing sentences ("The roofer found X, with Y and Z", "She later told me", "During my ... years, I saw").

2026-10-11, part 2: does a prompt alone do it? No.

• PROMPT.txt (editing rules distilled from the ablation) applied blind by GPT-5.5 to 3 texts x 2 samples: 0.51-0.96, all AI. GPT deletes the wrong sentences.
• Direct generation: GPT-5.5 0.997, GPT-4.1 0.983, gpt-5.6-sol 0.938/0.996, Claude 0.89-1.0. Generator model is irrelevant.
• ablate.py (score-guided deletion + GPT fact restore) on fresh AI emails: best 0.533 / 0.688. Not enough on its own.

What DOES generalise: score-oracle hill-climb (evolve.py + CANDIDATE_PROMPT.txt)

Loop: rank sentences by single-deletion impact (n credits) -> for the top 4, have an LLM write 5 structurally different replacements -> score each in full-text context -> accept the lowest -> repeat. b3/gptA.txt (GPT-5.5, 0.997 AI) -> 0.644 Mixed after 70 credits -> 0.251 Human after 57 more (Sol as candidate writer). Output: FINAL_gptA_evolved.txt. Quality cost: two sentences became telegraphic semicolon lists, one got a parenthetical. Requirements for the attack: deterministic continuous score per submission, unlimited near-duplicate submissions. ~130 credits/text.

What Pangram can patch

1. API: the per-window ai_assistance_score is deterministic to 3 decimals. Quantise it, add per-key noise, or return label-only for keys that submit many near-duplicates (shingle/MinHash on consecutive submissions catches single-sentence edits trivially).
2. Model: the moves that lowered the score were (a) deleting evidential/framing sentences ("the roofer found", "she told me", "in my experience"), (b) semicolon-separated noun-phrase fragments, (c) facts stuffed into parentheticals after a short main clause. Those three structures should be sampled into the AI-class training data.
3. Nothing stylistic works without the oracle: humanizing, typos, persona prompts, translation, genre, unicode all fail.

2026-10-11, part 3: one-prompt attempt (REWRITE_ONE.txt), blind, GPT-5.5

Rules distilled from the winning oracle moves (delete framing sentences, no reporting verbs, rotate 4 syntactic shapes, <110 words, sign-off). 4 fresh AI sources (roof email, kettle review, LinkedIn post, history essay) x 2 samples: essay 0.883/0.937, kettle 0.999/0.999, linkedin 1.000/1.000, roof 0.980/0.345(AI-assisted). 0 of 8 Human. Humanizer head starts firing (0.08-0.23) when the fragment shape is applied by rule. The two roof outputs differ by a few clauses and score 0.98 vs 0.35: the decision surface is not smooth, so fixed rules cannot target it; search can.

Deliverable: the series of steps that does work (needs their API as the oracle)

1. Split the text into sentences. Score the full text (1 credit).
2. Score the text with each single sentence deleted (n credits). Rank sentences by how much the score drops.
3. For the top 4 sentences, have an LLM write 5 structurally different replacements that keep the same facts (CANDIDATE_PROMPT.txt). Score each replacement inside the full text (20 credits). Accept the lowest if it improves.
4. Repeat 2-3 until prediction_short == "Human" (score <= ~0.35). Typical cost 100-150 credits for 130 words. Implementation: evolve.py. Demonstrated: 0.997 -> 0.251 Human on b3/gptA.txt.

2026-10-12: lean version (evolve_lean.py + CANDIDATE_PROMPT_LEAN.txt)

Changes: one LLM call classifies fact-free sentences; rank once; delete fact-free sentences using the ranking scores (free); 3 candidates (parenthetical, <8 words, no reporting/evaluating verb); rotate out sentences that fail twice; re-rank only after a round that gains <0.05; stop at the Human label. | text | old calls -> score | lean calls -> score | |---|---|---| | b3/gptA.txt (0.997) | 127 -> 0.251 Human | 68 -> 0.170 Human | | jenn.txt (0.834) | 78 -> 0.152 Human | 22 -> 0.345 Human | "calls" = distinct texts scored, i.e. what a fresh run pays. AGENT_PROMPT.md updated to the lean steps. Note: label is not a pure function of the max window score (0.314 came back "Mixed"), so loop on the label.