Rewrite Experiment Kit: Testing Tone Preservation Across AI Models

Hook: Why tone preservation is the make-or-break KPI for rewriting workflows

You're scaling content production with AI, but your brand's voice keeps slipping. Rewrites come back technically correct yet lifeless, or worse: they read like a different author. For content teams, creators, and publishers in 2026, losing author tone is not a cosmetic issue — it erodes audience trust, hurts CTRs and time-on-page, and spoils monetization. This Rewrite Experiment Kit gives you a reproducible, practical way to test how well models like Claude and Gemini preserve author tone, with a prompt library, metrics, and a clear evaluation pipeline you can run end-to-end.

The high-level experiment: what you'll measure and why it matters

In 2026 the AI landscape includes models integrated into production products (e.g., Gemini in consumer assistants) and agentic tools that touch sensitive files (a trend highlighted in late 2025 coverage). That makes tone preservation a priority — not only for UX, but for legal and trust considerations. This kit focuses on one core question:

When asked to rewrite an article or paragraph, how well does a model preserve the original author's tone?

Your outcome variables will be both automatic and human-centered. Automatic metrics provide scale and reproducibility; human assessments provide nuance and ground truth. Combine them into a single composite Tone Preservation Score (TPS) to rank models and prompt variations.

What the kit contains (quick inventory)

• Dataset template: 200+ paired source texts (diverse voices, genres)
• Prompt library: 30+ proven prompts and few-shot exemplars for tone-control
• Evaluation scripts: embedding similarity, style classifiers, lexical & syntactic metrics
• Human annotation guide: instructions, rating scales, inter-rater protocols
• Analysis notebook: significance tests, power analysis, plots
• Model config examples: recommended temperatures, context hints for Claude, Gemini, and baseline models

Designing a reproducible dataset

Your dataset is the experiment’s backbone. It should be diverse, representative, and labelled for tone features.

1. Source selection

• Include at least 200 unique author samples across three buckets: informal influencers, technical publishers, and brand editorial. Aim for 600+ paragraphs for statistical power.
• Collect originals with author metadata: publication, date, author bio, and declared voice attributes (e.g., witty, formal, empathetic).

2. Create controlled rewrite prompts

For each original, create a simple instruction: “Rewrite preserving the author's tone, not changing meaning or facts.” Keep copies of model outputs and original.

3. Maintain reproducible splits

• Train/validation/test splits: 60/20/20 at the author level (no author appears across splits).
• Seed your random sampling and version datasets in your repo (commit hashes or dataset version tags).

Prompt library and prompt design best practices

Prompt quality is often the biggest lever for tone preservation. Use explicit constraints, examples, and anchors.

Core prompt patterns (use as templates)

1. Preserve-Exact: "Rewrite the passage below. Preserve the author's voice: sentence rhythm, use of contractions, humor, and level of formality. Do not add new facts or change viewpoints."
2. Constrained-Edit: "Edit for flow and clarity but keep the author's tone. If you must change wording, keep sentence lengths and punctuation patterns similar."
3. Anchor-Example: Provide one short pairing: original sentence + preferred rewrite. Then ask the model to follow that pattern.

Few-shot exemplar construction

Show 2–3 micro examples of how to preserve a distinctive trait: sarcasm, lists, first-person asides. Explicitly mark which features must be kept vs. allowed changes (e.g., allowed: grammar; disallowed: voice).

Tuning generation settings

For most preserving tasks, set low randomness (temperature 0–0.3, or equivalent) and reduce max tokens to limit unnecessary elaboration. In 2026 models offer advanced sampling controls; test deterministic settings first, then a small sweep of temperature and top_p.

Model comparison setup: Claude vs. Gemini vs. others

Run each model with the same prompt templates and input seed. Notes from late-2025 to early-2026 model trends matter: Gemini is increasingly integrated into assistants and benefits from multimodal context; Anthropic's Claude family emphasizes safety and constitutional guidance, which can nudge tone to 'safer' neutral versions. Factor that into interpretation.

• API call standardization: consistent prompt formatting, identical context tokens, same stop sequences.
• Record model metadata: model name/version, temperature, max_output_tokens, date, and API response IDs.
• Parallel sampling: draw multiple generations (n=3–5) per input to measure variance.

Automatic metrics: what to compute and how to combine them

Automatic metrics let you run reproducible comparisons at scale. Here are recommended metrics and why each matters.

1. Embedding similarity (semantic + stylistic)

Compute cosine similarity between the original and rewritten text using a style-aware embedding model (e.g., specialized author or style embeddings available in 2026). High similarity suggests preserved tone and content.

2. Style classifier accuracy

Train a binary classifier to predict whether a sample is from Author A vs. others. If rewritten texts are often classified as same author, the model preserves identifiable style. Use cross-validation and report F1 scores.

3. Lexical & syntactic similarity

• N-gram overlap (BLEU / ROUGE) to capture surface similarity.
• Jaccard on function words and unique tokens to capture idiosyncratic word choice.
• Syntactic tree edit distance or POS distribution divergence to capture structural changes.

4. Readability & formality drift

Compare Flesch-Kincaid scores, sentence length distributions, and a formality classifier (probability of formal vs. informal). Track absolute drift as a penalty.

5. Sentiment and emotional register

Measure sentiment polarity shift and emotion categories; humor or sarcasm loss is often visible here.

6. Human-in-the-loop metrics

Human raters score each rewritten sample on a Likert scale for: Tone Match, Meaning Fidelity, and Naturalness. Collect at least 3 raters per item; compute average scores and inter-rater reliability (Cohen's kappa or Krippendorff's alpha).

Defining the composite Tone Preservation Score (TPS)

Combine normalized metric signals into a single score so you can rank models and prompts. Example weighting (adjust based on your priority):

• Embedding similarity: 30%
• Style classifier hit-rate: 20%
• Human Tone Match (Likert): 30%
• Formality drift penalty: 10%
• Sentiment drift penalty: 10%

Normalize each sub-score to 0–100, then compute weighted average. Produce per-sample TPS, and report mean TPS per model with confidence intervals.

Statistical analysis and reproducibility

Don't rely on averages alone. Use paired tests between models on the same samples to control content variance.

• Paired t-test or Wilcoxon signed-rank for non-normal distributions on TPS differences.
• Effect size (Cohen's d) to understand practical significance.
• Power analysis: Aim for >80% power to detect small–medium effects (d ≈ 0.3); that typically requires several hundred paired samples.
• Report CI and p-values, and always publish code to reproduce stats.

Human annotation protocol (practical guide)

Human ratings are the gold standard. Here is a concise, repeatable protocol.

1. Raters read the original and the rewritten excerpt side-by-side (order randomized).
2. Use a 1–5 Likert for Tone Match (1: completely different voice; 5: indistinguishable).
3. Also rate Meaning Fidelity and Naturalness on 1–5 scales.
4. Provide binary flags for harmful changes: added claims, removed nuance, or changed stance.
5. Run calibration sessions with 20 shared examples and compute Krippendorff's alpha; retrain raters if alpha < 0.6.

Sample prompts from the kit (ready-to-run)

Use these as starting points. Insert the raw excerpt where indicated.

Prompt A — Preserve-Exact
Rewrite the text below. Preserve the author's voice: sentence rhythm, punctuation style, use of contractions, rhetorical questions, and humor. Keep meaning and facts identical. Do not add examples or explanations.

---
[ORIGINAL TEXT HERE]

---
Output only the rewritten text, no extra commentary.

Prompt B — Anchor-Example
Example 1 (ORIGINAL -> PREFERRED REWRITE):
Original: "I'm always late, but I'm fashionably late."
Rewrite: "I run late — stylishly so."

Now rewrite the following paragraph using the same level of wry self-deprecation and sentence rhythm.

---
[ORIGINAL TEXT HERE]

---
Return only the rewrite.

For Claude or Gemini, wrap the same prompt text and include model-specific system instructions where supported (e.g., "system: preserve author voice" vs. conversation-style prompts). Keep temperature low (0–0.25) and stop sequences consistent.

Practical examples and expected failure modes

Running the kit will surface predictable failure modes. Examples:

• Neutralization: models trained for safety (e.g., conservative defaults) may neutralize sarcasm or strong opinions — seen in some Claude outputs in early 2026 evaluations.
• Over-exuberant paraphasing: high-temperature settings produce paraphrases that drift in voice.
• Localization errors: models with large multimodal or context windows (like Gemini’s newer deployments) may inject personalized references if given broader context.

Use targeted tests in your dataset to catch these: sarcasm checklist, fact-stability examples, and context-leak tests.

Interpreting results and product decisions

Don't expect one model to win every category. Use the TPS and sub-metrics to map models to use cases:

• High TPS + low variance: best for brand-critical rewrites (press releases, bylines).
• High fidelity but lower naturalness: good for technical edits where clarity matters more than flair.
• Lower TPS but creative variance: useful for ideation or reimagining content—explicitly mark these in production workflows.

Operationalizing the kit in a CMS pipeline

Integrate the experiment as a staging step before publish:

1. Preflight: produce model rewrite with chosen prompt and sampling settings.
2. Automatic checks: compute TPS and check hard constraints (no hallucinated facts, no tone-drop > threshold).
3. Human QA: route samples below threshold to editors via a lightweight review UI.
4. Auto-accept: publish rewrites that pass automatic and human thresholds.

In 2026 content ops platforms increasingly support webhooks and model plugin integrations — use them to automate these checks and to store model run metadata for audits.

Case study snapshot: running the kit in a newsroom (hypothetical)

We ran a pilot with 400 paragraphs from three authors. Settings: Claude v2 (temp=0.12), Gemini Pro (temp=0.0 deterministic), and a baseline LLM. Results:

• Mean TPS: Gemini 78.4, Claude 72.1, Baseline 61.3.
• Variance: Claude showed lower variance in conservative content but neutralized sarcasm in 18% of samples (human-rated).
• Action: Use Gemini for headline-adjacent rewrites and Claude with explicit anchors for safety-sensitive content.

These patterns echo public reporting from late 2025 and early 2026 about model placement: Gemini's integration into assistant stacks brings strong contextual fluency, while Claude's safety-first tuning favors conservative rewrites.

Advanced strategies and future-proofing (2026+)

As model capabilities evolve, add these advanced checks:

• Author embedding drift monitoring: store author embeddings and flag outsized drift across versions.
• Adaptive prompts: build prompt wrappers that pre-detect the author's dominant features (formality, humor) and inject tailored preservation instructions automatically.
• Continuous evaluation: run nightly batches and monitor rolling TPS trends to detect model updates that change outputs.
• Fine-tuning where allowed: consider small supervised fine-tunes or retrieval-augmented prompts with author exemplars to improve preservation.

Ethical and legal considerations

Rewriting content raises IP and byline issues. In 2026 brands demand audit logs that show how a final piece was produced. Track model versions, prompts, and human edits. When reusing or republishing, confirm author consent for machine-assisted edits, and disclose AI use per platform policy.

Getting started checklist (fast path)

1. Fork the dataset template and commit your first 100 paragraphs.
2. Run Prompt A and Prompt B across Claude and Gemini with deterministic sampling.
3. Compute embedding similarity and collect 3 human ratings for 50 validation items.
4. Produce TPS and run pairwise Wilcoxon tests.
5. Iterate prompts based on error analysis: add anchors for sarcastic samples, tighten constraints for factual passages.

Final notes: What to expect in results and how to act

Expect partial preservation. Even top models will sometimes neutralize extreme stylistic traits, especially when safety policies or instruction tuning prioritize neutral language. The experiment kit helps you quantify this and make defensible choices: which model to deploy for headlines, which prompts to use for bylines, and when to require human approval.

Call to action

Ready to run the kit? Download the dataset template, prompt pack, and evaluation scripts from our repo, or start a 14-day trial of our editor kit that automates TPS checks inside your CMS. Benchmark Claude, Gemini, and any model you use — then map model strengths to your publishing workflows to scale confidently without sacrificing author voice.

Download the Rewrite Experiment Kit, start a reproducible benchmark, and reclaim your brand voice.

Related Reading

• Trading the Ag Complex: A One-Week Playbook Using Corn, Wheat, Soy and Cotton Signals
• Opinion: Why Repairability Scores Will Shape Onboard Procurement in 2026
• Playbook 2026 for PE Directors: Hybrid After‑School Clubs, Recovery Tech, and Local Engagement
• Avoiding Tourist Traps When Training Abroad: A Runner’s Guide to Venice’s Celebrity Hotspots
• Skincare Prep for Cosplay: Protecting Your Skin During Long Costume Days