What survives when a tool edits a reviewed Word document
Two contract drafts with a live review history: comments and replies, other reviewers' pending tracked changes, footnotes and a header. Thirty ordinary requests: replace a phrase, reply to a comment, accept one revision, fix a footnote. Six document tools, python-docx on two code paths, and three coding agents ran them; an oracle that knows nothing about the checker compared every output with its source, and one output per kind of damage was opened in Word for Mac.
Claude Code, done and preserved
150 of 150
attempts, five captures
Codex, done and preserved
150 of 150
but Word hides some replies and refuses five files
docx-cli outputs damaged
38 of 56
attempts that ran
Correct edits the checker flags
130 → 12
of 537, once the requested change is declared (0.4.8)
What each tool did
Share of the attempts that ran. Done means the requested text is in place and recorded as asked, tracked or not. Preserved means nothing else changed: comments, anchors, threads, other authors' revisions with their author and date, notes, lists, tables. Operations a tool has no interface for are left out; the table has them.
- Done and preserved
- Preserved, not done as asked
- Damaged
Table with every count
| Tool | Ran | Unsupported | Rejected | Done and preserved | Preserved, not done | Damaged | Damaged and reported by 0.4.6 |
|---|---|---|---|---|---|---|---|
| Control | |||||||
| Reference control edits the XML the way Word records it | 60 | 0 | 0 | 60 | 0 | 0 | – |
| Document tools | |||||||
| python-docx 1.2, Run.text = the careful path | 28 | 32 | 0 | 20 | 8 | 0 | – |
| python-docx 1.2, Paragraph.text = the tutorial and MCP-wrapper path | 28 | 32 | 0 | 6 | 8 | 14 | 14 of 14 |
| adeu 3.0.6 tracked changes only | 34 | 24 | 2 | 30 | 0 | 4 | 4 of 4 |
| docx-cli 0.26.0 | 56 | 4 | 0 | 18 | 0 | 38 | 16 of 38 |
| Office-Word-MCP-Server 1.1.11 archived | 24 | 36 | 0 | 8 | 10 | 6 | 4 of 6 |
| docx-mcp 0.7.4 | 36 | 4 | 20 | 34 | 2 | 0 | – |
| docxengine 1.0.0 | 40 | 0 | 20 | 32 | 2 | 6 | 2 of 6 |
| Coding agents | |||||||
| Claude Code 2.1, Opus 5.5 five captures | 150 | 0 | 0 | 150 | 0 | 0 | – |
| Codex 0.160, gpt-6.1-sol five captures | 150 | 0 | 0 | 150 | 0 | 0 | – |
| OpenCode, Qwen3.8 27B (local) one capture | 30 | 0 | 0 | 29 | 0 | 1 | 1 of 1 |
What broke, as Word shows it
Each class below was found by the oracle and then opened in Word for Mac 16.113.2. Where a tool's maintainers can fix it, an issue is filed.
Bullets turn into •
Any write escapes the list's existing bullet reference a second time; Word prints the literal text instead of a bullet.
docx-cli, every output of the first document · issue, fixed in docx-cli 0.27.0
New words credited to another reviewer
A tracked replacement inside another author's pending insertion leaves the new words without a wrapper of their own, so Word shows them as that author's; next to a nested deletion it also deletes the wrong words.
docx-cli · issue, fixed in docx-cli 0.27.0 for tracked edits; the untracked edit of the wrong words is open
Replies nobody can see
The reply is in the comments part and linked to its thread, but has no anchor in the document. Word shows the reply neither in the margin nor in the Reviewing pane.
docx-mcp, docxengine, and the agents Codex and OpenCode · issues for docx-mcp and docxengine
A file Word will not open
The part that links replies to their threads is declared with a content type its relationship does not allow. Word reports unreadable content and opens the file once only that value is corrected; LibreOffice opens it either way. Found after the evaluation, in a Word check of repaired copies; the frozen oracle counts these outputs as correct.
Codex, all five outputs of one comment task · Word check
A comments part Word has to repair
A new comment uses a namespace prefix the part never declares. Word reports unreadable content; after repair the new comment is empty and has no author.
docxengine, on a classic comments part · issue
The title edited instead of the header
Asked to change the header, the tool replaced the same words in the document title, which it could reach, and left the header alone.
Office-Word-MCP-Server, docxengine
Comment anchors and footnote references gone
Setting a paragraph's or cell's text rebuilds its runs and drops the comment range and the footnote reference inside it; the note stays in its part and disappears from the page.
python-docx Paragraph.text = / _Cell.text =: 14 of 28 outputs
Another reviewer's change accepted silently
Editing inside a pending insertion, the agent removed the insertion's wrapper, so the other reviewer's proposed sentence now reads as agreed text.
OpenCode with a local Qwen3.8 27B, 1 of 30
What a checker can see
ooxml-integrity 0.4.6, the published version the protocol froze, compares an edited file with its source. It reported every comment anchor and footnote reference python-docx lost and the malformed comments part, and missed what it did not model: the escaped bullets, the misattributed words, the title edited instead of the header. 0.4.7 adds warnings for the first two.
Most of its findings on correct edits were the requested change itself: accepting a revision lowers the revision count, an untracked header edit changes the header. 0.4.8 lets a caller declare such a change: a matching finding is then expected, and an expectation nothing matches is an error, because the requested change did not happen. With expectations taken from each task's declaration, flagged correct outputs fell from 130 to 12 of 537, each of those left a true report outside the oracle's scope, and caught damaged outputs rose from 59 to 63 of 69: the title edited instead of the header now shows.
A correct tracked edit loses nothing, so it draws no error to expect. 0.4.9 lists the tracked changes an edit added, per part of the document and author (FID012), and an expectation can require one in the header. Expecting that for every tracked task as well, caught damaged outputs rise to 65 of 69 and flagged correct outputs stay at 12: a tool that made its tracked edit in the document title instead of the header now fails too. Analysis, run with the published 0.4.8 and 0.4.9.
0.5.1 also reports a tracked replacement that deletes and inserts again more unchanged words than it changes (FID013): docx-cli edits a footnote by striking its whole text and inserting it again with one phrase changed, which nothing reported before because nothing is lost. With it, caught damaged outputs reach 67 of 69, and flagged correct outputs stay at 12. The two left are one tool's untracked edit of the wrong words inside another reviewer's pending insertion, in both captures.
0.7.0 reports a part declared with a content type its relationship does not allow (PKG010): the five Codex outputs Word refuses. They were already flagged for their unanchored reply, so the counts stay. Its fix command repairs both, and Word opens the copy with the reply in its thread.
0.8.0 lists the paragraphs an edit changed, with their text before and after (FID014), so a caller can expect the text the edit was asked to produce. Expecting each replacement's text, caught damaged outputs reach 69 of 69 and flagged correct outputs stay at 12: the untracked edit of the wrong words inside another reviewer's insertion, which no loss rule reports, now fails. A recheck of docx-cli 0.27.0 shows its fixes removed the escaped bullets and the misattributed tracked edit; that untracked edit is unchanged.
How it was run
- The protocol and its 30 task declarations were frozen, with their hashes, before the first capture; later changes are dated amendments, made before the captures they affect.
- The first source is a synthetic contract; the second is a review record written by Word for Mac: two reviewers' comments, a threaded reply and tracked changes, including a deletion inside another reviewer's insertion.
- Each third-party document tool ran in its own container with no network, python-docx in the development environment. Each agent ran in a container whose only route out led to its model, with web tools off and no retries. Document tools were captured twice, agents five times, OpenCode once.
- The oracle reads both files into review facts and never imports the checker; a reference control that edits the XML the way Word does passes all 60 attempts.
- All captures, transcripts, receipts and the evaluation are in the evidence folder.
Limits: two documents, one version of each tool and model, and counts of attempts rather than independent documents. Only one output per kind of damage was opened, in one build of Word for Mac.