PyMuPDF redaction/replacement behavior changes when pymupdf4llm is imported

Description

There appears to be an interaction between pymupdf and pymupdf4llm that affects text replacement using PyMuPDF redaction annotations.

The replacement code itself uses only pymupdf. It does not call pymupdf4llm.

The same PyMuPDF replacement operation behaves differently depending on whether pymupdf4llm has been imported in the Python process.

Versions

I installed latest versions and some other versions with same problem.

  • pymupdf v1.26.7 and v1.27.1
  • pymupdf4llm v0.2.9 and v0.3.4
  • pymupdf v1.26.7 and v1.27.1

Observed behavior

With only pymupdf imported:

  • The target text is found.
  • The redaction annotation is applied.
  • The replacement text is inserted correctly.

After importing pymupdf4llm:

  • The same replacement code is executed.
  • For the attached PDF, the original text is removed. input.pdf (11.6 KB)
  • The replacement text is NOT inserted.

This is particularly problematic because the operation does not raise an exception; it result in silent data loss.

NOTE: For PDFs that I created using save as PDF from Word, this didn’t happened. This PDF and many others I have are generated using a proprietary software in Neurology field and I am unaware of the back-end PDF engine they use. The PDF opens perfectly fine in windows, Not sure what is specific about this PDF files.

Minimal reproducer

The following script demonstrates the issue with the attached PDF.

import pymupdf


INPUT = r"input.pdf"

ORIGINAL_TEXT = "3/4/2017"
REPLACEMENT_TEXT = "3/4/2037"


def test(label: str) -> None:
    print()
    print("=" * 70)
    print(label)
    print("=" * 70)

    with pymupdf.open(INPUT) as doc:
        page = doc[0]

        for x0, y0, x1, y1, text, *_ in page.get_text("words"):
            if text != ORIGINAL_TEXT:
                continue

            print("WORD:   ", repr(text))
            print("RECT:   ", (x0, y0, x1, y1))
            print("HEIGHT: ", y1 - y0)

            rect = pymupdf.Rect(x0, y0, x1, y1)

            page.add_redact_annot(
                rect,
                text=REPLACEMENT_TEXT,
                fontname="helv",
                fontsize=8,
                fill=None,
                text_color=(0, 0, 0),
                cross_out=False,
            )

            page.apply_redactions()

            result = page.get_text()

            replacement_found = REPLACEMENT_TEXT in result
            original_found = ORIGINAL_TEXT in result

            print(
                f"  {'✅' if replacement_found else '❌'} "
                f"Replacement present: {replacement_found}"
            )
            print(
                f"  {'❌' if original_found else '✅'} "
                f"Original still present: {original_found}"
            )

            return


# ---------------------------------------------------------------------------
# 1. Baseline: PyMuPDF only
# ---------------------------------------------------------------------------

test("1. PyMuPDF only")


# ---------------------------------------------------------------------------
# 2. Import PyMuPDF4LLM and perform exactly the same operation
# ---------------------------------------------------------------------------

import pymupdf4llm

test("2. After importing PyMuPDF4LLM")

Welcome @EulerLeibniz !

The root of the problem is the silly handling of fonts in this PDF: The font file binary contains 0.86 and -0.14 as values for ascender, respectively descender. So the total built-in / natural height is 1 = 0.86 - (-0.14).

(I am assuming you are aware of what these values mean.)

This means that text written with a font size of 8.25 in fact also appears with a height of 8.25 points.

The PDF creator however decided to overwrite these two values here


The two integers are divided by 1000 and thus result in the effective values

  • ascender = 0.220
  • descender = -0.036

Therefore, text written with a font size of 8.25 will appear within a bbox of height 8.25 * (0.22 + 0.036) = 2.112 points.

PyMuPDF contains a number corrective tools to deal with these things. The one applying here by default is a function that recalculates the height such that it never is smaller than the font size.
For several reasons, PyMuPDF4LLM must disable this function.
This is the effect you are observing.

We are going to publish the next version of PyMuPDF which will contain PyMuPDF4LLM. As a result, the behavior will always show the mentioned corrective function disabled.

This also means that you will never be able to successfully insert replacement text in a redaction area of height 2.112 using a font size of 8!

If you read the documentation carefully, you will detect the warning that the replacement text is not guaranteed to appear.
It is always safer to just remove old text with a redaction and insert the desired new text in a separate step. This is the only way to have complete control over its appearance.

To activate the mentioned corrective function, always execute pymupdf.TOOLS.unset_quad_corrections(False) before extracting text.

But let me repeat:
For text replacement, you are using a font (Helvetica) with entirely different metrics than the original MS Gothic. While for the original the sum of ascender and descender is 1, that value for Helvetica is 1.374.
Therefore, text written with font size 8 and Helvetica will result in a line height of ~11, not 8!
Even when you get text replacement working, this sort of thing will look qite ugly.

1 Like

Thanks for the detailed explanation! So to summarize for myself:

  • The issue was caused by pymupdf4llm disabling PyMuPDF’s quad correction for reasons you deem necessary. I should not really change that.
  • Replacement text is not guaranteed to appear when using page.add_redact_annot()
  • Recommendation: Remove old text with a redaction and insert the desired new text in a separate step.

This works correctly for my case. Thanks again

1 Like

Thanks for understanding!