Layout mode silently drops the text of blocks classified as "formula" (to_markdown / to_text)

PyMuPDF
1

Environment

  • pymupdf4llm 1.28.0, PyMuPDF 1.28.0, pymupdf-layout 1.28.0
  • Python 3.11, Linux container on arm64
  • Also checked against the 1.28.2 wheels: same behaviour

Summary

In layout mode, blocks classified as formula lose their text entirely in the output. Blocks classified as picture keep their text, but formula blocks do not, even with force_text=True (the default). When write_images and embed_images are both False (the defaults), no image is produced either, so the block contributes nothing at all.

Combined with a classification issue, this silently removed 50 paragraphs of body text from one of our documents.

What we observe

Our documents are Korean legal texts (statutes and insurance policy terms). The layout analyzer classifies the “supplementary provisions” sections as formula.

import pymupdf, pymupdf4llm

doc = pymupdf.open("statute.pdf")
page = doc[58]
page.get_layout(return_raw=True)

for b in page.layout_information:
    if b["class_name"] == "formula":
        print(repr(page.get_textbox(pymupdf.Rect(b["group_bbox"])).strip()))

Output:

'부   칙  <2010. 5. 7.>'
'제1조(시행일) 이 규정은 2010년 5월 7일부터 시행한다. 부  칙 <2015. 7. 16.>'
'제1조(시행일) 이 규정은 세종특별자치시가 조합에 가입한 날부터 효력을 발생한다. ...'
'제1조(시행일) 이 규정은 조합회의 의결이 있는 날부터 시행한다.'

This is ordinary body text. It states the effective dates of the regulation. None of it appears in the result of pymupdf4llm.to_markdown(doc).

Where it happens

In pymupdf4llm/helpers/document_layout.py, the markdown emitter (1.28.0 around line 814, 1.28.2 around line 1007):

if btype in ("picture", "formula"):
    ...                                   # write or embed the image
    # output text in image if requested
    if box.textlines:
        if btype == "picture":            # formula is not included here
            md_string += picture_text_to_md(...)
    string_lengths.append(len(md_string))
    continue

box.textlines is populated for formula blocks as well, but it is only emitted fo path has the same shape (1.28.2 around line 1138).

Since we do not write or embed images, the branch above appends only "\n\n" and tho warning or log entry.

Impact

In a 179 page document with 138 text pages, 50 paragraphs were missing from the marked because we compare the output against the raw text layer. In legal and insurancedocuments, the affected sections carry effective dates and transitional rules, so losing them is not acceptable for us.

For comparison, running the same document in legacy mode (use_layout(False)) does not lose these paragraphs.

Questions

  1. Is it intentional that formula blocks do not emit their text while picture blocks do? If so, what is the reasoning, given that force_text defaults to True?
  2. Is there any option to have the text of formula blocks included in the output?
  3. Is it known that this kind of layout, a short heading line followed by one or two short provisions, tends to be classified as formula? We see it consistently across several Korean statute documents.

We are happy to provide a small sample PDF that reproduces this if that helps.

2

Hi @lsj6924 Welcome to the forum - yes please , if you could please share a PDF that would be perfect and then we can investigate further! CC’ing @HaraldLieder

1 Like
3

As an initial reaction from a maintainer:
Yes, we intentionally handle “formula” regions differently than “picture” ones.
This was based on our observation that real formulas reliably generate nonsense output under plain text extraction (which is effectively what happens for pictures and force_text=True).

We may have to re-consider this, so an example file would indeed be valuable.

1 Like
4

Signature blocks in legal documents are classified formula, so their text is dropped

First, thank you for pymupdf4llm, and for keeping it free. We run several thousand
legal determinations and court filings through it, and the layout work in
particular does something we would have no realistic way of doing ourselves —
get_text() gets us characters, and this gets us a document. That is exactly why
I’d like to see this case handled.

Adding a second example to the formula behaviour discussed above, since Harald
mentioned being open to reconsidering given example files. Here are two minimal
ones where the misclassified content is unambiguously prose rather than anything
mathematical.

Versions: pymupdf4llm 1.28.2, PyMuPDF 1.28.2, Windows, Python 3.12.

What happens

These are Pennsylvania Office of Open Records final determinations. Each ends with
the adjudicator’s signature block:

Issued by:

/s/ Joshua Young
__________________
Deputy Chief Counsel
Joshua Young, Esq.

The layout model classifies that block as boxclass='formula', and its text never
reaches the output.

import sys, json, pymupdf, pymupdf4llm
from pymupdf4llm.ocr import OCRMode

for path in sys.argv[1:]:
    print(f"\n=== {path} ===")
    page = pymupdf.open(path)[0]

    # 1. The text is in the PDF and trivially extractable.
    box = None
    for b in json.loads(pymupdf4llm._layout_to_json(path, use_ocr=OCRMode.NEVER))["pages"][0]["boxes"]:
        rect = pymupdf.Rect(b["x0"], b["y0"], b["x1"], b["y1"])
        if "/s/" in page.get_textbox(rect):
            box = b
            break

    rect = pymupdf.Rect(box["x0"], box["y0"], box["x1"], box["y1"])
    print(f"  boxclass          : {box['boxclass']!r}")
    print(f"  y position        : {box['y0']:.0f} of {page.rect.height:.0f}"
          f"  ({box['y0']/page.rect.height*100:.0f}% down the page)")
    print(f"  textlines         : {box.get('textlines')!r}")
    print(f"  page.get_textbox(): {page.get_textbox(rect).strip()[:70]!r}")

    # 2. to_markdown omits it, with force_text=True and regardless of OCR mode.
    for ft in (True, False):
        md = "".join(c["text"] for c in pymupdf4llm.to_markdown(
            path, page_chunks=True, use_ocr=OCRMode.NEVER, force_text=ft))
        print(f"  to_markdown(force_text={ft!s:<5}) contains '/s/': {'/s/' in md}")

Output on the two attached files:

=== 20230013.pdf ===
  boxclass          : 'formula'
  y position        : 487 of 792  (61% down the page)
  textlines         : None
  page.get_textbox(): '/s/ Joshua Young \n__________________ \nDeputy Chief Counsel \nJoshua You'
  to_markdown(force_text=True ) contains '/s/': False
  to_markdown(force_text=False) contains '/s/': False

=== 20232162.pdf ===
  boxclass          : 'formula'
  y position        : 488 of 792  (62% down the page)
  textlines         : None
  page.get_textbox(): '/s/ Joshua Young \n______________'
  to_markdown(force_text=True ) contains '/s/': False
  to_markdown(force_text=False) contains '/s/': False

Three things I think make this worth a second look:

  1. The content is plain prose — a person’s name and job title. Whatever heuristic
    is firing, it isn’t seeing mathematics. My guess is the run of underscores that
    forms the signature rule, possibly combined with the name being set in italic
    (TimesNewRomanPS-ItalicMT), and the block being visually isolated from the
    surrounding paragraphs.

  2. It is not a footer or page furniture. The block sits at 61% of page height,
    mid-page, well above the actual page-footer box which the model identifies
    correctly and separately on the same page.

  3. 20232162.pdf shows it splitting a single logical block. There, only the
    /s/ Joshua Young line is classified formula; the underscore rule and the
    title immediately below it are classified text and survive. The output keeps
    Issued by: and Deputy Chief Counsel Joshua Young, Esq. but loses the
    signature line between them — so the result reads as complete while the
    signature is gone.

Why it is hard to notice

textlines is None for these boxes, so nothing downstream can tell that text was
present and discarded. force_text=True does not help, because the gate is earlier
than the emit step — in helpers/document_layout.py, textlines is only populated
when boxclass == "picture" (~line 1569), so formula boxes never carry text into
the markdown/text writers at ~lines 1007 and 1138.

There is also no warning or log line, and the surrounding text is unaffected, so the
document looks well-converted. In our corpus this silently removes the name of the
adjudicating officer from roughly 15% of determinations on the current template —
information we use downstream, and whose absence is not otherwise detectable.

Suggestions, in the order I’d prefer them

  1. Treat formula like picture for text purposes when force_text=True.
    Locally, changing the two gates to in ("picture", "formula") — populating
    textlines, and emitting them — recovers all three of our affected files with no
    other visible change. If formula text really is nonsense for genuine formulas,
    force_text=True seems like the right place for the caller to say “I’d rather
    have imperfect text than none”.

  2. Or make it opt-in/opt-out, e.g. a parameter naming which box classes may have
    their text dropped, so callers whose documents contain no mathematics can exclude
    formula.

  3. At minimum, make it visible. Populating textlines even when the text isn’t
    emitted, or emitting a warning when a formula box contains extractable text,
    would let callers detect the loss. Silence is the part that makes this expensive.

Happy to supply more samples — we have a few thousand documents on this template,
and I’m glad to test any patch against them and report back. Thanks again for the
work on this.

Attached

  • 20230013.pdf — whole signature block classified formula, entirely dropped
  • 20232162.pdf — only the /s/ line classified formula; the block splits and
    the surrounding lines survive, so the loss is much less obvious

Both are public records, published by the Pennsylvania Office of Open Records.

20230013.pdf (47.8 KB)

20232162.pdf (47.4 KB)

5

As announced, I experimented a bit and handled “formula” boxes just like “picture” ones.

  • Your first example 20230013.pdf works correctly anyway in 4LLM v. 1.28.2
  • The second example 20232162.pdf also works with this fix and delivers the is MD:
    20232162.md (2.2 KB)

So I will add this fix to the next version.

1 Like
6

Thanks for looking into this. Since the original PDF is a public-sector document, I created a sanitized reproduction by replacing the institution names, dates, article numbers, and content with fictional values while preserving the original layout, spacing, fonts, and PDF text structure.

I confirmed that the sanitized PDF reproduces the same issue: with Layout mode enabled, the supplementary provisions are classified as formula blocks and all of their text is omitted from both to_markdown() and to_text(). The text is still present in page.get_text() and legacy Markdown extraction.

I’ve attached the sanitized reproduction PDF.

Also, if you’re able to share, is this something that could potentially be addressed in an upcoming release, or is it still too early to say? No urgency I’m mainly curious about how this issue might be handled.

formula_text_drop_lsj6924.pdf (613.6 KB)