Apply_redactions() — link /URI values survive redaction, plus large output-size inflation

PyMuPDF
1

Hi, we’re testing PyMuPDF (1.28.0) for a document redaction tool (still evaluating) and found two separate issues in apply_redactions(). Posting a short summary, happy to describe more detail if useful.

  1. Link annotation /URI values aren’t scrubbed. If a sensitive value appears in page text and in a link’s /URI (common — Word/Acrobat/LaTeX auto-linkify emails and URLs), apply_redactions() removes it from the visible text but leaves the URI intact and recoverable via page.get_links(). Redaction reports success; the value is still in the file. Measured across 39 real documents: 26 leaked this way, 444 recoverable values total, 0 leaks on documents with no links.
  2. Default image mode inflates output size heavily. apply_redactions() defaults to images=PDF_REDACT_IMAGE_PIXELS, which re-encodes every image on any touched page — roughly +24MB per page touched in our testing, one document going from 24MB to 4.8GB. Switching to images=PDF_REDACT_IMAGE_REMOVE avoids it with no loss of redaction correctness, but it’s a rough default to hit unknowingly.
2

Looking into this …

3

The second point first:
If you accept the default image processing, then “holes” are punched into image regions that intersect a redaction. The underlying technique converts the respective image into a PNG (!) - which may quite likely be larger than the original format … just think of CCITTFaxDecode or JPX or even normal JPEG.

This is unavoidable!

Choose a different image handling: either not touching images at all or removing images entirely when they intersect a redaction.

For the first problem we need an example PDF exhibiting the problem. Just to be sure, please repeat the text with the most recent PyMuPDF version.

4

Out of curiosity: Which save options did you use for the redacted PDFs?

5

Thanks for the quick response on the image handling, that makes sense as a mechanism, and switching to remove-on-intersect is exactly what we’re doing on our side.

Save options: plain doc.tobytes(), no garbage= or clean= options passed for either the marked or redacted output.

Example PDF: rather than sharing one of our real corpus documents, here’s a minimal synthetic snippet that builds the problem PDF in-memory should reproduce standalone:

import pymupdf

TARGET = "j.doe@example.com"

doc = pymupdf.open()
page = doc.new_page(width=400, height=300)
page.insert_text((50, 100), f"Contact: {TARGET}", fontsize=12)
page.insert_text((50, 200), "Unrelated line of text", fontsize=12)
page.insert_link({
    "kind": pymupdf.LINK_URI,
    "from": pymupdf.Rect(50, 190, 250, 210),   # over the unrelated line, not the text
    "uri": f"mailto:{TARGET}",
})
buf = doc.tobytes()
doc.close()

doc = pymupdf.open("pdf", buf)
page = doc[0]
for r in page.search_for(TARGET):
    page.add_redact_annot(r, fill=(0, 0, 0))
page.apply_redactions()
out = doc.tobytes()
doc.close()

d = pymupdf.open("pdf", out)
p = d[0]
print("visible text has target:", TARGET in p.get_text())
print("links remaining:", len(p.get_links()))
print("recoverable from URI:", any(TARGET in (L.get("uri") or "") for L in p.get_links()))

On 1.28.0 this prints visible text has target: False, links remaining: 1, recoverable from URI: True, the redaction removes the text but the link survives with the value still in its URI.

We’ll also confirm against the latest release and report back.

6

Well, not specifying garbage nor deflate contradicts your announced intention of data protection:
Without garbage collection, old content will be present in the physical PDF file as “ghost” objects, thus invalidating your voiced intentions to protect data. Every PDF expert will be able to extract this ghost information. ALWAYS SPECIFY GARBAGE > 0 after redacting! In addition, your code will not only be inefficient in terms of data protection, but also in terms of bloating the file size.
Not requesting compression (deflate=True at a minimum) will in addition negate any chance to achieve acceptable file sizes.

Therefore, always use something like doc.ez_save(...) if you used redactions.
Only then we have a worthwhile basis for discussing remaining file size issues.

7

@Jaymish_Patel - I am still waiting for a reproducing PDF where hyperlinks fail to be removed under redactions.

Using your suggested script …!

8

@Jaymish_Patel
The riddle of the non-redacted hyperlink has a simple solution:
You specified the wrong redaction rectangle! I modified your script somewhat to make the link “from” rectangle visible:

import pymupdf

TARGET = "j.doe@example.com"
link_rect = pymupdf.Rect(50, 190, 250, 210)  # this will never be redacted!!!

doc = pymupdf.open()
page = doc.new_page(width=400, height=300)
page.insert_text((50, 100), f"Contact: {TARGET}", fontsize=12)
page.insert_text((50, 200), "Unrelated line of text", fontsize=12)
page.insert_link(
    {
        "kind": pymupdf.LINK_URI,
        "from": link_rect,  # over the unrelated line, not the text
        "uri": f"mailto:{TARGET}",
    }
)
page.draw_rect(link_rect, color=(1, 0, 0))
doc.save("test0.pdf")

doc = pymupdf.open("test0.pdf")
page = doc[0]
for r in page.search_for(TARGET):  # this is NOT where the link lives!
    page.add_redact_annot(r, fill=(0, 0, 0))
page.apply_redactions()
...

Here is how your PDF looks like:

image
image672×516 8 KB

The hyperlink rectangle is not redacted and thus survives.
You must redact a hyperlink’s “from” rectangle to remove a link.