PDFplumber extract_tables() recognizes two cells when 1 cell contains multiple lines

[bloktom]

I use the extract_tables() function on pages that i extract with PDFplumber. However the tables I use sometimes contain multiple lines of text. I guess it recognizes an hard enter. I tried using different sizes for the rectangles of the cells but nothing worked. Is there an easy fix for this issue?

[Ald392]

I am also facing the same problem. In my case it not only splits the single row to multiple but the column with the problem itself gets splitted. And it's only occurring for the first row. I tried recreating the issue in other rows but this was not occuring

This is the pdf i am using (Its just a random pdf i created for testing pdf plumber) : PATIENTRECORD12.pdf
The following is the script i used

import pdfplumber
import pandas as pd
import re
import io

def extract_pdf_content(pdf_path):
    content = ""
    
    with pdfplumber.open(pdf_path) as pdf:
        for count, page in enumerate(pdf.pages, start=1):
            # Extract text
            text = page.extract_text(x_tolerance=3, x_tolerance_ratio=None, y_tolerance=3, layout=True, x_density=7.25, y_density=13, line_dir_render=None, char_dir_render=None)
            if text:
                content += text+"\n"
    
    return re.sub(r'\n\s*\n+', '\n\n', content)

def clean_special_characters(df):
    cleaned_df = df.copy()

    def clean_string(x):
        if isinstance(x, str):
            # Replace all special characters with a space
            # This includes: punctuation, symbols, line breaks, tabs, multiple spaces
            cleaned = re.sub(r'\n|\t|\r', ' ', str(x))
            # Replace multiple spaces with a single space
            cleaned = re.sub(r'\s+', ' ', cleaned)
            return cleaned.strip()
        return x
    
    cleaned_df = cleaned_df.applymap(clean_string)
    
    return cleaned_df

def extract_tables(filename):
    with pdfplumber.open(filename) as pdf:
        tables = []
        for page in pdf.pages:  
            tables.extend(page.extract_tables())

    # Save each table to a unique CSV file
    for j, table in enumerate(tables):
        df = pd.DataFrame(table)  # Convert the extracted table to a DataFrame
        print(df)
        df = clean_special_characters(df)
        csv_from_table = io.StringIO()
        df.to_csv(csv_from_table, index=False)
        csv_from_table = csv_from_table.getvalue()
        print("\n\n\n")
        print(csv_from_table)

pdf_path = ""
print(extract_pdf_content(pdf_path))

extract_tables(pdf_path)

And this is the output i got for the first table

• Dataframe

• CSV

@jsvine Can you please look into this.

[bloktom]

@Ald392 This perfectly captures the challenges I'm facing too. However I was not aware this only counts for the first row. Did you try using the same data from the first row in other rows? Maybe the tool identifies an header of each table which is handeled differently. I will let you know if I know more.

[jsvine]

@Ald392 Thank you for providing the PDF, which is necessary for diagnosing such issues. In this case:

import pdfplumber
pdf = pdfplumber.open("path/to/PATIENTRECORD12.pdf")
page = pdf.pages[0]
im = page.to_image()
im.debug_tablefinder()

... reveals that there are extra, non-visible rects present in the PDF, interfering with the table extraction:

pd.DataFrame(page.rects)["non_stroking_color"].value_counts()

... suggests that the non_stroking_color attribute may be related to this:

non_stroking_color
(0, 0, 0)    206
(1, 1, 1)     27
Name: count, dtype: int64

... and this:

im.reset().draw_rects([ r for r in page.rects if r["non_stroking_color"] == (1, 1, 1)])

... suggest's that it's indeed the case:

That said, we want to keep the larger ones and get rid of the smaller ones, so let's set up a filtering function that accomplishes that:

def not_stray_rect(obj):
    if obj["object_type"] == "rect":
        return (
            (obj["non_stroking_color"] == (0, 0, 0))
            or (obj["height"] > 20)
        )
    return True

filtered = page.filter(not_stray_rect)

... and then test it:

filtered.to_image().debug_tablefinder()

... returns what we'd prefer:

... and this:

pd.DataFrame(filtered.extract_table())

... does too:

[Ald392]

Thanks @jsvine for the detailed analysis and solution.

[bloktom]

@jsvine Thanks alot, this also works for me! One exra tip to make it robust for different pdf documents, filter non_stroking_color on the most common color. This way it works for almost all PDF's:

def get_most_common_color(pages):
    colors = {}
    for page in pages:
        for rect in page.rects:
            color = rect["non_stroking_color"]
            colors[color] = colors.get(color, 0) + 1
    return max(colors, key=colors.get)

def not_stray_rect(obj, most_common_color):
    if obj["object_type"] == "rect":
        return (
            (obj["non_stroking_color"] == most_common_color)
            or (obj["height"] > 20)
        )
    return True

most_common_color = get_most_common_color(pages)
page = page.filter(lambda obj: not_stray_rect(obj, most_common_color))