Automating PDF-related Structural Testing

Please note that this tutorial assumes that you have at least a rudimentary knowledge of Windows PowerShell scripting. If not, there are plenty of tutorials on the web to get you started. No other prerequisites will be required to follow along. One more thing to note is that although PowerShell enables a programmer to write a compact script through the use of aliased commands, I will be using the longer form of many commands to make the script a little easier to understand for the beginner.

Introduction

One of the frequent challenges especially in document-centric organizations where production of documents is often highly automated is to ensure that these documents are correct both structurally and semantically before they are transmitted both for internal and external consumption by other systems or users. Unlike XML which uses text format, PDF uses a binary format which stores both layout as well as content within the same document. Automated verification of the content within a PDF is relatively easy as you can leverage any of the number of PDF parsers available on the market. Structural verification during regression testing on the other hand is a different story altogether. Ensuring the document has not changed in appearance (fonts, labeling, table border widths, etc.) when any new functionality is added or when defects in a system are addressed is of utmost importance to business users especially in the client facing side. Over the years, both through practice and observation, I have discovered and implemented several interesting ways to overcome these challenges and wanted to share one such idea in this article. I will show how to set up a simple but robust framework for testing your PDF documents in a matter of minutes using several freely available tools on the Internet.

Absolute Minimum You Need to Know about PDF

Almost everyone who uses the Internet has heard about or used a PDF document. PDF stands for “Portable Document Format”, and was invented by Adobe, a large software technology company that has pioneered other interesting technologies such as PostScript, Flash, etc. A PDF document is a self-contained document which can be opened on almost any computer or mobile device using a PDF “reader” software specific to that platform. Adobe releases its own reader software called “Adobe Reader”, but you can use hundreds if not thousands of PDF readers produced by third party manufacturers around the world. Adobe’s original goal in releasing this technology was for the document to look the same on the screen when viewed or when printed regardless of the kind of computer or printer being used. Before the PDF standard was invented, it was nearly impossible to distribute a document to others on the Internet since most document standards available then only permitted them to be viewed on specific operating systems. PDF changed all that. Everything including formatting information such as fonts, styles, images, video, audio and text can all be embedded in a highly compressed form allowing for consistent rendering as well as distribution of the PDF document much more easily.

“If you wish to make an apple pie from scratch, you must first invent the universe.” ~ Carl Sagan

Over the last two decades, the PDF standard has evolved from its original form to support the rich functionality such as encryption, digital signatures, interactive forms, and rich media support including audio and video information. Since 2008, the PDF standard is no longer a proprietary standard. This is because Adobe released it as an open standard to make it much more accessible for collaboration and innovation. For the purposes of this tutorial, this is pretty much you need to know about PDF. I will illustrate how you can transform a PDF document into a series of images allowing you to take advantage of some extremely powerful image manipulation libraries which should permit us to implement extremely accurate testing of the structural aspects of a PDF document. However, if you want to dive deeper into how the PDF document is structured internally, please refer to this excellent article here.

Regression Testing

Before we can proceed and see how to write these tests let us take a quick minute to review what regression testing really means. A common problem and scenario that exists with any software implementation (especially ones that don’t have a well written suite of automated tests) is that something is unintentionally modified because of making another change in the software. Regression testing is a type of testing done to ensure that these unintended changes are caught and addressed before the software is released for use by the end users. Several challenges exist when doing regression testing manually. The main ones are, 1) it is extremely time consuming and tedious 2) it is inconsistent, meaning that the same test or tests may be done differently depending on who is testing it leading to different interpretations, and 3) it is subject to human frailty, by which I mean that when you are doing the same thing over and over, you lose focus and things simply slip your attention. This is where test automation helps. By making the tests scripted to run automatically, they can be run often, run the same way every time, and they catch things that would slip through even the most diligent tester.

Tools - PowerShell, PDFTK and ImageMagick Library and GhostScript

A note from the future (September 2026): This article was written in 2010 and the tooling landscape has moved on considerably since then, although the overall approach (render the pages, compare the pixels, report the differences) is as valid as ever. I have kept the original tool chain in the article, but if you are building something new today, please consider the following before you copy this set-up verbatim.

  • PDFtk is no longer needed. The “burst” step exists only because ImageMagick handled multi-page PDFs poorly at the time. PDFtk Server has not seen a release since 2013. Modern renderers accept a multi-page PDF directly, so Steps 1 and 2 collapse into a single command.
  • Consider a renderer that does not depend on GhostScript. ImageMagick delegates PDF rendering to GhostScript, and since the 2018 GhostScript security advisories most ImageMagick installations ship with PDF handling disabled in policy.xml, which trips up almost everyone following this tutorial. MuPDF’s mutool (a single executable: mutool draw -r 150 -o page_%04d.png input.pdf) or Poppler’s pdftoppm render a PDF straight to images with no GhostScript at all, and both run on Windows, Linux and macOS.
  • ImageMagick is still a fine choice for the comparison and stitching steps. Use version 7, where the tools are invoked as magick compare and magick convert (this also avoids the clash with Windows’ own convert.exe). Its -metric AE and -fuzz options give you a numeric count of differing pixels and a tolerance for anti-aliasing “noise”, which turns the visual report into an automated pass/fail.
  • Wrap the checks in a test framework. With PowerShell 7 and Pester each PDF becomes a test case that fails when the differing pixel count exceeds a threshold, which makes the whole thing straightforward to run in a CI pipeline such as GitHub Actions.
  • If you would rather not script it at all, diff-pdf compares two PDFs in one command and produces a highlighted difference PDF, and if you are comfortable with Python, PyMuPDF can render, compare and stitch in a single library with no external tools to install.

The approach we will look at in the article for testing PDFs for structural changes will be to convert the pages in the document to a series of images, and then use an image comparison program to detect the differences between a set of images from before and after. This is where PDFtk comes in. This is an cross-platform open source software that allows you to manipulate PDF documents. This tool has several other capabilities including splitting, merging, encrypting, decrypting, compressing, uncompressing, and repairing PDF documents although we will focus primarily on its ability to convert a PDF document into a series of individual PDF pages for the purposes of this tutorial. This tool is fairly straight forward to use and can run on a number of operating systems.

In addition to PDFtk, we also need another piece of software called ImageMagick. This is needed to convert the series of individual PDF pages (generated from the main document using PDFtk) into a series of images which can then be used for a number of image processing operations to follow including image comparison, overlays and stitching. ImageMagick is an open source image processing library that has proven track record of over 15 years or so. The documentation is well written, and there are plenty of examples to use or emulate. There are a number of published books on this tool as well. ImageMagick requires the use of another piece of software called GhostScript for PDF and PostScript-related processing. Something I want to mention here is that ImageMagick provides direct support for converting a multipage PDF document into a series of individual images. However, I find PDFtk much more powerful in manipulating PDF pages. I will leave it to you to decide the route to follow when you build your production-worthy application.

Let us now proceed to seeing how these tools can be “stitched together” to help build an automated regression testing framework to enable quality assurance activities around structural aspects for PDF (and PDF-exportable) documents. However, download the following software and install them on your machine before you proceed.

  1. Download and install PDFtk from here

  2. Download and install ImageMagick from here

  3. Download and Install GhostScript from here. Ensure that you GhostScript is included in your environment path variable

  4. Ensure you have PowerShell enabled on your operating system

  5. Download the full source code used in this tutorial from GitHub here.

Common Setup: Configuration and a Helper for Running External Tools

Every step in this tutorial boils down to running an external program (PDFtk or one of the ImageMagick tools) with a handful of arguments. Rather than assembling a command line as a string and running it through Invoke-Expression (which is fragile whenever a path contains a space or a quote, and which is a well-known security anti-pattern), we will invoke the programs directly using PowerShell’s call operator (&) and pass the arguments as an array. PowerShell takes care of quoting each argument for us. Another thing to keep in mind is that external programs do not throw PowerShell exceptions when they fail; they simply return a non-zero exit code. A small helper function called Invoke-ExternalTool shown below takes care of checking the exit code and turning a failure into a terminating error so that the rest of the script can rely on normal try/catch handling. Progress messages are written using Write-Verbose rather than Write-Host so that they can be switched on when you want to follow along (set $VerbosePreference to Continue before running the script, or call the functions with -Verbose) and stay silent otherwise. The settings block at the top holds the location of the tools and the input and output folders. Please note that I am ignoring some best practices such as passing program settings using a configuration file for the purposes of keeping this tutorial short and focused.

#Requires -Version 5.1
Set-StrictMode -Version Latest
$ErrorActionPreference = 'Stop'

# Location of the external tools. Use full paths: Windows ships its own "convert.exe"
# (a file system utility) which would otherwise be picked up instead of ImageMagick's.
# Note: ImageMagick 7 replaces convert.exe/compare.exe with a single "magick.exe";
# in that case point both variables at magick.exe and prefix the arguments with
# "convert" or "compare" respectively.
$Script:ImageMagickConvert = 'C:\PdfDiffWork\ImageMagick\convert.exe'
$Script:ImageMagickCompare = 'C:\PdfDiffWork\ImageMagick\compare.exe'
$Script:Pdftk              = 'C:\PdfDiffWork\PDFtkServer\bin\pdftk.exe'

# Location of the first ("reference") set of PDF files
$Script:InputFolder1 = 'C:\PdfDiffWork\input1'
# Location of the second ("candidate") set of PDF files
$Script:InputFolder2 = 'C:\PdfDiffWork\input2'
# Location where the results of the comparison operation will be archived
$Script:OutputFolder = 'C:\PdfDiffWork\output'

# Runs an external program and fails loudly if it returns an unexpected exit code
function Invoke-ExternalTool
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $FilePath,

        [Parameter(Mandatory)]
        [string[]] $ArgumentList,

        # Exit codes that should be treated as success.
        # ImageMagick's compare, for example, returns 1 when the images differ.
        [int[]] $SuccessExitCode = @(0)
    )

    Write-Verbose "Running: $FilePath $($ArgumentList -join ' ')"

    # The call operator (&) runs the program; splatting (@) passes each array element as a separate argument
    $output = & $FilePath @ArgumentList
    $output | ForEach-Object { Write-Verbose $_ }

    if ($SuccessExitCode -notcontains $LASTEXITCODE)
    {
        throw "'$FilePath' failed with exit code $LASTEXITCODE. Arguments: $($ArgumentList -join ' ')"
    }
}

Step 1: Convert PDF documents into a series of individual PDF pages

In this step, we are going to “burst” the two PDF documents to be compared into a series of single page PDF documents. This will enable us to subsequently convert these PDF pages into images and then compare the image equivalents of these pages using the ImageMagick image processing library. The PDFtk command that does this operation is called “burst”. You have the option of leaving PDFtk to provide default filenames or take control of how the individual files are named. The default naming format generates files with names such as pg_0001.pdf, pg_0002.pdf, etc. We will supply our own, very similar, pattern (page_0001.pdf, page_0002.pdf, etc.) so that the page files sort correctly in a directory listing. The PowerShell function that enables this is shown below. Notice that the function takes everything it needs as parameters and uses Join-Path to build file paths instead of concatenating strings. Once the two multi-page PDF documents have been converted to a series of individual PDF pages, they are used as inputs for the next step in the automation which is to convert them into images. Running this script will result in creation of a folder with the name of the document and will create two folders underneath the main output folder showing the main PDF document and the document being compared against broken into individual PDF pages. The screenshot showing the results of the operation is shown below this step’s code listing as well. The full source code, including this function, is available on GitHub.

# This function bursts open a PDF file into individual single-page PDF files
function Split-PdfIntoPages
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $InputFile,

        [Parameter(Mandatory)]
        [string] $DestinationFolder
    )

    # -Force makes New-Item succeed quietly when the folder already exists
    New-Item -Path $DestinationFolder -ItemType Directory -Force | Out-Null

    # PDFtk names each page using the printf-style pattern supplied after "output"
    $pagePattern = Join-Path -Path $DestinationFolder -ChildPath 'page_%04d.pdf'

    Invoke-ExternalTool -FilePath $Script:Pdftk -ArgumentList $InputFile, 'burst', 'output', $pagePattern

    Write-Verbose "The PDF file '$InputFile' was successfully burst open into '$DestinationFolder'"
}

Screenshot showing the output of operation is seen below:

PDF Busrt Operation Screenshot

Step 2: Convert PDF pages into Images

The next step is to convert the individual PDF pages into images so that we can then use an image compare program to compare the pair of pages to be checked for differences. Both these steps can be easily accomplished using the ImageMagick image processing library. This is accomplished by the “convert” program that is included with the library. This program takes the horizontal and vertical density of the output image desired, the input PDF document and the name of the output image file desired. A small PowerShell function to accomplish this is shown below. The convert program takes numerous other parameters (nearly 100 arguments) and although many of them are very useful, I won’t cover them here as they are not required for the purposes of this tutorial. Notice that the function returns the path of the image it created so that the calling code can simply hand that value to the next step. You can see how these functions are wired together by the main routine in the full source code on GitHub. The screenshot showing the results of the operation are shown below this step’s code listing as well. You should now see some JPEG format files that were created as the result of conversion of each of the individual PDF pages within both sets of documents being compared.

# This function converts an individual PDF page file into a JPEG image to enable image comparison operations
function Convert-PdfPageToJpeg
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $PdfFile,

        # Horizontal x vertical resolution (dots per inch) used to render the page
        [string] $Density = '400x300'
    )

    # The image name will be the PDF file name with ".jpg" appended to the end
    $jpegFile = "$PdfFile.jpg"

    Invoke-ExternalTool -FilePath $Script:ImageMagickConvert -ArgumentList '-quiet', '-density', $Density, $PdfFile, $jpegFile

    if (-not (Test-Path -Path $jpegFile))
    {
        throw "Conversion of PDF file '$PdfFile' to JPEG format did not produce '$jpegFile'"
    }

    Write-Verbose "Conversion of PDF file '$PdfFile' to JPEG format was successful"
    return $jpegFile
}

Screenshot showing the output of operation is seen below:

PDF conversion to JPEG Images Screenshot

Step 3: Compare Images/Generate Difference Image

Now that the PDF pages have been converted to JPG images, you can easily compare them using the “compare” program included as part of the ImageMagick library. This program takes the two images to be compared and the output filename as arguments and generates a “difference image” with any image pixels that are different shown in red colour. The pixels that are not different between the two images appear white. The only caveat to this whole operation is that the images that are being compared should be of the same size. The output format of the difference image is determined by the filename extension specified for the output filename. One subtle point worth knowing is that the compare program uses its exit code to report the result: it returns 0 when the images are identical, 1 when they differ, and 2 when something went wrong. Because we want the difference image regardless of whether the pages match, the function below tells our helper that both 0 and 1 are acceptable. The PowerShell function that helps compare two images and generates an output difference image is shown below. Running the modified script will result in the creation of a “diff” folder underneath the output folder created for the document and will contain “difference” images generated by the ImageMagick compare program. A difference image generated due to the variation within the first pages of both PDF documents being compared against each other is shown below the code listing as well.

# This function compares two image files and writes the differences to another image file
function Compare-ImageFile
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $ReferenceImage,

        [Parameter(Mandatory)]
        [string] $CandidateImage,

        [Parameter(Mandatory)]
        [string] $DiffImage
    )

    # compare exits with 0 when the images are identical and 1 when they differ; only 2 means an error
    Invoke-ExternalTool -FilePath $Script:ImageMagickCompare -ArgumentList $ReferenceImage, $CandidateImage, $DiffImage -SuccessExitCode 0, 1

    if (-not (Test-Path -Path $DiffImage))
    {
        throw "Comparison of '$ReferenceImage' and '$CandidateImage' did not produce '$DiffImage'"
    }

    Write-Verbose "Difference operation was successful. Difference image '$DiffImage' was created"
    return $DiffImage
}

“Don’t let the fear that testing can’t catch all bugs stop you from writing the tests that will catch most bugs” ~ Martin Fowler

Step 4: Stitch Original, New and Difference Images Together

To enable easy visualization of the differences between the original and the new PDF being compared, the three images namely, the original PDF image, the new PDF image and the difference image generated from the compare operation can be stitched together. This is achieved using the “convert” application which takes the three images and the stitched image name as arguments. The “+append” option of the convert program is specified when this operation is performed. The code needed in the main routine to invoke the function below is included in the full source code on GitHub. Running this modified code will result in the creation of a “quickview” folder which contains the results of the stitch operation. One of the stitched images as well as the folder view of the results are included below the code listing as well. The full source code, including this function, is available on GitHub.

# This function stitches the two page images compared and the diff image generated into a single three-way image
function New-ThreeWayImage
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $ReferenceImage,

        [Parameter(Mandatory)]
        [string] $CandidateImage,

        [Parameter(Mandatory)]
        [string] $DiffImage,

        [Parameter(Mandatory)]
        [string] $OutputFile
    )

    # "+append" places the images side by side (left to right)
    Invoke-ExternalTool -FilePath $Script:ImageMagickConvert -ArgumentList $ReferenceImage, $CandidateImage, $DiffImage, '+append', $OutputFile

    if (-not (Test-Path -Path $OutputFile))
    {
        throw "Stitching '$ReferenceImage', '$CandidateImage' and '$DiffImage' did not produce '$OutputFile'"
    }

    Write-Verbose "Stitch operation was successful. Stitched image '$OutputFile' was created"
    return $OutputFile
}

Screenshot showing the output of operation is seen below:

Image Side by Side Stitch Operation Screenshot

Step 5: Create Complete Diff Report

The last step is to stitch the three-way diff view images together into a complete report for each set of PDF documents considered. In this step, the previously stitched three-way images are then stitched one more time (this time vertically) to create a full listing of all pages compared as well as the differences if any. This is achieved by using the convert program used in the previous step as well, this time with the “-append” option. A small PowerShell function that performs this operation is shown below. Note that the images are sorted by name before they are passed to convert so that the pages appear in the report in the correct order, and that the full path of each file (FullName) is used rather than relying on how PowerShell happens to render a file object as a string. The full diff report produced is also shown below. As you can see, such a report enables one to quickly see all the differences for every PDF document compared very easily.

# This function creates one single image which lists all the individual
# side by side comparison images vertically
function New-FullDiffReport
{
    [CmdletBinding()]
    param
    (
        [Parameter(Mandatory)]
        [string] $StitchedImageFolder,

        [Parameter(Mandatory)]
        [string] $OutputFile
    )

    $stitchedImages = @(Get-ChildItem -Path $StitchedImageFolder -Filter '*.jpg' | Sort-Object -Property Name)
    Write-Verbose "Number of stitched images to compile into single image for diff view: $($stitchedImages.Count)"

    if ($stitchedImages.Count -eq 0)
    {
        Write-Warning "No diff images found in '$StitchedImageFolder' to stitch together for full diff view operation"
        return
    }

    # "-append" places the images one below the other (top to bottom)
    $arguments = @($stitchedImages.FullName) + '-append' + $OutputFile
    Invoke-ExternalTool -FilePath $Script:ImageMagickConvert -ArgumentList $arguments

    if (-not (Test-Path -Path $OutputFile))
    {
        throw "Full diff view stitch operation did not produce '$OutputFile'"
    }

    Write-Verbose "Full diff view stitch operation was successful. Full diff view image '$OutputFile' was created"
    return $OutputFile
}

Screenshot showing the output of operation is seen below:

PDF Full Diff Report Screenshot

Source Code

You can also find the full source code used in the tutorial from GitHub here. Please ensure that you make the necessary adjustments to this code to make it production worthy before using it. Something I want to add here is that although this article talked about difference testing of PDF documents, in theory, you can use this tool to compare documents created originally in any other standard as along as they are ultimately converted to the PDF format before this script is used. Also, there is a port of the ImageMagick project called GraphicsMagick which can be substituted for ImageMagick if need be. However, as the two implementations are growing apart day by day, I cannot be 100% certain that the framework will work the same way with both these libraries.

Other Considerations

There are many advanced features that you can implement on top of the framework I describe here. This includes scenarios such as when the set of PDF files being compared against have different number of pages, when the pages are mismatched (meaning, you want to specify that for a certain PDF document, page 2 needs to be really compared against page 3 for instance), when the differences are so tiny and yet need to be caught, how to ignore minor pixel variations due to rendering issues or PDF driver variations which we would simply call “noise”, how to catch text variations only within the documents while ignoring other differences, etc. Several of these (in particular the “noise” problem) are much easier to solve with the more recent tools mentioned in the note at the start of the Tools section, so please look there first before extending this script.

Credits

Over the last decade, I have developed several testing frameworks while working on various projects for my clients, employers or for my own use, and the inspiration for developing those frameworks were often drawn from an article, a blog or a discussion forum on the Internet. The approach I describe is here is no exception. I was inspired to try out the approach to testing I describe here after reading an article from Adobe written by Timothy Oey many years ago (the article I referenced appears to be no longer accessible on the Adobe website). Please feel free to take the sample code provided here. Use it, modify or extend it, and share your enhancements to the community as well. If you have any questions or comments regarding these articles/tutorials, please feel free to send me an email. Please note that I may not get back to you right away due to work and other commitments.