Watch folders: react to new files with Python

Many office automations begin with a file arriving: the scanner drops a PDF into a network folder, the management system exports a CSV, a colleague saves a document to be processed. You can schedule a script to check the folder every hour, but sometimes an immediate reaction is needed. A watch folder is an always-on program that notices every new file and processes it straight away.

Two approaches: events or periodic checks

Operating systems offer mechanisms that report file changes: a program registers and receives a notice when a file is created, modified, moved or deleted. This is the most efficient approach, because it uses no resources when nothing happens.

The alternative is periodic checking: every few seconds the script lists the folder and compares the result with the previous pass. It is less elegant but more robust in some cases, particularly with network folders, where operating system notifications don't always arrive reliably.

The Python library watchdog supports both, with the same interface.

pip install watchdog

The skeleton of a watch folder

import time
from pathlib import Path
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

FOLDER = Path("C:/Scans/Incoming")

class NewFileHandler(FileSystemEventHandler):
    def on_created(self, event):
        if event.is_directory:
            return
        path = Path(event.src_path)
        if path.suffix.lower() == ".pdf":
            process(path)

observer = Observer()
observer.schedule(NewFileHandler(), str(FOLDER), recursive=False)
observer.start()
try:
    while True:
        time.sleep(1)
except KeyboardInterrupt:
    observer.stop()
observer.join()

The observer runs in a separate thread and calls on_created for every new file. The while loop only keeps the program alive. To use periodic checking, just replace Observer with PollingObserver, imported from watchdog.observers.polling.

The problem of the unfinished file

This is the main trap. The creation event arrives the moment the file comes into existence, not when it is finished. A scanner saving a twenty-page PDF, or a network copy of a large file, takes seconds to finish writing. If the script opens it immediately, it reads a truncated file and fails or, worse, processes an incomplete document.

The simplest solution is to wait for the size to settle:

def wait_until_complete(path, pause=2, attempts=30):
    previous_size = -1
    for _ in range(attempts):
        try:
            size = path.stat().st_size
        except FileNotFoundError:
            return False
        if size == previous_size and size > 0:
            return True
        previous_size = size
        time.sleep(pause)
    return False

On Windows you can add a further check: try to open the file for writing. If another program is still writing it, opening fails. Some systems adopt an even cleaner convention: whoever produces the file writes it with a temporary extension, such as .tmp, and renames it only at the end. The script watches only for the final extension.

Moving files after processing

A well-designed watch folder has three subfolders alongside the incoming one:

After processing, the script moves the file to the appropriate folder. The incoming folder thus stays always empty or nearly so, and anyone can see the state of the work at a glance. A file in errors is a clear sign that intervention is needed, while a file left in incoming for too long indicates that the script has stopped.

Duplicate events and files already present

Some applications, when saving a file, generate several events in quick succession: creation, then one or more modifications. If the script also reacts to modifications, it risks processing the same file several times. Listening only to on_created, and perhaps on_moved for renamed files, reduces the problem; keeping a set of paths already being processed eliminates it.

Remember too that the observer only sees files created after it starts. If the script is restarted after an interruption, files that arrived in the meantime just sit there. That is why, at start-up, it is always wise to process whatever is already in the incoming folder before starting to listen.

Keeping the script running

A script launched by hand stops when the session closes. For a permanent service the options are a Windows Task Scheduler task that starts when the computer boots, a systemd service on Linux or a small dedicated server. In every case you need a log file: in my work I record every file received, the outcome and the time taken, so I can reconstruct what happened even days later.

In short

• watchdog reports the creation of new files; PollingObserver is more reliable on network folders.
• Wait until the file is complete before processing it, checking that its size has settled.
• Move files to processed and errors folders to make the state visible.
• Handle duplicate events and process files already present at start-up.
• Run the script as a permanent service and keep a detailed log.