Python scripts

Why this exists

Log files never stop growing on their own. Left alone, an application or IIS log folder eventually fills the disk, usually at the worst possible time. The fix is a scheduled cleanup with a retention window that differs by environment: lower environments (dev, SIT, UAT, Stage) can afford to keep less, production needs a longer window in case an incident from a few weeks back needs investigating.

The script

#!/usr/bin/env python3
"""Delete log files older than an environment-aware retention window."""

import argparse
import logging
import sys
import time
from pathlib import Path

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger("log-cleanup")

RETENTION_DAYS = {
    "dev": 60,
    "sit": 60,
    "uat": 60,
    "stage": 60,
    "prod": 90,
}


def parse_args():
    parser = argparse.ArgumentParser(description="Clean up old log files.")
    parser.add_argument("--environment", required=True, choices=RETENTION_DAYS.keys())
    parser.add_argument("--log-dir", required=True, type=Path, help="Root folder to scan")
    parser.add_argument("--pattern", default="*.log", help="Glob pattern for log files")
    parser.add_argument("--dry-run", action="store_true", help="List what would be deleted, delete nothing")
    return parser.parse_args()


def main():
    args = parse_args()
    retention_days = RETENTION_DAYS[args.environment]
    cutoff = time.time() - (retention_days * 86400)

    if not args.log_dir.is_dir():
        log.error("log directory does not exist: %s", args.log_dir)
        sys.exit(1)

    deleted_count = 0
    freed_bytes = 0

    for path in args.log_dir.rglob(args.pattern):
        if not path.is_file():
            continue
        try:
            stat = path.stat()
        except OSError as exc:
            log.warning("could not stat %s: %s", path, exc)
            continue

        # Never touch a file written to in the last hour — that's very likely
        # today's active log, still open by IIS or the application itself.
        if stat.st_mtime > time.time() - 3600:
            continue

        if stat.st_mtime < cutoff:
            age_days = (time.time() - stat.st_mtime) / 86400
            if args.dry_run:
                log.info("[dry-run] would delete %s (%.0f days old, %d bytes)", path, age_days, stat.st_size)
                continue
            try:
                path.unlink()
                deleted_count += 1
                freed_bytes += stat.st_size
                log.info("deleted %s (%.0f days old)", path, age_days)
            except OSError as exc:
                log.warning("failed to delete %s: %s", path, exc)

    log.info(
        "done: environment=%s retention=%dd deleted=%d freed=%.1fMB dry_run=%s",
        args.environment, retention_days, deleted_count, freed_bytes / 1_048_576, args.dry_run,
    )


if __name__ == "__main__":
    main()

How it's wired into Jenkins

This runs as its own Jenkins job with a cron trigger, not tied to a deploy pipeline:

pipeline {
  agent { label 'windows' }
  triggers {
    cron('H 2 * * *')   // once a day, spread across a window so every job doesn't fire at :00
  }
  stages {
    stage('Clean up logs') {
      steps {
        bat "python log_cleanup.py --environment %ENVIRONMENT% --log-dir C:\\inetpub\\logs\\LogFiles"
      }
    }
  }
}

One job per environment (or a parameterised job run once per environment), each passing its own --environment value so the retention window is picked automatically.

Before you run this

• Run with --dry-run first against a real log folder and read the output before ever removing that flag.
• The default pattern is *.log. If logs are named like app.log.2026-09-01, change --pattern to match, or the script will silently clean up nothing.
• If a compliance or audit requirement needs logs kept longer than 90 days, don't shorten this script's window to work around it — archive them somewhere cheaper first, then let this delete from the live folder only.

A real incident

A dashboard that tailed the current day's log started showing gaps every night around 2am. The very first version of this script had no "last modified in the last hour" guard, and its retention math happened to land exactly on files still being appended to right at the moment IIS rotated them. Deleting a file that's still open doesn't fail loudly on Windows — the process keeps writing to it, but anything trying to open it fresh (like the dashboard's tailer) gets nothing. The one-hour freshness check exists specifically because "old enough by the clock" and "safe to delete" turned out to be two different things.