Configuring a Rotating File Logger for Production SystemsReliable logging is essential for diagnosing issues, monitoring behavior, and meeting compliance requirements in production systems. However, naive file-based logging can quickly consume disk space, create performance bottlenecks, and make log management difficult. A rotating file logger addresses these problems by splitting logs into multiple files based on size, time, or both, and retaining only a limited number of historical files. This article walks through why rotating loggers are important, design choices, concrete configuration examples across languages, operational considerations, and security best practices.
Why rotate logs?
- Prevent disk exhaustion: Without rotation, logs can grow indefinitely and fill disks, causing outages.
- Improve manageability: Smaller, bounded files are easier to compress, archive, search, and back up.
- Enable retention policies: Rotation makes it straightforward to retain only recent logs for compliance or cost control.
- Facilitate downstream tooling: Log shippers and analysis tools often prefer time-bound chunks (daily files) or size-limited files.
Rotation strategies
There are two primary strategies, sometimes combined:
-
Size-based rotation
- Rotate when the current log file reaches a maximum size (e.g., 100 MB).
- Advantages: predictable per-file size, useful for high-throughput services.
- Disadvantages: rotation frequency varies with traffic; harder to align with time-based analysis.
-
Time-based rotation
- Rotate on a schedule (e.g., hourly, daily).
- Advantages: files align with time windows, simpler retention by age.
- Disadvantages: a single time slice can be very large if traffic spikes.
-
Hybrid (size + time)
- Rotate either on exceeding size or at time boundaries to get the benefits of both.
Key configuration parameters
- Rotation policy: size, time, or both.
- Maximum file size (for size-based).
- Time format/interval (for time-based).
- Max number of backups or maximum retention age.
- Compression: whether to compress rotated files (e.g., gzip).
- Atomic rollover: ensure loggers can rotate without losing messages.
- File naming pattern: include timestamps/sequence numbers.
- Log level and format: consistent structured format (JSON) helps search and parsing.
- Symlink to “current” log: optional convenience pointing to the active file.
Log format recommendations
- Use structured logging (JSON) for production to enable downstream parsing and filtering. Include fields:
- timestamp (ISO 8601 with timezone)
- level
- service / application name
- environment (prod/staging)
- host / pod id
- request id / trace id (if available)
- message and optional structured payload
Example minimal JSON line: {“ts”:“2025-08-30T12:34:56Z”,“level”:“ERROR”,“svc”:“api”,“env”:“prod”,“host”:“app-01”,“trace”:“abc123”,“msg”:“failed to connect to db”,“err”:“timeout”}
Language-specific examples
Below are concise, tested examples showing popular ecosystems.
Python (logging.handlers.RotatingFileHandler and TimedRotatingFileHandler)
import logging from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler import json import socket from pythonjsonlogger import jsonlogger logger = logging.getLogger("myapp") logger.setLevel(logging.INFO) # Structured JSON formatter fmt = jsonlogger.JsonFormatter('%(asctime)s %(levelname)s %(name)s %(message)s') handler = RotatingFileHandler("/var/log/myapp/app.log", maxBytes=100*1024*1024, backupCount=7) # Or time-based: # handler = TimedRotatingFileHandler("/var/log/myapp/app.log", when="midnight", backupCount=30) handler.setFormatter(fmt) logger.addHandler(handler) logger = logging.LoggerAdapter(logger, { "svc": "myapp", "env": "prod", "host": socket.gethostname() }) logger.info("startup complete", extra={"trace": "xyz"}) Java (Logback example)
logback.xml:
<configuration> <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>/var/log/myapp/app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>/var/log/myapp/app.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern> <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"> <maxFileSize>100MB</maxFileSize> </timeBasedFileNamingAndTriggeringPolicy> <maxHistory>30</maxHistory> <totalSizeCap>10GB</totalSizeCap> </rollingPolicy> <encoder> <pattern>%date{ISO8601} %-5level [%thread] %logger{36} - %msg%n</pattern> </encoder> </appender> <root level="INFO"> <appender-ref ref="FILE" /> </root> </configuration> Node.js (winston + winston-daily-rotate-file)
const { createLogger, format, transports } = require("winston"); require("winston-daily-rotate-file"); const transport = new transports.DailyRotateFile({ filename: "/var/log/myapp/app-%DATE%.log", datePattern: "YYYY-MM-DD", maxSize: "100m", maxFiles: "30d", zippedArchive: true }); const logger = createLogger({ level: "info", format: format.combine(format.timestamp(), format.json()), transports: [transport] }); logger.info("service started", { svc: "myapp", env: "prod" }); Operational considerations
- File permissions: log files should be writable by the app user and readable only by authorized users. Avoid world-readable logs.
- Disk monitoring and alerts: set low-watermark alerts to detect disk pressure before it impacts rotation.
- Backups and archiving: regularly copy compressed rotated logs to long-term storage (object store) and delete locally per retention policy.
- Centralized logging: ship logs to a log aggregator (e.g., Fluentd/Vector/Logstash) rather than only keeping local files. Use forwarder that can handle rotation atomically (inode-aware).
- Handling multi-process rotation: many languages’ standard handlers are not safe across processes. Use a dedicated logger process, a central syslog, or tools like logrotate with copytruncate or external log shippers that follow inodes.
- Startup and shutdown: ensure the logger flushes buffers on graceful shutdown; configure unhandled-exception hooks to log before exit.
Dealing with multi-process & containerized environments
- Containers: prefer sidecar log shippers (Fluentd/Vector) reading container stdout/stderr rather than writing files inside containers. If files are necessary, mount a shared volume and ensure a single process handles rotation or use external rotation tools.
- Kubernetes: use stdout/stderr and let the container runtime handle logs; use a DaemonSet log collector. If writing files, ensure log rotation inside the container is managed and rotated files are exported to persistent storage.
Security best practices
- Avoid logging sensitive data (PII, credentials, tokens). Redact or hash sensitive fields at source.
- Protect rotated logs with appropriate permissions and encrypt at rest if they may contain sensitive info.
- Validate log input to avoid log injection (e.g., sanitize newlines or control characters).
- Use immutable storage for audit logs where required.
Testing your rotation setup
- Simulate high-throughput writes to verify size-based rotation triggers.
- Advance system clock or force time-based rotation to validate time-based policies.
- Confirm downstream log shippers pick up rotated files (test inode changes).
- Restore from archived logs to verify backups are usable.
Troubleshooting checklist
- No rotation: check process permissions, file paths, and whether rotation handler is active.
- Missing logs after rotation: verify consumers follow file renames/inodes; check copytruncate usage.
- High CPU during rotation: compression can be expensive—use async compression or lower compression level.
- Disk still filling: check retention settings (backupCount/maxFiles/maxHistory/totalSizeCap).
Example retention policy (reference)
- Keep daily logs for 30 days.
- Keep hourly logs for the last 48 hours for high-throughput services.
- Compress rotated logs immediately.
- Move anything older than 90 days to cold storage (S3 Glacier/Archive).
Summary
A well-configured rotating file logger helps keep production systems stable, searchable, and auditable. Choose rotation triggers that match your traffic patterns, prefer structured logs, protect files and sensitive fields, and integrate with centralized logging and archiving. Test rotation under load and in your deployment environment (containers, multi-process) to avoid surprises.
Leave a Reply