Send logs to a central system (ELK, Loki, or managed service). Set retention and rotation on the server so local disks do not fill. Use structured logging (JSON) for easier querying and alerting.
Centralize logs
- Why: Search and correlate across servers; single place for retention and alerting. Essential for debugging and security.
- Options: Self-hosted (ELK, Loki, Graylog) or managed (Datadog, Logtail, CloudWatch Logs). Choose by cost, scale, and retention needs.
- Shipping: Agents (Filebeat, Fluentd, Promtail) or syslog; send over TLS. Buffer locally if the central system is down.
Retention and rotation
- On the server: Rotate logs (logrotate, journald) so local disks do not fill. Keep only a short window (e.g. 1–7 days) locally; archive or send the rest to central storage.
- Central: Define retention (e.g. 30 days hot, 1 year cold). Compress and tier to control cost. Comply with any regulatory retention requirements.
- Deletion: Have a clear policy; automate so you do not retain forever by default.
Structured logging
- JSON: Log in JSON so fields (level, message, user_id, request_id) are queryable. Easier to build dashboards and alerts (e.g. "count where status >= 500").
- Consistent schema: Use the same field names across apps so queries and alerts are reusable.
Summary
Centralize logs; set retention and rotation on server and in central store; use structured (e.g. JSON) logging for querying and alerting.
