How to audit production system security without breaking it

Auditing security in production is different from a lab: every query, scan, or change can affect real users and real money. That does not mean postponing until the next redesign. It means using a sequence that prioritizes observation over intrusion, and small changes over big bang.

This guide summarizes how we approach legacy systems and owned SaaS platforms when the business cannot shut down—even for a weekend.

Phase 1: passive reconnaissance (zero impact)

Start without touching the system. Collect DNS, TLS certificates, HTTP headers, public subdomains, already-exposed registry data, leaked repos, and internal docs. Goal: attack surface map without aggressive traffic.

• Inventory of public endpoints and allowed HTTP methods.
• Security header review (HSTS, CSP, cookies).
• TLS and certificate chain verification.
• Secret search in repos and commit history.

Phase 2: configuration and access review

With read-only access to cloud consoles, IAM, and logs, evaluate who can do what. Look for over-permissive roles, unrotated service keys, and buckets or databases exposed by misconfiguration. On AWS, Azure, or GCP this often reveals more risk than application code.

In production, misconfiguration beats application bugs as a breach cause.

Phase 3: dependency and runtime secret analysis

Generate an SBOM from build or container. Cross versions against CVE databases. Review env vars, vaults, and server config files—without restarting services unless in an agreed window. Prioritize dependencies with public exploits and internet exposure.

Phase 4: controlled active testing

Only after prior phases, run gentle DAST scans or manual tests during low traffic, with rate limits and exclusions. Never run aggressive fuzzing against production without a mirror environment. If no staging exists, create a canary or read replica for destructive tests.

1. Agree on window and on-call contact.
2. Define forbidden URLs and actions (payments, mass deletes).
3. Monitor latency and 5xx errors in real time.
4. Stop at first measurable user impact.

Phase 5: interviews and tacit knowledge

Incomplete technical audits ignore the person who knows a service must restart on Mondays or that a forgotten SSH tunnel exists. Interview ops, support, and the longest-tenured dev. Document informal runbooks before they leave.

Deliverables that actually change risk

• Exposure map with business severity.
• Top 10 findings with owner and target date.
• Low-risk quick wins (MFA, close port, rotate leaked key).
• Roadmap for changes needing maintenance windows.

Metrics to measure progress

Track: percentage of accounts with MFA, mean time to offboard (hours from last day to revocation), admin count vs total users, active exceptions without expiry. Reviewing three numbers monthly convinces leadership better than a 40-page report.

Mistakes that would break production

Avoid: patching Friday without rollback, changing firewall rules without out-of-band console, scanning with admin creds at peak, or testing denial-of-service exploits on the main cluster. Defensive auditing reduces risk—it does not create it.

When to escalate to formal audit

The five-phase methodology covers 80% of value for most SMBs. Escalate to formal audit or pentest when: you pursue certification, a client requires external evidence, or phase 4 findings include exploitable criticals in production. Do not mix urgent remediation with simultaneous audit—stop the bleeding first, then measure again.

At Devson Labs we audit production systems that cannot fail: exposure inventory, secrets, dependencies, and controls without stopping operations. If you inherited a platform and do not know where to start, request a Radiography or explore our cybersecurity service.