How a DDoS Attack Appeared to Corrupt PostgreSQL — But the Real Problem Was Somewhere Else

on

This week we experienced an interesting infrastructure incident on one of Belgium's largest job aggregators. What initially looked like a PostgreSQL database corruption after a large DDoS attack turned into a deep investigation involving PostgreSQL recovery, Linux I/O, filesystems, storage diagnostics and backup verification.

The good news: no production data was lost.

Background

During a large DDoS attack the server became unstable and eventually stopped responding. The operating system itself became unresponsive and could not be restarted remotely.

Neither a normal reboot nor a hardware reset from the control panel worked. The machine simply stopped during reboot. Eventually the Hetzner datacenter technicians had to intervene manually and restart the server from their side.

After the server finally booted, PostgreSQL entered crash recovery as expected after an unclean shutdown.

Initial Symptoms

  • Website response time increased from milliseconds to 20–60 seconds.
  • Django migrations appeared to freeze.
  • Simple SQL queries became extremely slow.
  • Many PostgreSQL sessions returned:
FATAL: the database system is starting up

Initially everything pointed towards database corruption.

Recovery Process

PostgreSQL recovery logs looked completely normal.

database system was not properly shut down
automatic recovery in progress
redo starts...
redo done...

Eventually PostgreSQL finished recovery successfully.

However...

The database remained painfully slow.

Database Investigation

We started eliminating every possible database-related cause.

  • Checked PostgreSQL recovery status.
  • Verified WAL replay.
  • Checked active locks.
  • Verified long-running transactions.
  • Analyzed autovacuum.
  • Inspected migration locks.
  • Reviewed PostgreSQL background processes.

Nothing suggested actual database corruption.

The Strange Discovery

Eventually we noticed something unexpected.

Almost every PostgreSQL process was blocked in:

io_schedule

Including:

  • checkpointer
  • COMMIT
  • SELECT
  • UPDATE
  • autovacuum

Even more surprising...

The Linux command below completely froze:

sync

At this point it became obvious that PostgreSQL itself was not the bottleneck.

Storage Diagnostics

Disk statistics showed something very unusual.

Disk utilization: 100%
Write throughput: only a few hundred KB/sec
Average write latency: 30–80 seconds

In other words, almost no data was being written, yet every write operation was waiting tens of seconds before completing.

That explained why every COMMIT inside PostgreSQL appeared to hang.

SMART Looked Perfect

The SSD itself reported no hardware problems.

SMART: PASSED

Reallocated sectors: 0
Pending sectors: 0
CRC errors: 0
Filesystem: clean
RAID: healthy

Everything suggested that the hardware itself was healthy.

Backup Verification

Before making any risky recovery decisions we verified our external backups.

Fortunately all scheduled backups were still available on the external BackupBox.

The previous daily backups were complete, confirming that no production data had been lost.

This immediately reduced the risk of further troubleshooting.

The Final Fix

After the operating system had been properly restarted by the datacenter technicians, the storage subsystem finally returned to normal.

The easiest verification was simply:

time sync

real    0.034s

Immediately afterwards:

  • PostgreSQL became responsive.
  • COMMIT operations completed normally.
  • Disk latency returned to normal.
  • The website recovered.

No database restore was required.

Lessons Learned

  • PostgreSQL recovery after an unexpected shutdown does not automatically mean database corruption.
  • If sync blocks, the problem is often below the database layer.
  • Healthy SMART statistics do not guarantee a healthy storage stack.
  • Always verify backups before attempting destructive recovery actions.
  • Linux tools such as iostat, iotop, vmstat and PostgreSQL process states are invaluable during incident analysis.
  • Sometimes the database is only the victim, while the real problem exists inside the operating system's storage subsystem.

Next Improvements

  • Implement storage latency monitoring.
  • Add alerts for PostgreSQL I/O wait conditions.
  • Improve reboot and recovery procedures.
  • Continue validating backup integrity automatically.
  • Review filesystem and storage configuration after unexpected shutdowns.

In the end, what initially appeared to be a catastrophic PostgreSQL failure turned out to be an operating system storage issue triggered after an unclean shutdown during a DDoS incident.

Most importantly, no production data was lost, backups remained intact, and the investigation provided valuable insights that will improve the platform's resilience in the future.

DDoS Mitigation

One positive outcome of this incident was that the actual DDoS attack was mitigated surprisingly quickly.

After identifying the attacking IP ranges, I deployed my existing firewall automation project and imported the generated block list directly into the UFW firewall. The entire mitigation process took approximately 15–20 minutes, including generating the rules, importing them and reloading the firewall.

Once the firewall rules were applied, the attack traffic dropped almost immediately and the website became reachable again.

The firewall rules were generated using the open-source project:

DropIPsByCountry
https://github.com/OnlineSolutionsGroupBV/DropIPsByCountry.git

The DDoS attack itself was therefore not the biggest challenge. The more difficult part came afterwards.

Although the attack had already been stopped, the operating system had become unstable during the incident. The server required a reboot, but neither a standard reboot nor a remote hardware reset was able to recover it. Intervention from the Hetzner datacenter technicians was required before the machine could boot again.

After the reboot, PostgreSQL performed crash recovery successfully, but an unexpected storage-layer issue caused extremely high disk write latency. That secondary problem turned out to be significantly more difficult to diagnose than the DDoS attack itself.

This was a good reminder that solving the security incident is not always the end of the story. Sometimes the largest amount of engineering work begins only after the attack has already been mitigated.

Comments

Post a Comment