350,000 Rows Recovered After a Large-Scale Accidental Data Deletion
At around 14:00 on a weekend, more than 95% of the rows in a PostgreSQL 17 table at a healthcare organization were accidentally deleted, leaving only a few thousand of roughly 380,000 rows. WAL archiving was not enabled, but the customer stopped the database service immediately after discovering the incident. Analysis began close to 16:00 and found about 350,000 usable DELETE records still present in pg_wal. PDU recovered the data and completed import and validation in about three hours.
Main Timeline
- 1A healthcare industry customer was running PostgreSQL 17, and the target table originally contained roughly 380,000 rows.
- 2At around 14:00 on a weekend, a large-scale accidental deletion removed more than 95% of the data and left only a few thousand rows.
- 3WAL archiving was not enabled, and recovery analysis began close to 16:00.
- 1DELETE does not immediately erase the original rows from heap pages, but deleted tuples can later be pruned and cleaned.
- 2The deletion volume was far above a normal autovacuum threshold, so continued database activity would progressively reduce the recovery window.
- 3Without archived WAL, the recoverable amount depended on how many usable DELETE records remained in pg_wal.
- 1The customer stopped the PostgreSQL service immediately after discovering the incident to avoid further changes to the site.
- 2The pg_wal range around the incident was inspected to locate DELETE records that were still present.
- 3PDU parsed the relevant WAL and reconstructed the pre-deletion data.
- 4The recovered results were imported into the database and validated for row count and usability.
- 1About 350,000 usable DELETE records were found in pg_wal.
- 2About 350,000 of the roughly 380,000 affected rows were ultimately recovered.
- 3Recovery took about three hours from the start of analysis; the data was imported, validated, and confirmed usable by the customer.