How I recovered a damaged Debian 13 container after exhausting my Proxmox thin pool, without rebuilding the system or reinstalling Drupal.
This article documents a real recovery I performed on my Proxmox VE host in November 2026. I'm keeping it as a record of the incident, the commands I used, the decisions I made and, hopefully, a useful reference for anyone who runs into a similar problem.
1. The incident
I had an LXC container (ID 108) running Debian 13.7, Apache and Drupal 10. It was working normally until my Proxmox LVM-thin storage pool ran out of available space.
The container's root filesystem was stored on:
local-lvm:vm-108-disk-0,size=10G
The underlying logical volume was:
/dev/pve/vm-108-disk-0
The important detail is that the container had a 10 GB virtual disk, but it was stored inside an LVM-thin pool shared with other virtual disks.
When the pool reached 100% usage, the guest filesystem started reporting I/O errors. The ext4 journal aborted and the filesystem was remounted read-only.
This wasn't simply a case of the container running out of disk space. The underlying Proxmox storage had run out of room to allocate blocks.
My first lesson from this incident: a container can report plenty of free filesystem space while the host has no remaining thin-pool capacity.
2. Understanding my storage layout
My relevant Proxmox storage configuration looked like this:
dir: local
path /var/lib/vz
content backup,vztmpl,iso
lvmthin: local-lvm
thinpool data
vgname pve
content images,rootdir
dir: s200g
path /mnt/data
content rootdir,iso,images
prune-backups keep-all=1
shared 0
I also had two ESXi import storages configured, but they weren't involved in this recovery.
The distinction between these storage types turned out to be important:
- local: a directory storage at /var/lib/vz, supporting backups, ISO images and container templates.
- local-lvm: an LVM-thin pool used for VM disks and container root filesystems.
- s200g: a directory storage mounted at /mnt/data, configured for images, root filesystems and ISOs, but not backups.
I initially had to establish exactly where everything was stored, rather than assume that all available storage destinations could hold backups.
These were the commands I used:
cat /etc/pve/storage.cfg
pvesm status
To inspect the LVM layout:
lvs -a -o +devices
vgs
lvs
3. Making room in the thin pool
The initial diagnosis showed that the LVM-thin pool named data had reached 100% data usage. Metadata usage was considerably lower.
The relevant volume group was pve, and the thin pool was:
pve/data
Fortunately, there was still free space in the volume group. I decided to extend the thin pool by 30 GB:
lvextend -L +30G pve/data
After the extension, the pool had a size of approximately 66.5 GiB and usage dropped to 54.88%.
This gave me enough breathing room to continue investigating and recovering the container.
One important distinction: extending the thin pool does not repair the guest filesystem. It only restores capacity to the underlying storage.
What I learned: when the storage pool itself is full, I need to address that first. Attempting filesystem repairs while the underlying storage is unable to allocate blocks is not a good starting point.
4. Identifying the damaged filesystem
The affected container was LXC 108.
Its original logical volume was:
/dev/pve/vm-108-disk-0
It was a 10 GiB thin-provisioned volume.
The filesystem had suffered ext4 journal and I/O errors following the storage exhaustion.
At this point, I wanted to preserve the data before attempting any repair.
I deliberately avoided formatting the volume, creating a new filesystem or immediately running repair operations against the original logical volume.
I wanted an untouched copy first.
5. Creating an untouched master image
I copied the original logical volume to a separate image on my directory storage mounted at /mnt/data.
I used GNU ddrescue:
ddrescue /dev/pve/vm-108-disk-0
/mnt/data/vm108-disk-0.img
/mnt/data/vm108-disk-0.map
The copy completed successfully:
- Approximately 10,737 MB rescued.
- 100% completed.
- Zero read errors.
- No bad areas reported.
The resulting image was:
/mnt/data/vm108-disk-0.img
I kept this as my untouched master copy.
I chose ddrescue because it is designed to copy data from potentially damaged storage while tracking progress and read errors in a map file. Even though my source was readable, it provided a reliable copy and a record of the operation.
My recovery rule: preserve the original image and perform repairs on a separate working copy.
6. Repairing a working copy
I created a second image for filesystem repair:
/mnt/data/vm108-repair.img
The repair was performed on this working copy, leaving the original image untouched.
I ran:
e2fsck -f -y /mnt/data/vm108-repair.img
The options mean:
- -f: force a full filesystem check.
- -y: automatically answer yes to repair prompts.
The repair recovered the ext4 journal and corrected filesystem inconsistencies.
Afterwards, I ran another check in read-only mode:
e2fsck -f -n /mnt/data/vm108-repair.img
The result was clean across all five filesystem checks:
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
102899/655360 files
487794/2621440 blocks
There were no further filesystem errors reported.
I now had a repaired image that passed the filesystem checks, while still retaining the original image in case something went wrong.
7. Inspecting the recovered data
Before writing anything back to the Proxmox logical volume, I mounted the repaired image read-only.
I used a dedicated mount point:
/mnt/recovery108
I wanted to verify that the filesystem was actually usable and that the important data was present.
My inspection confirmed:
- Debian 13.7 system files were present.
- The Drupal installation was intact.
- The expected filesystem structure was present.
- The lost+found directory was empty.
This was an important checkpoint. A clean filesystem check alone doesn't prove that the application data is complete or usable.
8. Restoring the repaired image
Once I was satisfied with the repaired image, I stopped the container.
I then copied the repaired image back to the original logical volume:
dd if=/mnt/data/vm108-repair.img
of=/dev/pve/vm-108-disk-0
bs=16M status=progress conv=fsync
sync
[b]Be careful with this operation.[b]
It overwrites the destination logical volume. The input and output paths must be checked before running it.
In my case:
- Source: repaired image.
- Destination: original LXC 108 logical volume.
- Both: 10 GiB.
After the copy completed, I performed another read-only filesystem check directly against the restored logical volume:
e2fsck -f -n /dev/pve/vm-108-disk-0
It passed all five stages without reporting filesystem errors.
This confirmed that the restored volume contained the repaired filesystem.
9. Starting the container and checking the services
With the filesystem restored and verified, I started the container:
pct start 108
pct status 108
The status reported:
status: running
I checked Apache from the Proxmox host:
pct exec 108 -- systemctl is-active apache2
Result:
active
I also checked the container's network interface:
pct exec 108 -- ip -br addr
The container had its expected network configuration.
The database was hosted separately in LXC 109, running MariaDB. I verified that the recovered container could reach the database server on port 3306.
Finally, I tested Apache locally, supplying the Drupal site's hostname:
pct exec 108 -- curl -sS -I --max-time 10
-H 'Host: your-domain.example'
http://127.0.0.1/
The server returned:
HTTP/1.1 200 OK
X-Generator: Drupal 10
Content-Type: text/html
This confirmed that Apache was responding and serving a Drupal 10 page.
I also checked the container's filesystem usage:
pct exec 108 -- df -h /
The result was approximately:
Filesystem Size Used Avail Use%
/dev/root 9.8G 1.7G 7.7G 18%
The filesystem itself was only 18% full.
At this point, the container was back online without rebuilding Debian or reinstalling Drupal.
10. The thin-provisioning surprise
After the recovery, I noticed something that initially looked rather strange.
The container's filesystem was only 18% full, but its Proxmox logical volume showed 100% allocation:
vm-108-disk-0 10.00g thin Data% 100.00
The thin pool itself showed approximately:
Pool size: 66.50 GiB
Data usage: 66.71%
Metadata usage: 2.89%
Both sets of numbers were correct. They simply measured different things.
Inside the container: ext4 reports how much filesystem space is occupied by files.
On the Proxmox host: LVM-thin reports how much of the logical volume's virtual capacity has actually been allocated from the thin pool.
When I copied the repaired image back using dd, I wrote the entire 10 GiB image to the logical volume. Consequently, all of its blocks became allocated in the thin pool.
That explains why the logical volume reports 100% allocation even though the filesystem only uses 18%.
This is not a filesystem-full condition.
It does mean that LXC 108's entire 10 GiB virtual volume is now allocated from the pool. It cannot grow beyond its configured 10 GiB capacity without a separate volume and filesystem resize.
11. Why I still need to monitor the thin pool
My thin pool contains several other guest volumes.
The approximate layout after recovery was:
LV Virtual size Data%
vm-103-disk-0 8.00 GiB 91.45%
vm-104-disk-0 20.00 GiB 45.25%
vm-104-disk-1 1.00 GiB 99.40%
vm-105-disk-0 20.00 GiB 77.88%
vm-107-disk-0 10.00 GiB 14.24%
vm-108-disk-0 10.00 GiB 100.00%
The combined virtual capacity is approximately 69 GiB, while the thin pool is approximately 66.5 GiB.
In other words, I had overprovisioned the pool.
This isn't necessarily a problem while actual allocations remain below the pool's capacity. However, if several guests grow their allocated storage at the same time, the pool can run out of space again.
At the time of my final checks:
- Thin pool data usage was approximately 66.71%.
- Metadata usage was approximately 2.89%.
- There was still free capacity in the pool.
- The pool was not currently full, but it needed monitoring.
I decided that 80% should be a warning threshold and 90% a critical threshold for my own monitoring.
These are operational guidelines, not Proxmox-enforced limits.
12. Discard and TRIM considerations
My LVM-thin pool was configured with discard passdown.
This allows discarded blocks to be passed down to the underlying storage where supported.
I attempted to run TRIM inside the unprivileged container:
pct exec 108 -- fstrim -v /
It failed with:
FITRIM ioctl failed: Operation not permitted
This was not evidence of filesystem corruption. The unprivileged container did not have permission to perform the operation.
We discussed a possible host-side approach involving stopping the container and mounting its filesystem directly, but I did not perform it as part of this recovery.
My takeaway: a failed TRIM command inside an unprivileged LXC does not automatically mean that the underlying filesystem or storage is broken.
13. Finding and checking the existing Proxmox backup
During the recovery, I wanted to establish whether I already had a Proxmox backup of the affected container.
I inspected my storage configuration and discovered that the backup was stored on the local directory storage, not on my separate data storage.
The backup was:
/var/lib/vz/dump/vzdump-lxc-108-2026_11_04-18_15_45.tar.zst
Its details were:
- Container: 108
- Backup date: November 4, 2026, 18:15
- File size: approximately 409 MB
- Format: tar.zst
- Proxmox storage: local
I checked its compression integrity using:
zstd -t /var/lib/vz/dump/vzdump-lxc-108-2026_11_04-18_15_45.tar.zst
The command completed successfully and reported 1,419,509,760 decompressed bytes.
I also checked that Proxmox recognized the backup:
pvesm list local --content backup
The archive appeared as:
local:backup/vzdump-lxc-108-2026_11_04-18_15_45.tar.zst
What I verified: the Zstandard compression stream passed its integrity test, and Proxmox recognized the archive as a backup for CT 108.
What I had not yet verified: that every file inside the archive could be restored successfully. A test restore would provide stronger confirmation.
This was also a useful reminder that a directory storage needs to have backup content enabled before Proxmox will use it as a backup destination.
14. Keeping the recovery files
I stored the recovery images outside the LVM-thin pool, on my directory storage mounted at /mnt/data.
The two important files were:
/mnt/data/vm108-disk-0.img
/mnt/data/vm108-repair.img
Their purposes:
- vm108-disk-0.img: untouched copy of the original logical volume.
- vm108-repair.img: working copy with the repaired ext4 filesystem, also used to restore the original logical volume.
I am keeping both images for now.
Just because the container is running again doesn't mean I should immediately delete the recovery material.
I want a fresh Proxmox backup, with its integrity checked, before considering this recovery fully closed.
15. What went wrong, and what I learned
1. Thin provisioning can hide a capacity problem.
A guest can report plenty of free filesystem space while the host has no remaining thin-pool capacity.
2. Thin-pool exhaustion can damage guest filesystems.
When storage allocations fail, the guest may experience I/O errors, journal aborts and read-only remounts. This is more serious than an ordinary full filesystem.
3. Restore storage capacity before repairing the filesystem.
I extended the thin pool before attempting filesystem repair. This restored the underlying storage capacity and gave me room to work.
4. Preserve the original disk before attempting repairs.
I made an untouched image of the original logical volume and performed all repair work on a separate copy.
5. A successful filesystem check is not the same as a successful application recovery.
I checked the filesystem, inspected the recovered files, restored the image, started the container, verified Apache, tested database connectivity and checked the HTTP response.
6. Guest filesystem usage and thin-pool allocation are different measurements.
My recovered container used approximately 18% of its filesystem, while its thin logical volume showed 100% allocation. Both figures were correct.
7. Backups need verification too.
A backup file existing on disk is not enough. Compression integrity is a useful first check, but an actual restore test provides stronger confidence.
8. Know where Proxmox stores backups.
My local directory storage supported backup content, while my separate data storage did not. A directory storage does not automatically become a backup destination simply because it has plenty of free space.
16. My future storage strategy: moving away from thin provisioning
One of the main lessons from this incident is that thin provisioning introduces a capacity-management risk that I don't particularly want in my environment.
My previous experience with VMware ESXi involved file-based virtual disks with fully allocated storage. I prefer knowing that the physical capacity required by a virtual disk has already been reserved instead of relying on dynamic allocation and hoping that several guests won't grow at the same time.
For this reason, I am considering moving my Proxmox guests toward file-based storage with explicit full allocation.
Proxmox directory storage can hold virtual disk image files and container root filesystems.
For virtual machines, a raw image with full preallocation is one possible approach. A qcow2 image, by contrast, can remain dynamically allocated unless configured otherwise.
For LXC containers, directory-backed root filesystems are another option.
My migration plan needs to take several things into account:
- Identify the filesystem and capacity available on the destination storage.
- Decide which guests should use directory-backed storage.
- Choose fully allocated raw images where appropriate.
- Ensure the destination has enough physical capacity for the actual allocated disks.
- Migrate one guest at a time.
- Verify each guest after migration.
- Maintain backups before removing the original volumes.
Moving away from LVM-thin doesn't eliminate the need to monitor free storage. It does, however, make the relationship between allocated guest storage and available physical capacity much more predictable when full allocation is used.
17. Useful diagnostic command reference
These are the commands I used during the investigation and recovery, collected here for convenience.
Check Proxmox storage status:
pvesm status
Inspect Proxmox storage configuration:
cat /etc/pve/storage.cfg
Inspect LVM volumes:
lvs -a -o +devices
vgs
Check LXC status:
pct status 108
Check services inside the container:
pct exec 108 -- systemctl is-active apache2
Check container filesystem usage:
pct exec 108 -- df -h /
Check a filesystem without making changes:
e2fsck -f -n /path/to/filesystem-or-image
Check a Zstandard-compressed backup:
zstd -t /path/to/backup.tar.zst
List Proxmox backups:
pvesm list local --content backup
18. Final status of my recovery
| Component | Result |
|---|---|
| LVM-thin pool | Capacity restored |
| Original disk image | Copied successfully |
| Repaired image | Filesystem checks clean |
| Restored LXC volume | Filesystem checks clean |
| LXC 108 | Running |
| Apache | Active |
| MariaDB connectivity | Verified |
| Drupal HTTP response | HTTP 200 OK |
| Existing backup archive | Zstandard integrity passed |
| Backup restore test | Not yet performed |
| Recovery images | Preserved |
I recovered the container without rebuilding the Debian installation or reinstalling Drupal.
I preserved the original filesystem, repaired a separate image, verified it, and restored it to the original Proxmox logical volume.
A storage pool running out of space is not just a storage warning. In a thin-provisioned environment, it can become a filesystem recovery incident.
Monitor the pool, understand what is allocated, and keep backups that you have actually tested.
Document status: initial recovery report, November 6, 2026. I will update this article when I complete the backup restore test and the planned storage migration.
- Log in to post comments
- 25 views