Bug #225
crash with recursive zfs destroy on a snapshot
| Status: | New | Start: | 08/04/2010 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assigned to: | - | % Done: | 0% |
|
| Category: | - | Spent time: | - | |
| Target version: | - |
Description
I was testing a replication script using snapshots and zfs send/receive. It is supposed to delete the old snapshot after the replication was complete.
It wasn't deleting the old snapshots, so I tried deleting them myself. I tried various combinations with the -R -r -f -d options and kept getting a message that "dataset does not exist":
root@thinkmate2:~# zfs destroy -rd thinkmate2@rep20100804131546
cannot destroy 'thinkmate2@rep20100804131546': dataset does not exist
no snapshots destroyed
root@thinkmate2:~# man zfs
Reformatting zfs(1m), please wait...
root@thinkmate2:~# zfs destroy -Rf thinkmate2
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy -Rf thinkmate2
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy thinkmate2
cannot destroy 'thinkmate2': operation does not apply to pools
use 'zfs destroy -r thinkmate2' to destroy all datasets in the pool
use 'zpool destroy thinkmate2' to destroy the pool itself
root@thinkmate2:~# zfs destroy -r thinkmate2
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy thinkmate2/staff/jstrunk@rep20100804131142
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy -r thinkmate2/staff/jstrunk
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy -rf thinkmate2/staff/jstrunk
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
root@thinkmate2:~# zfs destroy -R thinkmate2/staff/jstrunk
cannot destroy 'thinkmate2/staff/jstrunk@rep20100804131142': dataset does not exist
I created a new snapshot and was able to delete it with just zfs destroy -r.
root@thinkmate2:~# zfs snapshot -r thinkmate2@test
root@thinkmate2:~# zfs destroy -r thinkmate2@test
root@thinkmate2:~# zfs snapshot -r thinkmate2@test
root@thinkmate2:~# zfs send -R thinkmate2@test | ssh root@thinkmate1 zfs receive -Fd thinkmate2
root@thinkmate2:~# zfs snapshot -r thinkmate2@test2
root@thinkmate2:~# zfs destroy -r thinkmate2@test
root@thinkmate2:~# zfs destroy -r thinkmate2@test2
Then I tried to delete the old snapshots again and the system crashed:
root@thinkmate2:~# zfs destroy -rd thinkmate2@rep20100804131228
Write failed: Broken pipe
That snapshot still existed after the crash caused a reboot. I was able to destroy it normally.
History
Updated by Jeff Strunk about 1 year ago
I forgot to mention, this is with RC3. All of the zpools have been upgraded.
Updated by Jeff Strunk about 1 year ago
I can't trigger this again, maybe the system needs to be on for a while.
Updated by Jeff Strunk about 1 year ago
- File mdb.thinkmate1 added
- File zfsreplicator.sh added
My replication script using zfs send/receive seems to be causing problems. I had the servers mirroring to each other hourly overnight to test. One ran the script on the hour, and the other ran it on the half hour.
This morning, I came in to find that 'zfs list' was hanging. 'zpool status' returned fine. I didn't know I should run mdb in that case, so I tried rebooting with the reboot command. I waited about 15 minutes while I research more about the problem, and it still hadn't shutdown. I had to press the reset button on the server.
Recently, one of the servers crashed during the destroy old snapshots part of the script. I have attached the the output from mdb and an updated version of my script.
Updated by Jeff Strunk about 1 year ago
- File zfsdestroy.truss added
I have attached the output of truss for zfs destroy -r of a snapshot when it says "dataset does not exist".
Updated by Kim Lundgren about 1 year ago
I had a similar issue today, at least I assume they're releated.
Whenever trying to destroy a zfs or zpool that is already in use, the system incorrectly responds with:
cannot destroy 'X': dataset does not exist
Updated by Jeff Strunk about 1 year ago
I think I found the issue. I just found out about zfs holds and that zfs send will hold the snapshot being sent. It looks like some of the holds are not getting released.
Here is a log of my test:
root@thinkmate2:~# zfs list -t all -r thinkmate2
NAME USED AVAIL REFER MOUNTPOINT
thinkmate2 15.1M 7.13T 61.9K /thinkmate2
thinkmate2@r0001 0 - 61.9K -
thinkmate2@r0002 0 - 61.9K -
thinkmate2/bar 55.9K 7.13T 55.9K /thinkmate2/bar
thinkmate2/bar@r0001 0 - 55.9K -
thinkmate2/bar@r0002 0 - 55.9K -
thinkmate2/baz 267K 7.13T 61.9K /thinkmate2/baz
thinkmate2/baz@r0001 0 - 61.9K -
thinkmate2/baz@r0002 0 - 61.9K -
thinkmate2/baz/1 55.9K 7.13T 55.9K /thinkmate2/baz/1
thinkmate2/baz/1@r0001 0 - 55.9K -
thinkmate2/baz/1@r0002 0 - 55.9K -
thinkmate2/baz/2 55.9K 7.13T 55.9K /thinkmate2/baz/2
thinkmate2/baz/2@r0001 0 - 55.9K -
thinkmate2/baz/2@r0002 0 - 55.9K -
thinkmate2/baz/3 92.9K 7.13T 56.9K /thinkmate2/baz/3
thinkmate2/baz/3@r0001 36.0K - 55.9K -
thinkmate2/baz/3@r0002 0 - 56.9K -
thinkmate2/foo 151K 7.13T 58.9K /thinkmate2/foo
thinkmate2/foo@r0001 36.0K - 57.9K -
thinkmate2/foo@r0002 0 - 58.9K -
thinkmate2/foo/1 55.9K 7.13T 55.9K /thinkmate2/foo/1
thinkmate2/foo/1@r0001 0 - 55.9K -
thinkmate2/foo/1@r0002 0 - 55.9K -
root@thinkmate2:~# zfs send -R -i thinkmate2@r0001 thinkmate2@r0002 | zfs receive -d thinkmate1
root@thinkmate2:~# zfs list -t all -r thinkmate1
NAME USED AVAIL REFER MOUNTPOINT
thinkmate1 1.73M 7.13T 61.9K /thinkmate1
thinkmate1@r0001 0 - 61.9K -
thinkmate1@r0002 0 - 61.9K -
thinkmate1/bar 55.9K 7.13T 55.9K /thinkmate1/bar
thinkmate1/bar@r0001 0 - 55.9K -
thinkmate1/bar@r0002 0 - 55.9K -
thinkmate1/baz 267K 7.13T 61.9K /thinkmate1/baz
thinkmate1/baz@r0001 0 - 61.9K -
thinkmate1/baz@r0002 0 - 61.9K -
thinkmate1/baz/1 55.9K 7.13T 55.9K /thinkmate1/baz/1
thinkmate1/baz/1@r0001 0 - 55.9K -
thinkmate1/baz/1@r0002 0 - 55.9K -
thinkmate1/baz/2 55.9K 7.13T 55.9K /thinkmate1/baz/2
thinkmate1/baz/2@r0001 0 - 55.9K -
thinkmate1/baz/2@r0002 0 - 55.9K -
thinkmate1/baz/3 92.9K 7.13T 56.9K /thinkmate1/baz/3
thinkmate1/baz/3@r0001 36.0K - 55.9K -
thinkmate1/baz/3@r0002 0 - 56.9K -
thinkmate1/foo 151K 7.13T 58.9K /thinkmate1/foo
thinkmate1/foo@r0001 36.0K - 57.9K -
thinkmate1/foo@r0002 0 - 58.9K -
thinkmate1/foo/1 55.9K 7.13T 55.9K /thinkmate1/foo/1
thinkmate1/foo/1@r0001 0 - 55.9K -
thinkmate1/foo/1@r0002 0 - 55.9K -
root@thinkmate2:~# zfs destroy -r thinkmate2@r0001
cannot destroy 'thinkmate2@r0001': dataset does not exist
no snapshots destroyed
root@thinkmate2:~# zfs holds -r thinkmate2@r0001
NAME TAG TIMESTAMP
thinkmate2/baz/2@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
thinkmate2/foo/1@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
thinkmate2@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
As you can see, only some of the filesystems still had holds on them. I don't know of anything special about them. After 'zfs release', 'zfs destroy' succeeded.
Further attempts to replicate this seem to show that only the initial send has this problem. zfs send -R thinkmate2@r0007 | zfs receive -F -d thinkmate1
In a later case where I destroyed thinkmate1 and started over with snapshot r0007, the same filesystems had the bad holds:
root@thinkmate2:~# zfs holds -r thinkmate2@r0007NAME TAG TIMESTAMP
thinkmate2/baz/2@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
thinkmate2/foo/1@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
thinkmate2@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
I still don't know why this could cause a crash.
Updated by Jeff Strunk about 1 year ago
I forgot to add the pre tags to that last messaged.
root@thinkmate2:~# zfs list -t all -r thinkmate2
NAME USED AVAIL REFER MOUNTPOINT
thinkmate2 15.1M 7.13T 61.9K /thinkmate2
thinkmate2@r0001 0 - 61.9K -
thinkmate2@r0002 0 - 61.9K -
thinkmate2/bar 55.9K 7.13T 55.9K /thinkmate2/bar
thinkmate2/bar@r0001 0 - 55.9K -
thinkmate2/bar@r0002 0 - 55.9K -
thinkmate2/baz 267K 7.13T 61.9K /thinkmate2/baz
thinkmate2/baz@r0001 0 - 61.9K -
thinkmate2/baz@r0002 0 - 61.9K -
thinkmate2/baz/1 55.9K 7.13T 55.9K /thinkmate2/baz/1
thinkmate2/baz/1@r0001 0 - 55.9K -
thinkmate2/baz/1@r0002 0 - 55.9K -
thinkmate2/baz/2 55.9K 7.13T 55.9K /thinkmate2/baz/2
thinkmate2/baz/2@r0001 0 - 55.9K -
thinkmate2/baz/2@r0002 0 - 55.9K -
thinkmate2/baz/3 92.9K 7.13T 56.9K /thinkmate2/baz/3
thinkmate2/baz/3@r0001 36.0K - 55.9K -
thinkmate2/baz/3@r0002 0 - 56.9K -
thinkmate2/foo 151K 7.13T 58.9K /thinkmate2/foo
thinkmate2/foo@r0001 36.0K - 57.9K -
thinkmate2/foo@r0002 0 - 58.9K -
thinkmate2/foo/1 55.9K 7.13T 55.9K /thinkmate2/foo/1
thinkmate2/foo/1@r0001 0 - 55.9K -
thinkmate2/foo/1@r0002 0 - 55.9K -
root@thinkmate2:~# zfs send -R -i thinkmate2@r0001 thinkmate2@r0002 | zfs receive -d thinkmate1
root@thinkmate2:~# zfs list -t all -r thinkmate1
NAME USED AVAIL REFER MOUNTPOINT
thinkmate1 1.73M 7.13T 61.9K /thinkmate1
thinkmate1@r0001 0 - 61.9K -
thinkmate1@r0002 0 - 61.9K -
thinkmate1/bar 55.9K 7.13T 55.9K /thinkmate1/bar
thinkmate1/bar@r0001 0 - 55.9K -
thinkmate1/bar@r0002 0 - 55.9K -
thinkmate1/baz 267K 7.13T 61.9K /thinkmate1/baz
thinkmate1/baz@r0001 0 - 61.9K -
thinkmate1/baz@r0002 0 - 61.9K -
thinkmate1/baz/1 55.9K 7.13T 55.9K /thinkmate1/baz/1
thinkmate1/baz/1@r0001 0 - 55.9K -
thinkmate1/baz/1@r0002 0 - 55.9K -
thinkmate1/baz/2 55.9K 7.13T 55.9K /thinkmate1/baz/2
thinkmate1/baz/2@r0001 0 - 55.9K -
thinkmate1/baz/2@r0002 0 - 55.9K -
thinkmate1/baz/3 92.9K 7.13T 56.9K /thinkmate1/baz/3
thinkmate1/baz/3@r0001 36.0K - 55.9K -
thinkmate1/baz/3@r0002 0 - 56.9K -
thinkmate1/foo 151K 7.13T 58.9K /thinkmate1/foo
thinkmate1/foo@r0001 36.0K - 57.9K -
thinkmate1/foo@r0002 0 - 58.9K -
thinkmate1/foo/1 55.9K 7.13T 55.9K /thinkmate1/foo/1
thinkmate1/foo/1@r0001 0 - 55.9K -
thinkmate1/foo/1@r0002 0 - 55.9K -
root@thinkmate2:~# zfs destroy -r thinkmate2@r0001
cannot destroy 'thinkmate2@r0001': dataset does not exist
no snapshots destroyed
root@thinkmate2:~# zfs holds -r thinkmate2@r0001
NAME TAG TIMESTAMP
thinkmate2/baz/2@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
thinkmate2/foo/1@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
thinkmate2@r0001 .send-571-0 Wed Sep 22 15:07:46 2010
As you can see, only some of the filesystems still had holds on them. I don't know of anything special about them. After 'zfs release', 'zfs destroy' succeeded.
Further attempts to replicate this seem to show that only the initial send has this problem.
zfs send -R thinkmate2@r0007 | zfs receive -F -d thinkmate1
In a later case where I destroyed thinkmate1 and started over with snapshot r0007, the same filesystems had the bad holds:
root@thinkmate2:~# zfs holds -r thinkmate2@r0007
NAME TAG TIMESTAMP
thinkmate2/baz/2@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
thinkmate2/foo/1@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
thinkmate2@r0007 .send-718-0 Wed Sep 22 15:47:32 2010
I still don't know why this could cause a crash.
Updated by Jeff Strunk about 1 year ago
- File mdb.thinkmate1 added
While testing this workaround of releasing holds with a home directory on nfs, the system storing the home directory, thinkmate1, crashed. The cron job on thinkmate1 that runs the replication script runs every half hour on the 15 and the 45. It takes a couple of minutes to run. thinkmate1 crashed at 1:17pm on Friday. That seems to be during the portion of the script where all the holds are released for particular snapshots and those snapshots are destroyed.
This must mean that zfs send is not cleaning up more that just a few holds.
Updated by Jeff Strunk about 1 year ago
According to bslbb on IRC, http://bugs.opensolaris.org/viewbug.do?bugid=6884007 fixes this problem. I'd like to test this fix on my systems.
<bslbb> jstrunk: there are some known bugs related to zfs snapshot holds, in particular http://bugs.opensolaris.org/view_bug.do?bug_id=6884007
<jstrunk> a ha
<bslbb> this one in particular was hanging my kernel in NCP3 when doing a zfs send of my syspool
<jstrunk> bslbb: if the fix is released on that one, is there a way to test it in nexenta?
<xinkeT> only if that fix has been backported
<bslbb> not easily without integrating the fix and rebuilding the kernel (which I did, and it fixed my problem)
Updated by Jeff Strunk about 1 year ago
- File mdb.thinkmate1-gp added
I got a different crash today while testing.
fmdump still reports that /var/fm/fmd/fltlog is empty.
Updated by Jeff Strunk about 1 year ago
I just tracked down the crash to issue #224.
When I was testing over nfs, the file indexer had been accessing the .zfs directory and this crashed the server very quickly. I set snapdir to hidden, and the server seemed stable.
To verify this, I ran "find .zfs" over nfs, and the server crashed. It's too bad snapdir=hidden doesn't prevent knowledgeable users from crashing the server.
How long until the fixes from snv142 for this holds bug and #224 get backported?
Updated by Jeff Strunk about 1 year ago
onnv changeset 693dd2cad55f fails to apply for usr/src/uts/common/fs/zfs/zvol.c . The function the patch fails in does not exist in 134.
I think too much has changed for me to test it.