Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zfs list -t snapshot: pointer being freed was not allocated #123

Closed
jhale opened this issue Feb 26, 2014 · 27 comments
Closed

zfs list -t snapshot: pointer being freed was not allocated #123

jhale opened this issue Feb 26, 2014 · 27 comments

Comments

@jhale
Copy link

jhale commented Feb 26, 2014

jack.hale@ijhale:~$ sudo zfs list -t snapshot
real_zfs(813,0x7fff7bdbd310) malloc: *** error for object 0x7feacb504800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Mavericks using the last few nightly builds from lundman's ftp. This is the 20140224 build. Can you give me some pointers on how to debug it?

@ilovezfs
Copy link
Contributor

I suggest trying a source-build and see if you get same error.

@jhale
Copy link
Author

jhale commented Feb 26, 2014

Didn't fix the problem.

@lundman
Copy link
Contributor

lundman commented Feb 26, 2014

Be interesting to see what gdb or lldb says from either the core, or from running it

@jhale
Copy link
Author

jhale commented Feb 26, 2014

Current executable set to '/usr/local/sbin/zfs' (x86_64).
(lldb) r list -t snapshot
Process 919 launched: '/usr/local/sbin/zfs' (x86_64)
Process 919 stopped and restarted: thread 1 received signal: SIGCHLD
zfs(919,0x7fff78ad9310) malloc: *** error for object 0x100404800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Process 919 stopped
* thread #1: tid = 0x9982, 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff93ade866:  jae    0x7fff93ade870            ; __pthread_kill + 20
   0x7fff93ade868:  movq   %rax, %rdi
   0x7fff93ade86b:  jmpq   0x7fff93adb175            ; cerror_nocancel
   0x7fff93ade870:  ret   

I'm not great with lldb, what should I be doing?

@ilovezfs
Copy link
Contributor

Does it happen if you do zfs list -t all

@rottegift
Copy link
Contributor

I do not see this in a test system, however the master branch currently suffers from openzfs/zfs#2094 and the pools I have readily available to it (and of course created by it) all suffer from their ZFS Errata # 1 http://zfsonlinux.org/msg/ZFS-8000-ER/

Is it possible that this affects only pools that do not have the on-disk Errata # 1 incompatibility ?

A test would be to create a test pool with whatever build is giving you the bug:

$ pwd
/Users/you
$ mkfile -v 2g foo
$ sudo zpool create erratum1pool /Users/you/foo
$ sudo zfs snapshot erratum1pool@a
$ sudo zfs list -t snapshot erratum1pool

Alternatively or additionally it would be interesting to know how many numbers are in the "scan =" line returned from

$ sudo zdb -U /etc/zfs/zpool.cache.tmp -dddd yourregularpool 1

in partcular whether there are 24 or 25, or whether there is no "scan =" line at all.

(You can safely interrupt the zdb process; it can take a long time to complete, but would emit the relevant line within not much more than a few seconds.)

There will be 25 for any pool that has been scrubbed, resilvered, or created since the bug was imported into zfs-osx (probably on and since 24 Jan 2014). Other pools should have 24 numbers or no "scan =" line at all.

@ilovezfs
Copy link
Contributor

I'd be surprised if that had anything to do with the price of bread. If it does happen to be related, switching to the branch upstream-20140225 should fix it.

@jhale
Copy link
Author

jhale commented Feb 27, 2014

The behaviour seems variable:

jack.hale@ijhale:~$ sudo zfs list -t all
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana                            425G  1.35T   784K  /banana
banana@20140131-new               112K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code                      1.80G  1.35T  1.80G  /banana/code
banana/code@20140131-new         1.06M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/doc                        241M  1.35T   241M  /banana/doc
jack.hale@ijhale:~$ sudo zfs list -t all
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana                            425G  1.35T   784K  /banana
banana@20140131-new               112K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code                      1.80G  1.35T  1.80G  /banana/code
banana/code@20140131-new         1.06M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/doc                        241M  1.35T   241M  /banana/doc
banana/documents                 35.7G  1.35T  21.5G  /banana/documents
banana/documents@20140131-new    14.2G      -  35.5G  /banana/documents/.zfs/snapshot/20140131-new
banana/gaia                       396M  1.35T   396M  /banana/gaia
banana/gaia@20140131-new          120K      -      0  /banana/gaia/.zfs/snapshot/20140131-new
banana/software                  14.5G  1.35T  14.5G  /banana/software
banana/src                       4.70G  1.35T  4.70G  /banana/src
banana/src@20140131-new           840K      -  4.69G  /banana/src/.zfs/snapshot/20140131-new
banana/timemachine                304G  1.35T   301G  /banana/timemachine
banana/timemachine@20140131-new  2.66G      -   273G  /banana/timemachine/.zfs/snapshot/20140131-new
banana/vms                       64.3G  1.35T  64.3G  /banana/vms
banana/vms@20140131-new           184K      -  64.0G  /banana/vms/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t all
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana                            425G  1.35T   784K  /banana
banana@20140131-new               112K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code                      1.80G  1.35T  1.80G  /banana/code
banana/code@20140131-new         1.06M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/doc                        241M  1.35T   241M  /banana/doc
banana/documents                 35.7G  1.35T  21.5G  /banana/documents
banana/documents@20140131-new    14.2G      -  35.5G  /banana/documents/.zfs/snapshot/20140131-new
banana/gaia                       396M  1.35T   396M  /banana/gaia
banana/gaia@20140131-new          120K      -      0  /banana/gaia/.zfs/snapshot/20140131-new
banana/software                  14.5G  1.35T  14.5G  /banana/software
banana/src                       4.70G  1.35T  4.70G  /banana/src
banana/src@20140131-new           840K      -  4.69G  /banana/src/.zfs/snapshot/20140131-new
banana/timemachine                304G  1.35T   301G  /banana/timemachine
banana/timemachine@20140131-new  2.66G      -   273G  /banana/timemachine/.zfs/snapshot/20140131-new
banana/vms                       64.3G  1.35T  64.3G  /banana/vms
banana/vms@20140131-new           184K      -  64.0G  /banana/vms/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t snapshot
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana@20140131-new               112K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code@20140131-new         1.06M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/documents@20140131-new    14.2G      -  35.5G  /banana/documents/.zfs/snapshot/20140131-new
banana/gaia@20140131-new          120K      -   396M  /banana/gaia/.zfs/snapshot/20140131-new
banana/src@20140131-new           840K      -  4.69G  /banana/src/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t snapshot
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana@20140131-new               112K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code@20140131-new         1.06M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/documents@20140131-new    14.2G      -  35.5G  /banana/documents/.zfs/snapshot/20140131-new
banana/gaia@20140131-new          120K      -   396M  /banana/gaia/.zfs/snapshot/20140131-new
banana/src@20140131-new           840K      -  4.69G  /banana/src/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs(1535,0x7fff78ad9310) malloc: *** error for object 0x7ff7da504800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

@lundman
Copy link
Contributor

lundman commented Feb 27, 2014

Oh in lldb, can you do "bt" which is backtrace.

@jhale
Copy link
Author

jhale commented Feb 27, 2014

Here you are.

(lldb) r list -t snapshot
Process 2975 launched: '/usr/local/sbin/zfs' (x86_64)
Process 2975 stopped and restarted: thread 1 received signal: SIGCHLD
zfs(2975,0x7fff78ad9310) malloc: *** error for object 0x100404800: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Process 2975 stopped
* thread #1: tid = 0x56eca, 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill + 10:
-> 0x7fff93ade866:  jae    0x7fff93ade870            ; __pthread_kill + 20
   0x7fff93ade868:  movq   %rax, %rdi
   0x7fff93ade86b:  jmpq   0x7fff93adb175            ; cerror_nocancel
   0x7fff93ade870:  ret    
(lldb) bt
* thread #1: tid = 0x56eca, 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff93ade866 libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff8941a35c libsystem_pthread.dylib`pthread_kill + 92
    frame #2: 0x00007fff8d06cb1a libsystem_c.dylib`abort + 125
    frame #3: 0x00007fff91c6607f libsystem_malloc.dylib`free + 411
    frame #4: 0x000000010002726d libnvpair.1.dylib`nv_free_sys(nva=0x00000001000343c8, buf=0x0000000100404800, size=1684370309) + 29 at nvpair_alloc_system.c:42
    frame #5: 0x0000000100027a14 libnvpair.1.dylib`nv_mem_free(nvp=0x0000000100404310, buf=0x0000000100404800, size=1684370309) + 68 at nvpair.c:202
    frame #6: 0x0000000100027cae libnvpair.1.dylib`nvp_buf_free(nvl=0x0000000100404890, nvp=0x0000000100404810) + 78 at nvpair.c:350
    frame #7: 0x0000000100027aef libnvpair.1.dylib`nvlist_free(nvl=0x0000000100404890) + 127 at nvpair.c:588
    frame #8: 0x0000000100210831 libzfs.2.dylib`zfs_close(zhp=0x0000000100406880) + 65 at libzfs_dataset.c:603
    frame #9: 0x00000001000021ca zfs`zfs_callback(zhp=0x0000000100406880, data=0x00007fff5fbff9f0) + 698 at zfs_iter.c:145
    frame #10: 0x000000010022307c libzfs.2.dylib`zfs_iter_filesystems(zhp=0x0000000100400530, func=0x0000000100001f10, data=0x00007fff5fbff9f0) + 268 at libzfs_iter.c:130
    frame #11: 0x000000010000214a zfs`zfs_callback(zhp=0x0000000100400530, data=0x00007fff5fbff9f0) + 570 at zfs_iter.c:135
    frame #12: 0x000000010020f8e2 libzfs.2.dylib`zfs_iter_root(hdl=0x0000000101000000, func=0x0000000100001f10, data=0x00007fff5fbff9f0) + 162 at libzfs_config.c:456
    frame #13: 0x00000001000015ea zfs`zfs_for_each(argc=0, argv=0x00007fff5fbffc58, flags=2, types=ZFS_TYPE_SNAPSHOT, sortcol=0x0000000000000000, proplist=0x00007fff5fbffb80, limit=0, callback=0x00000001000116c0, data=0x00007fff5fbffb70) + 650 at zfs_iter.c:434
    frame #14: 0x0000000100005399 zfs`zfs_do_list(argc=0, argv=0x00007fff5fbffc58) + 1289 at zfs_main.c:3087
    frame #15: 0x000000010000275f zfs`main(argc=4, argv=0x00007fff5fbffc38) + 623 at zfs_main.c:6543
    frame #16: 0x00007fff916f45fd libdyld.dylib`start + 1
    frame #17: 0x00007fff916f45fd libdyld.dylib`start + 1
(lldb) 

@ilovezfs ilovezfs mentioned this issue Mar 8, 2014
@stefarossi
Copy link

I'm having the same problem. My pool (which was created with ZFSOnLinux) has this scan line:

scan = 2 2 0 3 1420634 1400139 1420634 1394171979 1394186833 410855520768 411015283200 0 411015283200 0 1 2 0 8 281485365257240 0 0 0 0 0

This is my lldb:

(lldb) r list -t snapshot
r list -t snapshot
Process 2417 launched: '/usr/zfsbuild/zfs/cmd/zfs/.libs/zfs' (x86_64)
Process 2417 stopped and restarted: thread 1 received signal: SIGCHLD
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
RAID1@0                                          26K      -  9.53G  /RAID1/.zfs/snapshot/0
RAID1@1                                          32K      -  9.53G  /RAID1/.zfs/snapshot/1
[many mores snapshots are shown here]
zfs(2417) malloc: *** error for object 0x100312a00: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
Process 2417 stopped
* thread #1: tid = 0x651ab, 0x00007fff8d41bd46 libsystem_kernel.dylib`__kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff8d41bd46 libsystem_kernel.dylib`__kill + 10
libsystem_kernel.dylib`__kill + 10:
-> 0x7fff8d41bd46:  jae    0x7fff8d41bd4d            ; __kill + 17
   0x7fff8d41bd48:  jmpq   0x7fff8d41d4d4            ; cerror_nocancel
   0x7fff8d41bd4d:  ret    
   0x7fff8d41bd4e:  nop 
(lldb) bt
* thread #1: tid = 0x651ab, 0x00007fff8d41bd46 libsystem_kernel.dylib`__kill + 10, queue = 'com.apple.main-thread, stop reason = signal SIGABRT
    frame #0: 0x00007fff8d41bd46 libsystem_kernel.dylib`__kill + 10
    frame #1: 0x00007fff8503af83 libsystem_c.dylib`abort + 177
    frame #2: 0x00007fff8500e989 libsystem_c.dylib`free + 392
    frame #3: 0x000000010002836d libnvpair.1.dylib`nv_free_sys(nva=0x00000001000383c8, buf=0x0000000100312a00, size=3219616) + 29 at nvpair_alloc_system.c:42
    frame #4: 0x0000000100028b14 libnvpair.1.dylib`nv_mem_free(nvp=0x0000000100311f60, buf=0x0000000100312a00, size=3219616) + 68 at nvpair.c:202
    frame #5: 0x0000000100028dae libnvpair.1.dylib`nvp_buf_free(nvl=0x0000000100311f90, nvp=0x0000000100312a10) + 78 at nvpair.c:350
    frame #6: 0x0000000100028bef libnvpair.1.dylib`nvlist_free(nvl=0x0000000100311f90) + 127 at nvpair.c:588
    frame #7: 0x0000000100230f81 libzfs.2.dylib`zfs_close(zhp=0x0000000100312680) + 65 at libzfs_dataset.c:603
    frame #8: 0x0000000100001530 zfs`zfs_for_each(argc=0, argv=0x00007fff5fbffab0, flags=2, types=ZFS_TYPE_SNAPSHOT, sortcol=0x0000000000000000, proplist=0x00007fff5fbff9e0, limit=0, callback=0x0000000100011470, data=0x00007fff5fbff9d0) + 1200 at zfs_iter.c:482
    frame #9: 0x0000000100005089 zfs`zfs_do_list(argc=0, argv=0x00007fff5fbffab0) + 1289 at zfs_main.c:3087
    frame #10: 0x000000010000245f zfs`main(argc=4, argv=0x00007fff5fbffa90) + 623 at zfs_main.c:6543
    frame #11: 0x00007fff8c6187e1 libdyld.dylib`start + 1

@lundman
Copy link
Contributor

lundman commented Mar 8, 2014

It is possible this is related to:

openzfs/zfs#2157

@stefarossi
Copy link

Now it happened with 'zfs inherit':

$ zfs inherit -r mountpoint RAID1/TRASH           
zfs(1747) malloc: *** error for object 0x7f9a08412300: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

@lundman
Copy link
Contributor

lundman commented Mar 11, 2014

Could one, or all, of you try e41afca

To see if it fixes it, or, what the debug print outputs

@jhale
Copy link
Author

jhale commented Mar 11, 2014

A few runs for you...

jack.hale@ijhale:~$ sudo zpool import banana
zfs_close(0x7f8369f07f70): zfs_props 0x7f8369f08210
zfs_close(0x7f8369e00120): zfs_props 0x7f8369e00000
zfs_close(0x7f8369e01660): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369e005a0): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369e00180): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369e015b0): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369e005a0): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369e00180): zfs_props 0x7f8369e009b0
zfs_close(0x7f8369f04f50): zfs_props 0x7f8369f04ce0
zfs_close(0x7f8369d02610): zfs_props 0x7f8369d028b0
zfs_close(0x7f8369d079f0): zfs_props 0x7f8369d07c90
zfs_close(0x7f8369f02af0): zfs_props 0x7f8369f02d90
zfs_close(0x7f8369d08f60): zfs_props 0x7f8369d09200
zfs_close(0x7f8369d06480): zfs_props 0x7f8369d06720
zfs_close(0x7f8369d0a4d0): zfs_props 0x7f8369d0a770
zfs_close(0x7f8369d04f10): zfs_props 0x7f8369d051b0
zfs_close(0x7f8369f01580): zfs_props 0x7f8369f01820
zfs_close(0x7f8369f00050): zfs_props 0x7f8369f00030
jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs_close(0x7fe10ac06e90): zfs_props 0x7fe10ac07130
zfs_close(0x7fe10ad012b0): zfs_props 0x7fe10ad00e30
zfs_close(0x7fe10ad012b0): zfs_props 0x7fe10ad026b0
zfs_close(0x7fe10ad03380): zfs_props 0x7fe10ad020d0
zfs_close(0x7fe10ad03380): zfs_props 0x7fe10ad02f10
zfs_close(0x7fe10ad03380): zfs_props 0x7fe10ad02f10
zfs_close(0x7fe10ad03380): zfs_props 0x7fe10ad02f10
zfs_close(0x7fe10ad03380): zfs_props 0x7fe10ad044d0
zfs_close(0x7fe10ac03df0): zfs_props 0x7fe10ac04060
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana@20140131-new               120K      -   224K  /banana/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs_close(0x7fea93d035b0): zfs_props 0x7fea93d03850
zfs_close(0x7fea93d035b0): zfs_props 0x7fea93d04b00
zfs_close(0x7fea93d035b0): zfs_props 0x7fea93d04960
zfs_close(0x7fea93d068e0): zfs_props 0x7fea93d048f0
zfs(23061,0x7fff7d539310) malloc: *** error for object 0x7fea93d04900: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs_close(0x7fd079c06e90): zfs_props 0x7fd079c07130
zfs_close(0x7fd079c06e90): zfs_props 0x7fd079c083e0
zfs_close(0x7fd079c06e90): zfs_props 0x7fd079c08240
zfs_close(0x7fd079c0a1c0): zfs_props 0x7fd079c081d0
zfs_close(0x7fd079c0a1c0): zfs_props 0x7fd079c083e0
zfs_close(0x7fd079c0a1c0): zfs_props 0x7fd079c083e0
zfs_close(0x7fd079c0a1c0): zfs_props 0x7fd079c083e0
zfs_close(0x7fd079c0a1c0): zfs_props 0x7fd079c0b350
zfs_close(0x7fd079c03df0): zfs_props 0x7fd079c04060
NAME                              USED  AVAIL  REFER  MOUNTPOINT
banana@20140131-new               120K      -   224K  /banana/.zfs/snapshot/20140131-new
banana/code@20140131-new         1.07M      -  1.27G  /banana/code/.zfs/snapshot/20140131-new
banana/documents@20140131-new    14.2G      -  35.5G  /banana/documents/.zfs/snapshot/20140131-new
banana/gaia@20140131-new          120K      -   396M  /banana/gaia/.zfs/snapshot/20140131-new
banana/src@20140131-new           840K      -  4.69G  /banana/src/.zfs/snapshot/20140131-new
jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs_close(0x7fe750d035b0): zfs_props 0x7fe750d03850
zfs_close(0x7fe750d035b0): zfs_props 0x7fe750d04b00
zfs_close(0x7fe750d035b0): zfs_props 0x7fe750d04960
zfs_close(0x7fe750d068e0): zfs_props 0x7fe750d048f0
zfs(23070,0x7fff7d539310) malloc: *** error for object 0x7fe750d04900: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

@lundman
Copy link
Contributor

lundman commented Mar 11, 2014

Well huh, that's not at all as expected.

Now we dump the nvlist as well, could you show the output of a failed lsting

e2f9ae5

@jhale
Copy link
Author

jhale commented Mar 11, 2014

No problem:

jack.hale@ijhale:~$ sudo zfs list -t snapshot
zfs_close(0x7fc3215035b0): zfs_props 0x7fc321503850
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    refcompressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end refcompressratio)

    logicalreferenced = (embedded nvlist)
    nvlist version: 0
        value = 0x126d98400
    (end logicalreferenced)

    used = (embedded nvlist)
    nvlist version: 0
        value = 0x12f84d000
    (end used)

    quota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end quota)

    reservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end reservation)

    compressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end compressratio)

    logicalused = (embedded nvlist)
    nvlist version: 0
        value = 0x126e11600
    (end logicalused)

    usedbysnapshots = (embedded nvlist)
    nvlist version: 0
        value = 0xd2000
    (end usedbysnapshots)

    usedbydataset = (embedded nvlist)
    nvlist version: 0
        value = 0x12f77b000
    (end usedbydataset)

    usedbyrefreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbyrefreservation)

    usedbychildren = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbychildren)

    available = (embedded nvlist)
    nvlist version: 0
        value = 0x15836272000
    (end available)

    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x12f77b000
    (end referenced)

    creation = (embedded nvlist)
    nvlist version: 0
        value = 0x52ebc335
    (end creation)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

    refquota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refquota)

    refreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refreservation)

    guid = (embedded nvlist)
    nvlist version: 0
        value = 0xbd201885718f95a0
    (end guid)

    unique = (embedded nvlist)
    nvlist version: 0
        value = 0x34f3000
    (end unique)

    objsetid = (embedded nvlist)
    nvlist version: 0
        value = 0x5e
    (end objsetid)

    userrefs = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end userrefs)

    defer_destroy = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end defer_destroy)

    written = (embedded nvlist)
    nvlist version: 0
        value = 0x34f3000
    (end written)

    type = (embedded nvlist)
    nvlist version: 0
        value = 0x2
    (end type)

    useraccounting = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end useraccounting)

zfs_close(0x7fc3215035b0): zfs_user_props 0x7fc321504b00
zfs_close(0x7fc3215035b0): zfs_props 0x7fc321504b00
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    refcompressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end refcompressratio)

    logicalreferenced = (embedded nvlist)
    nvlist version: 0
        value = 0x18b36e00
    (end logicalreferenced)

    used = (embedded nvlist)
    nvlist version: 0
        value = 0x18c0a000
    (end used)

    quota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end quota)

    reservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end reservation)

    compressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end compressratio)

    logicalused = (embedded nvlist)
    nvlist version: 0
        value = 0x18b3a800
    (end logicalused)

    usedbysnapshots = (embedded nvlist)
    nvlist version: 0
        value = 0x1e000
    (end usedbysnapshots)

    usedbydataset = (embedded nvlist)
    nvlist version: 0
        value = 0x18bec000
    (end usedbydataset)

    usedbyrefreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbyrefreservation)

    usedbychildren = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbychildren)

    available = (embedded nvlist)
    nvlist version: 0
        value = 0x15836272000
    (end available)

    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x18bec000
    (end referenced)

    creation = (embedded nvlist)
    nvlist version: 0
        value = 0x52ebc3d5
    (end creation)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

    refquota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refquota)

    refreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refreservation)

    guid = (embedded nvlist)
    nvlist version: 0
        value = 0x6789514c8dea6d17
    (end guid)

    unique = (embedded nvlist)
    nvlist version: 0
        value = 0x81000
    (end unique)

    objsetid = (embedded nvlist)
    nvlist version: 0
        value = 0x65
    (end objsetid)

    userrefs = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end userrefs)

    defer_destroy = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end defer_destroy)

    written = (embedded nvlist)
    nvlist version: 0
        value = 0x81000
    (end written)

    type = (embedded nvlist)
    nvlist version: 0
        value = 0x2
    (end type)

    useraccounting = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end useraccounting)

zfs_close(0x7fc3215035b0): zfs_user_props 0x7fc321504960
zfs_close(0x7fc3215035b0): zfs_props 0x7fc321504960
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    refcompressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end refcompressratio)

    logicalreferenced = (embedded nvlist)
    nvlist version: 0
        value = 0x75506000
    (end logicalreferenced)

    used = (embedded nvlist)
    nvlist version: 0
        value = 0x7c873000
    (end used)

    quota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end quota)

    reservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end reservation)

    compressratio = (embedded nvlist)
    nvlist version: 0
        value = 0x64
    (end compressratio)

    logicalused = (embedded nvlist)
    nvlist version: 0
        value = 0x7555f200
    (end logicalused)

    usedbysnapshots = (embedded nvlist)
    nvlist version: 0
        value = 0x111000
    (end usedbysnapshots)

    usedbydataset = (embedded nvlist)
    nvlist version: 0
        value = 0x7c762000
    (end usedbydataset)

    usedbyrefreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbyrefreservation)

    usedbychildren = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end usedbychildren)

    available = (embedded nvlist)
    nvlist version: 0
        value = 0x15836272000
    (end available)

    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x7c762000
    (end referenced)

    creation = (embedded nvlist)
    nvlist version: 0
        value = 0x52eba856
    (end creation)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

    refquota = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refquota)

    refreservation = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = 
    (end refreservation)

    guid = (embedded nvlist)
    nvlist version: 0
        value = 0x8026c81a0ff82a24
    (end guid)

    unique = (embedded nvlist)
    nvlist version: 0
        value = 0x2b7a9000
    (end unique)

    objsetid = (embedded nvlist)
    nvlist version: 0
        value = 0x34
    (end objsetid)

    userrefs = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end userrefs)

    defer_destroy = (embedded nvlist)
    nvlist version: 0
        value = 0x0
    (end defer_destroy)

    written = (embedded nvlist)
    nvlist version: 0
        value = 0x2b7a9000
    (end written)

    type = (embedded nvlist)
    nvlist version: 0
        value = 0x2
    (end type)

    useraccounting = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end useraccounting)

zfs_close(0x7fc3215035b0): zfs_user_props 0x7fc321504b00
zfs_close(0x7fc3215068e0): zfs_props 0x7fc3215048f0
nvlist version: 0
 unknown data type (0)zfs(42727,0x7fff7d539310) malloc: *** error for object 0x7fc321504900: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

@ilovezfs
Copy link
Contributor

what happens if you do

zfs get atime

@jhale
Copy link
Author

jhale commented Mar 11, 2014

stdout:

NAME                             PROPERTY  VALUE  SOURCE
banana                           atime     off    local
banana@20140131-new              atime     -      -
banana/code                      atime     off    inherited from banana
banana/code@20140131-new         atime     -      -
banana/doc                       atime     off    inherited from banana
banana/documents                 atime     off    inherited from banana
banana/documents@20140131-new    atime     -      -
banana/gaia                      atime     off    inherited from banana
banana/gaia@20140131-new         atime     -      -
banana/software                  atime     off    inherited from banana
banana/src                       atime     off    inherited from banana
banana/src@20140131-new          atime     -      -
banana/timemachine               atime     off    inherited from banana
banana/timemachine@20140131-new  atime     -      -
banana/vms                       atime     off    inherited from banana
banana/vms@20140131-new          atime     -      -

stderr:

zfs_close(0x7f9e20403d50): zfs_props 0x7f9e20403bc0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7f9e20403d50): zfs_user_props 0x7f9e20405be0
zfs_close(0x7f9e2040a560): zfs_props 0x7f9e20409ec0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e2040a560): zfs_user_props 0x7f9e2040a460
zfs_close(0x7f9e204077b0): zfs_props 0x7f9e204057a0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7f9e204077b0): zfs_user_props 0x7f9e20407200
zfs_close(0x7f9e20407d50): zfs_props 0x7f9e20407760
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20407d50): zfs_user_props 0x7f9e20408600
zfs_close(0x7f9e20409f30): zfs_props 0x7f9e20409ea0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7f9e20409f30): zfs_user_props 0x7f9e20409940
zfs_close(0x7f9e20408050): zfs_props 0x7f9e20407ff0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7f9e20408050): zfs_user_props 0x7f9e20404680
zfs_close(0x7f9e20408620): zfs_props 0x7f9e20408c10
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20408620): zfs_user_props 0x7f9e20408e70
zfs_close(0x7f9e204054b0): zfs_props 0x7f9e20405720
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7f9e204054b0): zfs_user_props 0x7f9e20405ac0
zfs_close(0x7f9e20407270): zfs_props 0x7f9e20405780
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20407270): zfs_user_props 0x7f9e20407d30
zfs_close(0x7f9e20408920): zfs_props 0x7f9e20408bc0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7f9e20408920): zfs_user_props 0x7f9e20408e50
zfs_close(0x7f9e20405090): zfs_props 0x7f9e20405b80
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7f9e20405090): zfs_user_props 0x7f9e20405ae0
zfs_close(0x7f9e20406db0): zfs_props 0x7f9e20405b00
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20406db0): zfs_user_props 0x7f9e204046a0
zfs_close(0x7f9e20409260): zfs_props 0x7f9e204091b0
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7f9e20409260): zfs_user_props 0x7f9e2040a280
zfs_close(0x7f9e20409b40): zfs_props 0x7f9e20409160
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20409b40): zfs_user_props 0x7f9e2040a4c0
zfs_close(0x7f9e20408e90): zfs_props 0x7f9e20408e30
nvlist version: 0
    atime = (embedded nvlist)
    nvlist version: 0
        value = 0x0
        source = banana
    (end atime)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7f9e20408e90): zfs_user_props 0x7f9e20404640
zfs_close(0x7f9e20409570): zfs_props 0x7f9e20409980
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f9e20409570): zfs_user_props 0x7f9e20407700

@ilovezfs
Copy link
Contributor

zfs get name
zfs get used
zfs get mountpoint
zfs get available
zfs get referenced
zfs get mountpoint

please

@jhale
Copy link
Author

jhale commented Mar 11, 2014

zfs_close(0x7f8430403d50): zfs_props 0x7f8430403bc0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7f8430403d50): zfs_user_props 0x7f8430405be0
zfs_close(0x7f8430409ea0): zfs_props 0x7f8430407ae0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8430409ea0): zfs_user_props 0x7f8430404620
zfs_close(0x7f84304071b0): zfs_props 0x7f84304074a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7f84304071b0): zfs_user_props 0x7f8430404660
zfs_close(0x7f84304054b0): zfs_props 0x7f84304074c0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f84304054b0): zfs_user_props 0x7f8430404060
zfs_close(0x7f8430409900): zfs_props 0x7f8430409080
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7f8430409900): zfs_user_props 0x7f8430409cd0
zfs_close(0x7f8430407d60): zfs_props 0x7f84304074e0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7f8430407d60): zfs_user_props 0x7f8430407c60
zfs_close(0x7f84304081f0): zfs_props 0x7f84304088c0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f84304081f0): zfs_user_props 0x7f84304076e0
zfs_close(0x7f84304057a0): zfs_props 0x7f8430405b20
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7f84304057a0): zfs_user_props 0x7f8430404680
zfs_close(0x7f8430404230): zfs_props 0x7f8430407970
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8430404230): zfs_user_props 0x7f8430407530
zfs_close(0x7f8430408540): zfs_props 0x7f84304084e0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7f8430408540): zfs_user_props 0x7f8430407130
zfs_close(0x7f8430405090): zfs_props 0x7f8430405b50
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7f8430405090): zfs_user_props 0x7f84304046a0
zfs_close(0x7f8430406db0): zfs_props 0x7f8430405b00
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8430406db0): zfs_user_props 0x7f8430404080
zfs_close(0x7f8430408e10): zfs_props 0x7f8430407910
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7f8430408e10): zfs_user_props 0x7f8430405aa0
zfs_close(0x7f84304095e0): zfs_props 0x7f8430409480
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f84304095e0): zfs_user_props 0x7f8430407680
zfs_close(0x7f8430408a50): zfs_props 0x7f8430408490
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7f8430408a50): zfs_user_props 0x7f8430407cf0
zfs_close(0x7f8430409210): zfs_props 0x7f8430409500
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8430409210): zfs_user_props 0x7f8430408dd0
NAME                             PROPERTY  VALUE                            SOURCE
banana                           name      banana                           -
banana@20140131-new              name      banana@20140131-new              -
banana/code                      name      banana/code                      -
banana/code@20140131-new         name      banana/code@20140131-new         -
banana/doc                       name      banana/doc                       -
banana/documents                 name      banana/documents                 -
banana/documents@20140131-new    name      banana/documents@20140131-new    -
banana/gaia                      name      banana/gaia                      -
banana/gaia@20140131-new         name      banana/gaia@20140131-new         -
banana/software                  name      banana/software                  -
banana/src                       name      banana/src                       -
banana/src@20140131-new          name      banana/src@20140131-new          -
banana/timemachine               name      banana/timemachine               -
banana/timemachine@20140131-new  name      banana/timemachine@20140131-new  -
banana/vms                       name      banana/vms                       -
banana/vms@20140131-new          name      banana/vms@20140131-new          -
zfs_close(0x7f8473e00260): zfs_props 0x7f8473e004d0
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x6c99d8b000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7f8473e00260): zfs_user_props 0x7f8473e02150
zfs_close(0x7f8473e06a50): zfs_props 0x7f8473e06080
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x1e000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e06a50): zfs_user_props 0x7f8473e062d0
zfs_close(0x7f8473e03d10): zfs_props 0x7f8473e03c60
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x7c873000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7f8473e03d10): zfs_user_props 0x7f8473e049d0
zfs_close(0x7f8473e037c0): zfs_props 0x7f8473e04940
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x111000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e037c0): zfs_user_props 0x7f8473e00b40
zfs_close(0x7f8473e06680): zfs_props 0x7f8473e069c0
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x19ab1000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7f8473e06680): zfs_user_props 0x7f8473e062f0
zfs_close(0x7f8473e042f0): zfs_props 0x7f8473e04960
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x8f3754000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7f8473e042f0): zfs_user_props 0x7f8473e04620
zfs_close(0x7f8473e04a10): zfs_props 0x7f8473e05300
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x38c2cc000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e04a10): zfs_user_props 0x7f8473e04860
zfs_close(0x7f8473e01cc0): zfs_props 0x7f8473e01f30
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x18c0a000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7f8473e01cc0): zfs_user_props 0x7f8473e04090
zfs_close(0x7f8473e01a20): zfs_props 0x7f8473e006e0
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x1e000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e01a20): zfs_user_props 0x7f8473e042d0
zfs_close(0x7f8473e04c80): zfs_props 0x7f8473e04600
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x39fb3a000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7f8473e04c80): zfs_user_props 0x7f8473e00b20
zfs_close(0x7f8473e01600): zfs_props 0x7f8473e020f0
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x12f84d000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7f8473e01600): zfs_user_props 0x7f8473e02050
zfs_close(0x7f8473e007c0): zfs_props 0x7f8473e02030
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0xd2000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e007c0): zfs_user_props 0x7f8473e00c10
zfs_close(0x7f8473e059b0): zfs_props 0x7f8473e04190
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x4e130bb000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7f8473e059b0): zfs_user_props 0x7f8473e05960
zfs_close(0x7f8473e06330): zfs_props 0x7f8473e069a0
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0xb47af000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e06330): zfs_user_props 0x7f8473e060a0
zfs_close(0x7f8473e05360): zfs_props 0x7f8473e05290
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x10148fa000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7f8473e05360): zfs_user_props 0x7f8473e052b0
zfs_close(0x7f8473e05c80): zfs_props 0x7f8473e04f20
nvlist version: 0
    used = (embedded nvlist)
    nvlist version: 0
        value = 0x2e000
    (end used)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7f8473e05c80): zfs_user_props 0x7f8473e06310
NAME                             PROPERTY  VALUE  SOURCE
banana                           used      434G   -
banana@20140131-new              used      120K   -
banana/code                      used      1.95G  -
banana/code@20140131-new         used      1.07M  -
banana/doc                       used      411M   -
banana/documents                 used      35.8G  -
banana/documents@20140131-new    used      14.2G  -
banana/gaia                      used      396M   -
banana/gaia@20140131-new         used      120K   -
banana/software                  used      14.5G  -
banana/src                       used      4.74G  -
banana/src@20140131-new          used      840K   -
banana/timemachine               used      312G   -
banana/timemachine@20140131-new  used      2.82G  -
banana/vms                       used      64.3G  -
banana/vms@20140131-new          used      184K   -
zfs_close(0x7febf2500410): zfs_props 0x7febf2500680
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7febf2500410): zfs_user_props 0x7febf2502300
zfs_close(0x7febf25073f0): zfs_props 0x7febf2500910
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25073f0): zfs_user_props 0x7febf25040d0
zfs_close(0x7febf25043f0): zfs_props 0x7febf2500d70
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7febf25043f0): zfs_user_props 0x7febf2504140
zfs_close(0x7febf2504c70): zfs_props 0x7febf2505320
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2504c70): zfs_user_props 0x7febf25055b0
zfs_close(0x7febf2506c30): zfs_props 0x7febf25067a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7febf2506c30): zfs_user_props 0x7febf2503b40
zfs_close(0x7febf2504fb0): zfs_props 0x7febf2505340
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7febf2504fb0): zfs_user_props 0x7febf2504120
zfs_close(0x7febf25055d0): zfs_props 0x7febf2505360
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25055d0): zfs_user_props 0x7febf2505570
zfs_close(0x7febf2500a80): zfs_props 0x7febf2500d20
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7febf2500a80): zfs_user_props 0x7febf2504bb0
zfs_close(0x7febf25036f0): zfs_props 0x7febf2503990
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25036f0): zfs_user_props 0x7febf2504740
zfs_close(0x7febf2505840): zfs_props 0x7febf2505ab0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7febf2505840): zfs_user_props 0x7febf2500a10
zfs_close(0x7febf25006a0): zfs_props 0x7febf25036d0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7febf25006a0): zfs_user_props 0x7febf2500dc0
zfs_close(0x7febf2503ca0): zfs_props 0x7febf2504370
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2503ca0): zfs_user_props 0x7febf2502200
zfs_close(0x7febf25069c0): zfs_props 0x7febf25062a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7febf25069c0): zfs_user_props 0x7febf25040f0
zfs_close(0x7febf2507180): zfs_props 0x7febf2506f00
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2507180): zfs_user_props 0x7febf2503b60
zfs_close(0x7febf2505ce0): zfs_props 0x7febf25062d0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7febf2505ce0): zfs_user_props 0x7febf2506260
zfs_close(0x7febf2505f50): zfs_props 0x7febf2506280
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2505f50): zfs_user_props 0x7febf2503b80
NAME                             PROPERTY    VALUE                                           SOURCE
banana                           mountpoint  /banana                                         default
banana@20140131-new              mountpoint  /banana/.zfs/snapshot/20140131-new              default
banana/code                      mountpoint  /banana/code                                    default
banana/code@20140131-new         mountpoint  /banana/code/.zfs/snapshot/20140131-new         default
banana/doc                       mountpoint  /banana/doc                                     default
banana/documents                 mountpoint  /banana/documents                               default
banana/documents@20140131-new    mountpoint  /banana/documents/.zfs/snapshot/20140131-new    default
banana/gaia                      mountpoint  /banana/gaia                                    default
banana/gaia@20140131-new         mountpoint  /banana/gaia/.zfs/snapshot/20140131-new         default
banana/software                  mountpoint  /banana/software                                default
banana/src                       mountpoint  /banana/src                                     default
banana/src@20140131-new          mountpoint  /banana/src/.zfs/snapshot/20140131-new          default
banana/timemachine               mountpoint  /banana/timemachine                             default
banana/timemachine@20140131-new  mountpoint  /banana/timemachine/.zfs/snapshot/20140131-new  default
banana/vms                       mountpoint  /banana/vms                                     default
banana/vms@20140131-new          mountpoint  /banana/vms/.zfs/snapshot/20140131-new          default
zfs_close(0x7febf2500410): zfs_props 0x7febf2500680
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7febf2500410): zfs_user_props 0x7febf2502300
zfs_close(0x7febf25073f0): zfs_props 0x7febf2500910
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25073f0): zfs_user_props 0x7febf25040d0
zfs_close(0x7febf25043f0): zfs_props 0x7febf2500d70
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7febf25043f0): zfs_user_props 0x7febf2504140
zfs_close(0x7febf2504c70): zfs_props 0x7febf2505320
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2504c70): zfs_user_props 0x7febf25055b0
zfs_close(0x7febf2506c30): zfs_props 0x7febf25067a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7febf2506c30): zfs_user_props 0x7febf2503b40
zfs_close(0x7febf2504fb0): zfs_props 0x7febf2505340
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7febf2504fb0): zfs_user_props 0x7febf2504120
zfs_close(0x7febf25055d0): zfs_props 0x7febf2505360
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25055d0): zfs_user_props 0x7febf2505570
zfs_close(0x7febf2500a80): zfs_props 0x7febf2500d20
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7febf2500a80): zfs_user_props 0x7febf2504bb0
zfs_close(0x7febf25036f0): zfs_props 0x7febf2503990
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf25036f0): zfs_user_props 0x7febf2504740
zfs_close(0x7febf2505840): zfs_props 0x7febf2505ab0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7febf2505840): zfs_user_props 0x7febf2500a10
zfs_close(0x7febf25006a0): zfs_props 0x7febf25036d0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7febf25006a0): zfs_user_props 0x7febf2500dc0
zfs_close(0x7febf2503ca0): zfs_props 0x7febf2504370
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2503ca0): zfs_user_props 0x7febf2502200
zfs_close(0x7febf25069c0): zfs_props 0x7febf25062a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7febf25069c0): zfs_user_props 0x7febf25040f0
zfs_close(0x7febf2507180): zfs_props 0x7febf2506f00
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2507180): zfs_user_props 0x7febf2503b60
zfs_close(0x7febf2505ce0): zfs_props 0x7febf25062d0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7febf2505ce0): zfs_user_props 0x7febf2506260
zfs_close(0x7febf2505f50): zfs_props 0x7febf2506280
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7febf2505f50): zfs_user_props 0x7febf2503b80
NAME                             PROPERTY    VALUE                                           SOURCE
banana                           mountpoint  /banana                                         default
banana@20140131-new              mountpoint  /banana/.zfs/snapshot/20140131-new              default
banana/code                      mountpoint  /banana/code                                    default
banana/code@20140131-new         mountpoint  /banana/code/.zfs/snapshot/20140131-new         default
banana/doc                       mountpoint  /banana/doc                                     default
banana/documents                 mountpoint  /banana/documents                               default
banana/documents@20140131-new    mountpoint  /banana/documents/.zfs/snapshot/20140131-new    default
banana/gaia                      mountpoint  /banana/gaia                                    default
banana/gaia@20140131-new         mountpoint  /banana/gaia/.zfs/snapshot/20140131-new         default
banana/software                  mountpoint  /banana/software                                default
banana/src                       mountpoint  /banana/src                                     default
banana/src@20140131-new          mountpoint  /banana/src/.zfs/snapshot/20140131-new          default
banana/timemachine               mountpoint  /banana/timemachine                             default
banana/timemachine@20140131-new  mountpoint  /banana/timemachine/.zfs/snapshot/20140131-new  default
banana/vms                       mountpoint  /banana/vms                                     default
banana/vms@20140131-new          mountpoint  /banana/vms/.zfs/snapshot/20140131-new          default
zfs_close(0x7fb38b500410): zfs_props 0x7fb38b500680
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7fb38b500410): zfs_user_props 0x7fb38b502300
zfs_close(0x7fb38b506f70): zfs_props 0x7fb38b506320
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b506f70): zfs_user_props 0x7fb38b5047c0
zfs_close(0x7fb38b503f80): zfs_props 0x7fb38b503ed0
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7fb38b503f80): zfs_user_props 0x7fb38b5044a0
zfs_close(0x7fb38b504510): zfs_props 0x7fb38b504bb0
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b504510): zfs_user_props 0x7fb38b504e40
zfs_close(0x7fb38b506950): zfs_props 0x7fb38b501810
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7fb38b506950): zfs_user_props 0x7fb38b506e80
zfs_close(0x7fb38b504900): zfs_props 0x7fb38b504b70
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7fb38b504900): zfs_user_props 0x7fb38b504e20
zfs_close(0x7fb38b504e60): zfs_props 0x7fb38b504b90
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b504e60): zfs_user_props 0x7fb38b504d50
zfs_close(0x7fb38b500b50): zfs_props 0x7fb38b500ad0
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7fb38b500b50): zfs_user_props 0x7fb38b503920
zfs_close(0x7fb38b503a20): zfs_props 0x7fb38b504260
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b503a20): zfs_user_props 0x7fb38b5044c0
zfs_close(0x7fb38b5051e0): zfs_props 0x7fb38b500990
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7fb38b5051e0): zfs_user_props 0x7fb38b501830
zfs_close(0x7fb38b501bd0): zfs_props 0x7fb38b5022a0
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7fb38b501bd0): zfs_user_props 0x7fb38b500dc0
zfs_close(0x7fb38b501e40): zfs_props 0x7fb38b502250
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b501e40): zfs_user_props 0x7fb38b500840
zfs_close(0x7fb38b505b40): zfs_props 0x7fb38b505a10
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7fb38b505b40): zfs_user_props 0x7fb38b502110
zfs_close(0x7fb38b506560): zfs_props 0x7fb38b504c40
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b506560): zfs_user_props 0x7fb38b506ea0
zfs_close(0x7fb38b5057a0): zfs_props 0x7fb38b504bd0
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7fb38b5057a0): zfs_user_props 0x7fb38b505de0
zfs_close(0x7fb38b506010): zfs_props 0x7fb38b506280
nvlist version: 0
    available = (embedded nvlist)
    nvlist version: 0
        value = 0x1583626c000
    (end available)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fb38b506010): zfs_user_props 0x7fb38b504bf0
NAME                             PROPERTY   VALUE  SOURCE
banana                           available  1.34T  -
banana@20140131-new              available  -      -
banana/code                      available  1.34T  -
banana/code@20140131-new         available  -      -
banana/doc                       available  1.34T  -
banana/documents                 available  1.34T  -
banana/documents@20140131-new    available  -      -
banana/gaia                      available  1.34T  -
banana/gaia@20140131-new         available  -      -
banana/software                  available  1.34T  -
banana/src                       available  1.34T  -
banana/src@20140131-new          available  -      -
banana/timemachine               available  1.34T  -
banana/timemachine@20140131-new  available  -      -
banana/vms                       available  1.34T  -
banana/vms@20140131-new          available  -      -
zfs_close(0x7ff3e9403d50): zfs_props 0x7ff3e9403bc0
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0xe8000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7ff3e9403d50): zfs_user_props 0x7ff3e9405be0
zfs_close(0x7ff3e940a8e0): zfs_props 0x7ff3e9409b40
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x38000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e940a8e0): zfs_user_props 0x7ff3e940a020
zfs_close(0x7ff3e9407880): zfs_props 0x7ff3e9404230
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x7c762000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7ff3e9407880): zfs_user_props 0x7ff3e94053e0
zfs_close(0x7ff3e9407bf0): zfs_props 0x7ff3e94084b0
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x510ca000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e9407bf0): zfs_user_props 0x7ff3e94053c0
zfs_close(0x7ff3e940a210): zfs_props 0x7ff3e9408e30
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x19ab1000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7ff3e940a210): zfs_user_props 0x7ff3e940a770
zfs_close(0x7ff3e9407e60): zfs_props 0x7ff3e9404150
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x567488000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7ff3e9407e60): zfs_user_props 0x7ff3e94050f0
zfs_close(0x7ff3e9408540): zfs_props 0x7ff3e94084d0
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x8e1b89000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e9408540): zfs_user_props 0x7ff3e94044a0
zfs_close(0x7ff3e9406db0): zfs_props 0x7ff3e9404390
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x18bec000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7ff3e9406db0): zfs_user_props 0x7ff3e94046a0
zfs_close(0x7ff3e94074c0): zfs_props 0x7ff3e9407b60
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x18b89000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e94074c0): zfs_user_props 0x7ff3e9407840
zfs_close(0x7ff3e9408ae0): zfs_props 0x7ff3e9408340
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x39fb3a000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7ff3e9408ae0): zfs_user_props 0x7ff3e9408270
zfs_close(0x7ff3e94054b0): zfs_props 0x7ff3e9405b50
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x12f77b000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7ff3e94054b0): zfs_user_props 0x7ff3e9405ae0
zfs_close(0x7ff3e9405720): zfs_props 0x7ff3e9405b00
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x12c35a000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e9405720): zfs_user_props 0x7ff3e9404420
zfs_close(0x7ff3e94094d0): zfs_props 0x7ff3e9409740
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x4d5e90c000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7ff3e94094d0): zfs_user_props 0x7ff3e9404350
zfs_close(0x7ff3e9409c70): zfs_props 0x7ff3e9409b20
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x445e279000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e9409c70): zfs_user_props 0x7ff3e940a790
zfs_close(0x7ff3e9408e90): zfs_props 0x7ff3e9404460
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0x10148cc000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7ff3e9408e90): zfs_user_props 0x7ff3e94045d0
zfs_close(0x7ff3e9409230): zfs_props 0x7ff3e9404480
nvlist version: 0
    referenced = (embedded nvlist)
    nvlist version: 0
        value = 0xfff0e2000
    (end referenced)

    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7ff3e9409230): zfs_user_props 0x7ff3e9409b70
NAME                             PROPERTY    VALUE  SOURCE
banana                           referenced  928K   -
banana@20140131-new              referenced  224K   -
banana/code                      referenced  1.94G  -
banana/code@20140131-new         referenced  1.27G  -
banana/doc                       referenced  411M   -
banana/documents                 referenced  21.6G  -
banana/documents@20140131-new    referenced  35.5G  -
banana/gaia                      referenced  396M   -
banana/gaia@20140131-new         referenced  396M   -
banana/software                  referenced  14.5G  -
banana/src                       referenced  4.74G  -
banana/src@20140131-new          referenced  4.69G  -
banana/timemachine               referenced  309G   -
banana/timemachine@20140131-new  referenced  273G   -
banana/vms                       referenced  64.3G  -
banana/vms@20140131-new          referenced  64.0G  -
zfs_close(0x7fd559c03d50): zfs_props 0x7fd559c03bc0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1
    (end createtxg)

zfs_close(0x7fd559c03d50): zfs_user_props 0x7fd559c05be0
zfs_close(0x7fd559c0a9f0): zfs_props 0x7fd559c07930
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c0a9f0): zfs_user_props 0x7fd559c07490
zfs_close(0x7fd559c080c0): zfs_props 0x7fd559c07240
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x21c
    (end createtxg)

zfs_close(0x7fd559c080c0): zfs_user_props 0x7fd559c08050
zfs_close(0x7fd559c084c0): zfs_props 0x7fd559c08c80
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c084c0): zfs_user_props 0x7fd559c08870
zfs_close(0x7fd559c0a4f0): zfs_props 0x7fd559c08b20
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x2d64
    (end createtxg)

zfs_close(0x7fd559c0a4f0): zfs_user_props 0x7fd559c0a190
zfs_close(0x7fd559c088b0): zfs_props 0x7fd559c07290
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1b
    (end createtxg)

zfs_close(0x7fd559c088b0): zfs_user_props 0x7fd559c08c30
zfs_close(0x7fd559c08dc0): zfs_props 0x7fd559c09470
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c08dc0): zfs_user_props 0x7fd559c04330
zfs_close(0x7fd559c043a0): zfs_props 0x7fd559c04640
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x14e5
    (end createtxg)

zfs_close(0x7fd559c043a0): zfs_user_props 0x7fd559c07eb0
zfs_close(0x7fd559c06fd0): zfs_props 0x7fd559c08430
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c06fd0): zfs_user_props 0x7fd559c07c00
zfs_close(0x7fd559c09090): zfs_props 0x7fd559c09490
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x16346
    (end createtxg)

zfs_close(0x7fd559c09090): zfs_user_props 0x7fd559c07e90
zfs_close(0x7fd559c03fc0): zfs_props 0x7fd559c059a0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x1477
    (end createtxg)

zfs_close(0x7fd559c03fc0): zfs_user_props 0x7fd559c05ae0
zfs_close(0x7fd559c07580): zfs_props 0x7fd559c07c50
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c07580): zfs_user_props 0x7fd559c07ed0
zfs_close(0x7fd559c09d20): zfs_props 0x7fd559c09ff0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x658
    (end createtxg)

zfs_close(0x7fd559c09d20): zfs_user_props 0x7fd559c0a260
zfs_close(0x7fd559c0a280): zfs_props 0x7fd559c0a010
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c0a280): zfs_user_props 0x7fd559c0a240
zfs_close(0x7fd559c095b0): zfs_props 0x7fd559c094b0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x24b
    (end createtxg)

zfs_close(0x7fd559c095b0): zfs_user_props 0x7fd559c04290
zfs_close(0x7fd559c09820): zfs_props 0x7fd559c09fd0
nvlist version: 0
    createtxg = (embedded nvlist)
    nvlist version: 0
        value = 0x156b
    (end createtxg)

zfs_close(0x7fd559c09820): zfs_user_props 0x7fd559c09c40
NAME                             PROPERTY    VALUE                                           SOURCE
banana                           mountpoint  /banana                                         default
banana@20140131-new              mountpoint  /banana/.zfs/snapshot/20140131-new              default
banana/code                      mountpoint  /banana/code                                    default
banana/code@20140131-new         mountpoint  /banana/code/.zfs/snapshot/20140131-new         default
banana/doc                       mountpoint  /banana/doc                                     default
banana/documents                 mountpoint  /banana/documents                               default
banana/documents@20140131-new    mountpoint  /banana/documents/.zfs/snapshot/20140131-new    default
banana/gaia                      mountpoint  /banana/gaia                                    default
banana/gaia@20140131-new         mountpoint  /banana/gaia/.zfs/snapshot/20140131-new         default
banana/software                  mountpoint  /banana/software                                default
banana/src                       mountpoint  /banana/src                                     default
banana/src@20140131-new          mountpoint  /banana/src/.zfs/snapshot/20140131-new          default
banana/timemachine               mountpoint  /banana/timemachine                             default
banana/timemachine@20140131-new  mountpoint  /banana/timemachine/.zfs/snapshot/20140131-new  default
banana/vms                       mountpoint  /banana/vms                                     default
banana/vms@20140131-new          mountpoint  /banana/vms/.zfs/snapshot/20140131-new          default

@ilovezfs
Copy link
Contributor

I wonder if it could be related to one of the properties retrieved but not listed. Perhaps if you run

zfs get all

it can trigger it.

@jhale
Copy link
Author

jhale commented Mar 11, 2014

ilovezfs added a commit that referenced this issue Mar 14, 2014
@ilovezfs
Copy link
Contributor

@jhale Can you please try again with 20312c8

@jhale
Copy link
Author

jhale commented Mar 14, 2014

Here you go:

http://pastebin.com/gcpSXg4v

@lundman
Copy link
Contributor

lundman commented Apr 3, 2014

Thanks to dch for letting me have access to his broken pool, I have spent a while checking over our userland code. It appears to be all ok. This suggests the problem is in the kernel where we build the nvlists to return.

Once interesting thing I noticed was that any combination of props would work;

zfs list -t all -o  referenced,name,used,available,compressratio,mounted,quota,recordsize,compression,atime,checksum,readonly,canmount,dedup,written

as long as the list did not contain mountpoint. If it had mountpoint, it would frequently core.

Since we did do some changes to let our snapshots have mountpoints (due to the way we have to mount them) it could be that we fail to fill in the mountpoint property under some circumstances.

lundman added a commit that referenced this issue Apr 3, 2014
causing amusing mayhem. Hopefully addresses issue #123
@ilovezfs ilovezfs closed this as completed Apr 3, 2014
@jhale
Copy link
Author

jhale commented Apr 3, 2014

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants