NFS服務運行一段時間後導致客戶端被卡死的問題分析及解決 原

問題描述

在現場環境中，多個節點上運行CentOS7 X64發行版，其中一個節點上運行NFS服務，其他節點作爲NFS客戶端，共享訪問NFS服務節點上的文件。系統上還運行有其他很多軟件，但是由於系統龐大，參與開發的人員較多，我們無法通過詢問得知具體情況（通過多次詢問，才得知他們NFS的服務端根目錄使用過多次bind操作）。但是系統中出現NFS服務有時候運行30分鐘左右會導致NFS客戶端被卡死，且無法通過Kill -9終止的現象，期間也沒有錯誤日誌，也無法在現場環境進行。

初步分析

• 在此之前，我們對NFS的瞭解也僅限於簡單的使用，所以此時我們應快速瞭解其基本架構、原理，以便進一步分析。

• 經初步資料查閱發下：NFS是一個網絡文件系統，其關鍵部分運行在內核中，用戶態只有一個簡單的守護進程。多節點之間使用基於TCP協議的Sun RPC進行通訊，使用portmap服務進行服務發現。

• 使用kill -9都無法終止程序，說明程序是在內核態產生死鎖。對於上層而言，NFS和普通文件系統並無差別，關鍵應該是在執行底層的RPC調用時被阻塞。根據經驗，我們可以初步推測，這個死鎖應該是發生在RPC遠程調用的等待時。

• 進一步查閱資料發現：NFS的mount有soft和hard模式之分，其錯誤時等待方式不同，問題是否出在這裏？下面我們將從這裏入手，並配合一定的測試，來初步瞭解其特性和調試手段，然後再進行有針對性的測試來暴露問題，最終找到問題及其解決方法。

獲取源碼包

準備環境

$ yum install rpmdevtools yum-utils
$ rpmdev-setuptree

查看軟件包信息

$ which nfsstat 
/usr/sbin/nfsstat

$ rpm -qf /usr/sbin/nfsstat
nfs-utils-1.3.0-0.21.el7_2.1.x86_64

$ yum info nfs-utils
...
已安裝的軟件包
名稱    ：nfs-utils
架構    ：x86_64
時期       ：1
版本    ：1.3.0
發佈    ：0.21.el7_2.1
大小    ：1.0 M
源    ：installed
來自源：updates
簡介    ： NFS utilities and supporting clients and daemons for the kernel NFS server
網址    ：http://sourceforge.net/projects/nfs
協議    ： MIT and GPLv2 and GPLv2+ and BSD
描述    ： The nfs-utils package provides a daemon for the kernel NFS server and
         : related tools, which provides a much higher level of performance than the
         : traditional Linux NFS server used by most users.
         : 
         : This package also contains the showmount program.  Showmount queries the
         : mount daemon on a remote host for information about the NFS (Network File
         : System) server on the remote host.  For example, showmount can display the
         : clients which are mounted on that host.
         : 
         : This package also contains the mount.nfs and umount.nfs program.
...

下載源碼包

$ cd ~/rpmbuild/
$ yumdownloader --source nfs-utils

$ rpm -ivh nfs-utils-1.3.0-0.21.el7_2.1.src.rpm

編譯源碼包

提取源碼

$ yum install libevent-devel libnfsidmap-devel libtirpc-devel fedfs-utils-devel
$ rpmbuild -bp SPECS/nfs-utils.spec

$ mkdir -pv /opt/nfs
$ cp -rv BUILD/nfs-utils-1.3.0/ /opt/nfs/

配置

$ cd /opt/nfs/nfs-utils-1.3.0/
$ ./autogen.sh
$ ./configure \
	--build=x86_64-redhat-linux-gnu \
	--host=x86_64-redhat-linux-gnu \
	--disable-dependency-tracking \
	--enable-mountconfig \
	--enable-ipv6 \
	--enable-libmount-mount \
	--prefix=/opt/nfs/target \
'CFLAGS=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fpie -Wl,-z,relro,-z,now -D_FILE_OFFSET_BITS=64' \
CPPFLAGS= \
LDFLAGS=-pie

編譯安裝

$ make all
$ make install

查看目標文件

$ cd ../target/
$ tree
.
├── sbin
│   ├── blkmapd			pNFS的塊映射服務。
│   ├── exportfs			NFS文件系統導出。
│   ├── mountstats		查看mount的文件系統的統計信息。
│   ├── nfsdcltrack		NFS狀態持久化工具。
│   ├── nfsidmap			NFS的ID映射服務。
│   ├── nfsiostat		查看IO狀態。
│   ├── nfsstat			查看NFS狀態。
│   ├── rpcdebug			調試功能能設置工具。
│   ├── rpc.gssd			GSS認證服務。
│   ├── rpc.idmapd
│   ├── rpc.mountd
│   ├── rpc.nfsd
│   ├── rpc.statd
│   ├── rpc.svcgssd
│   ├── showmount		顯示NFS服務器mount信息。
│   ├── sm-notify		向NFS端點發送重啓信息。
│   └── start-statd
└── share
    └── man
        ├── man5
        │   ├── exports.5
        │   ├── nfs.5
        │   └── nfsmount.conf.5
        ├── man7
        │   └── nfsd.7
        └── man8
            ├── blkmapd.8
            ├── exportfs.8
            ├── gssd.8
            ├── idmapd.8
            ├── mountd.8
            ├── mount.nfs.8
            ├── mountstats.8
            ├── nfsd.8
            ├── nfsdcltrack.8
            ├── nfsidmap.8
            ├── nfsiostat.8
            ├── nfsstat.8
            ├── rpcdebug.8
            ├── rpc.gssd.8 -> gssd.8
            ├── rpc.idmapd.8 -> idmapd.8
            ├── rpc.mountd.8 -> mountd.8
            ├── rpc.nfsd.8 -> nfsd.8
            ├── rpc.sm-notify.8 -> sm-notify.8
            ├── rpc.statd.8 -> statd.8
            ├── rpc.svcgssd.8 -> svcgssd.8
            ├── showmount.8
            ├── sm-notify.8
            ├── statd.8
            ├── svcgssd.8
            └── umount.nfs.8

6 directories, 46 files

hard與soft模式異常測試

NFS服務器爲192.168.195.136，配置如下：

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)

服務端開啓NFS調試信息：

$ rpcdebug -m nfsd -s all

客戶端開啓NFS調試信息：

$ rpcdebug -m nfs -s all

使用hard模式mount

$ mount -t nfs 192.168.195.136:/opt /opt -osoft,timeo=5,retrans=2

網絡正常時，查看ls命令的日誌

$ ls /opt
$ dmesg -c
[1109522.499944] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109522.499947] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109522.500454] decode_attr_files_avail: files avail=5126859
[1109522.500456] decode_attr_files_free: files free=5126859
[1109522.500457] decode_attr_files_total: files total=5242880
[1109522.500458] decode_attr_space_avail: space avail=48261320704
[1109522.500458] decode_attr_space_free: space free=52573065216
[1109522.500459] decode_attr_space_total: space total=84417077248
[1109522.500460] decode_statfs: xdr returned 0!
[1109522.500462] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.467099] NFS: revalidating (0:44/2)
[1109523.467108] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.467109] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.467617] decode_attr_type: type=040000
[1109523.467619] decode_attr_change: change attribute=6364596956768490675
[1109523.467620] decode_attr_size: file size=4096
[1109523.467621] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.467621] decode_attr_fileid: fileid=2
[1109523.467622] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.467623] decode_attr_mode: file mode=0755
[1109523.467624] decode_attr_nlink: nlink=21
[1109523.467626] decode_attr_owner: uid=0
[1109523.467626] decode_attr_group: gid=0
[1109523.467627] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.467627] decode_attr_space_used: space used=4096
[1109523.467628] decode_attr_time_access: atime=1481873345
[1109523.467629] decode_attr_time_metadata: ctime=1481873206
[1109523.467629] decode_attr_time_modify: mtime=1481873206
[1109523.467630] decode_attr_mounted_on_fileid: fileid=524289
[1109523.467630] decode_getfattr_attrs: xdr returned 0
[1109523.467631] decode_getfattr_generic: xdr returned 0
[1109523.467633] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.467639] NFS: nfs_update_inode(0:44/2 fh_crc=0x41dc8638 ct=2 info=0x427e7f)
[1109523.467641] NFS: (0:44/2) revalidation complete
[1109523.467658] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.467659] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.468301] decode_attr_type: type=00
[1109523.468302] decode_attr_change: change attribute=6364596956768490675
[1109523.468303] decode_attr_size: file size=4096
[1109523.468304] decode_attr_fsid: fsid=(0x0/0x0)
[1109523.468305] decode_attr_fileid: fileid=0
[1109523.468305] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.468306] decode_attr_mode: file mode=00
[1109523.468306] decode_attr_nlink: nlink=1
[1109523.468307] decode_attr_owner: uid=-2
[1109523.468308] decode_attr_group: gid=-2
[1109523.468308] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.468309] decode_attr_space_used: space used=0
[1109523.468309] decode_attr_time_access: atime=0
[1109523.468333] decode_attr_time_metadata: ctime=1481873206
[1109523.468335] decode_attr_time_modify: mtime=1481873206
[1109523.468335] decode_attr_mounted_on_fileid: fileid=0
[1109523.468336] decode_getfattr_attrs: xdr returned 0
[1109523.468337] decode_getfattr_generic: xdr returned 0
[1109523.468338] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.468343] NFS: nfs_update_inode(0:44/2 fh_crc=0x41dc8638 ct=2 info=0x26040)
[1109523.468346] NFS: permission(0:44/2), mask=0x24, res=0
[1109523.468348] NFS: open dir(/)
[1109523.468349] NFS: revalidating (0:44/2)
[1109523.468351] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.468352] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.468563] decode_attr_type: type=040000
[1109523.468564] decode_attr_change: change attribute=6364596956768490675
[1109523.468565] decode_attr_size: file size=4096
[1109523.468566] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.468566] decode_attr_fileid: fileid=2
[1109523.468567] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.468567] decode_attr_mode: file mode=0755
[1109523.468568] decode_attr_nlink: nlink=21
[1109523.468569] decode_attr_owner: uid=0
[1109523.468569] decode_attr_group: gid=0
[1109523.468570] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.468571] decode_attr_space_used: space used=4096
[1109523.468571] decode_attr_time_access: atime=1481873345
[1109523.468572] decode_attr_time_metadata: ctime=1481873206
[1109523.468572] decode_attr_time_modify: mtime=1481873206
[1109523.468573] decode_attr_mounted_on_fileid: fileid=524289
[1109523.468573] decode_getfattr_attrs: xdr returned 0
[1109523.468574] decode_getfattr_generic: xdr returned 0
[1109523.468575] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.468578] NFS: nfs_update_inode(0:44/2 fh_crc=0x41dc8638 ct=2 info=0x427e7f)
[1109523.468579] NFS: (0:44/2) revalidation complete
[1109523.468590] NFS: readdir(/) starting at cookie 0
[1109523.468593] NFS: nfs_do_filldir() filling ended @ cookie 9223372036854775807; returning = 0
[1109523.468594] NFS: readdir(/) returns 0
[1109523.468600] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.468602] NFS: nfs_lookup_revalidate(/android) is valid
[1109523.468604] NFS: dentry_delete(/android, 10800cc)
[1109523.468605] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.468606] NFS: nfs_lookup_revalidate(/zfs) is valid
[1109523.468607] NFS: dentry_delete(/zfs, 10800cc)
[1109523.468608] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.468609] NFS: nfs_lookup_revalidate(/test.txt) is valid
[1109523.468611] NFS: revalidating (0:44/13)
[1109523.468612] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.468613] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.469745] decode_attr_type: type=0100000
[1109523.469746] decode_attr_change: change attribute=6364598550564246077
[1109523.469747] decode_attr_size: file size=2050
[1109523.469748] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.469748] decode_attr_fileid: fileid=13
[1109523.469749] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.469749] decode_attr_mode: file mode=0644
[1109523.469750] decode_attr_nlink: nlink=1
[1109523.469751] decode_attr_owner: uid=0
[1109523.469752] decode_attr_group: gid=0
[1109523.469752] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.469753] decode_attr_space_used: space used=4096
[1109523.469753] decode_attr_time_access: atime=1481873587
[1109523.469754] decode_attr_time_metadata: ctime=1481873577
[1109523.469754] decode_attr_time_modify: mtime=1481873577
[1109523.469755] decode_attr_mounted_on_fileid: fileid=13
[1109523.469755] decode_getfattr_attrs: xdr returned 0
[1109523.469756] decode_getfattr_generic: xdr returned 0
[1109523.469757] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.469760] NFS: nfs_update_inode(0:44/13 fh_crc=0x37ac62f7 ct=1 info=0x427e7f)
[1109523.469761] NFS: (0:44/13) revalidation complete
[1109523.469762] NFS: dentry_delete(/test.txt, 40800cc)
[1109523.469778] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.469780] NFS: nfs_lookup_revalidate(/test.txt) is valid
[1109523.469783] NFS: dentry_delete(/test.txt, 40800cc)
[1109523.469788] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.469789] NFS: nfs_lookup_revalidate(/CentOS-7-x86_64-Everything-1511.iso) is valid
[1109523.469790] NFS: revalidating (0:44/11)
[1109523.469791] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.469792] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.470356] decode_attr_type: type=0100000
[1109523.470357] decode_attr_change: change attribute=6364524539288547030
[1109523.470358] decode_attr_size: file size=420741120
[1109523.470359] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.470359] decode_attr_fileid: fileid=11
[1109523.470360] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.470360] decode_attr_mode: file mode=0755
[1109523.470361] decode_attr_nlink: nlink=1
[1109523.470362] decode_attr_owner: uid=0
[1109523.470362] decode_attr_group: gid=0
[1109523.470363] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.470363] decode_attr_space_used: space used=411308032
[1109523.470364] decode_attr_time_access: atime=1481855120
[1109523.470364] decode_attr_time_metadata: ctime=1481856345
[1109523.470365] decode_attr_time_modify: mtime=1481856345
[1109523.470365] decode_attr_mounted_on_fileid: fileid=11
[1109523.470366] decode_getfattr_attrs: xdr returned 0
[1109523.470366] decode_getfattr_generic: xdr returned 0
[1109523.470367] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.470370] NFS: nfs_update_inode(0:44/11 fh_crc=0x8e8a0486 ct=1 info=0x427e7f)
[1109523.470371] NFS: (0:44/11) revalidation complete
[1109523.470372] NFS: dentry_delete(/CentOS-7-x86_64-Everything-1511.iso, 40800cc)
[1109523.470377] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470378] NFS: nfs_lookup_revalidate(/lvm2) is valid
[1109523.470379] NFS: dentry_delete(/lvm2, 10800cc)
[1109523.470380] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470381] NFS: nfs_lookup_revalidate(/cluster) is valid
[1109523.470382] NFS: dentry_delete(/cluster, 10800cc)
[1109523.470386] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470387] NFS: nfs_lookup_revalidate(/syscall.sh) is valid
[1109523.470388] NFS: revalidating (0:44/14)
[1109523.470389] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.470390] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.470621] decode_attr_type: type=0100000
[1109523.470622] decode_attr_change: change attribute=6353487525572665064
[1109523.470623] decode_attr_size: file size=553
[1109523.470623] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.470624] decode_attr_fileid: fileid=14
[1109523.470624] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.470625] decode_attr_mode: file mode=0755
[1109523.470625] decode_attr_nlink: nlink=1
[1109523.470626] decode_attr_owner: uid=0
[1109523.470627] decode_attr_group: gid=0
[1109523.470627] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.470628] decode_attr_space_used: space used=4096
[1109523.470628] decode_attr_time_access: atime=1475207632
[1109523.470629] decode_attr_time_metadata: ctime=1479286590
[1109523.470629] decode_attr_time_modify: mtime=1475207569
[1109523.470630] decode_attr_mounted_on_fileid: fileid=14
[1109523.470630] decode_getfattr_attrs: xdr returned 0
[1109523.470631] decode_getfattr_generic: xdr returned 0
[1109523.470632] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.470635] NFS: nfs_update_inode(0:44/14 fh_crc=0xdd939116 ct=1 info=0x427e7f)
[1109523.470635] NFS: (0:44/14) revalidation complete
[1109523.470637] NFS: dentry_delete(/syscall.sh, 40800cc)
[1109523.470642] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470643] NFS: nfs_lookup_revalidate(/openstack) is valid
[1109523.470644] NFS: dentry_delete(/openstack, 10800cc)
[1109523.470646] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470646] NFS: nfs_lookup_revalidate(/dm-dynamic) is valid
[1109523.470649] NFS: dentry_delete(/dm-dynamic, 10800cc)
[1109523.470650] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470651] NFS: nfs_lookup_revalidate(/hello) is valid
[1109523.470652] NFS: dentry_delete(/hello, 10800cc)
[1109523.470653] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470654] NFS: nfs_lookup_revalidate(/stack) is valid
[1109523.470654] NFS: dentry_delete(/stack, 10800cc)
[1109523.470655] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470656] NFS: nfs_lookup_revalidate(/vm) is valid
[1109523.470657] NFS: dentry_delete(/vm, 10800cc)
[1109523.470658] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470659] NFS: nfs_lookup_revalidate(/multipath) is valid
[1109523.470659] NFS: dentry_delete(/multipath, 10800cc)
[1109523.470660] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470661] NFS: nfs_lookup_revalidate(/Software) is valid
[1109523.470662] NFS: dentry_delete(/Software, 10800cc)
[1109523.470663] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470664] NFS: nfs_lookup_revalidate(/systemd) is valid
[1109523.470664] NFS: dentry_delete(/systemd, 10800cc)
[1109523.470665] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470666] NFS: nfs_lookup_revalidate(/python) is valid
[1109523.470667] NFS: dentry_delete(/python, 10800cc)
[1109523.470668] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470668] NFS: nfs_lookup_revalidate(/progress) is valid
[1109523.470669] NFS: dentry_delete(/progress, 10800cc)
[1109523.470670] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470671] NFS: nfs_lookup_revalidate(/ref) is valid
[1109523.470671] NFS: dentry_delete(/ref, 10800cc)
[1109523.470672] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470673] NFS: nfs_lookup_revalidate(/nfs) is valid
[1109523.470674] NFS: dentry_delete(/nfs, 10800cc)
[1109523.470675] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470676] NFS: nfs_lookup_revalidate(/eclipse) is valid
[1109523.470676] NFS: dentry_delete(/eclipse, 10800cc)
[1109523.470677] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470678] NFS: nfs_lookup_revalidate(/scsi) is valid
[1109523.470679] NFS: dentry_delete(/scsi, 10800cc)
[1109523.470680] NFS: permission(0:44/2), mask=0x81, res=0
[1109523.470680] NFS: nfs_lookup_revalidate(/yum) is valid
[1109523.470681] NFS: dentry_delete(/yum, 10800cc)
[1109523.470682] NFS: readdir(/) starting at cookie 24
[1109523.470687] NFS: (0:44/2) data cache invalidated
[1109523.470696] _nfs4_proc_readdir: dentry = /, cookie = 0
[1109523.470698] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109523.470698] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109523.470701] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b0a23a:00000000
[1109523.470702] nfs4_xdr_enc_readdir: inlined page args = (60, ffff88000087fce8, 80, 32688)
[1109523.473190] decode_readdir: verifier = 00000000:00000000
[1109523.473193] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109523.473196] _nfs4_proc_readdir: returns 4584
[1109523.473198] decode_attr_type: type=00
[1109523.473199] decode_attr_change: change attribute=0
[1109523.473199] decode_attr_size: file size=0
[1109523.473200] decode_attr_fsid: fsid=(0x0/0x0)
[1109523.473201] decode_attr_fileid: fileid=2
[1109523.473201] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473202] decode_attr_mode: file mode=00
[1109523.473202] decode_attr_nlink: nlink=1
[1109523.473203] decode_attr_owner: uid=-2
[1109523.473203] decode_attr_group: gid=-2
[1109523.473204] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473204] decode_attr_space_used: space used=0
[1109523.473205] decode_attr_time_access: atime=0
[1109523.473205] decode_attr_time_metadata: ctime=0
[1109523.473206] decode_attr_time_modify: mtime=0
[1109523.473206] decode_attr_mounted_on_fileid: fileid=0
[1109523.473207] decode_getfattr_attrs: xdr returned 0
[1109523.473208] decode_attr_type: type=00
[1109523.473208] decode_attr_change: change attribute=0
[1109523.473209] decode_attr_size: file size=0
[1109523.473209] decode_attr_fsid: fsid=(0x0/0x0)
[1109523.473226] decode_attr_fileid: fileid=2
[1109523.473228] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473228] decode_attr_mode: file mode=00
[1109523.473229] decode_attr_nlink: nlink=1
[1109523.473229] decode_attr_owner: uid=-2
[1109523.473230] decode_attr_group: gid=-2
[1109523.473230] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473230] decode_attr_space_used: space used=0
[1109523.473231] decode_attr_time_access: atime=0
[1109523.473231] decode_attr_time_metadata: ctime=0
[1109523.473232] decode_attr_time_modify: mtime=0
[1109523.473232] decode_attr_mounted_on_fileid: fileid=0
[1109523.473233] decode_getfattr_attrs: xdr returned 0
[1109523.473234] decode_attr_type: type=040000
[1109523.473235] decode_attr_change: change attribute=6346343096919327641
[1109523.473235] decode_attr_size: file size=4096
[1109523.473236] decode_attr_fsid: fsid=(0xa983386a266c40c2/0x886bf94fb379d906)
[1109523.473236] decode_attr_fileid: fileid=2
[1109523.473237] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473237] decode_attr_mode: file mode=0755
[1109523.473238] decode_attr_nlink: nlink=11
[1109523.473238] decode_attr_owner: uid=0
[1109523.473239] decode_attr_group: gid=0
[1109523.473240] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473240] decode_attr_space_used: space used=4096
[1109523.473241] decode_attr_time_access: atime=1481953801
[1109523.473241] decode_attr_time_metadata: ctime=1477623148
[1109523.473242] decode_attr_time_modify: mtime=1477623148
[1109523.473242] decode_attr_mounted_on_fileid: fileid=4718593
[1109523.473242] decode_getfattr_attrs: xdr returned 0
[1109523.473244] NFS: dentry_delete(/android, 10800cc)
[1109523.473244] decode_attr_type: type=040000
[1109523.473245] decode_attr_change: change attribute=6353487525598664995
[1109523.473245] decode_attr_size: file size=4096
[1109523.473246] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473247] decode_attr_fileid: fileid=1181159
[1109523.473247] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473248] decode_attr_mode: file mode=0755
[1109523.473258] decode_attr_nlink: nlink=15
[1109523.473259] decode_attr_owner: uid=0
[1109523.473259] decode_attr_group: gid=0
[1109523.473260] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473260] decode_attr_space_used: space used=4096
[1109523.473261] decode_attr_time_access: atime=1479286629
[1109523.473261] decode_attr_time_metadata: ctime=1479286590
[1109523.473262] decode_attr_time_modify: mtime=1467185801
[1109523.473262] decode_attr_mounted_on_fileid: fileid=1181159
[1109523.473263] decode_getfattr_attrs: xdr returned 0
[1109523.473265] NFS: nfs_update_inode(0:44/1181159 fh_crc=0x6176c546 ct=1 info=0x427e7f)
[1109523.473265] NFS: dentry_delete(/zfs, 10800cc)
[1109523.473266] decode_attr_type: type=0100000
[1109523.473267] decode_attr_change: change attribute=6364598550564246077
[1109523.473267] decode_attr_size: file size=2050
[1109523.473268] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473268] decode_attr_fileid: fileid=13
[1109523.473269] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473269] decode_attr_mode: file mode=0644
[1109523.473269] decode_attr_nlink: nlink=1
[1109523.473270] decode_attr_owner: uid=0
[1109523.473270] decode_attr_group: gid=0
[1109523.473271] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473271] decode_attr_space_used: space used=4096
[1109523.473272] decode_attr_time_access: atime=1481873587
[1109523.473272] decode_attr_time_metadata: ctime=1481873577
[1109523.473273] decode_attr_time_modify: mtime=1481873577
[1109523.473273] decode_attr_mounted_on_fileid: fileid=13
[1109523.473274] decode_getfattr_attrs: xdr returned 0
[1109523.473275] NFS: nfs_update_inode(0:44/13 fh_crc=0x37ac62f7 ct=1 info=0x427e7f)
[1109523.473276] NFS: dentry_delete(/test.txt, 40800cc)
[1109523.473276] decode_attr_type: type=0100000
[1109523.473277] decode_attr_change: change attribute=6364524539288547030
[1109523.473277] decode_attr_size: file size=420741120
[1109523.473278] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473278] decode_attr_fileid: fileid=11
[1109523.473300] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473301] decode_attr_mode: file mode=0755
[1109523.473301] decode_attr_nlink: nlink=1
[1109523.473302] decode_attr_owner: uid=0
[1109523.473303] decode_attr_group: gid=0
[1109523.473303] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473303] decode_attr_space_used: space used=411308032
[1109523.473304] decode_attr_time_access: atime=1481855120
[1109523.473304] decode_attr_time_metadata: ctime=1481856345
[1109523.473305] decode_attr_time_modify: mtime=1481856345
[1109523.473305] decode_attr_mounted_on_fileid: fileid=11
[1109523.473306] decode_getfattr_attrs: xdr returned 0
[1109523.473307] NFS: nfs_update_inode(0:44/11 fh_crc=0x8e8a0486 ct=1 info=0x427e7f)
[1109523.473308] NFS: dentry_delete(/CentOS-7-x86_64-Everything-1511.iso, 40800cc)
[1109523.473309] decode_attr_type: type=040000
[1109523.473309] decode_attr_change: change attribute=6363103974682233166
[1109523.473309] decode_attr_size: file size=4096
[1109523.473310] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473311] decode_attr_fileid: fileid=1179650
[1109523.473311] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473311] decode_attr_mode: file mode=0755
[1109523.473312] decode_attr_nlink: nlink=5
[1109523.473325] decode_attr_owner: uid=0
[1109523.473325] decode_attr_group: gid=0
[1109523.473326] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473326] decode_attr_space_used: space used=4096
[1109523.473326] decode_attr_time_access: atime=1481525595
[1109523.473327] decode_attr_time_metadata: ctime=1481525594
[1109523.473327] decode_attr_time_modify: mtime=1481525594
[1109523.473328] decode_attr_mounted_on_fileid: fileid=1179650
[1109523.473328] decode_getfattr_attrs: xdr returned 0
[1109523.473330] NFS: nfs_update_inode(0:44/1179650 fh_crc=0xc26ca614 ct=1 info=0x427e7f)
[1109523.473330] NFS: dentry_delete(/lvm2, 10800cc)
[1109523.473331] decode_attr_type: type=040000
[1109523.473331] decode_attr_change: change attribute=6353487242054997306
[1109523.473332] decode_attr_size: file size=4096
[1109523.473332] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473333] decode_attr_fileid: fileid=1181155
[1109523.473333] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473333] decode_attr_mode: file mode=0755
[1109523.473334] decode_attr_nlink: nlink=8
[1109523.473334] decode_attr_owner: uid=0
[1109523.473335] decode_attr_group: gid=0
[1109523.473335] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473336] decode_attr_space_used: space used=4096
[1109523.473336] decode_attr_time_access: atime=1479286638
[1109523.473337] decode_attr_time_metadata: ctime=1479286524
[1109523.473337] decode_attr_time_modify: mtime=1467265348
[1109523.473338] decode_attr_mounted_on_fileid: fileid=1181155
[1109523.473338] decode_getfattr_attrs: xdr returned 0
[1109523.473339] NFS: nfs_update_inode(0:44/1181155 fh_crc=0xb3b072b8 ct=1 info=0x427e7f)
[1109523.473339] NFS: dentry_delete(/cluster, 10800cc)
[1109523.473340] decode_attr_type: type=0100000
[1109523.473340] decode_attr_change: change attribute=6353487525572665064
[1109523.473341] decode_attr_size: file size=553
[1109523.473341] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473342] decode_attr_fileid: fileid=14
[1109523.473342] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473353] decode_attr_mode: file mode=0755
[1109523.473354] decode_attr_nlink: nlink=1
[1109523.473354] decode_attr_owner: uid=0
[1109523.473355] decode_attr_group: gid=0
[1109523.473355] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473356] decode_attr_space_used: space used=4096
[1109523.473356] decode_attr_time_access: atime=1475207632
[1109523.473356] decode_attr_time_metadata: ctime=1479286590
[1109523.473357] decode_attr_time_modify: mtime=1475207569
[1109523.473357] decode_attr_mounted_on_fileid: fileid=14
[1109523.473358] decode_getfattr_attrs: xdr returned 0
[1109523.473359] NFS: nfs_update_inode(0:44/14 fh_crc=0xdd939116 ct=1 info=0x427e7f)
[1109523.473359] NFS: dentry_delete(/syscall.sh, 40800cc)
[1109523.473370] decode_attr_type: type=040000
[1109523.473371] decode_attr_change: change attribute=6339664126708629689
[1109523.473371] decode_attr_size: file size=4096
[1109523.473372] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473372] decode_attr_fileid: fileid=655361
[1109523.473373] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473373] decode_attr_mode: file mode=0755
[1109523.473374] decode_attr_nlink: nlink=5
[1109523.473374] decode_attr_owner: uid=1001
[1109523.473375] decode_attr_group: gid=1001
[1109523.473375] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473376] decode_attr_space_used: space used=4096
[1109523.473376] decode_attr_time_access: atime=1478830795
[1109523.473376] decode_attr_time_metadata: ctime=1476068079
[1109523.473377] decode_attr_time_modify: mtime=1476068079
[1109523.473377] decode_attr_mounted_on_fileid: fileid=655361
[1109523.473378] decode_getfattr_attrs: xdr returned 0
[1109523.473379] NFS: nfs_update_inode(0:44/655361 fh_crc=0x3b9f1a18 ct=1 info=0x427e7f)
[1109523.473380] NFS: dentry_delete(/openstack, 10800cc)
[1109523.473380] decode_attr_type: type=040000
[1109523.473381] decode_attr_change: change attribute=6353487255295890364
[1109523.473381] decode_attr_size: file size=4096
[1109523.473382] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473382] decode_attr_fileid: fileid=1185085
[1109523.473383] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473383] decode_attr_mode: file mode=0755
[1109523.473384] decode_attr_nlink: nlink=13
[1109523.473384] decode_attr_owner: uid=0
[1109523.473385] decode_attr_group: gid=0
[1109523.473385] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473385] decode_attr_space_used: space used=4096
[1109523.473386] decode_attr_time_access: atime=1480579008
[1109523.473386] decode_attr_time_metadata: ctime=1479286527
[1109523.473387] decode_attr_time_modify: mtime=1479110336
[1109523.473387] decode_attr_mounted_on_fileid: fileid=1185085
[1109523.473388] decode_getfattr_attrs: xdr returned 0
[1109523.473389] NFS: nfs_update_inode(0:44/1185085 fh_crc=0xa63c794a ct=1 info=0x427e7f)
[1109523.473389] NFS: dentry_delete(/dm-dynamic, 10800cc)
[1109523.473390] decode_attr_type: type=040000
[1109523.473390] decode_attr_change: change attribute=6353487525572665064
[1109523.473391] decode_attr_size: file size=4096
[1109523.473391] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473392] decode_attr_fileid: fileid=1185087
[1109523.473392] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473393] decode_attr_mode: file mode=0755
[1109523.473393] decode_attr_nlink: nlink=3
[1109523.473393] decode_attr_owner: uid=0
[1109523.473394] decode_attr_group: gid=0
[1109523.473394] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473395] decode_attr_space_used: space used=4096
[1109523.473395] decode_attr_time_access: atime=1479287767
[1109523.473396] decode_attr_time_metadata: ctime=1479286590
[1109523.473396] decode_attr_time_modify: mtime=1468911344
[1109523.473397] decode_attr_mounted_on_fileid: fileid=1185087
[1109523.473397] decode_getfattr_attrs: xdr returned 0
[1109523.473398] NFS: nfs_update_inode(0:44/1185087 fh_crc=0xc87109d7 ct=1 info=0x427e7f)
[1109523.473399] NFS: dentry_delete(/hello, 10800cc)
[1109523.473399] decode_attr_type: type=040000
[1109523.473400] decode_attr_change: change attribute=6340121355456166012
[1109523.473400] decode_attr_size: file size=4096
[1109523.473401] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473401] decode_attr_fileid: fileid=3932161
[1109523.473401] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473402] decode_attr_mode: file mode=0700
[1109523.473402] decode_attr_nlink: nlink=12
[1109523.473403] decode_attr_owner: uid=1001
[1109523.473403] decode_attr_group: gid=1001
[1109523.473404] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473404] decode_attr_space_used: space used=4096
[1109523.473404] decode_attr_time_access: atime=1478830795
[1109523.473405] decode_attr_time_metadata: ctime=1476174536
[1109523.473405] decode_attr_time_modify: mtime=1476174536
[1109523.473407] decode_attr_mounted_on_fileid: fileid=3932161
[1109523.473407] decode_getfattr_attrs: xdr returned 0
[1109523.473408] NFS: nfs_update_inode(0:44/3932161 fh_crc=0x7267b98e ct=1 info=0x427e7f)
[1109523.473409] NFS: dentry_delete(/stack, 10800cc)
[1109523.473409] decode_attr_type: type=040000
[1109523.473410] decode_attr_change: change attribute=6339664307157381939
[1109523.473410] decode_attr_size: file size=4096
[1109523.473411] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473411] decode_attr_fileid: fileid=4456449
[1109523.473412] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473412] decode_attr_mode: file mode=0755
[1109523.473413] decode_attr_nlink: nlink=6
[1109523.473413] decode_attr_owner: uid=0
[1109523.473413] decode_attr_group: gid=0
[1109523.473414] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473414] decode_attr_space_used: space used=4096
[1109523.473415] decode_attr_time_access: atime=1478830795
[1109523.473415] decode_attr_time_metadata: ctime=1476068121
[1109523.473416] decode_attr_time_modify: mtime=1476068121
[1109523.473416] decode_attr_mounted_on_fileid: fileid=4456449
[1109523.473417] decode_getfattr_attrs: xdr returned 0
[1109523.473418] NFS: nfs_update_inode(0:44/4456449 fh_crc=0x8999fd48 ct=1 info=0x427e7f)
[1109523.473418] NFS: dentry_delete(/vm, 10800cc)
[1109523.473419] decode_attr_type: type=040000
[1109523.473419] decode_attr_change: change attribute=6358661118912613391
[1109523.473419] decode_attr_size: file size=12288
[1109523.473420] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473420] decode_attr_fileid: fileid=4194305
[1109523.473421] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473421] decode_attr_mode: file mode=0755
[1109523.473422] decode_attr_nlink: nlink=9
[1109523.473422] decode_attr_owner: uid=0
[1109523.473423] decode_attr_group: gid=0
[1109523.473423] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473423] decode_attr_space_used: space used=12288
[1109523.473424] decode_attr_time_access: atime=1481078966
[1109523.473424] decode_attr_time_metadata: ctime=1480491161
[1109523.473425] decode_attr_time_modify: mtime=1480491161
[1109523.473425] decode_attr_mounted_on_fileid: fileid=4194305
[1109523.473426] decode_getfattr_attrs: xdr returned 0
[1109523.473427] NFS: nfs_update_inode(0:44/4194305 fh_crc=0x9f21f0d6 ct=1 info=0x427e7f)
[1109523.473427] NFS: dentry_delete(/multipath, 10800cc)
[1109523.473428] decode_attr_type: type=040000
[1109523.473428] decode_attr_change: change attribute=6308931294901908761
[1109523.473429] decode_attr_size: file size=4096
[1109523.473429] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473430] decode_attr_fileid: fileid=131073
[1109523.473430] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473438] decode_attr_mode: file mode=0755
[1109523.473449] decode_attr_nlink: nlink=4
[1109523.473450] decode_attr_owner: uid=0
[1109523.473450] decode_attr_group: gid=0
[1109523.473451] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473451] decode_attr_space_used: space used=4096
[1109523.473452] decode_attr_time_access: atime=1479286483
[1109523.473452] decode_attr_time_metadata: ctime=1468912534
[1109523.473453] decode_attr_time_modify: mtime=1465806943
[1109523.473453] decode_attr_mounted_on_fileid: fileid=131073
[1109523.473453] decode_getfattr_attrs: xdr returned 0
[1109523.473455] NFS: nfs_update_inode(0:44/131073 fh_crc=0xecf5aafb ct=1 info=0x427e7f)
[1109523.473455] NFS: dentry_delete(/Software, 10800cc)
[1109523.473456] decode_attr_type: type=040000
[1109523.473457] decode_attr_change: change attribute=6361553229585786199
[1109523.473457] decode_attr_size: file size=4096
[1109523.473457] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473458] decode_attr_fileid: fileid=917505
[1109523.473458] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473459] decode_attr_mode: file mode=0755
[1109523.473459] decode_attr_nlink: nlink=6
[1109523.473460] decode_attr_owner: uid=0
[1109523.473460] decode_attr_group: gid=0
[1109523.473477] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473477] decode_attr_space_used: space used=4096
[1109523.473478] decode_attr_time_access: atime=1481266299
[1109523.473478] decode_attr_time_metadata: ctime=1481164533
[1109523.473478] decode_attr_time_modify: mtime=1481164533
[1109523.473479] decode_attr_mounted_on_fileid: fileid=917505
[1109523.473479] decode_getfattr_attrs: xdr returned 0
[1109523.473480] NFS: nfs_update_inode(0:44/917505 fh_crc=0xce6be67c ct=1 info=0x427e7f)
[1109523.473481] NFS: dentry_delete(/systemd, 10800cc)
[1109523.473481] decode_attr_type: type=040000
[1109523.473482] decode_attr_change: change attribute=6358287748949713371
[1109523.473482] decode_attr_size: file size=4096
[1109523.473483] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473483] decode_attr_fileid: fileid=5111809
[1109523.473484] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473484] decode_attr_mode: file mode=0755
[1109523.473485] decode_attr_nlink: nlink=4
[1109523.473485] decode_attr_owner: uid=0
[1109523.473485] decode_attr_group: gid=0
[1109523.473486] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473486] decode_attr_space_used: space used=4096
[1109523.473487] decode_attr_time_access: atime=1481858139
[1109523.473487] decode_attr_time_metadata: ctime=1480404229
[1109523.473488] decode_attr_time_modify: mtime=1480404229
[1109523.473488] decode_attr_mounted_on_fileid: fileid=5111809
[1109523.473488] decode_getfattr_attrs: xdr returned 0
[1109523.473489] NFS: nfs_update_inode(0:44/5111809 fh_crc=0x41e8e646 ct=1 info=0x427e7f)
[1109523.473490] NFS: dentry_delete(/python, 10800cc)
[1109523.473490] decode_attr_type: type=040000
[1109523.473491] decode_attr_change: change attribute=6363782746754615169
[1109523.473491] decode_attr_size: file size=4096
[1109523.473492] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473492] decode_attr_fileid: fileid=4718602
[1109523.473493] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473493] decode_attr_mode: file mode=0755
[1109523.473494] decode_attr_nlink: nlink=3
[1109523.473494] decode_attr_owner: uid=0
[1109523.473494] decode_attr_group: gid=0
[1109523.473495] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473495] decode_attr_space_used: space used=4096
[1109523.473496] decode_attr_time_access: atime=1481683676
[1109523.473496] decode_attr_time_metadata: ctime=1481683633
[1109523.473497] decode_attr_time_modify: mtime=1481683633
[1109523.473497] decode_attr_mounted_on_fileid: fileid=4718602
[1109523.473498] decode_getfattr_attrs: xdr returned 0
[1109523.473498] NFS: nfs_update_inode(0:44/4718602 fh_crc=0xcf53986f ct=1 info=0x427e7f)
[1109523.473499] NFS: dentry_delete(/progress, 10800cc)
[1109523.473500] decode_attr_type: type=040000
[1109523.473500] decode_attr_change: change attribute=6353487435452406902
[1109523.473500] decode_attr_size: file size=4096
[1109523.473501] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473501] decode_attr_fileid: fileid=1310721
[1109523.473502] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473502] decode_attr_mode: file mode=0755
[1109523.473503] decode_attr_nlink: nlink=11
[1109523.473503] decode_attr_owner: uid=0
[1109523.473503] decode_attr_group: gid=0
[1109523.473504] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473504] decode_attr_space_used: space used=4096
[1109523.473505] decode_attr_time_access: atime=1480661269
[1109523.473505] decode_attr_time_metadata: ctime=1479286569
[1109523.473506] decode_attr_time_modify: mtime=1479286569
[1109523.473506] decode_attr_mounted_on_fileid: fileid=1310721
[1109523.473506] decode_getfattr_attrs: xdr returned 0
[1109523.473507] NFS: nfs_update_inode(0:44/1310721 fh_crc=0xe1dbe710 ct=1 info=0x427e7f)
[1109523.473508] NFS: dentry_delete(/ref, 10800cc)
[1109523.473508] decode_attr_type: type=040000
[1109523.473509] decode_attr_change: change attribute=6364248145473236575
[1109523.473509] decode_attr_size: file size=4096
[1109523.473510] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473510] decode_attr_fileid: fileid=524289
[1109523.473520] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473521] decode_attr_mode: file mode=0755
[1109523.473522] decode_attr_nlink: nlink=5
[1109523.473522] decode_attr_owner: uid=0
[1109523.473523] decode_attr_group: gid=0
[1109523.473523] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473524] decode_attr_space_used: space used=4096
[1109523.473524] decode_attr_time_access: atime=1481867719
[1109523.473524] decode_attr_time_metadata: ctime=1481791992
[1109523.473525] decode_attr_time_modify: mtime=1481791992
[1109523.473525] decode_attr_mounted_on_fileid: fileid=524289
[1109523.473526] decode_getfattr_attrs: xdr returned 0
[1109523.473527] NFS: nfs_update_inode(0:44/524289 fh_crc=0x1ff30610 ct=1 info=0x427e7f)
[1109523.473528] NFS: dentry_delete(/nfs, 10800cc)
[1109523.473528] decode_attr_type: type=040000
[1109523.473529] decode_attr_change: change attribute=6296346614982177361
[1109523.473529] decode_attr_size: file size=4096
[1109523.473530] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473530] decode_attr_fileid: fileid=393217
[1109523.473531] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473531] decode_attr_mode: file mode=0755
[1109523.473532] decode_attr_nlink: nlink=4
[1109523.473532] decode_attr_owner: uid=0
[1109523.473533] decode_attr_group: gid=0
[1109523.473533] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473534] decode_attr_space_used: space used=4096
[1109523.473534] decode_attr_time_access: atime=1478830795
[1109523.473535] decode_attr_time_metadata: ctime=1465982435
[1109523.473535] decode_attr_time_modify: mtime=1465982435
[1109523.473535] decode_attr_mounted_on_fileid: fileid=393217
[1109523.473536] decode_getfattr_attrs: xdr returned 0
[1109523.473537] NFS: nfs_update_inode(0:44/393217 fh_crc=0x9037e86d ct=1 info=0x427e7f)
[1109523.473537] NFS: dentry_delete(/eclipse, 10800cc)
[1109523.473538] decode_attr_type: type=040000
[1109523.473538] decode_attr_change: change attribute=6353487461794193390
[1109523.473539] decode_attr_size: file size=4096
[1109523.473539] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473540] decode_attr_fileid: fileid=2884911
[1109523.473540] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473540] decode_attr_mode: file mode=0755
[1109523.473541] decode_attr_nlink: nlink=5
[1109523.473541] decode_attr_owner: uid=0
[1109523.473542] decode_attr_group: gid=0
[1109523.473542] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473543] decode_attr_space_used: space used=4096
[1109523.473543] decode_attr_time_access: atime=1479287767
[1109523.473544] decode_attr_time_metadata: ctime=1479286575
[1109523.473544] decode_attr_time_modify: mtime=1479286575
[1109523.473544] decode_attr_mounted_on_fileid: fileid=2884911
[1109523.473545] decode_getfattr_attrs: xdr returned 0
[1109523.473546] NFS: nfs_update_inode(0:44/2884911 fh_crc=0x06b07062 ct=1 info=0x427e7f)
[1109523.473546] NFS: dentry_delete(/scsi, 10800cc)
[1109523.473547] decode_attr_type: type=040000
[1109523.473547] decode_attr_change: change attribute=6348575243338301752
[1109523.473548] decode_attr_size: file size=4096
[1109523.473548] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109523.473549] decode_attr_fileid: fileid=4325377
[1109523.473549] decode_attr_fs_locations: fs_locations done, error = 0
[1109523.473549] decode_attr_mode: file mode=0755
[1109523.473550] decode_attr_nlink: nlink=3
[1109523.473550] decode_attr_owner: uid=0
[1109523.473551] decode_attr_group: gid=0
[1109523.473551] decode_attr_rdev: rdev=(0x0:0x0)
[1109523.473552] decode_attr_space_used: space used=4096
[1109523.473552] decode_attr_time_access: atime=1478830795
[1109523.473552] decode_attr_time_metadata: ctime=1478142860
[1109523.473553] decode_attr_time_modify: mtime=1478142860
[1109523.473553] decode_attr_mounted_on_fileid: fileid=4325377
[1109523.473554] decode_getfattr_attrs: xdr returned 0
[1109523.473555] NFS: nfs_update_inode(0:44/4325377 fh_crc=0x5aa00cdd ct=1 info=0x427e7f)
[1109523.473555] NFS: dentry_delete(/yum, 10800cc)
[1109523.473558] NFS: readdir(/) returns 0
[1109524.838283] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109524.838291] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109524.838754] decode_attr_files_avail: files avail=5126859
[1109524.838756] decode_attr_files_free: files free=5126859
[1109524.838757] decode_attr_files_total: files total=5242880
[1109524.838758] decode_attr_space_avail: space avail=48261320704
[1109524.838758] decode_attr_space_free: space free=52573065216
[1109524.838759] decode_attr_space_total: space total=84417077248
[1109524.838759] decode_statfs: xdr returned 0!
[1109524.838761] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109524.838767] NFS: revalidating (0:44/2)
[1109524.838769] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1109524.838770] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1109524.838965] decode_attr_type: type=040000
[1109524.838966] decode_attr_change: change attribute=6364596956768490675
[1109524.838967] decode_attr_size: file size=4096
[1109524.838968] decode_attr_fsid: fsid=(0x25fc7e9117764101/0x94fa84be75ffc4ec)
[1109524.838969] decode_attr_fileid: fileid=2
[1109524.838970] decode_attr_fs_locations: fs_locations done, error = 0
[1109524.838970] decode_attr_mode: file mode=0755
[1109524.838971] decode_attr_nlink: nlink=21
[1109524.838972] decode_attr_owner: uid=0
[1109524.838988] decode_attr_group: gid=0
[1109524.838989] decode_attr_rdev: rdev=(0x0:0x0)
[1109524.838989] decode_attr_space_used: space used=4096
[1109524.838990] decode_attr_time_access: atime=1481873345
[1109524.838991] decode_attr_time_metadata: ctime=1481873206
[1109524.838991] decode_attr_time_modify: mtime=1481873206
[1109524.838992] decode_attr_mounted_on_fileid: fileid=524289
[1109524.838992] decode_getfattr_attrs: xdr returned 0
[1109524.838993] decode_getfattr_generic: xdr returned 0
[1109524.838994] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1109524.838998] NFS: nfs_update_inode(0:44/2 fh_crc=0x41dc8638 ct=2 info=0x427e7f)
[1109524.838999] NFS: (0:44/2) revalidation complete
[1109524.964453] nfs4_renew_state: start
[1109524.964462] nfs4_renew_state: done
[1109524.964788] nfs4_schedule_state_renewal: requeueing work. Lease period = 60
[1109524.964790] --> nfs_put_client({2})

斷開網絡時，再次查看ls命令的日誌

$ ls /opt
$ dmesg -c
[71282.004255] nfs4_handle_reclaim_lease_error: handled error -110 for server 192.168.195.136
[71282.004262] NFS call  setclientid auth=UNIX, 'Linux NFSv4.0 192.168.195.132/192.168.195.136 tcp'
[71283.507284] nfs: server 192.168.195.136 not responding, timed out
[71283.507292] NFS reply setclientid: -110
[71284.509164] nfs4_handle_reclaim_lease_error: handled error -110 for server 192.168.195.136
[71284.509171] NFS call  setclientid auth=UNIX, 'Linux NFSv4.0 192.168.195.132/192.168.195.136 tcp'
[71284.520140] nfs4_renew_state: start
[71284.520146] nfs4_renew_state: done
[71286.012153] nfs: server 192.168.195.136 not responding, timed out
[71286.012162] NFS reply setclientid: -110
[71286.020128] nfs: server 192.168.195.136 not responding, timed out
[71286.020132] nfs4_schedule_lease_recovery: scheduling lease recovery for server 192.168.195.136
[71286.020136] nfs4_schedule_state_renewal: requeueing work. Lease period = 5
[71286.020138] --> nfs_put_client({3})
[71287.014152] nfs4_handle_reclaim_lease_error: handled error -110 for server 192.168.195.136
[71287.014160] NFS call  setclientid auth=UNIX, 'Linux NFSv4.0 192.168.195.132/192.168.195.136 tcp'
[71288.518172] nfs: server 192.168.195.136 not responding, timed out
[71288.518192] NFS reply setclientid: -110
[71289.519071] nfs4_handle_reclaim_lease_error: handled error -110 for server 192.168.195.136
[71289.519079] NFS call  setclientid auth=UNIX, 'Linux NFSv4.0 192.168.195.132/192.168.195.136 tcp'
[71291.023090] nfs: server 192.168.195.136 not responding, timed out
[71291.023099] NFS reply setclientid: -110
[71291.035097] nfs4_renew_state: start

使用hard模式mount

$ mount -t nfs 192.168.195.136:/opt /opt -osoft,timeo=5,retrans=2

斷開網絡時，查看ls命令的日誌

$ dmesg -c
[71635.512640] NFS: revalidating (0:43/2)
[71635.512647] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[71635.512648] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[71637.016890] nfs: server 192.168.195.136 not responding, timed out
[71637.016895] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[71637.016903] nfs_revalidate_inode: (0:43/2) getattr failed, error=-5

使用hard模式，在重試2次後還繼續重試，且無法退出，有時候甚至kill -9也不行（可以通過連續執行兩次“umount -f /opt”後退出）。使用soft模式，在重試2次後，會直接報“輸入/輸出錯誤”而退出。

子目錄自動掛載測試

因爲現場環境中也使用了NFS的子目錄掛載功能，所以我們也在這裏進行一些相關測試，查看和分析其特性。

查看NFS服務器狀態

在服務器上查看本地目錄掛載情況：

$ df -h | grep /opt
/dev/sdb1                   79G   30G   45G   40% /opt
/dev/sdc1                   79G   58G   18G   77% /opt/android

在客戶端上查看與服務器的聯通情況：

$ showmount -e 192.168.195.136
Export list for 192.168.195.136:
/opt/android *
/opt         *

使用NFSv4掛載並查看子目錄

$ mount -t nfs 192.168.195.136:/opt /opt -osoft,timeo=5,retrans=2
$ mount | grep /opt
192.168.195.136:/opt on /opt type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,soft,proto=tcp,port=0,timeo=5,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)

$ ls /opt/android/
aoc  backup  fanlian  hky  projects  rk-tools  sdk  studio  tx
$ mount | grep /opt
192.168.195.136:/opt on /opt type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,soft,proto=tcp,port=0,timeo=5,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)
192.168.195.136:/opt/android on /opt/android type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)
$ umount /opt

在NFSv4中，會自動掛載子目錄。

使用NFSv3掛載並查看子目錄

$ mount -t nfs 192.168.195.136:/opt /opt -osoft,timeo=5,retrans=2,vers=3
$ mount | grep /opt
192.168.195.136:/opt on /opt type nfs (rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,soft,proto=tcp,timeo=5,retrans=2,sec=sys,mountaddr=192.168.195.136,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=192.168.195.136)
$ ls /opt/android/
$ umount /opt

在NFSv3中，使用默認參數不會自動掛載子目錄。

子目錄掛載順序測試

先啓動NFS，後掛載子目錄測試

服務端：

$ systemctl stop nfs-server.service
$ umount /opt/android
$ systemctl start nfs-server.service
$ mount /dev/sdc1 /opt/android

客戶端：

$ ls /opt/android/

子目錄不顯示。

先掛載子目錄，後啓動NFS服務

服務端：

$ systemctl restart nfs-server.service

客戶端：

$ ls /opt/android/
$ umount /opt
$ mount -t nfs 192.168.195.136:/opt /opt -osoft,timeo=5,retrans=2
$ ls /opt/android/
backup  projects  rk-tools  sdk  studio  tx

子目錄顯示正常。

使用“mount --bind”目錄作爲NFS子目錄的測試 1 (Block,LVM,EXT4,bind,NFSv4)

服務端：

$ pvcreate /dev/sdd1
  Physical volume "/dev/sdd1" successfully created
$ vgcreate my_vg /dev/sdd1
$ lvcreate -n nfs_test -L 10G my_vg

$ mkfs.ext4 /dev/my_vg/nfs_test
$ mkdir -pv /mnt/test
$ mount /dev/my_vg/nfs_test /mnt/test

$ mkdir /mnt/test/subdir1
$ mkdir /mnt/test/subdir2
$ mkdir /mnt/test/subdir3

$ mkdir /opt/export1
$ mkdir /opt/export2
$ mkdir /opt/export3

$ mount --bind /mnt/test/subdir1 /opt/export1
$ mount --bind /mnt/test/subdir2 /opt/export2
$ mount --bind /mnt/test/subdir3 /opt/export3

$ touch /mnt/test/subdir1/test1
$ touch /mnt/test/subdir2/test2
$ touch /mnt/test/subdir3/test3

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
/opt/export1 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export2 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export3 *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)

$ exportfs -av

$ cat /var/lib/nfs/etab
/opt/android	*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)
/opt/export3	*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)
/opt/export2	*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)
/opt/export1	*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)
/opt	*(rw,sync,wdelay,hide,nocrossmnt,secure,no_root_squash,no_all_squash,no_subtree_check,secure_locks,acl,no_pnfs,anonuid=65534,anongid=65534,sec=sys,rw,secure,no_root_squash,no_all_squash)

$ rpcdebug -m nfsd -s all

客戶端：

$ rpcdebug -m nfs -s all

$ showmount -e 192.168.195.136
Export list for 192.168.195.136:
/opt/android *
/opt/export3 *
/opt/export2 *
/opt/export1 *
$ umount /opt
$ mount -t nfs 192.168.195.136:/opt /opt

ls測試

客戶端：

$ ls /opt/export1/
test1
$ dmesg -c
[1113642.555296] NFS: permission(0:44/2), mask=0x81, res=0
[1113642.555301] NFS: nfs_lookup_revalidate(/export1) is valid
[1113642.555310] NFS: permission(0:44/2), mask=0x81, res=0
[1113642.555311] NFS: nfs_lookup_revalidate(/export1) is valid
[1113642.555339] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1113642.555341] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1113642.555922] decode_attr_type: type=00
[1113642.555924] decode_attr_change: change attribute=6364967891928841364
[1113642.555925] decode_attr_size: file size=4096
[1113642.555926] decode_attr_fsid: fsid=(0x0/0x0)
[1113642.555926] decode_attr_fileid: fileid=0
[1113642.555927] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.555928] decode_attr_mode: file mode=00
[1113642.555928] decode_attr_nlink: nlink=1
[1113642.555929] decode_attr_owner: uid=-2
[1113642.555930] decode_attr_group: gid=-2
[1113642.555930] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.555931] decode_attr_space_used: space used=0
[1113642.555931] decode_attr_time_access: atime=0
[1113642.555932] decode_attr_time_metadata: ctime=1481959571
[1113642.555933] decode_attr_time_modify: mtime=1481959571
[1113642.555933] decode_attr_mounted_on_fileid: fileid=0
[1113642.555934] decode_getfattr_attrs: xdr returned 0
[1113642.555935] decode_getfattr_generic: xdr returned 0
[1113642.555936] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1113642.555943] NFS: nfs_update_inode(0:43/524289 fh_crc=0xafcd4d98 ct=3 info=0x26040)
[1113642.555945] NFS: permission(0:43/524289), mask=0x24, res=0
[1113642.555947] NFS: open dir(/)
[1113642.555949] NFS: revalidating (0:43/524289)
[1113642.555951] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1113642.555952] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1113642.556187] decode_attr_type: type=040000
[1113642.556189] decode_attr_change: change attribute=6364967891928841364
[1113642.556189] decode_attr_size: file size=4096
[1113642.556190] decode_attr_fsid: fsid=(0xe6468c490ef04fb4/0xb59f3f06b1a42594)
[1113642.556191] decode_attr_fileid: fileid=524289
[1113642.556192] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.556192] decode_attr_mode: file mode=0755
[1113642.556193] decode_attr_nlink: nlink=2
[1113642.556194] decode_attr_owner: uid=0
[1113642.556195] decode_attr_group: gid=0
[1113642.556196] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.556196] decode_attr_space_used: space used=4096
[1113642.556197] decode_attr_time_access: atime=1481959806
[1113642.556197] decode_attr_time_metadata: ctime=1481959571
[1113642.556198] decode_attr_time_modify: mtime=1481959571
[1113642.556199] decode_attr_mounted_on_fileid: fileid=786480
[1113642.556199] decode_getfattr_attrs: xdr returned 0
[1113642.556200] decode_getfattr_generic: xdr returned 0
[1113642.556201] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1113642.556205] NFS: nfs_update_inode(0:43/524289 fh_crc=0xafcd4d98 ct=3 info=0x427e7f)
[1113642.556206] NFS: (0:43/524289) revalidation complete
[1113642.556231] NFS: readdir(/) starting at cookie 0
[1113642.556234] NFS: nfs_do_filldir() filling ended @ cookie 9223372036854775807; returning = 0
[1113642.556235] NFS: readdir(/) returns 0
[1113642.556243] NFS: permission(0:44/2), mask=0x81, res=0
[1113642.556244] NFS: nfs_lookup_revalidate(/export1) is valid
[1113642.556258] NFS: permission(0:43/524289), mask=0x81, res=0
[1113642.556259] NFS: nfs_lookup_revalidate(/test1) is valid
[1113642.556260] NFS: revalidating (0:43/524290)
[1113642.556262] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1113642.556263] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1113642.556601] decode_attr_type: type=0100000
[1113642.556603] decode_attr_change: change attribute=6364967891928841364
[1113642.556603] decode_attr_size: file size=0
[1113642.556604] decode_attr_fsid: fsid=(0xe6468c490ef04fb4/0xb59f3f06b1a42594)
[1113642.556605] decode_attr_fileid: fileid=524290
[1113642.556605] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.556606] decode_attr_mode: file mode=0644
[1113642.556606] decode_attr_nlink: nlink=1
[1113642.556608] decode_attr_owner: uid=0
[1113642.556608] decode_attr_group: gid=0
[1113642.556609] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.556609] decode_attr_space_used: space used=0
[1113642.556639] decode_attr_time_access: atime=1481959571
[1113642.556640] decode_attr_time_metadata: ctime=1481959571
[1113642.556640] decode_attr_time_modify: mtime=1481959571
[1113642.556641] decode_attr_mounted_on_fileid: fileid=524290
[1113642.556655] decode_getfattr_attrs: xdr returned 0
[1113642.556656] decode_getfattr_generic: xdr returned 0
[1113642.556657] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1113642.556662] NFS: nfs_update_inode(0:43/524290 fh_crc=0xf34c6e39 ct=1 info=0x427e7f)
[1113642.556663] NFS: (0:43/524290) revalidation complete
[1113642.556664] NFS: dentry_delete(/test1, 40840cc)
[1113642.556692] NFS: permission(0:44/2), mask=0x81, res=0
[1113642.556693] NFS: nfs_lookup_revalidate(/export1) is valid
[1113642.556710] NFS: permission(0:43/524289), mask=0x81, res=0
[1113642.556711] NFS: nfs_lookup_revalidate(/test1) is valid
[1113642.556714] NFS: dentry_delete(/test1, 40840cc)
[1113642.556719] NFS: readdir(/) starting at cookie 3
[1113642.556723] NFS: (0:43/524289) data cache invalidated
[1113642.556729] _nfs4_proc_readdir: dentry = /, cookie = 0
[1113642.556730] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1113642.556731] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1113642.556735] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b0a23a:00000000
[1113642.556736] nfs4_xdr_enc_readdir: inlined page args = (60, ffff880000af3ce8, 80, 32688)
[1113642.558673] decode_readdir: verifier = 00000000:00000000
[1113642.558676] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1113642.558679] _nfs4_proc_readdir: returns 300
[1113642.558682] decode_attr_type: type=00
[1113642.558682] decode_attr_change: change attribute=0
[1113642.558683] decode_attr_size: file size=0
[1113642.558684] decode_attr_fsid: fsid=(0x0/0x0)
[1113642.558684] decode_attr_fileid: fileid=524289
[1113642.558685] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.558685] decode_attr_mode: file mode=00
[1113642.558686] decode_attr_nlink: nlink=1
[1113642.558687] decode_attr_owner: uid=-2
[1113642.558687] decode_attr_group: gid=-2
[1113642.558688] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.558688] decode_attr_space_used: space used=0
[1113642.558689] decode_attr_time_access: atime=0
[1113642.558689] decode_attr_time_metadata: ctime=0
[1113642.558690] decode_attr_time_modify: mtime=0
[1113642.558690] decode_attr_mounted_on_fileid: fileid=0
[1113642.558691] decode_getfattr_attrs: xdr returned 0
[1113642.558692] decode_attr_type: type=00
[1113642.558693] decode_attr_change: change attribute=0
[1113642.558693] decode_attr_size: file size=0
[1113642.558694] decode_attr_fsid: fsid=(0x0/0x0)
[1113642.558694] decode_attr_fileid: fileid=524289
[1113642.558695] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.558695] decode_attr_mode: file mode=00
[1113642.558695] decode_attr_nlink: nlink=1
[1113642.558696] decode_attr_owner: uid=-2
[1113642.558696] decode_attr_group: gid=-2
[1113642.558697] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.558697] decode_attr_space_used: space used=0
[1113642.558698] decode_attr_time_access: atime=0
[1113642.558698] decode_attr_time_metadata: ctime=0
[1113642.558698] decode_attr_time_modify: mtime=0
[1113642.558699] decode_attr_mounted_on_fileid: fileid=0
[1113642.558699] decode_getfattr_attrs: xdr returned 0
[1113642.558700] decode_attr_type: type=0100000
[1113642.558701] decode_attr_change: change attribute=6364967891928841364
[1113642.558701] decode_attr_size: file size=0
[1113642.558702] decode_attr_fsid: fsid=(0xe6468c490ef04fb4/0xb59f3f06b1a42594)
[1113642.558702] decode_attr_fileid: fileid=524290
[1113642.558703] decode_attr_fs_locations: fs_locations done, error = 0
[1113642.558703] decode_attr_mode: file mode=0644
[1113642.558704] decode_attr_nlink: nlink=1
[1113642.558705] decode_attr_owner: uid=0
[1113642.558705] decode_attr_group: gid=0
[1113642.558706] decode_attr_rdev: rdev=(0x0:0x0)
[1113642.558706] decode_attr_space_used: space used=0
[1113642.558707] decode_attr_time_access: atime=1481959571
[1113642.558725] decode_attr_time_metadata: ctime=1481959571
[1113642.558727] decode_attr_time_modify: mtime=1481959571
[1113642.558727] decode_attr_mounted_on_fileid: fileid=524290
[1113642.558728] decode_getfattr_attrs: xdr returned 0
[1113642.558730] NFS: nfs_update_inode(0:43/524290 fh_crc=0xf34c6e39 ct=1 info=0x427e7f)
[1113642.558732] NFS: dentry_delete(/test1, 40840cc)
[1113642.558735] NFS: readdir(/) returns 0
[1113643.054220] nfs4_renew_state: start
[1113643.054230] nfs4_renew_state: done
[1113643.055726] nfs4_schedule_state_renewal: requeueing work. Lease period = 60
[1113643.055728] --> nfs_put_client({4})

服務端：

$ dmesg -c
[ 1441.035560] nfsd_dispatch: vers 4 proc 1
[ 1441.035567] nfsv4 compound op #1/3: 22 (OP_PUTFH)
[ 1441.035569] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.035581] nfsv4 compound op ffff880075f5a080 opcnt 3 #1: 22: status 0
[ 1441.035582] nfsv4 compound op #2/3: 3 (OP_ACCESS)
[ 1441.035584] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.035586] nfsv4 compound op ffff880075f5a080 opcnt 3 #2: 3: status 0
[ 1441.035587] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1441.035588] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.035591] nfsv4 compound op ffff880075f5a080 opcnt 3 #3: 9: status 0
[ 1441.035592] nfsv4 compound returned 0
[ 1441.036026] nfsd_dispatch: vers 4 proc 1
[ 1441.036030] nfsv4 compound op #1/2: 22 (OP_PUTFH)
[ 1441.036031] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036037] nfsv4 compound op ffff880075f58080 opcnt 2 #1: 22: status 0
[ 1441.036038] nfsv4 compound op #2/2: 9 (OP_GETATTR)
[ 1441.036039] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036045] nfsv4 compound op ffff880075f58080 opcnt 2 #2: 9: status 0
[ 1441.036045] nfsv4 compound returned 0
[ 1441.036409] nfsd_dispatch: vers 4 proc 1
[ 1441.036413] nfsv4 compound op #1/2: 22 (OP_PUTFH)
[ 1441.036415] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036423] nfsv4 compound op ffff880075f5a080 opcnt 2 #1: 22: status 0
[ 1441.036424] nfsv4 compound op #2/2: 9 (OP_GETATTR)
[ 1441.036425] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036456] nfsv4 compound op ffff880075f5a080 opcnt 2 #2: 9: status 0
[ 1441.036458] nfsv4 compound returned 0
[ 1441.036815] nfsd_dispatch: vers 4 proc 1
[ 1441.036818] nfsv4 compound op #1/2: 22 (OP_PUTFH)
[ 1441.036819] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036823] nfsv4 compound op ffff880075f5a080 opcnt 2 #1: 22: status 0
[ 1441.036823] nfsv4 compound op #2/2: 26 (OP_READDIR)
[ 1441.036825] nfsd: fh_verify(28: 00070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1441.036839] nfsd: fh_compose(exp fd:03/524289 subdir1/test1, ino=524290)
[ 1441.036856] nfsv4 compound op ffff880075f5a080 opcnt 2 #2: 26: status 0
[ 1441.036857] nfsv4 compound returned 0
[ 1441.535112] nfsd_dispatch: vers 4 proc 1
[ 1441.535119] nfsv4 compound op #1/1: 30 (OP_RENEW)
[ 1441.535121] process_renew(5854e49e/00000000): starting
[ 1441.535122] renewing client (clientid 5854e49e/00000000)
[ 1441.535124] nfsv4 compound op ffff880075f5a080 opcnt 1 #1: 30: status 0
[ 1441.535125] nfsv4 compound returned 0
[ 1441.535126] renewing client (clientid 5854e49e/00000000)

dd測試

客戶端：

$ dd if=/dev/zero of=/opt/export1/test.bin bs=8M count=1024
$ dmesg -c
[1114105.753138] NFS:       nfs_updatepage(/test.bin 4096@8586174464)
[1114105.753139] NFS:       nfs_updatepage returns 0 (isize 8586178560)
[1114105.753140] NFS: write_begin(/test.bin(524291), 4096@8586178560)
[1114105.753141] NFS: write_end(/test.bin(524291), 4096@8586178560)
[1114105.753141] NFS:       nfs_updatepage(/test.bin 4096@8586178560)
[1114105.753142] NFS:       nfs_updatepage returns 0 (isize 8586182656)
[1114105.753143] NFS: write_begin(/test.bin(524291), 4096@8586182656)
[1114105.753144] NFS: write_end(/test.bin(524291), 4096@8586182656)
[1114105.753145] NFS:       nfs_updatepage(/test.bin 4096@8586182656)
[1114105.753145] NFS:       nfs_updatepage returns 0 (isize 8586186752)
...
[1114105.756368] NFS: write_begin(/test.bin(524291), 4096@8589914112)
[1114105.756369] NFS: write_end(/test.bin(524291), 4096@8589914112)
[1114105.756370] NFS:       nfs_updatepage(/test.bin 4096@8589914112)
[1114105.756370] NFS:       nfs_updatepage returns 0 (isize 8589918208)
[1114105.756371] NFS: write_begin(/test.bin(524291), 4096@8589918208)
[1114105.756373] NFS: write_end(/test.bin(524291), 4096@8589918208)
[1114105.756373] NFS:       nfs_updatepage(/test.bin 4096@8589918208)
[1114105.756374] NFS:       nfs_updatepage returns 0 (isize 8589922304)
[1114105.756374] NFS: write_begin(/test.bin(524291), 4096@8589922304)
[1114105.756376] NFS: write_end(/test.bin(524291), 4096@8589922304)
[1114105.756376] NFS:       nfs_updatepage(/test.bin 4096@8589922304)
[1114105.756377] NFS:       nfs_updatepage returns 0 (isize 8589926400)
[1114105.756377] NFS: write_begin(/test.bin(524291), 4096@8589926400)
[1114105.756379] NFS: write_end(/test.bin(524291), 4096@8589926400)
[1114105.756379] NFS:       nfs_updatepage(/test.bin 4096@8589926400)
[1114105.756380] NFS:       nfs_updatepage returns 0 (isize 8589930496)
[1114105.756381] NFS: write_begin(/test.bin(524291), 4096@8589930496)
[1114105.756382] NFS: write_end(/test.bin(524291), 4096@8589930496)
[1114105.756382] NFS:       nfs_updatepage(/test.bin 4096@8589930496)
[1114105.756383] NFS:       nfs_updatepage returns 0 (isize 8589934592)
[1114105.756400] NFS: flush(/test.bin)
[1114105.756440] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8583462912)
[1114105.756448] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1114105.756449] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1114105.757716] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8583725056)
[1114105.757722] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.757723] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.758810] decode_attr_type: type=00
[1114105.758811] decode_attr_change: change attribute=6364971666868740273
[1114105.758812] decode_attr_size: file size=8583725056
[1114105.758813] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.758814] decode_attr_fileid: fileid=0
[1114105.758815] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.758815] decode_attr_mode: file mode=00
[1114105.758816] decode_attr_nlink: nlink=1
[1114105.758816] decode_attr_owner: uid=-2
[1114105.758817] decode_attr_group: gid=-2
[1114105.758818] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.758818] decode_attr_space_used: space used=0
[1114105.758819] decode_attr_time_access: atime=0
[1114105.758820] decode_attr_time_metadata: ctime=1481960450
[1114105.758820] decode_attr_time_modify: mtime=1481960450
[1114105.758821] decode_attr_mounted_on_fileid: fileid=0
[1114105.758821] decode_getfattr_attrs: xdr returned 0
[1114105.758822] decode_getfattr_generic: xdr returned 0
[1114105.758823] NFS: nfs_pgio_result:  2399, (status 262144)
[1114105.758825] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.758828] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.758829] NFS: isize change on server for file 0:43/524291 (8589934592 to 8583725056)
[1114105.758856] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8583987200)
[1114105.758859] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.758860] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.759865] decode_attr_type: type=00
[1114105.759866] decode_attr_change: change attribute=6364971666869740273
[1114105.759866] decode_attr_size: file size=8583987200
[1114105.759867] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.759868] decode_attr_fileid: fileid=0
[1114105.759868] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.759869] decode_attr_mode: file mode=00
[1114105.759870] decode_attr_nlink: nlink=1
[1114105.759870] decode_attr_owner: uid=-2
[1114105.759871] decode_attr_group: gid=-2
[1114105.759871] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.759872] decode_attr_space_used: space used=0
[1114105.759872] decode_attr_time_access: atime=0
[1114105.759873] decode_attr_time_metadata: ctime=1481960450
[1114105.759874] decode_attr_time_modify: mtime=1481960450
[1114105.759874] decode_attr_mounted_on_fileid: fileid=0
[1114105.759875] decode_getfattr_attrs: xdr returned 0
[1114105.759875] decode_getfattr_generic: xdr returned 0
[1114105.759876] NFS: nfs_pgio_result:  2400, (status 262144)
[1114105.759877] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.759880] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.759881] NFS: isize change on server for file 0:43/524291 (8589934592 to 8583987200)
[1114105.759906] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8584249344)
[1114105.759908] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.759909] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.760961] decode_attr_type: type=00
[1114105.760962] decode_attr_change: change attribute=6364971666870740272
[1114105.760962] decode_attr_size: file size=8584249344
[1114105.760963] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.760964] decode_attr_fileid: fileid=0
[1114105.760964] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.760965] decode_attr_mode: file mode=00
[1114105.760965] decode_attr_nlink: nlink=1
[1114105.760966] decode_attr_owner: uid=-2
[1114105.760966] decode_attr_group: gid=-2
[1114105.760967] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.760967] decode_attr_space_used: space used=0
[1114105.760968] decode_attr_time_access: atime=0
[1114105.760968] decode_attr_time_metadata: ctime=1481960450
[1114105.760969] decode_attr_time_modify: mtime=1481960450
[1114105.760969] decode_attr_mounted_on_fileid: fileid=0
[1114105.760970] decode_getfattr_attrs: xdr returned 0
[1114105.760970] decode_getfattr_generic: xdr returned 0
[1114105.760971] NFS: nfs_pgio_result:  2401, (status 262144)
[1114105.760973] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.760975] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.760976] NFS: isize change on server for file 0:43/524291 (8589934592 to 8584249344)
[1114105.761009] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8584511488)
[1114105.761012] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.761013] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.762240] decode_attr_type: type=00
[1114105.762241] decode_attr_change: change attribute=6364971666871740272
[1114105.762242] decode_attr_size: file size=8584511488
[1114105.762242] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.762243] decode_attr_fileid: fileid=0
[1114105.762243] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.762244] decode_attr_mode: file mode=00
[1114105.762244] decode_attr_nlink: nlink=1
[1114105.762245] decode_attr_owner: uid=-2
[1114105.762246] decode_attr_group: gid=-2
[1114105.762246] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.762247] decode_attr_space_used: space used=0
[1114105.762247] decode_attr_time_access: atime=0
[1114105.762248] decode_attr_time_metadata: ctime=1481960450
[1114105.762248] decode_attr_time_modify: mtime=1481960450
[1114105.762249] decode_attr_mounted_on_fileid: fileid=0
[1114105.762249] decode_getfattr_attrs: xdr returned 0
[1114105.762250] decode_getfattr_generic: xdr returned 0
[1114105.762251] NFS: nfs_pgio_result:  2402, (status 262144)
[1114105.762252] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.762254] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.762255] NFS: isize change on server for file 0:43/524291 (8589934592 to 8584511488)
[1114105.762279] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8584773632)
[1114105.762282] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.762282] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.763466] decode_attr_type: type=00
[1114105.763467] decode_attr_change: change attribute=6364971666872740272
[1114105.763468] decode_attr_size: file size=8584773632
[1114105.763469] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.763469] decode_attr_fileid: fileid=0
[1114105.763470] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.763470] decode_attr_mode: file mode=00
[1114105.763471] decode_attr_nlink: nlink=1
[1114105.763472] decode_attr_owner: uid=-2
[1114105.763472] decode_attr_group: gid=-2
[1114105.763473] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.763473] decode_attr_space_used: space used=0
[1114105.763474] decode_attr_time_access: atime=0
[1114105.763474] decode_attr_time_metadata: ctime=1481960450
[1114105.763475] decode_attr_time_modify: mtime=1481960450
[1114105.763475] decode_attr_mounted_on_fileid: fileid=0
[1114105.763476] decode_getfattr_attrs: xdr returned 0
[1114105.763477] decode_getfattr_generic: xdr returned 0
[1114105.763477] NFS: nfs_pgio_result:  2403, (status 262144)
[1114105.763479] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.763480] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.763481] NFS: isize change on server for file 0:43/524291 (8589934592 to 8584773632)
[1114105.763514] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8585035776)
[1114105.763516] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.763517] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.764651] decode_attr_type: type=00
[1114105.764652] decode_attr_change: change attribute=6364971666873740271
[1114105.764653] decode_attr_size: file size=8585035776
[1114105.764653] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.764654] decode_attr_fileid: fileid=0
[1114105.764655] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.764655] decode_attr_mode: file mode=00
[1114105.764656] decode_attr_nlink: nlink=1
[1114105.764656] decode_attr_owner: uid=-2
[1114105.764657] decode_attr_group: gid=-2
[1114105.764657] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.764658] decode_attr_space_used: space used=0
[1114105.764658] decode_attr_time_access: atime=0
[1114105.764659] decode_attr_time_metadata: ctime=1481960450
[1114105.764660] decode_attr_time_modify: mtime=1481960450
[1114105.764660] decode_attr_mounted_on_fileid: fileid=0
[1114105.764661] decode_getfattr_attrs: xdr returned 0
[1114105.764661] decode_getfattr_generic: xdr returned 0
[1114105.764662] NFS: nfs_pgio_result:  2404, (status 262144)
[1114105.764663] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.764665] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.764665] NFS: isize change on server for file 0:43/524291 (8589934592 to 8585035776)
[1114105.764690] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8585297920)
[1114105.764692] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.764693] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.765914] decode_attr_type: type=00
[1114105.765916] decode_attr_change: change attribute=6364971666875740270
[1114105.765916] decode_attr_size: file size=8585297920
[1114105.765917] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.765917] decode_attr_fileid: fileid=0
[1114105.765918] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.765918] decode_attr_mode: file mode=00
[1114105.765919] decode_attr_nlink: nlink=1
[1114105.765920] decode_attr_owner: uid=-2
[1114105.765920] decode_attr_group: gid=-2
[1114105.765921] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.765921] decode_attr_space_used: space used=0
[1114105.765922] decode_attr_time_access: atime=0
[1114105.765922] decode_attr_time_metadata: ctime=1481960450
[1114105.765923] decode_attr_time_modify: mtime=1481960450
[1114105.765923] decode_attr_mounted_on_fileid: fileid=0
[1114105.765924] decode_getfattr_attrs: xdr returned 0
[1114105.765924] decode_getfattr_generic: xdr returned 0
[1114105.765925] NFS: nfs_pgio_result:  2405, (status 262144)
[1114105.765926] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.765928] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.765929] NFS: isize change on server for file 0:43/524291 (8589934592 to 8585297920)
[1114105.765953] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8585560064)
[1114105.765956] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.765956] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.767092] decode_attr_type: type=00
[1114105.767094] decode_attr_change: change attribute=6364971666876740270
[1114105.767094] decode_attr_size: file size=8585560064
[1114105.767095] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.767096] decode_attr_fileid: fileid=0
[1114105.767097] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.767097] decode_attr_mode: file mode=00
[1114105.767098] decode_attr_nlink: nlink=1
[1114105.767098] decode_attr_owner: uid=-2
[1114105.767099] decode_attr_group: gid=-2
[1114105.767100] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.767100] decode_attr_space_used: space used=0
[1114105.767101] decode_attr_time_access: atime=0
[1114105.767101] decode_attr_time_metadata: ctime=1481960450
[1114105.767102] decode_attr_time_modify: mtime=1481960450
[1114105.767102] decode_attr_mounted_on_fileid: fileid=0
[1114105.767103] decode_getfattr_attrs: xdr returned 0
[1114105.767103] decode_getfattr_generic: xdr returned 0
[1114105.767104] NFS: nfs_pgio_result:  2406, (status 262144)
[1114105.767106] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.767108] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.767109] NFS: isize change on server for file 0:43/524291 (8589934592 to 8585560064)
[1114105.767136] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8585822208)
[1114105.767139] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.767140] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.768110] decode_attr_type: type=00
[1114105.768111] decode_attr_change: change attribute=6364971666877740269
[1114105.768112] decode_attr_size: file size=8585822208
[1114105.768112] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.768113] decode_attr_fileid: fileid=0
[1114105.768113] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.768114] decode_attr_mode: file mode=00
[1114105.768114] decode_attr_nlink: nlink=1
[1114105.768115] decode_attr_owner: uid=-2
[1114105.768116] decode_attr_group: gid=-2
[1114105.768116] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.768117] decode_attr_space_used: space used=0
[1114105.768117] decode_attr_time_access: atime=0
[1114105.768118] decode_attr_time_metadata: ctime=1481960450
[1114105.768118] decode_attr_time_modify: mtime=1481960450
[1114105.768119] decode_attr_mounted_on_fileid: fileid=0
[1114105.768119] decode_getfattr_attrs: xdr returned 0
[1114105.768120] decode_getfattr_generic: xdr returned 0
[1114105.768121] NFS: nfs_pgio_result:  2407, (status 262144)
[1114105.768122] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.768124] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.768125] NFS: isize change on server for file 0:43/524291 (8589934592 to 8585822208)
[1114105.768153] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8586084352)
[1114105.768156] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.768156] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.769265] decode_attr_type: type=00
[1114105.769266] decode_attr_change: change attribute=6364971666878740269
[1114105.769267] decode_attr_size: file size=8586084352
[1114105.769268] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.769268] decode_attr_fileid: fileid=0
[1114105.769269] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.769270] decode_attr_mode: file mode=00
[1114105.769270] decode_attr_nlink: nlink=1
[1114105.769271] decode_attr_owner: uid=-2
[1114105.769271] decode_attr_group: gid=-2
[1114105.769272] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.769272] decode_attr_space_used: space used=0
[1114105.769273] decode_attr_time_access: atime=0
[1114105.769274] decode_attr_time_metadata: ctime=1481960450
[1114105.769274] decode_attr_time_modify: mtime=1481960450
[1114105.769275] decode_attr_mounted_on_fileid: fileid=0
[1114105.769275] decode_getfattr_attrs: xdr returned 0
[1114105.769276] decode_getfattr_generic: xdr returned 0
[1114105.769277] NFS: nfs_pgio_result:  2408, (status 262144)
[1114105.769278] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.769280] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.769281] NFS: isize change on server for file 0:43/524291 (8589934592 to 8586084352)
[1114105.769312] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8586346496)
[1114105.769315] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.769316] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.770441] decode_attr_type: type=00
[1114105.770443] decode_attr_change: change attribute=6364971666879740269
[1114105.770443] decode_attr_size: file size=8586346496
[1114105.770444] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.770444] decode_attr_fileid: fileid=0
[1114105.770445] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.770446] decode_attr_mode: file mode=00
[1114105.770446] decode_attr_nlink: nlink=1
[1114105.770447] decode_attr_owner: uid=-2
[1114105.770447] decode_attr_group: gid=-2
[1114105.770448] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.770448] decode_attr_space_used: space used=0
[1114105.770449] decode_attr_time_access: atime=0
[1114105.770449] decode_attr_time_metadata: ctime=1481960450
[1114105.770450] decode_attr_time_modify: mtime=1481960450
[1114105.770450] decode_attr_mounted_on_fileid: fileid=0
[1114105.770451] decode_getfattr_attrs: xdr returned 0
[1114105.770451] decode_getfattr_generic: xdr returned 0
[1114105.770452] NFS: nfs_pgio_result:  2409, (status 262144)
[1114105.770453] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.770455] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.770456] NFS: isize change on server for file 0:43/524291 (8589934592 to 8586346496)
[1114105.770510] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8586608640)
[1114105.770515] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.770515] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.771581] decode_attr_type: type=00
[1114105.771582] decode_attr_change: change attribute=6364971666881740268
[1114105.771583] decode_attr_size: file size=8586608640
[1114105.771584] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.771584] decode_attr_fileid: fileid=0
[1114105.771585] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.771585] decode_attr_mode: file mode=00
[1114105.771586] decode_attr_nlink: nlink=1
[1114105.771586] decode_attr_owner: uid=-2
[1114105.771587] decode_attr_group: gid=-2
[1114105.771587] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.771588] decode_attr_space_used: space used=0
[1114105.771588] decode_attr_time_access: atime=0
[1114105.771589] decode_attr_time_metadata: ctime=1481960450
[1114105.771590] decode_attr_time_modify: mtime=1481960450
[1114105.771590] decode_attr_mounted_on_fileid: fileid=0
[1114105.771591] decode_getfattr_attrs: xdr returned 0
[1114105.771591] decode_getfattr_generic: xdr returned 0
[1114105.771592] NFS: nfs_pgio_result:  2410, (status 262144)
[1114105.771593] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.771595] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.771595] NFS: isize change on server for file 0:43/524291 (8589934592 to 8586608640)
[1114105.771624] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8586870784)
[1114105.771626] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.771627] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.773057] decode_attr_type: type=00
[1114105.773059] decode_attr_change: change attribute=6364971666882740267
[1114105.773060] decode_attr_size: file size=8586870784
[1114105.773061] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.773061] decode_attr_fileid: fileid=0
[1114105.773062] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.773063] decode_attr_mode: file mode=00
[1114105.773063] decode_attr_nlink: nlink=1
[1114105.773064] decode_attr_owner: uid=-2
[1114105.773064] decode_attr_group: gid=-2
[1114105.773065] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.773066] decode_attr_space_used: space used=0
[1114105.773066] decode_attr_time_access: atime=0
[1114105.773067] decode_attr_time_metadata: ctime=1481960450
[1114105.773067] decode_attr_time_modify: mtime=1481960450
[1114105.773068] decode_attr_mounted_on_fileid: fileid=0
[1114105.773069] decode_getfattr_attrs: xdr returned 0
[1114105.773069] decode_getfattr_generic: xdr returned 0
[1114105.773070] NFS: nfs_pgio_result:  2411, (status 262144)
[1114105.773072] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.773074] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.773075] NFS: isize change on server for file 0:43/524291 (8589934592 to 8586870784)
[1114105.773148] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8587132928)
[1114105.773153] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.773154] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.774320] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8587395072)
[1114105.774327] --> nfs4_alloc_slot used_slots=0003 highest_used=1 max_slots=1024
[1114105.774328] <-- nfs4_alloc_slot used_slots=0007 highest_used=2 slotid=2
[1114105.775645] decode_attr_type: type=00
[1114105.775648] decode_attr_change: change attribute=6364971666884740267
[1114105.775648] decode_attr_size: file size=8587132928
[1114105.775649] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.775650] decode_attr_fileid: fileid=0
[1114105.775651] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.775652] decode_attr_mode: file mode=00
[1114105.775652] decode_attr_nlink: nlink=1
[1114105.775653] decode_attr_owner: uid=-2
[1114105.775653] decode_attr_group: gid=-2
[1114105.775654] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.775655] decode_attr_space_used: space used=0
[1114105.775656] decode_attr_time_access: atime=0
[1114105.775656] decode_attr_time_metadata: ctime=1481960450
[1114105.775657] decode_attr_time_modify: mtime=1481960450
[1114105.775658] decode_attr_mounted_on_fileid: fileid=0
[1114105.775658] decode_getfattr_attrs: xdr returned 0
[1114105.775659] decode_getfattr_generic: xdr returned 0
[1114105.775660] NFS: nfs_pgio_result:  2412, (status 262144)
[1114105.775662] nfs4_free_slot: slotid 1 highest_used_slotid 2
[1114105.775665] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.775666] NFS: isize change on server for file 0:43/524291 (8589934592 to 8587132928)
[1114105.775671] decode_attr_type: type=00
[1114105.775671] decode_attr_change: change attribute=6364971666885740266
[1114105.775672] decode_attr_size: file size=8587395072
[1114105.775672] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.775673] decode_attr_fileid: fileid=0
[1114105.775673] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.775674] decode_attr_mode: file mode=00
[1114105.775674] decode_attr_nlink: nlink=1
[1114105.775675] decode_attr_owner: uid=-2
[1114105.775675] decode_attr_group: gid=-2
[1114105.775676] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.775676] decode_attr_space_used: space used=0
[1114105.775677] decode_attr_time_access: atime=0
[1114105.775677] decode_attr_time_metadata: ctime=1481960450
[1114105.775678] decode_attr_time_modify: mtime=1481960450
[1114105.775678] decode_attr_mounted_on_fileid: fileid=0
[1114105.775679] decode_getfattr_attrs: xdr returned 0
[1114105.775679] decode_getfattr_generic: xdr returned 0
[1114105.775680] NFS: nfs_pgio_result:  2413, (status 262144)
[1114105.775681] nfs4_free_slot: slotid 0 highest_used_slotid 2
[1114105.775682] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.775682] NFS: isize change on server for file 0:43/524291 (8589934592 to 8587395072)
[1114105.775715] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8587657216)
[1114105.775719] --> nfs4_alloc_slot used_slots=0004 highest_used=2 max_slots=1024
[1114105.775720] <-- nfs4_alloc_slot used_slots=0005 highest_used=2 slotid=0
[1114105.776722] decode_attr_type: type=00
[1114105.776723] decode_attr_change: change attribute=6364971666886740266
[1114105.776724] decode_attr_size: file size=8587657216
[1114105.776724] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.776725] decode_attr_fileid: fileid=0
[1114105.776726] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.776726] decode_attr_mode: file mode=00
[1114105.776727] decode_attr_nlink: nlink=1
[1114105.776728] decode_attr_owner: uid=-2
[1114105.776728] decode_attr_group: gid=-2
[1114105.776729] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.776729] decode_attr_space_used: space used=0
[1114105.776730] decode_attr_time_access: atime=0
[1114105.776730] decode_attr_time_metadata: ctime=1481960450
[1114105.776731] decode_attr_time_modify: mtime=1481960450
[1114105.776731] decode_attr_mounted_on_fileid: fileid=0
[1114105.776732] decode_getfattr_attrs: xdr returned 0
[1114105.776732] decode_getfattr_generic: xdr returned 0
[1114105.776733] NFS: nfs_pgio_result:  2414, (status 262144)
[1114105.776735] nfs4_free_slot: slotid 2 highest_used_slotid 0
[1114105.776737] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.776739] NFS: isize change on server for file 0:43/524291 (8589934592 to 8587657216)
[1114105.776772] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8587919360)
[1114105.776775] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.776776] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.778295] decode_attr_type: type=00
[1114105.778298] decode_attr_change: change attribute=6364971666887740265
[1114105.778299] decode_attr_size: file size=8587919360
[1114105.778300] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.778300] decode_attr_fileid: fileid=0
[1114105.778301] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.778302] decode_attr_mode: file mode=00
[1114105.778302] decode_attr_nlink: nlink=1
[1114105.778303] decode_attr_owner: uid=-2
[1114105.778304] decode_attr_group: gid=-2
[1114105.778304] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.778305] decode_attr_space_used: space used=0
[1114105.778305] decode_attr_time_access: atime=0
[1114105.778306] decode_attr_time_metadata: ctime=1481960450
[1114105.778307] decode_attr_time_modify: mtime=1481960450
[1114105.778307] decode_attr_mounted_on_fileid: fileid=0
[1114105.778308] decode_getfattr_attrs: xdr returned 0
[1114105.778309] decode_getfattr_generic: xdr returned 0
[1114105.778310] NFS: nfs_pgio_result:  2415, (status 262144)
[1114105.778312] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.778315] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.778316] NFS: isize change on server for file 0:43/524291 (8589934592 to 8587919360)
[1114105.778362] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8588181504)
[1114105.778366] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.778366] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.779448] decode_attr_type: type=00
[1114105.779450] decode_attr_change: change attribute=6364971666888740265
[1114105.779450] decode_attr_size: file size=8588181504
[1114105.779451] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.779452] decode_attr_fileid: fileid=0
[1114105.779452] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.779453] decode_attr_mode: file mode=00
[1114105.779453] decode_attr_nlink: nlink=1
[1114105.779454] decode_attr_owner: uid=-2
[1114105.779455] decode_attr_group: gid=-2
[1114105.779455] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.779456] decode_attr_space_used: space used=0
[1114105.779456] decode_attr_time_access: atime=0
[1114105.779457] decode_attr_time_metadata: ctime=1481960450
[1114105.779457] decode_attr_time_modify: mtime=1481960450
[1114105.779458] decode_attr_mounted_on_fileid: fileid=0
[1114105.779458] decode_getfattr_attrs: xdr returned 0
[1114105.779459] decode_getfattr_generic: xdr returned 0
[1114105.779460] NFS: nfs_pgio_result:  2416, (status 262144)
[1114105.779462] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.779464] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.779465] NFS: isize change on server for file 0:43/524291 (8589934592 to 8588181504)
[1114105.779519] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8588443648)
[1114105.779524] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.779525] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.780702] decode_attr_type: type=00
[1114105.780703] decode_attr_change: change attribute=6364971666889740264
[1114105.780704] decode_attr_size: file size=8588443648
[1114105.780704] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.780705] decode_attr_fileid: fileid=0
[1114105.780706] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.780706] decode_attr_mode: file mode=00
[1114105.780707] decode_attr_nlink: nlink=1
[1114105.780707] decode_attr_owner: uid=-2
[1114105.780708] decode_attr_group: gid=-2
[1114105.780708] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.780709] decode_attr_space_used: space used=0
[1114105.780709] decode_attr_time_access: atime=0
[1114105.780710] decode_attr_time_metadata: ctime=1481960450
[1114105.780710] decode_attr_time_modify: mtime=1481960450
[1114105.780711] decode_attr_mounted_on_fileid: fileid=0
[1114105.780711] decode_getfattr_attrs: xdr returned 0
[1114105.780712] decode_getfattr_generic: xdr returned 0
[1114105.780713] NFS: nfs_pgio_result:  2417, (status 262144)
[1114105.780714] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.780716] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.780717] NFS: isize change on server for file 0:43/524291 (8589934592 to 8588443648)
[1114105.780749] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8588705792)
[1114105.780752] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.780753] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.782264] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8588967936)
[1114105.782284] --> nfs4_alloc_slot used_slots=0003 highest_used=1 max_slots=1024
[1114105.782285] <-- nfs4_alloc_slot used_slots=0007 highest_used=2 slotid=2
[1114105.783400] decode_attr_type: type=00
[1114105.783401] decode_attr_change: change attribute=6364971666891740263
[1114105.783402] decode_attr_size: file size=8588705792
[1114105.783403] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.783404] decode_attr_fileid: fileid=0
[1114105.783405] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.783405] decode_attr_mode: file mode=00
[1114105.783406] decode_attr_nlink: nlink=1
[1114105.783406] decode_attr_owner: uid=-2
[1114105.783407] decode_attr_group: gid=-2
[1114105.783408] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.783408] decode_attr_space_used: space used=0
[1114105.783409] decode_attr_time_access: atime=0
[1114105.783409] decode_attr_time_metadata: ctime=1481960450
[1114105.783410] decode_attr_time_modify: mtime=1481960450
[1114105.783410] decode_attr_mounted_on_fileid: fileid=0
[1114105.783411] decode_getfattr_attrs: xdr returned 0
[1114105.783411] decode_getfattr_generic: xdr returned 0
[1114105.783412] NFS: nfs_pgio_result:  2418, (status 262144)
[1114105.783414] nfs4_free_slot: slotid 1 highest_used_slotid 2
[1114105.783417] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.783418] NFS: isize change on server for file 0:43/524291 (8589934592 to 8588705792)
[1114105.783422] decode_attr_type: type=00
[1114105.783422] decode_attr_change: change attribute=6364971666892740263
[1114105.783423] decode_attr_size: file size=8588967936
[1114105.783423] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.783424] decode_attr_fileid: fileid=0
[1114105.783424] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.783425] decode_attr_mode: file mode=00
[1114105.783425] decode_attr_nlink: nlink=1
[1114105.783426] decode_attr_owner: uid=-2
[1114105.783426] decode_attr_group: gid=-2
[1114105.783427] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.783427] decode_attr_space_used: space used=0
[1114105.783428] decode_attr_time_access: atime=0
[1114105.783428] decode_attr_time_metadata: ctime=1481960450
[1114105.783429] decode_attr_time_modify: mtime=1481960450
[1114105.783429] decode_attr_mounted_on_fileid: fileid=0
[1114105.783430] decode_getfattr_attrs: xdr returned 0
[1114105.783430] decode_getfattr_generic: xdr returned 0
[1114105.783431] NFS: nfs_pgio_result:  2419, (status 262144)
[1114105.783431] nfs4_free_slot: slotid 0 highest_used_slotid 2
[1114105.783432] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.783433] NFS: isize change on server for file 0:43/524291 (8589934592 to 8588967936)
[1114105.783473] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8589230080)
[1114105.783477] --> nfs4_alloc_slot used_slots=0004 highest_used=2 max_slots=1024
[1114105.783478] <-- nfs4_alloc_slot used_slots=0005 highest_used=2 slotid=0
[1114105.784682] decode_attr_type: type=00
[1114105.784683] decode_attr_change: change attribute=6364971666894740262
[1114105.784684] decode_attr_size: file size=8589230080
[1114105.784685] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.784685] decode_attr_fileid: fileid=0
[1114105.784686] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.784687] decode_attr_mode: file mode=00
[1114105.784687] decode_attr_nlink: nlink=1
[1114105.784688] decode_attr_owner: uid=-2
[1114105.784688] decode_attr_group: gid=-2
[1114105.784689] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.784689] decode_attr_space_used: space used=0
[1114105.784690] decode_attr_time_access: atime=0
[1114105.784690] decode_attr_time_metadata: ctime=1481960450
[1114105.784691] decode_attr_time_modify: mtime=1481960450
[1114105.784691] decode_attr_mounted_on_fileid: fileid=0
[1114105.784692] decode_getfattr_attrs: xdr returned 0
[1114105.784692] decode_getfattr_generic: xdr returned 0
[1114105.784693] NFS: nfs_pgio_result:  2420, (status 262144)
[1114105.784695] nfs4_free_slot: slotid 2 highest_used_slotid 0
[1114105.784697] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.784698] NFS: isize change on server for file 0:43/524291 (8589934592 to 8589230080)
[1114105.784728] NFS:     0 initiated pgio call (req 0:43/524291, 262144 bytes @ offset 8589492224)
[1114105.784731] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=1024
[1114105.784732] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[1114105.785827] decode_attr_type: type=00
[1114105.785828] decode_attr_change: change attribute=6364971666895740262
[1114105.785829] decode_attr_size: file size=8589492224
[1114105.785829] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.785830] decode_attr_fileid: fileid=0
[1114105.785831] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.785831] decode_attr_mode: file mode=00
[1114105.785831] decode_attr_nlink: nlink=1
[1114105.785832] decode_attr_owner: uid=-2
[1114105.785833] decode_attr_group: gid=-2
[1114105.785833] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.785834] decode_attr_space_used: space used=0
[1114105.785834] decode_attr_time_access: atime=0
[1114105.785835] decode_attr_time_metadata: ctime=1481960450
[1114105.785835] decode_attr_time_modify: mtime=1481960450
[1114105.785835] decode_attr_mounted_on_fileid: fileid=0
[1114105.785836] decode_getfattr_attrs: xdr returned 0
[1114105.785836] decode_getfattr_generic: xdr returned 0
[1114105.785837] NFS: nfs_pgio_result:  2421, (status 262144)
[1114105.785839] nfs4_free_slot: slotid 0 highest_used_slotid 1
[1114105.785841] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.785842] NFS: isize change on server for file 0:43/524291 (8589934592 to 8589492224)
[1114105.785864] NFS:     0 initiated pgio call (req 0:43/524291, 180224 bytes @ offset 8589754368)
[1114105.785870] --> nfs4_alloc_slot used_slots=0002 highest_used=1 max_slots=1024
[1114105.785870] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=0
[1114105.787575] decode_attr_type: type=00
[1114105.787588] decode_attr_change: change attribute=6364971666897740261
[1114105.787589] decode_attr_size: file size=8589754368
[1114105.787589] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.787590] decode_attr_fileid: fileid=0
[1114105.787591] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.787591] decode_attr_mode: file mode=00
[1114105.787592] decode_attr_nlink: nlink=1
[1114105.787592] decode_attr_owner: uid=-2
[1114105.787593] decode_attr_group: gid=-2
[1114105.787593] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.787594] decode_attr_space_used: space used=0
[1114105.787594] decode_attr_time_access: atime=0
[1114105.787595] decode_attr_time_metadata: ctime=1481960450
[1114105.787595] decode_attr_time_modify: mtime=1481960450
[1114105.787596] decode_attr_mounted_on_fileid: fileid=0
[1114105.787596] decode_getfattr_attrs: xdr returned 0
[1114105.787597] decode_getfattr_generic: xdr returned 0
[1114105.787598] NFS: nfs_pgio_result:  2422, (status 262144)
[1114105.787599] nfs4_free_slot: slotid 1 highest_used_slotid 0
[1114105.787601] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x7e0c0)
[1114105.787615] NFS: isize change on server for file 0:43/524291 (8589934592 to 8589754368)
[1114105.788659] systemd-journald[618]: /dev/kmsg buffer overrun, some messages lost.
[1114105.790553] decode_attr_type: type=00
[1114105.790555] decode_attr_change: change attribute=6364971666897740261
[1114105.790556] decode_attr_size: file size=8589934592
[1114105.790557] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.790557] decode_attr_fileid: fileid=0
[1114105.790558] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.790559] decode_attr_mode: file mode=00
[1114105.790559] decode_attr_nlink: nlink=1
[1114105.790560] decode_attr_owner: uid=-2
[1114105.790560] decode_attr_group: gid=-2
[1114105.790561] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.790561] decode_attr_space_used: space used=0
[1114105.790562] decode_attr_time_access: atime=0
[1114105.790563] decode_attr_time_metadata: ctime=1481960450
[1114105.790563] decode_attr_time_modify: mtime=1481960450
[1114105.790564] decode_attr_mounted_on_fileid: fileid=0
[1114105.790564] decode_getfattr_attrs: xdr returned 0
[1114105.790565] decode_getfattr_generic: xdr returned 0
[1114105.790566] NFS: nfs_pgio_result:  2423, (status 180224)
[1114105.790567] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1114105.801654] NFS: fsync file(/test.bin) datasync 0
[1114105.801825] NFS:     0 initiated commit call
[1114105.801832] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1114105.801833] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1114105.829570] NFS:  2424 nfs_commit_done (status 0)
[1114105.829576] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1114105.840569] NFS:       commit (0:43/524291 4096@8553123840)
[1114105.840572]  OK
[1114105.840575] NFS:       commit (0:43/524291 4096@8553127936)
[1114105.840575]  OK
...
[1114105.850031] NFS:       commit (0:43/524291 4096@8589918208)
[1114105.850031]  OK
[1114105.850031] NFS:       commit (0:43/524291 4096@8589922304)
[1114105.850032]  OK
[1114105.850032] NFS:       commit (0:43/524291 4096@8589926400)
[1114105.850033]  OK
[1114105.850033] NFS:       commit (0:43/524291 4096@8589930496)
[1114105.850034]  OK
[1114105.850048] NFS: release(/test.bin)
[1114105.850061] nfs4_close_prepare: begin!
[1114105.850063] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=1024
[1114105.850064] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[1114105.850065] nfs4_close_prepare: done!
[1114105.850177] systemd-journald[618]: /dev/kmsg buffer overrun, some messages lost.
[1114105.851595] decode_attr_type: type=00
[1114105.851596] decode_attr_change: change attribute=6364971666897740261
[1114105.851597] decode_attr_size: file size=8589934592
[1114105.851598] decode_attr_fsid: fsid=(0x0/0x0)
[1114105.851598] decode_attr_fileid: fileid=0
[1114105.851599] decode_attr_fs_locations: fs_locations done, error = 0
[1114105.851600] decode_attr_mode: file mode=00
[1114105.851600] decode_attr_nlink: nlink=1
[1114105.851601] decode_attr_owner: uid=-2
[1114105.851602] decode_attr_group: gid=-2
[1114105.851602] decode_attr_rdev: rdev=(0x0:0x0)
[1114105.851603] decode_attr_space_used: space used=0
[1114105.851603] decode_attr_time_access: atime=0
[1114105.851604] decode_attr_time_metadata: ctime=1481960450
[1114105.851605] decode_attr_time_modify: mtime=1481960450
[1114105.851605] decode_attr_mounted_on_fileid: fileid=0
[1114105.851606] decode_getfattr_attrs: xdr returned 0
[1114105.851606] decode_getfattr_generic: xdr returned 0
[1114105.851607] nfs4_close_done: begin!
[1114105.851609] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[1114105.851612] NFS: nfs_update_inode(0:43/524291 fh_crc=0xb084f9f0 ct=2 info=0x26040)
[1114105.851613] nfs4_close_done: done, ret = 0!
[1114105.862646] NFS: dentry_delete(/test.bin, 400408c)
[1114115.077218] nfs4_renew_state: start
[1114115.077222] nfs4_renew_state: failed to call renewd. Reason: lease not expired 
[1114115.077224] nfs4_schedule_state_renewal: requeueing work. Lease period = 51
[1114115.077225] nfs4_renew_state: done
[1114166.044757] nfs4_renew_state: start
[1114166.044767] nfs4_renew_state: done
[1114166.045170] nfs4_schedule_state_renewal: requeueing work. Lease period = 60
[1114166.045173] --> nfs_put_client({4})

服務端：

$ dmesg -c
[ 1902.093749] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1902.093752] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.093759] nfsv4 compound op ffff880075f5c080 opcnt 3 #3: 9: status 0
[ 1902.093760] nfsv4 compound returned 0
[ 1902.093762] renewing client (clientid 5854e49e/00000000)
[ 1902.094070] nfsd_dispatch: vers 4 proc 1
[ 1902.094086] nfsv4 compound op #1/3: 22 (OP_PUTFH)
[ 1902.094088] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.094096] nfsv4 compound op ffff880075f58080 opcnt 3 #1: 22: status 0
[ 1902.094097] nfsv4 compound op #2/3: 38 (OP_WRITE)
[ 1902.094099] renewing client (clientid 5854e49e/00000000)
[ 1902.094249] nfsd: write complete host_err=262144
[ 1902.094250] nfsv4 compound op ffff880075f58080 opcnt 3 #2: 38: status 0
[ 1902.094251] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1902.094252] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.094254] nfsv4 compound op ffff880075f58080 opcnt 3 #3: 9: status 0
[ 1902.094255] nfsv4 compound returned 0
[ 1902.094255] renewing client (clientid 5854e49e/00000000)
[ 1902.096240] nfsd_dispatch: vers 4 proc 1
[ 1902.096255] nfsv4 compound op #1/3: 22 (OP_PUTFH)
[ 1902.096257] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.096266] nfsv4 compound op ffff880075f5c080 opcnt 3 #1: 22: status 0
[ 1902.096267] nfsv4 compound op #2/3: 38 (OP_WRITE)
[ 1902.096269] renewing client (clientid 5854e49e/00000000)
[ 1902.096365] nfsd_dispatch: vers 4 proc 1
[ 1902.096392] nfsv4 compound op #1/3: 22 (OP_PUTFH)
[ 1902.096394] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.096399] nfsv4 compound op ffff880075f58080 opcnt 3 #1: 22: status 0
[ 1902.096400] nfsv4 compound op #2/3: 38 (OP_WRITE)
[ 1902.096401] renewing client (clientid 5854e49e/00000000)
[ 1902.096440] nfsd: write complete host_err=262144
[ 1902.096442] nfsv4 compound op ffff880075f5c080 opcnt 3 #2: 38: status 0
[ 1902.096443] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1902.096444] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.096447] nfsv4 compound op ffff880075f5c080 opcnt 3 #3: 9: status 0
[ 1902.096448] nfsv4 compound returned 0
[ 1902.096572] nfsd: write complete host_err=262144
[ 1902.096574] nfsv4 compound op ffff880075f58080 opcnt 3 #2: 38: status 0
[ 1902.096575] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1902.096576] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1902.096578] nfsv4 compound op ffff880075f58080 opcnt 3 #3: 9: status 0
[ 1902.096579] nfsv4 compound returned 0
[ 1902.096580] renewing client (clientid 5854e49e/00000000)
[ 1902.096693] nfsd_dispatch: vers 4 proc 1
[ 1902.096701] nfsv4 compound op #1/3: 22 (OP_PUTFH)
[ 1902.096702] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
...
[ 1904.329281] NFSD: nfsd4_close on file test.bin
[ 1904.329283] NFSD: nfs4_preprocess_seqid_op: seqid=5 stateid = (5854e49e/00000000/00000004/00000002)
[ 1904.329284] renewing client (clientid 5854e49e/00000000)
[ 1904.329287] NFSD: move_to_close_lru nfs4_openowner ffff880035074000
[ 1904.329293] nfsv4 compound op ffff880078cc5080 opcnt 3 #2: 4: status 0
[ 1904.329294] nfsv4 compound op #3/3: 9 (OP_GETATTR)
[ 1904.329295] nfsd: fh_verify(36: 01070001 00080001 00000000 498c46e6 b44ff00e 063f9fb5)
[ 1904.329298] nfsv4 compound op ffff880078cc5080 opcnt 3 #3: 9: status 0
[ 1904.329299] nfsv4 compound returned 0
[ 1904.329300] renewing client (clientid 5854e49e/00000000)
[ 1964.523277] nfsd_dispatch: vers 4 proc 1
[ 1964.523283] nfsv4 compound op #1/1: 30 (OP_RENEW)
[ 1964.523285] process_renew(5854e49e/00000000): starting
[ 1964.523287] renewing client (clientid 5854e49e/00000000)
[ 1964.523288] nfsv4 compound op ffff880078cc5080 opcnt 1 #1: 30: status 0
[ 1964.523289] nfsv4 compound returned 0
[ 1964.523290] renewing client (clientid 5854e49e/00000000)
[ 1973.438951] NFSD: laundromat service - starting
[ 1973.438955] NFSD: laundromat_main - sleeping for 21 seconds
[ 1994.493691] NFSD: laundromat service - starting
[ 1994.493732] NFSD: laundromat_main - sleeping for 60 seconds

使用“mount --bind”目錄作爲NFS子目錄的測試 2 (iSCSI,multipath,LVM,EXT4,bind,NFSv4)

服務端

$ yum install -y scsi-target-utils iscsi-initiator-utils
$ cp -v /etc/tgt/targets.conf /etc/tgt/targets.conf.bak
$ vi /etc/tgt/targets.conf
<target iqn.2016-10.org.lr:lvm>
  backing-store /dev/sdd1
  initiator-address 192.168.0.0/16
</target>
$ vi /etc/lvm/lvm.conf
global_filter = [ "r|.*/|" ]

$ systemctl enable tgtd.service
$ systemctl restart tgtd.service
$ systemctl status tgtd.service

$ tgtadm -L iscsi -m target -o show

$ systemctl enable iscsid.service
$ systemctl restart iscsid.service
$ systemctl status iscsid.service

$ iscsiadm -m node - l
$ iscsiadm -m discovery -t st -p 192.168.195.131
$ iscsiadm -m node -T iqn.2016-10.org.lr:lvm -p 192.168.195.131:3260 -l

$ yum install -y device-mapper*
$ modprobe dm-multipath

$ /lib/udev/scsi_id -g -u /dev/sde
360000000000000000e00000000010001

$ vi /etc/multipath.conf
blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^sd[a-d][0-9]*"
}

defaults {
  user_friendly_names yes
  path_grouping_policy multibus
  failback immediate
  no_path_retry fail
}

multipaths {
  multipath {
    wwid 360000000000000000e00000000010001
    alias iscsi
  }
}

$ mkdir /run/multipathd/ -pv
$ touch /run/multipathd/multipathd.pid
$ chmod 777 /run/multipathd/multipathd.pid

$ systemctl enable multipathd.service
$ systemctl restart multipathd.service
$ systemctl status multipathd.service

$ lvcreate -n nfs_test -L 10G storage
$ mkfs.ext4 /dev/storage/nfs_test

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export1 *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export2 *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export3 *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)
$ exportfs -av

$ umount /opt/export1
$ umount /opt/export2
$ umount /opt/export3

$ mkdir -pv /mnt/test
$ mount /dev/storage/nfs_test /mnt/test

$ mkdir /mnt/test/subdir1
$ mkdir /mnt/test/subdir2
$ mkdir /mnt/test/subdir3

$ mount --bind /mnt/test/subdir1 /opt/export1
$ mount --bind /mnt/test/subdir2 /opt/export2
$ mount --bind /mnt/test/subdir3 /opt/export3

$ touch /mnt/test/subdir1/test1
$ touch /mnt/test/subdir2/test2
$ touch /mnt/test/subdir3/test3

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
/opt/export1 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export2 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export3 *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)

$ exportfs -av
$ rpcdebug -m nfsd -s all

客戶端

$ rpcdebug -m nfs -s all

$ umount /opt
$ mount -t nfs 192.168.195.136:/opt /opt

$ dd if=/dev/zero of=/opt/export1/test.bin bs=8M count=1024

結果正常。

使用“mount --bind”目錄作爲NFS子目錄的測試 1 (iSCSI,XFS,bind,NFSv4)

iSCSI服務端：

$ vi /etc/tgt/targets.conf
<target iqn.2016-12.org.lr:nfs>
  backing-store /dev/sdd2
  initiator-address 192.168.0.0/16
</target>

$ systemctl reload tgtd.service
$ tgtadm -L iscsi -m target -o show

NFS服務端：

$ iscsiadm -m discovery -t st -p 192.168.195.131
192.168.195.131:3260,1 iqn.2016-12.org.lr:lvm
192.168.195.131:3260,1 iqn.2016-12.org.lr:nfs

$ ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sda3  /dev/sdb  /dev/sdb1  /dev/sdc  /dev/sdc1  /dev/sdd  /dev/sdd1  /dev/sde  /dev/sdf

$ iscsiadm -m node -T iqn.2016-12.org.lr:nfs -p 192.168.195.131:3260 -l

$ iscsiadm -m node –l

$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sdb   /dev/sdc   /dev/sdd   /dev/sde  /dev/sdg
/dev/sda1  /dev/sda3  /dev/sdb1  /dev/sdc1  /dev/sdd1  /dev/sdf

$ vi /etc/multipath.conf 
blacklist {
  ...
  devnode "^sdg"
}
...

$ multipath -F
$ multipath -v2
$ multipath -ll

$ mkfs.xfs /dev/sdg

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export1 *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export2 *(rw,sync,no_root_squash,no_subtree_check)
#/opt/export3 *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)
$ exportfs -av

$ umount /opt/export1
$ umount /opt/export2
$ umount /opt/export3
$ umount /mnt/test/

$ mkdir -pv /mnt/test
$ mount /dev/sdg /mnt/test
$ mount | grep sdg
/dev/sdg on /mnt/test type xfs (rw,relatime,attr2,inode64,noquota)

$ mkdir /mnt/test/subdir1
$ mkdir /mnt/test/subdir2
$ mkdir /mnt/test/subdir3

$ mount --bind /mnt/test/subdir1 /opt/export1
$ mount --bind /mnt/test/subdir2 /opt/export2
$ mount --bind /mnt/test/subdir3 /opt/export3

$ touch /mnt/test/subdir1/test1
$ touch /mnt/test/subdir2/test2
$ touch /mnt/test/subdir3/test3

$ vi /etc/exports
/opt *(rw,sync,no_root_squash,no_subtree_check)
/opt/export1 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export2 *(rw,sync,no_root_squash,no_subtree_check)
/opt/export3 *(rw,sync,no_root_squash,no_subtree_check)
/opt/android *(rw,sync,no_root_squash,no_subtree_check)

$ exportfs -av
$ rpcdebug -m nfsd -s all

$ dd if=/dev/zero of=/mnt/test/subdir1/test.bin bs=8M count=1024

$ while :; do date +"%s %c -- $i" | tee -a /mnt/test/subdir2/nfsd.log; sleep 1; (( ++i )); done

客戶端 :

$ rpcdebug -m nfs -s all

$ umount /opt
$ mount -t nfs 192.168.195.136:/opt /opt
$ df -h | grep opt
192.168.195.136:/opt   79G   30G   45G   40% /opt
$ mount | grep /opt
192.168.195.136:/opt on /opt type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)

$ ls /opt/export1
$ mount | grep /opt
192.168.195.136:/opt on /opt type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)
192.168.195.136:/opt/export1 on /opt/export1 type nfs4 (rw,relatime,vers=4.0,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.195.131,local_lock=none,addr=192.168.195.136)

$ dd if=/dev/zero of=/opt/export1/test.bin bs=8M count=1024

$ while :; do date +"%s %c -- $i" | tee -a /opt/export2/nfstest.log; sleep 1; (( ++i )); done

結果依舊正常，這說明子目錄、多路徑，LVM之類對NFS並沒有本質影響。此時我們根據目前的測試結果，據此再次詢問產品開發人員，是否使用除了上述描述之外的操作，看他們那裏有沒有新的線索。結果他們說對NFS服務端目錄使用了強制umount。此時我心裏一萬個XXX，問題或許就是在這，下面就是更新測試的結果。

最終測試

現場測試1

在NFS服務節點上，屏蔽掉與NFS目錄相關的檢測和掛載代碼後，在其他節點運行以下腳本：

$ while :; do date +"%s %c -- $i" | tee -a /opt/nfs/iso/remotefs/nfstest.log; sleep 1; (( ++i )); done
...
1482377978 2016年12月22日 星期四 11時39分38秒 -- 61913
...

在經歷61913秒後依舊正常。

現場測試2

在NFS服務點上，屏蔽掉與NFS目錄相關的檢測和掛載代碼後，在其他節點運行如下腳本：

$ cat test.py
#!/usr/bin/env python

import os, oslo_utils.timeutils,time


def update_magic_file(magic_file, magic=None):
    with open(magic_file, "w") as fd:
        if not magic:
            magic = oslo_utils.timeutils.strtime()

        fd.write(magic)
        fd.flush()

while True :
    print "%s" % oslo_utils.timeutils.strtime()
#    update_magic_file("/opt/nfs/iso/remotefs/.remote_magic")
    update_magic_file("/mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188/.remote_magic")
    
    os.system("mount /dev/sdf1 /mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188")
    os.system("mount --bind /mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188/image /opt/glance/images/remotefs")
    os.system("mount --bind /mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188/iso /opt/nfs/iso/remotefs")
    os.system("mount --bind /mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188/template /opt/template")
#    os.system("systemctl restart nfs")
#    os.system("exportfs -av")
    print "mount list"
    os.system("mount | grep sdf")
    print "\n"
    time.sleep(120)

    print "%s" % oslo_utils.timeutils.strtime()
    os.system("umount /opt/glance/images/remotefs -lf")
    os.system("umount /opt/nfs/iso/remotefs -lf")
    os.system("umount /opt/template -lf")
    os.system("umount /mnt/efc7d7b0-26fb-45bd-ba29-9792d2dcd188 -lf")
#    os.system("systemctl restart nfs")
#    os.system("exportfs -av")
    print "umount list"
    os.system("mount | grep sdf")
    print "\n\n"
    time.sleep(3)

在其他節點運行以下腳本：

$ while :; do date +"%s %c -- $i" | tee -a /opt/nfs/iso/remotefs/nfstest.log; sleep 1; (( ++i )); done
...
1482313628 2016年12月21日 星期三 17時47分08秒 -- 1631
...

多次測試，均在30分鐘左右客戶端會被卡死。

模擬測試

在自己搭建的測試環境中，NFS服務端執行如下代碼：

$ cat test.py
#!/usr/bin/env python

import os,oslo_utils.timeutils,time


def update_magic_file(magic_file, magic=None):
    with open(magic_file, "w") as fd:
        if not magic:
            magic = oslo_utils.timeutils.strtime()

        fd.write(magic)
        fd.flush()


while True :
    print "%s" % oslo_utils.timeutils.strtime()
    os.system("mount /dev/sdi /mnt/test")
    os.system("mount --bind /mnt/test/subdir1 /opt/export1")
    os.system("mount --bind /mnt/test/subdir2 /opt/export2")
    os.system("mount --bind /mnt/test/subdir3 /opt/export3")
#   os.system("systemctl restart nfs")
#   os.system("exportfs -av")
#   update_magic_file("/mnt/test/subdir1/remote_magic")
    update_magic_file("/mnt/test/remote_magic")

    print "mount list"
    os.system("mount | grep sdi")
    print "\n"
    time.sleep(3)

    print "%s" % oslo_utils.timeutils.strtime()
    os.system("umount /opt/export1 -lf")
    os.system("umount /opt/export2 -lf")
    os.system("umount /opt/export3 -lf")
    os.system("umount /mnt/test -lf")
#   os.system("systemctl restart nfs")
#   os.system("exportfs -av")
    print "umount list"
    os.system("mount | grep sdi")
    print "\n\n"
    time.sleep(3)

客戶端執行如下腳本：

$ while :; do date +"%s %c -- $i" | tee -a /opt/export2/nfstest.log; sleep 1; (( ++i )); done
...
1482476918 2016年12月23日 星期五 15時08分38秒 -- 1714
1482476919 2016年12月23日 星期五 15時08分39秒 -- 1715

大約30分鐘後也會卡死，至此，問題出處找到。

總結分析

使用帶多路徑或者不帶多路徑，帶LVM或者不帶LVM測試，對結果沒影響。同時使用Wireshark抓取NFS協議報分析發現，在客戶端被卡死時，NFS服務端均返回錯誤代碼10008。通過NFS協議規範查詢可以知道，這是NFS3ERR_DELAY錯誤，在NFS服務端導出目錄發生變動（遷移）時可能會出現這種錯誤。另外通過其他資料查詢得知，NFS4中對每個導出的文件系統都有一個fsid，對於每個文件會產生一個satsid，文件的改動會導致這些id的改動。因此問題主要出在強制umount被nfs導出的目錄會導致這些id的變更，出現問題。

解決方法

• 服務端在umount或者mount完nfs導出目錄後，應該執行exportfs -av命令或者重啓nfs服務。exportfs -av會導致客戶端讀寫直接返回錯誤，必須umount並從新mount後才能解決問題。重啓nfs服務會導致客戶端被阻塞100秒左右，但是除此之外一切正常。

• 對於讀寫大文件出錯的問題，建議合理設置nfs的超時時間及重試次數，以及重試失敗後的處理方式。

• 對於檢測iSCSI或者FC存儲設備的有效性的方法，建議使用DirectIO讀取塊設備第一扇區或者讀取塊設備的uevent信息的方法來檢測，而不是通過mount後的目錄。

最終問題在短時間內解決了（這個問題大概花了我4天時間，而在我處理之前，這個問題已經拖了兩三個月），但此時我們對NFS也僅限於瞭解狀態，只是知道他的基本框架和原理，以及一些簡單的調試手段。這其中，對計算機系統完整的理解，NFS框架和原理，之前相關經驗積累，和與研發人員的交流都是至關重要的。