NetApp A700 存储 使用 multipath 多路径管理软件

背景:

去年(2018年)正好赶上数据库需要存储扩容。在各方协调下,我们借到并测试了 NetApp A700 All Flash Arrays。

因为测试得出的性能报告,是根据我们特定的业务场景和具体的硬件设备测试所得,所以我这里测试得出的存储的性能报告未必对大家都受用。也就不公开分享了。

  存储性能的三大关键指标(IOPS、 Throughput 吞吐量、 Latency 访问时延)
  IOPS: 和盘的数量、数据块大小有关。
  Throughput 吞吐量: 和数据块大小也是相关的。
  Latency 访问时延: 是指块存储处理一个I/O需要的时间。
  所以,存储的性能数据需要有个基准,一般是需要具体测试的。

虽说由于种种原因最终没有采购使用,但还是非常感谢当时的领导、同事以及 NetApp 厂商、代理商的支持。

整理此文「NetApp A700 存储 使用 multipath 多路径管理软件」与大家分享。

Revision V1.4

No.DateAuthor/ModifierComments
1.02018-07-20谈权初稿
1.12018-08-23谈权完善自动生成各个配置文件的脚本
1.22018-08-24谈权增加【附2:配置 multipath 基本使用方法】
1.32018-09-03谈权增加【附3:删除LUN的操作】
1.42018-09-05谈权增加【附4:网卡配置中添加 hotplug=no 参数,
避免start_udev命令导致Oracle RAC 的vip漂移问题】

步骤1: 生成 lun_info.txt 文件。(格式化 sanlun lun show 输出,见附1)

# sanlun lun show | awk '{a[$2]=$3;b[$2]+=!c[$3]++;d[$2]=$(NF-1)}END{for(i in a){print i,a[i],b[i],d[i]}}' | sort -n | grep -v filename | grep -v device | grep -v '[\-]' | grep -v unknown > /root/test/20180801/lun_info_20180801.txt

步骤2: 生成多路径配置文件 /etc/multipath.conf

generate_multipath.sh 生成多路径配置文件( /etc/multipath.conf )

#!/bin/bash

lun_file=/root/test/20180801/lun_info_20180801.txt


echo "
# NetApp A700 `date '+%Y-%m-%d %H:%M:%S'`
defaults {
user_friendly_names no
max_fds max
flush_on_last_del yes
queue_without_daemon no
}

# All data under blacklist must be specific to your system.
blacklist {
devnode \"^hd[a-z]\"
devnode \"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*\"
devnode \"^cciss.*\"
}

devices {
device {
vendor \"NETAPP\"
product \"LUN\"
path_grouping_policy group_by_prio
features \"3 queue_if_no_path pg_init_retries 50\"
prio \"alua\"
path_checker tur
failback immediate
path_selector \"round-robin 0\"
hardware_handler \"1 alua\"
rr_weight uniform
rr_min_io 128
getuid_callout \"/lib/udev/scsi_id -g -u -d /dev/%n\"
}
}

"
{
echo "multipaths {"
cat $lun_file | awk '{gsub(/.*\//,"",$1);print $1,$2}' | while read t1 t2;do a=$(/lib/udev/scsi_id --whitelisted --device=$t2); echo "
  multipath {
  wwid ${a} 
  alias netapp-${t1}
  }";done
echo "}"
} | grep -v "^$"

生成 /etc/multipath.conf 多路径配置文件

[root@dbtan21: ~/test/20180801]# sh generate_multipath.sh > /etc/multipath.conf

步骤2-1: 格式化输出 multipath -ll 信息,方便查看链路状态

format_multipath-ll.sh

[root@dbtan22: ~/test/20180808]# cat format_multipath-ll.sh     
#!/bin/bash

multipath_file=/root/test/20180808/dbtan22_multipath-ll.txt


cat ${multipath_file} |awk $'{
        if($0~/^netapp/)
        {
                line=$0
                getline tmp
                line=line" "tmp
                gsub(/features=.*wp=/,"wp=",line)
                if(status!="")
                {
                        print status" active_count="int(active_count)" failed_count="int(failed_count)
                }
                status=""
                active_count=""
                failed_count=""
                print line
        }
        if($0~/status=active/ || $0~/status=enabled/)
        {
                if(status!="")
                {
                        print status" active_count="int(active_count)" failed_count="int(failed_count)
                }
                status=$0
                active_count=""
                failed_count=""
        }
        if($0~/active ready running/)
        {
                active_count++
        }
        if($0~/failed faulty running/)
        {
                failed_count++
        }

}
END{
                if(status!="")
                {
                        print status" active_count="int(active_count)" failed_count="int(failed_count)
                }
}'

[root@dbtan22: ~/test/20180808]#

执行 format_multipath-ll.sh 脚本,统计每个 LUN 的主/备(active/enabled) 的活跃/失败(active_count/failed_count) 链路个数

[root@dbtan22: ~/test/20180808]# sh format_multipath-ll.sh    
netapp-lun19 (3600a09803830475a4c2b4d3059494955) dm-10 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun36 (3600a09803830475a4d3f4d3072414248) dm-28 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun21 (3600a09803830475a4c2b4d3059494957) dm-12 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun68 (3600a09803830475a4c2b4d305949496a) dm-59 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun53 (3600a09803830475a4d3f4d3072414259) dm-44 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun70 (3600a09803830475a4c2b4d305949496c) dm-61 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd12 (3600a09803830475a4d3f4d3072414273) dm-84 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun18 (3600a09803830475a4c2b4d3059494954) dm-7 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun35 (3600a09803830475a4d3f4d3072414247) dm-27 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun20 (3600a09803830475a4c2b4d3059494956) dm-11 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun67 (3600a09803830475a4d3f4d307241426c) dm-58 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun52 (3600a09803830475a4d3f4d3072414258) dm-43 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd09 (3600a09803830475a4d3f4d3072414271) dm-80 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd11 (3600a09803830475a4d3f4d3072414274) dm-83 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun17 (3600a09803830475a4c2b4d3059494953) dm-9 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun49 (3600a09803830475a4d3f4d3072414255) dm-40 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun34 (3600a09803830475a4c2b4d3059494969) dm-26 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun66 (3600a09803830475a4d3f4d307241426b) dm-57 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun51 (3600a09803830475a4d3f4d3072414257) dm-42 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd08 (3600a09803830475a4d3f4d3072414270) dm-79 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd10 (3600a09803830475a4d3f4d3072414272) dm-81 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun16 (3600a09803830475a4c2b4d3059494952) dm-8 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun48 (3600a09803830475a4d3f4d3072414254) dm-39 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun33 (3600a09803830475a4c2b4d3059494968) dm-24 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun65 (3600a09803830475a4d3f4d307241426a) dm-56 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun50 (3600a09803830475a4d3f4d3072414256) dm-41 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd07 (3600a09803830475a4d3f4d307241426f) dm-78 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun15 (3600a09803830475a4c2b4d3059494951) dm-6 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun47 (3600a09803830475a4d3f4d3072414253) dm-38 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun32 (3600a09803830475a4c2b4d3059494967) dm-25 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun64 (3600a09803830475a4d3f4d3072414269) dm-55 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd06 (3600a09803830475a4c2b4d305949497a) dm-77 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun29 (3600a09803830475a4c2b4d3059494964) dm-22 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun14 (3600a09803830475a4c2b4d3059494950) dm-5 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun46 (3600a09803830475a4d3f4d3072414252) dm-37 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun31 (3600a09803830475a4c2b4d3059494966) dm-23 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun63 (3600a09803830475a4d3f4d3072414268) dm-54 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd05 (3600a09803830475a4c2b4d3059494979) dm-76 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun28 (3600a09803830475a4c2b4d3059494963) dm-19 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun13 (3600a09803830475a4c2b4d305949494f) dm-3 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun45 (3600a09803830475a4d3f4d3072414251) dm-36 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun30 (3600a09803830475a4c2b4d3059494965) dm-21 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun103 (3600a09803830475a4d3f4d307241426e) dm-71 NETAPP,LUN C-Mode size=10G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun77 (3600a09803830475a4c2b4d3059494973) dm-68 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun62 (3600a09803830475a4d3f4d3072414267) dm-53 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd04 (3600a09803830475a4c2b4d3059494978) dm-75 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun27 (3600a09803830475a4c2b4d3059494962) dm-18 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun12 (3600a09803830475a4c2b4d305949494e) dm-2 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun59 (3600a09803830475a4d3f4d3072414264) dm-50 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun44 (3600a09803830475a4d3f4d3072414250) dm-35 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun102 (3600a09803830475a4d3f4d307241426d) dm-70 NETAPP,LUN C-Mode size=10G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun76 (3600a09803830475a4c2b4d3059494972) dm-67 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun61 (3600a09803830475a4d3f4d3072414266) dm-52 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd03 (3600a09803830475a4c2b4d3059494977) dm-74 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun26 (3600a09803830475a4c2b4d3059494961) dm-17 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun11 (3600a09803830475a4c2b4d305949494d) dm-4 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun58 (3600a09803830475a4d3f4d3072414263) dm-49 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun43 (3600a09803830475a4d3f4d307241424f) dm-34 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun101 (3600a09803830475a4c2b4d3059494974) dm-69 NETAPP,LUN C-Mode size=10G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun75 (3600a09803830475a4c2b4d3059494971) dm-66 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun60 (3600a09803830475a4d3f4d3072414265) dm-51 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd02 (3600a09803830475a4c2b4d3059494976) dm-73 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun25 (3600a09803830475a4c2b4d305949492f) dm-13 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun57 (3600a09803830475a4d3f4d3072414262) dm-48 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun42 (3600a09803830475a4d3f4d307241424e) dm-33 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun74 (3600a09803830475a4c2b4d3059494970) dm-65 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lunvd01 (3600a09803830475a4c2b4d3059494975) dm-72 NETAPP,LUN C-Mode size=80G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun39 (3600a09803830475a4d3f4d307241424b) dm-20 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun24 (3600a09803830475a4c2b4d305949495a) dm-16 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun56 (3600a09803830475a4d3f4d3072414261) dm-47 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun41 (3600a09803830475a4d3f4d307241424d) dm-32 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun73 (3600a09803830475a4c2b4d305949496f) dm-64 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun38 (3600a09803830475a4d3f4d307241424a) dm-29 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun23 (3600a09803830475a4c2b4d3059494959) dm-15 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun55 (3600a09803830475a4d3f4d307241422f) dm-46 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun40 (3600a09803830475a4d3f4d307241424c) dm-31 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun72 (3600a09803830475a4c2b4d305949496e) dm-63 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun37 (3600a09803830475a4d3f4d3072414249) dm-30 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun22 (3600a09803830475a4c2b4d3059494958) dm-14 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun69 (3600a09803830475a4c2b4d305949496b) dm-60 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun54 (3600a09803830475a4d3f4d307241425a) dm-45 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
netapp-lun71 (3600a09803830475a4c2b4d305949496d) dm-62 NETAPP,LUN C-Mode size=400G wp=rw
|-+- policy='round-robin 0' prio=50 status=active active_count=4 failed_count=0
`-+- policy='round-robin 0' prio=10 status=enabled active_count=4 failed_count=0
[root@dbtan22: ~/test/20180808]# 

步骤3:生成绑定文件 /etc/multipath/bindings (可不做,由于多路径配置文件中 user_friendly_names no

cat generate_bindings.sh 生成绑定文件 (/etc/multipath/bindings)

#!/bin/bash

lun_file=/root/test/20180801/lun_info_20180801.txt

{
echo "# NetApp A700 `date '+%Y-%m-%d %H:%M:%S'`"
cat $lun_file | awk '{gsub(/.*\//,"",$1);print $1,$2}' | while read t1 t2;do a=$(/lib/udev/scsi_id --whitelisted --device=$t2); echo "
mpath${t1} ${a}
";done
} | grep -v "^$"

生成 /etc/multipath/bindings 多路径绑定配置文件

[root@dbtan21: ~/test/20180801]# sh generate_bindings.sh >> /etc/multipath/bindings

步骤4:生成 99-oracle-asmdevices.rules 规则

generate_99-oracle-asmdevices.rules.sh

#!/bin/bash

lun_file=/root/test/20180801/lun_info_20180801.txt

{
cat $lun_file | awk '{gsub(/.*\//,"",$1);print $1,$2}' | while read t1 t2;do a=$(/lib/udev/scsi_id --whitelisted --device=$t2); echo "
KERNEL==\"dm*\",SUBSYSTEM==\"block\", PROGRAM==\"/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/$name\", RESULT==\"$a\", NAME=\"asm-$t1\", OWNER=\"grid\", GROUP=\"asmadmin\", MODE=\"0660\"
";done
} | grep -v "^$"

生成 /etc/udev/rules.d/99-oracle-asmdevices.rules 规则

[root@dbtan21: ~/test/20180801]# sh generate_99-oracle-asmdevices.rules.sh | grep -v asm-lunvd > /etc/udev/rules.d/99-oracle-asmdevices.rules 
[root@dbtan21: ~/test/20180801]# 

步骤5:对比2个节点的 /dev/asm-lunX 块设备(block)的 wwid

generate_ASM_wwid.sh

#!/bin/bash

lun_file=/root/test/20180801/lun_info_20180801.txt

{
cat $lun_file | awk '{gsub(/.*\//,"",$1);print $1,$2}' | while read t1 t2;do a=$(/lib/udev/scsi_id --whitelisted --device=$t2); echo "
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-$t1
`/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-$t1`
";done
} | grep -v "^$"

生成节点1(dbtan21) /dev/asm-lunX 块设备(block)的 wwid

[root@dbtan21: ~/test/20180801]# sh generate_ASM_wwid.sh 
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun101
3600a09803830475a4c2b4d3059494974
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun102
3600a09803830475a4d3f4d307241426d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun103
3600a09803830475a4d3f4d307241426e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun11
3600a09803830475a4c2b4d305949494d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun12
3600a09803830475a4c2b4d305949494e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun13
3600a09803830475a4c2b4d305949494f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun14
3600a09803830475a4c2b4d3059494950
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun15
3600a09803830475a4c2b4d3059494951
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun16
3600a09803830475a4c2b4d3059494952
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun17
3600a09803830475a4c2b4d3059494953
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun18
3600a09803830475a4c2b4d3059494954
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun19
3600a09803830475a4c2b4d3059494955
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun20
3600a09803830475a4c2b4d3059494956
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun21
3600a09803830475a4c2b4d3059494957
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun22
3600a09803830475a4c2b4d3059494958
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun23
3600a09803830475a4c2b4d3059494959
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun24
3600a09803830475a4c2b4d305949495a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun25
3600a09803830475a4c2b4d305949492f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun26
3600a09803830475a4c2b4d3059494961
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun27
3600a09803830475a4c2b4d3059494962
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun28
3600a09803830475a4c2b4d3059494963
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun29
3600a09803830475a4c2b4d3059494964
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun30
3600a09803830475a4c2b4d3059494965
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun31
3600a09803830475a4c2b4d3059494966
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun32
3600a09803830475a4c2b4d3059494967
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun33
3600a09803830475a4c2b4d3059494968
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun34
3600a09803830475a4c2b4d3059494969
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun35
3600a09803830475a4d3f4d3072414247
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun36
3600a09803830475a4d3f4d3072414248
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun37
3600a09803830475a4d3f4d3072414249
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun38
3600a09803830475a4d3f4d307241424a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun39
3600a09803830475a4d3f4d307241424b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun40
3600a09803830475a4d3f4d307241424c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun41
3600a09803830475a4d3f4d307241424d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun42
3600a09803830475a4d3f4d307241424e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun43
3600a09803830475a4d3f4d307241424f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun44
3600a09803830475a4d3f4d3072414250
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun45
3600a09803830475a4d3f4d3072414251
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun46
3600a09803830475a4d3f4d3072414252
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun47
3600a09803830475a4d3f4d3072414253
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun48
3600a09803830475a4d3f4d3072414254
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun49
3600a09803830475a4d3f4d3072414255
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun50
3600a09803830475a4d3f4d3072414256
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun51
3600a09803830475a4d3f4d3072414257
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun52
3600a09803830475a4d3f4d3072414258
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun53
3600a09803830475a4d3f4d3072414259
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun54
3600a09803830475a4d3f4d307241425a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun55
3600a09803830475a4d3f4d307241422f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun56
3600a09803830475a4d3f4d3072414261
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun57
3600a09803830475a4d3f4d3072414262
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun58
3600a09803830475a4d3f4d3072414263
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun59
3600a09803830475a4d3f4d3072414264
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun60
3600a09803830475a4d3f4d3072414265
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun61
3600a09803830475a4d3f4d3072414266
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun62
3600a09803830475a4d3f4d3072414267
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun63
3600a09803830475a4d3f4d3072414268
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun64
3600a09803830475a4d3f4d3072414269
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun65
3600a09803830475a4d3f4d307241426a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun66
3600a09803830475a4d3f4d307241426b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun67
3600a09803830475a4d3f4d307241426c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun68
3600a09803830475a4c2b4d305949496a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun69
3600a09803830475a4c2b4d305949496b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun70
3600a09803830475a4c2b4d305949496c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun71
3600a09803830475a4c2b4d305949496d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun72
3600a09803830475a4c2b4d305949496e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun73
3600a09803830475a4c2b4d305949496f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun74
3600a09803830475a4c2b4d3059494970
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun75
3600a09803830475a4c2b4d3059494971
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun76
3600a09803830475a4c2b4d3059494972
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun77
3600a09803830475a4c2b4d3059494973
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd01
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd02
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd03
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd04
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd05
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd06
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd07
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd08
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd09
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd10
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd11
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd12
[root@dbtan21: ~/test/20180801]# 

生成节点2(dbtan22) /dev/asm-lunX 块设备(block)的 wwid

[root@dbtan22: ~/test/20180801]# sh generate_ASM_wwid.sh 
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun101
3600a09803830475a4c2b4d3059494974
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun102
3600a09803830475a4d3f4d307241426d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun103
3600a09803830475a4d3f4d307241426e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun11
3600a09803830475a4c2b4d305949494d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun12
3600a09803830475a4c2b4d305949494e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun13
3600a09803830475a4c2b4d305949494f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun14
3600a09803830475a4c2b4d3059494950
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun15
3600a09803830475a4c2b4d3059494951
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun16
3600a09803830475a4c2b4d3059494952
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun17
3600a09803830475a4c2b4d3059494953
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun18
3600a09803830475a4c2b4d3059494954
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun19
3600a09803830475a4c2b4d3059494955
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun20
3600a09803830475a4c2b4d3059494956
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun21
3600a09803830475a4c2b4d3059494957
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun22
3600a09803830475a4c2b4d3059494958
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun23
3600a09803830475a4c2b4d3059494959
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun24
3600a09803830475a4c2b4d305949495a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun25
3600a09803830475a4c2b4d305949492f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun26
3600a09803830475a4c2b4d3059494961
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun27
3600a09803830475a4c2b4d3059494962
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun28
3600a09803830475a4c2b4d3059494963
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun29
3600a09803830475a4c2b4d3059494964
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun30
3600a09803830475a4c2b4d3059494965
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun31
3600a09803830475a4c2b4d3059494966
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun32
3600a09803830475a4c2b4d3059494967
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun33
3600a09803830475a4c2b4d3059494968
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun34
3600a09803830475a4c2b4d3059494969
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun35
3600a09803830475a4d3f4d3072414247
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun36
3600a09803830475a4d3f4d3072414248
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun37
3600a09803830475a4d3f4d3072414249
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun38
3600a09803830475a4d3f4d307241424a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun39
3600a09803830475a4d3f4d307241424b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun40
3600a09803830475a4d3f4d307241424c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun41
3600a09803830475a4d3f4d307241424d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun42
3600a09803830475a4d3f4d307241424e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun43
3600a09803830475a4d3f4d307241424f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun44
3600a09803830475a4d3f4d3072414250
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun45
3600a09803830475a4d3f4d3072414251
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun46
3600a09803830475a4d3f4d3072414252
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun47
3600a09803830475a4d3f4d3072414253
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun48
3600a09803830475a4d3f4d3072414254
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun49
3600a09803830475a4d3f4d3072414255
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun50
3600a09803830475a4d3f4d3072414256
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun51
3600a09803830475a4d3f4d3072414257
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun52
3600a09803830475a4d3f4d3072414258
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun53
3600a09803830475a4d3f4d3072414259
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun54
3600a09803830475a4d3f4d307241425a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun55
3600a09803830475a4d3f4d307241422f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun56
3600a09803830475a4d3f4d3072414261
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun57
3600a09803830475a4d3f4d3072414262
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun58
3600a09803830475a4d3f4d3072414263
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun59
3600a09803830475a4d3f4d3072414264
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun60
3600a09803830475a4d3f4d3072414265
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun61
3600a09803830475a4d3f4d3072414266
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun62
3600a09803830475a4d3f4d3072414267
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun63
3600a09803830475a4d3f4d3072414268
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun64
3600a09803830475a4d3f4d3072414269
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun65
3600a09803830475a4d3f4d307241426a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun66
3600a09803830475a4d3f4d307241426b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun67
3600a09803830475a4d3f4d307241426c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun68
3600a09803830475a4c2b4d305949496a
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun69
3600a09803830475a4c2b4d305949496b
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun70
3600a09803830475a4c2b4d305949496c
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun71
3600a09803830475a4c2b4d305949496d
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun72
3600a09803830475a4c2b4d305949496e
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun73
3600a09803830475a4c2b4d305949496f
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun74
3600a09803830475a4c2b4d3059494970
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun75
3600a09803830475a4c2b4d3059494971
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun76
3600a09803830475a4c2b4d3059494972
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lun77
3600a09803830475a4c2b4d3059494973
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd01
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd02
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd03
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd04
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd05
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd06
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd07
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd08
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd09
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd10
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd11
/sbin/scsi_id --whitelisted --replace-whitespace --device=/dev/asm-lunvd12
[root@dbtan22: ~/test/20180801]# 

附1:查看格式化后的 sanlun lun show 输出

[root@dbtan22: ~/test/20180801]# cat lun_info_20180801.txt 
/vol/dbtan2122_ocr1/lun101 /dev/sdbr 8 10g
/vol/dbtan2122_ocr2/lun102 /dev/sdbv 8 10g
/vol/dbtan2122_ocr3/lun103 /dev/sdbw 8 10g
/vol/dbtan2122_vol11/lun11 /dev/sdb 8 400.0g
/vol/dbtan2122_vol12/lun12 /dev/sdd 8 400.0g
/vol/dbtan2122_vol13/lun13 /dev/sde 8 400.0g
/vol/dbtan2122_vol14/lun14 /dev/sdf 8 400.0g
/vol/dbtan2122_vol15/lun15 /dev/sdg 8 400.0g
/vol/dbtan2122_vol16/lun16 /dev/sdh 8 400.0g
/vol/dbtan2122_vol17/lun17 /dev/sdi 8 400.0g
/vol/dbtan2122_vol18/lun18 /dev/sdj 8 400.0g
/vol/dbtan2122_vol19/lun19 /dev/sdk 8 400.0g
/vol/dbtan2122_vol20/lun20 /dev/sdl 8 400.0g
/vol/dbtan2122_vol21/lun21 /dev/sdm 8 400.0g
/vol/dbtan2122_vol22/lun22 /dev/sdn 8 400.0g
/vol/dbtan2122_vol23/lun23 /dev/sdo 8 400.0g
/vol/dbtan2122_vol24/lun24 /dev/sdp 8 400.0g
/vol/dbtan2122_vol25/lun25 /dev/sdq 8 400.0g
/vol/dbtan2122_vol26/lun26 /dev/sdr 8 400.0g
/vol/dbtan2122_vol27/lun27 /dev/sds 8 400.0g
/vol/dbtan2122_vol28/lun28 /dev/sdt 8 400.0g
/vol/dbtan2122_vol29/lun29 /dev/sdu 8 400.0g
/vol/dbtan2122_vol30/lun30 /dev/sdv 8 400.0g
/vol/dbtan2122_vol31/lun31 /dev/sdw 8 400.0g
/vol/dbtan2122_vol32/lun32 /dev/sdx 8 400.0g
/vol/dbtan2122_vol33/lun33 /dev/sdy 8 400.0g
/vol/dbtan2122_vol34/lun34 /dev/sdz 8 400.0g
/vol/dbtan2122_vol35/lun35 /dev/sdaa 8 400.0g
/vol/dbtan2122_vol36/lun36 /dev/sdab 8 400.0g
/vol/dbtan2122_vol37/lun37 /dev/sdac 8 400.0g
/vol/dbtan2122_vol38/lun38 /dev/sdad 8 400.0g
/vol/dbtan2122_vol39/lun39 /dev/sdae 8 400.0g
/vol/dbtan2122_vol40/lun40 /dev/sdaf 8 400.0g
/vol/dbtan2122_vol41/lun41 /dev/sdag 8 400.0g
/vol/dbtan2122_vol42/lun42 /dev/sdah 8 400.0g
/vol/dbtan2122_vol43/lun43 /dev/sdai 8 400.0g
/vol/dbtan2122_vol44/lun44 /dev/sdaj 8 400.0g
/vol/dbtan2122_vol45/lun45 /dev/sdak 8 400.0g
/vol/dbtan2122_vol46/lun46 /dev/sdal 8 400.0g
/vol/dbtan2122_vol47/lun47 /dev/sdam 8 400.0g
/vol/dbtan2122_vol48/lun48 /dev/sdan 8 400.0g
/vol/dbtan2122_vol49/lun49 /dev/sdao 8 400.0g
/vol/dbtan2122_vol50/lun50 /dev/sdap 8 400.0g
/vol/dbtan2122_vol51/lun51 /dev/sdaq 8 400.0g
/vol/dbtan2122_vol52/lun52 /dev/sdar 8 400.0g
/vol/dbtan2122_vol53/lun53 /dev/sdas 8 400.0g
/vol/dbtan2122_vol54/lun54 /dev/sdat 8 400.0g
/vol/dbtan2122_vol55/lun55 /dev/sdau 8 400.0g
/vol/dbtan2122_vol56/lun56 /dev/sdav 8 400.0g
/vol/dbtan2122_vol57/lun57 /dev/sdaw 8 400.0g
/vol/dbtan2122_vol58/lun58 /dev/sdax 8 400.0g
/vol/dbtan2122_vol59/lun59 /dev/sday 8 400.0g
/vol/dbtan2122_vol60/lun60 /dev/sdaz 8 400.0g
/vol/dbtan2122_vol61/lun61 /dev/sdbb 8 400.0g
/vol/dbtan2122_vol62/lun62 /dev/sdbc 8 400.0g
/vol/dbtan2122_vol63/lun63 /dev/sdbd 8 400.0g
/vol/dbtan2122_vol64/lun64 /dev/sdbe 8 400.0g
/vol/dbtan2122_vol65/lun65 /dev/sdbg 8 400.0g
/vol/dbtan2122_vol66/lun66 /dev/sdbh 8 400.0g
/vol/dbtan2122_vol67/lun67 /dev/sdbi 8 400.0g
/vol/dbtan2122_vol68/lun68 /dev/sdbj 8 400.0g
/vol/dbtan2122_vol69/lun69 /dev/sdbk 8 400.0g
/vol/dbtan2122_vol70/lun70 /dev/sdbl 8 400.0g
/vol/dbtan2122_vol71/lun71 /dev/sdbm 8 400.0g
/vol/dbtan2122_vol72/lun72 /dev/sdbn 8 400.0g
/vol/dbtan2122_vol73/lun73 /dev/sdbo 8 400.0g
/vol/dbtan2122_vol74/lun74 /dev/sdbp 8 400.0g
/vol/dbtan2122_vol75/lun75 /dev/sdc 8 400.0g
/vol/dbtan2122_vol76/lun76 /dev/sdba 8 400.0g
/vol/dbtan2122_vol77/lun77 /dev/sdbf 8 400.0g
/vol/vdbench01/lunvd01 /dev/sdbx 8 80.0g
/vol/vdbench02/lunvd02 /dev/sdby 8 80.0g
/vol/vdbench03/lunvd03 /dev/sdbz 8 80.0g
/vol/vdbench04/lunvd04 /dev/sdca 8 80.0g
/vol/vdbench05/lunvd05 /dev/sdcb 8 80.0g
/vol/vdbench06/lunvd06 /dev/sdcc 8 80.0g
/vol/vdbench07/lunvd07 /dev/sdcd 8 80.0g
/vol/vdbench08/lunvd08 /dev/sdce 8 80.0g
/vol/vdbench09/lunvd09 /dev/sdcf 8 80.0g
/vol/vdbench10/lunvd10 /dev/sdcg 8 80.0g
/vol/vdbench11/lunvd11 /dev/sdxr 8 80.0g
/vol/vdbench12/lunvd12 /dev/sdxs 8 80.0g
[root@dbtan22: ~/test/20180801]# 

附2:配置 multipath 基本使用方法

1. 查看主机或者存储交换机上的WWN号,在存储上将LUN映射给需要的主机

[root@dbtan21: ~]# cat /sys/class/fc_host/host*/port_name
0x100000109b1b2c72
0x100000109b1b2c73
0x100000109b176552
0x100000109b176553
[root@dbtan21: ~]# 

2. 在系统内执行扫盘命令,没有命令先安装 sg3_utils

# yum install sg3_utils
rescan-scsi-bus.sh
[root@dbtan21: ~]# rpm -qa | grep sg3_utils
sg3_utils-1.28-13.el6.x86_64
sg3_utils-libs-1.28-13.el6.x86_64
[root@dbtan21: ~]# 
[root@dbtan21: ~]# ll /usr/bin/rescan-scsi-bus.sh 
-rwxr-xr-x. 1 root root 33968 Jun 19 23:22 /usr/bin/rescan-scsi-bus.sh
[root@dbtan21: ~]# 

3. 查看是否映射到对应的

# fdisk -l
# lsblk -f

4. 查看是否安装了 multipath

# yum install device-mapper-multipath
[root@dbtan21: ~]# rpm -qa | grep device-mapper
device-mapper-libs-1.02.117-7.el6.x86_64
device-mapper-event-1.02.117-7.el6.x86_64
device-mapper-persistent-data-0.6.2-0.1.rc7.el6.x86_64
device-mapper-1.02.117-7.el6.x86_64
device-mapper-multipath-libs-0.4.9-93.el6.x86_64
device-mapper-multipath-0.4.9-93.el6.x86_64
device-mapper-event-libs-1.02.117-7.el6.x86_64
[root@dbtan21: ~]# 

5. 拷贝默认的 multipath.conf/etc 目录下,也可以使用mpathconf命令创建默认模板

# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
# mpathconf --enable --with_multipathd y

6. 查看本地存储wwid

[root@dbtan21: ~]# /lib/udev/scsi_id --whitelisted --device=/dev/sda
36101b5442bcc700022bf914a0cca39f5
[root@dbtan21: ~]# /lib/udev/scsi_id --whitelisted --device=/dev/mapper/netapp-lun11
3600a09803830475a4c2b4d305949494d
[root@dbtan21: ~]# 
[root@dbtan21: ~]# ll /dev/mapper/*
crw-rw---- 1 root root 10, 58 Aug 13 18:38 /dev/mapper/control
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun101 -> ../dm-69
lrwxrwxrwx 1 root root      8 Aug 16 20:50 /dev/mapper/netapp-lun102 -> ../dm-70
lrwxrwxrwx 1 root root      8 Aug 16 20:50 /dev/mapper/netapp-lun103 -> ../dm-71
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun11 -> ../dm-3
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun12 -> ../dm-4
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun13 -> ../dm-5
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun14 -> ../dm-6
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun15 -> ../dm-9
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun16 -> ../dm-7
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun17 -> ../dm-14
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun18 -> ../dm-8
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun19 -> ../dm-10
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun20 -> ../dm-15
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun21 -> ../dm-11
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun22 -> ../dm-13
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun23 -> ../dm-12
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun24 -> ../dm-17
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun25 -> ../dm-16
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun26 -> ../dm-18
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun27 -> ../dm-20
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun28 -> ../dm-19
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun29 -> ../dm-21
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun30 -> ../dm-23
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun31 -> ../dm-22
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun32 -> ../dm-24
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun33 -> ../dm-25
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun34 -> ../dm-26
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun35 -> ../dm-27
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun36 -> ../dm-28
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun37 -> ../dm-30
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun38 -> ../dm-29
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun39 -> ../dm-31
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun40 -> ../dm-34
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun41 -> ../dm-33
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun42 -> ../dm-32
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun43 -> ../dm-36
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun44 -> ../dm-38
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun45 -> ../dm-35
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun46 -> ../dm-39
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun47 -> ../dm-44
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun48 -> ../dm-41
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun49 -> ../dm-37
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun50 -> ../dm-40
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun51 -> ../dm-42
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun52 -> ../dm-43
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun53 -> ../dm-46
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun54 -> ../dm-45
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun55 -> ../dm-47
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun56 -> ../dm-49
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun57 -> ../dm-48
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun58 -> ../dm-50
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun59 -> ../dm-51
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun60 -> ../dm-52
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun61 -> ../dm-54
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun62 -> ../dm-55
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun63 -> ../dm-56
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun64 -> ../dm-58
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun65 -> ../dm-59
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun66 -> ../dm-60
lrwxrwxrwx 1 root root      8 Aug 16 20:30 /dev/mapper/netapp-lun67 -> ../dm-66
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun68 -> ../dm-62
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun69 -> ../dm-63
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun70 -> ../dm-61
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun71 -> ../dm-64
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun72 -> ../dm-65
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun73 -> ../dm-67
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun74 -> ../dm-68
lrwxrwxrwx 1 root root      7 Aug 16 20:28 /dev/mapper/netapp-lun75 -> ../dm-2
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun76 -> ../dm-53
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun77 -> ../dm-57
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun78 -> ../dm-93
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun79 -> ../dm-94
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun80 -> ../dm-95
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun81 -> ../dm-96
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun82 -> ../dm-97
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun83 -> ../dm-98
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lun84 -> ../dm-99
lrwxrwxrwx 1 root root      9 Aug 16 20:28 /dev/mapper/netapp-lun85 -> ../dm-100
lrwxrwxrwx 1 root root      9 Aug 16 20:28 /dev/mapper/netapp-lun86 -> ../dm-101
lrwxrwxrwx 1 root root      9 Aug 16 20:28 /dev/mapper/netapp-lun87 -> ../dm-102
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lunvd01 -> ../dm-72
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd02 -> ../vdbench-lunvd02
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd03 -> ../vdbench-lunvd03
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd04 -> ../vdbench-lunvd04
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd05 -> ../vdbench-lunvd05
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lunvd06 -> ../dm-77
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lunvd07 -> ../dm-78
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd08 -> ../vdbench-lunvd08
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd09 -> ../vdbench-lunvd09
lrwxrwxrwx 1 root root     18 Aug 16 20:28 /dev/mapper/netapp-lunvd10 -> ../vdbench-lunvd10
lrwxrwxrwx 1 root root      8 Aug 16 20:28 /dev/mapper/netapp-lunvd11 -> ../dm-83
lrwxrwxrwx 1 root root      8 Aug 16 20:29 /dev/mapper/netapp-lunvd12 -> ../dm-84
lrwxrwxrwx 1 root root     18 Aug 16 20:29 /dev/mapper/netapp-lunvd13 -> ../vdbench-lunvd13
lrwxrwxrwx 1 root root     18 Aug 16 20:29 /dev/mapper/netapp-lunvd14 -> ../vdbench-lunvd14
lrwxrwxrwx 1 root root     18 Aug 16 20:29 /dev/mapper/netapp-lunvd15 -> ../vdbench-lunvd15
lrwxrwxrwx 1 root root     18 Aug 16 20:29 /dev/mapper/netapp-lunvd16 -> ../vdbench-lunvd16
lrwxrwxrwx 1 root root      8 Aug 16 20:29 /dev/mapper/netapp-lunvd17 -> ../dm-89
lrwxrwxrwx 1 root root     18 Aug 16 20:29 /dev/mapper/netapp-lunvd18 -> ../vdbench-lunvd18
lrwxrwxrwx 1 root root      8 Aug 16 20:29 /dev/mapper/netapp-lunvd19 -> ../dm-91
lrwxrwxrwx 1 root root      8 Aug 16 20:29 /dev/mapper/netapp-lunvd20 -> ../dm-92
lrwxrwxrwx 1 root root      7 Aug 13 18:38 /dev/mapper/VolGroup-LogVol00 -> ../dm-1
lrwxrwxrwx 1 root root      7 Aug 13 18:38 /dev/mapper/VolGroup-LogVol01 -> ../dm-0
lrwxrwxrwx 1 root root      8 Aug 13 18:38 /dev/mapper/VolGroup-lv_app -> ../dm-82
[root@dbtan21: ~]# 

7. 查看存储参数

[root@dbtan21: ~]# cat /sys/block/sdb/device/vendor 
NETAPP  
[root@dbtan21: ~]# cat /sys/block/sdb/device/model
LUN C-Mode      
[root@dbtan21: ~]# 

8. 添加本地盘到 blacklist 内,不同的存储和系统参考官方的最佳实践

已经可以脚本自动生成 multipath 配置文件,参见:
【步骤2: 生成多路径配置文件 /etc/multipath.conf
【步骤3:生成绑定文件 /etc/multipath/bindings (可不做,由于多路径配置文件中 user_friendly_names no )】

9. 清空已有的 multipath 记录

# multipath -F

说明:multipath -F 不会清空使用中的链路。

下列举例,multipath -F 全部清理了,是因为此时所以链路都已经没有在使用了。

清理后,重启 multipath 服务(/etc/init.d/multipathd restart),即可重新聚合链路。

我们可以发现,在写好 /etc/udev/rules.d/99-oracle-asmdevices.rules 规则后,每次重启 multipath 服务,/dev/mapper/netapp-lun 的符号链接会切换指向 /dev/dm-/dev/asm-lun 的块设备(block)。

通过分析,/dev/mapper/netapp-lun 的符号链接会切换,是与执行 partprobe 有关。(例:partprobe /dev/mapper/netapp-lun101

[root@dbtan21: ~]# multipath -F
[root@dbtan21: ~]# multipath -ll
[root@dbtan21: ~]# ll /dev/mapper/
total 0
crw-rw---- 1 root root 10, 58 Aug 13 18:38 control
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol00 -> ../dm-1
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol01 -> ../dm-0
lrwxrwxrwx 1 root root      8 Aug 13 18:38 VolGroup-lv_app -> ../dm-82
[root@dbtan21: ~]# /etc/init.d/multipathd restart
ok
Stopping multipathd daemon:                                [  OK  ]
Starting multipathd daemon:                                [  OK  ]
[root@dbtan21: ~]# ll /dev/mapper/               
total 0
crw-rw---- 1 root root 10, 58 Aug 13 18:38 control
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun102 -> ../dm-80
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun103 -> ../dm-81
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun11 -> ../dm-2
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun12 -> ../dm-3
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun13 -> ../dm-4
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun14 -> ../dm-5
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun15 -> ../dm-6
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun16 -> ../dm-7
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun17 -> ../dm-8
lrwxrwxrwx 1 root root      7 Aug 24 15:43 netapp-lun18 -> ../dm-9
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun19 -> ../dm-10
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun20 -> ../dm-11
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun21 -> ../dm-12
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun22 -> ../dm-13
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun23 -> ../dm-14
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun24 -> ../dm-15
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun25 -> ../dm-16
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun26 -> ../dm-17
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun27 -> ../dm-18
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun28 -> ../dm-19
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun29 -> ../dm-20
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun30 -> ../dm-21
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun31 -> ../dm-22
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun32 -> ../dm-23
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun33 -> ../dm-24
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun34 -> ../dm-25
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun35 -> ../dm-26
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun36 -> ../dm-27
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun37 -> ../dm-28
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun38 -> ../dm-29
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun39 -> ../dm-30
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun40 -> ../dm-31
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun41 -> ../dm-32
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun42 -> ../dm-33
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun43 -> ../dm-34
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun44 -> ../dm-35
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun45 -> ../dm-36
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun46 -> ../dm-37
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun47 -> ../dm-38
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun48 -> ../dm-39
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun49 -> ../dm-40
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun50 -> ../dm-41
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun51 -> ../dm-42
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun52 -> ../dm-43
lrwxrwxrwx 1 root root     12 Aug 24 15:43 netapp-lun53 -> ../asm-lun53
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun54 -> ../dm-45
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun55 -> ../dm-46
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun56 -> ../dm-47
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun57 -> ../dm-48
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun58 -> ../dm-49
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun59 -> ../dm-50
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun60 -> ../dm-51
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun61 -> ../dm-52
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun62 -> ../dm-53
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun63 -> ../dm-54
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun64 -> ../dm-55
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun65 -> ../dm-56
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun66 -> ../dm-57
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun67 -> ../dm-58
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun68 -> ../dm-59
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun69 -> ../dm-60
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun70 -> ../dm-61
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun71 -> ../dm-62
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun72 -> ../dm-63
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun73 -> ../dm-64
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun74 -> ../dm-65
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun75 -> ../dm-66
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun76 -> ../dm-67
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun77 -> ../dm-68
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun78 -> ../dm-69
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun79 -> ../dm-70
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun80 -> ../dm-71
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun81 -> ../dm-72
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun83 -> ../dm-74
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun84 -> ../dm-75
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun85 -> ../dm-76
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lun87 -> ../dm-78
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lunvd01 -> ../dm-83
lrwxrwxrwx 1 root root      8 Aug 24 15:43 netapp-lunvd03 -> ../dm-85
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol00 -> ../dm-1
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol01 -> ../dm-0
lrwxrwxrwx 1 root root      8 Aug 13 18:38 VolGroup-lv_app -> ../dm-82
[root@dbtan21: ~]# 
[root@dbtan21: ~]# /etc/init.d/multipathd restart
ok
Stopping multipathd daemon:                                [  OK  ]
Starting multipathd daemon:                                [  OK  ]
[root@dbtan21: ~]#
[root@dbtan21: ~]# ll /dev/mapper/
total 0
crw-rw---- 1 root root 10, 58 Aug 13 18:38 control
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun101 -> ../asm-lun78
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun102 -> ../asm-lun79
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun103 -> ../asm-lun80
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun11 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun12 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun13 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun14 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun15 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun16 -> ../asm-lun16
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun17 -> ../asm-lun23
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun18 -> ../asm-lun18
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun19 -> ../asm-lun19
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun20 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun21 -> ../asm-lun21
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun22 -> ../asm-lun22
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun23 -> ../asm-lun23
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun24 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun25 -> ../asm-lun25
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun26 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun27 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun28 -> ../asm-lun28
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun29 -> ../asm-lun29
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun30 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun31 -> ../asm-lun31
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun32 -> ../asm-lun32
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun33 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun34 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun35 -> ../asm-lun35
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun36 -> ../asm-lun36
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun37 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun38 -> ../asm-lun38
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun39 -> ../asm-lun39
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun40 -> ../asm-lun40
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun41 -> ../asm-lun41
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun42 -> ../asm-lun42
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun43 -> ../asm-lun43
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun44 -> ../asm-lun53
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun45 -> ../asm-lun45
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun46 -> ../asm-lun50
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun47 -> ../asm-lun53
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun48 -> ../asm-lun50
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun49 -> ../asm-lun49
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun50 -> ../asm-lun50
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun51 -> ../asm-lun51
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun52 -> ../asm-lun52
lrwxrwxrwx 1 root root      8 Aug 24 15:46 netapp-lun53 -> ../dm-44
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun54 -> ../asm-lun54
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun55 -> ../asm-lun56
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun56 -> ../asm-lun56
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun57 -> ../asm-lun57
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun58 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun59 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun60 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun61 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun62 -> ../asm-lun75
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun63 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun64 -> ../asm-lun75
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun65 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun66 -> ../asm-lun77
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun67 -> ../asm-lun75
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun68 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun69 -> ../asm-lun77
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun70 -> ../asm-lun70
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun71 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun72 -> ../asm-lun77
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun73 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun74 -> ../asm-lun77
lrwxrwxrwx 1 root root     12 Aug 24 15:46 netapp-lun75 -> ../asm-lun75
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun76 -> ../asm-lun76
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun77 -> ../asm-lun77
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun78 -> ../asm-lun78
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun79 -> ../asm-lun79
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun80 -> ../asm-lun80
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun81 -> ../asm-lun81
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun82 -> ../asm-lun82
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun83 -> ../asm-lun83
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun84 -> ../asm-lun84
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun85 -> ../asm-lun85
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun86 -> ../asm-lun86
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lun87 -> ../asm-lun87
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd01 -> ../asm-lun81
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd02 -> ../asm-lun82
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd03 -> ../asm-lun83
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd04 -> ../asm-lun84
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd05 -> ../asm-lun85
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd06 -> ../asm-lun86
lrwxrwxrwx 1 root root     12 Aug 24 15:47 netapp-lunvd07 -> ../asm-lun87
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd08 -> ../vdbench-lunvd08
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd09 -> ../vdbench-lunvd09
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd10 -> ../vdbench-lunvd10
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd11 -> ../vdbench-lunvd11
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd12 -> ../vdbench-lunvd12
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd13 -> ../vdbench-lunvd13
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd14 -> ../vdbench-lunvd14
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd15 -> ../vdbench-lunvd15
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd16 -> ../vdbench-lunvd16
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd17 -> ../vdbench-lunvd17
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd18 -> ../vdbench-lunvd18
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd19 -> ../vdbench-lunvd19
lrwxrwxrwx 1 root root     18 Aug 24 15:47 netapp-lunvd20 -> ../vdbench-lunvd20
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol00 -> ../dm-1
lrwxrwxrwx 1 root root      7 Aug 13 18:38 VolGroup-LogVol01 -> ../dm-0
lrwxrwxrwx 1 root root      8 Aug 13 18:38 VolGroup-lv_app -> ../dm-82
[root@dbtan21: ~]# 
[root@dbtan21: ~]# ll /dev/mapper/netapp-lun10*
lrwxrwxrwx 1 root root 12 Aug 24 15:47 /dev/mapper/netapp-lun101 -> ../asm-lun78
lrwxrwxrwx 1 root root 12 Aug 24 15:47 /dev/mapper/netapp-lun102 -> ../asm-lun79
lrwxrwxrwx 1 root root 12 Aug 24 15:47 /dev/mapper/netapp-lun103 -> ../asm-lun80
[root@dbtan21: ~]# partprobe /dev/mapper/netapp-lun101
[root@dbtan21: ~]# partprobe /dev/mapper/netapp-lun102
[root@dbtan21: ~]# partprobe /dev/mapper/netapp-lun103
[root@dbtan21: ~]# 
[root@dbtan21: ~]# ll /dev/mapper/netapp-lun10*       
lrwxrwxrwx 1 root root 8 Aug 24 15:52 /dev/mapper/netapp-lun101 -> ../dm-79
lrwxrwxrwx 1 root root 8 Aug 24 15:52 /dev/mapper/netapp-lun102 -> ../dm-80
lrwxrwxrwx 1 root root 8 Aug 24 15:52 /dev/mapper/netapp-lun103 -> ../dm-81
[root@dbtan21: ~]# 

10. 打印诊断信息

# multipath -v3

11. 启用多路径守护程序以在引导时启动

# mpathconf --enable

12. 启动多路径服务

# service multipathd start
或
# /etc/init.d/multipathd restart

13. 如果在启动 multipath 守护程序后更改多路径配置文件,请运行以下命令以使更改生效。

# service multipathd reload 

14. 重启系统测试

说明:首次配置好 multipath 后,需要重启服务器。

# init 6

15. 查看多路径当前状态

# multipath -ll

附3:删除LUN的操作

通过匹配 lsblk -s 的结果分别执行

echo 1 > /sys/block/sdX/device/delete
multipath -f mpathc

从而达到删除LUN设备的目的。

下例,在 dbtan21/dbtan22 主机删除12个测试LUN( netapp-lunvd01 ~ netapp-lunvd12

dbtan21 节点:

--  dbtan21
echo 1 > /sys/block/sdbx/device/delete
echo 1 > /sys/block/sddl/device/delete
echo 1 > /sys/block/sdiq/device/delete
echo 1 > /sys/block/sduf/device/delete
echo 1 > /sys/block/sdfg/device/delete
echo 1 > /sys/block/sdrd/device/delete
echo 1 > /sys/block/sdnp/device/delete
echo 1 > /sys/block/sdxh/device/delete
multipath -f netapp-lunvd01

echo 1 > /sys/block/sdby/device/delete
echo 1 > /sys/block/sddw/device/delete
echo 1 > /sys/block/sdis/device/delete
echo 1 > /sys/block/sdug/device/delete
echo 1 > /sys/block/sdfh/device/delete
echo 1 > /sys/block/sdre/device/delete
echo 1 > /sys/block/sdnr/device/delete
echo 1 > /sys/block/sdxi/device/delete
multipath -f netapp-lunvd02

echo 1 > /sys/block/sdbz/device/delete
echo 1 > /sys/block/sdeh/device/delete
echo 1 > /sys/block/sdit/device/delete
echo 1 > /sys/block/sduh/device/delete
echo 1 > /sys/block/sdfi/device/delete
echo 1 > /sys/block/sdrf/device/delete
echo 1 > /sys/block/sdns/device/delete
echo 1 > /sys/block/sdxj/device/delete
multipath -f netapp-lunvd03

echo 1 > /sys/block/sdca/device/delete
echo 1 > /sys/block/sder/device/delete
echo 1 > /sys/block/sdiu/device/delete
echo 1 > /sys/block/sdui/device/delete
echo 1 > /sys/block/sdfj/device/delete
echo 1 > /sys/block/sdrg/device/delete
echo 1 > /sys/block/sdnu/device/delete
echo 1 > /sys/block/sdxk/device/delete
multipath -f netapp-lunvd04

echo 1 > /sys/block/sdcb/device/delete
echo 1 > /sys/block/sdfd/device/delete
echo 1 > /sys/block/sdiw/device/delete
echo 1 > /sys/block/sduj/device/delete
echo 1 > /sys/block/sdfk/device/delete
echo 1 > /sys/block/sdrh/device/delete
echo 1 > /sys/block/sdnv/device/delete
echo 1 > /sys/block/sdxl/device/delete
multipath -f netapp-lunvd05

echo 1 > /sys/block/sdcc/device/delete
echo 1 > /sys/block/sdfq/device/delete
echo 1 > /sys/block/sdix/device/delete
echo 1 > /sys/block/sduk/device/delete
echo 1 > /sys/block/sdfl/device/delete
echo 1 > /sys/block/sdri/device/delete
echo 1 > /sys/block/sdnw/device/delete
echo 1 > /sys/block/sdxm/device/delete
multipath -f netapp-lunvd06

echo 1 > /sys/block/sdfm/device/delete
echo 1 > /sys/block/sdrj/device/delete
echo 1 > /sys/block/sdnz/device/delete
echo 1 > /sys/block/sdxn/device/delete
echo 1 > /sys/block/sdcd/device/delete
echo 1 > /sys/block/sdgc/device/delete
echo 1 > /sys/block/sdja/device/delete
echo 1 > /sys/block/sdul/device/delete
multipath -f netapp-lunvd07

echo 1 > /sys/block/sdfn/device/delete
echo 1 > /sys/block/sdrk/device/delete
echo 1 > /sys/block/sdoa/device/delete
echo 1 > /sys/block/sdxo/device/delete
echo 1 > /sys/block/sdce/device/delete
echo 1 > /sys/block/sdgo/device/delete
echo 1 > /sys/block/sdjb/device/delete
echo 1 > /sys/block/sdum/device/delete
multipath -f netapp-lunvd08

echo 1 > /sys/block/sdfo/device/delete
echo 1 > /sys/block/sdrl/device/delete
echo 1 > /sys/block/sdob/device/delete
echo 1 > /sys/block/sdxp/device/delete
echo 1 > /sys/block/sdcf/device/delete
echo 1 > /sys/block/sdhb/device/delete
echo 1 > /sys/block/sdjd/device/delete
echo 1 > /sys/block/sdun/device/delete
multipath -f netapp-lunvd09

echo 1 > /sys/block/sdfp/device/delete
echo 1 > /sys/block/sdrm/device/delete
echo 1 > /sys/block/sdod/device/delete
echo 1 > /sys/block/sdxq/device/delete
echo 1 > /sys/block/sdcg/device/delete
echo 1 > /sys/block/sdhn/device/delete
echo 1 > /sys/block/sdje/device/delete
echo 1 > /sys/block/sduo/device/delete
multipath -f netapp-lunvd10

echo 1 > /sys/block/sdxt/device/delete
echo 1 > /sys/block/sdyb/device/delete
echo 1 > /sys/block/sdxx/device/delete
echo 1 > /sys/block/sdyf/device/delete
echo 1 > /sys/block/sdxr/device/delete
echo 1 > /sys/block/sdxz/device/delete
echo 1 > /sys/block/sdxv/device/delete
echo 1 > /sys/block/sdyd/device/delete
multipath -f netapp-lunvd11

echo 1 > /sys/block/sdxu/device/delete
echo 1 > /sys/block/sdyc/device/delete
echo 1 > /sys/block/sdxy/device/delete
echo 1 > /sys/block/sdyg/device/delete
echo 1 > /sys/block/sdxs/device/delete
echo 1 > /sys/block/sdya/device/delete
echo 1 > /sys/block/sdxw/device/delete
echo 1 > /sys/block/sdye/device/delete
multipath -f netapp-lunvd12

dbtan22 节点:

-- dbtan22
echo 1 > /sys/block/sdbt/device/delete
echo 1 > /sys/block/sdob/device/delete
echo 1 > /sys/block/sdhx/device/delete
echo 1 > /sys/block/sduf/device/delete
echo 1 > /sys/block/sdev/device/delete
echo 1 > /sys/block/sdrd/device/delete
echo 1 > /sys/block/sdkz/device/delete
echo 1 > /sys/block/sdxh/device/delete
multipath -f netapp-lunvd01

echo 1 > /sys/block/sdbu/device/delete
echo 1 > /sys/block/sdoc/device/delete
echo 1 > /sys/block/sdhy/device/delete
echo 1 > /sys/block/sdug/device/delete
echo 1 > /sys/block/sdew/device/delete
echo 1 > /sys/block/sdre/device/delete
echo 1 > /sys/block/sdla/device/delete
echo 1 > /sys/block/sdxi/device/delete
multipath -f netapp-lunvd02

echo 1 > /sys/block/sdbv/device/delete
echo 1 > /sys/block/sdod/device/delete
echo 1 > /sys/block/sdhz/device/delete
echo 1 > /sys/block/sduh/device/delete
echo 1 > /sys/block/sdex/device/delete
echo 1 > /sys/block/sdrf/device/delete
echo 1 > /sys/block/sdlb/device/delete
echo 1 > /sys/block/sdxj/device/delete
multipath -f netapp-lunvd03

echo 1 > /sys/block/sdbw/device/delete
echo 1 > /sys/block/sdoe/device/delete
echo 1 > /sys/block/sdia/device/delete
echo 1 > /sys/block/sdui/device/delete
echo 1 > /sys/block/sdey/device/delete
echo 1 > /sys/block/sdrg/device/delete
echo 1 > /sys/block/sdlc/device/delete
echo 1 > /sys/block/sdxk/device/delete
multipath -f netapp-lunvd04

echo 1 > /sys/block/sdbx/device/delete
echo 1 > /sys/block/sdof/device/delete
echo 1 > /sys/block/sdib/device/delete
echo 1 > /sys/block/sduj/device/delete
echo 1 > /sys/block/sdez/device/delete
echo 1 > /sys/block/sdrh/device/delete
echo 1 > /sys/block/sdld/device/delete
echo 1 > /sys/block/sdxl/device/delete
multipath -f netapp-lunvd05

echo 1 > /sys/block/sdby/device/delete
echo 1 > /sys/block/sdog/device/delete
echo 1 > /sys/block/sdic/device/delete
echo 1 > /sys/block/sduk/device/delete
echo 1 > /sys/block/sdfa/device/delete
echo 1 > /sys/block/sdri/device/delete
echo 1 > /sys/block/sdle/device/delete
echo 1 > /sys/block/sdxm/device/delete
multipath -f netapp-lunvd06

echo 1 > /sys/block/sdfb/device/delete
echo 1 > /sys/block/sdrj/device/delete
echo 1 > /sys/block/sdlf/device/delete
echo 1 > /sys/block/sdxn/device/delete
echo 1 > /sys/block/sdbz/device/delete
echo 1 > /sys/block/sdoh/device/delete
echo 1 > /sys/block/sdid/device/delete
echo 1 > /sys/block/sdul/device/delete
multipath -f netapp-lunvd07

echo 1 > /sys/block/sdfc/device/delete
echo 1 > /sys/block/sdrk/device/delete
echo 1 > /sys/block/sdlg/device/delete
echo 1 > /sys/block/sdxo/device/delete
echo 1 > /sys/block/sdca/device/delete
echo 1 > /sys/block/sdoi/device/delete
echo 1 > /sys/block/sdie/device/delete
echo 1 > /sys/block/sdum/device/delete
multipath -f netapp-lunvd08

echo 1 > /sys/block/sdfd/device/delete
echo 1 > /sys/block/sdrl/device/delete
echo 1 > /sys/block/sdlh/device/delete
echo 1 > /sys/block/sdxp/device/delete
echo 1 > /sys/block/sdcb/device/delete
echo 1 > /sys/block/sdoj/device/delete
echo 1 > /sys/block/sdif/device/delete
echo 1 > /sys/block/sdun/device/delete
multipath -f netapp-lunvd09

echo 1 > /sys/block/sdfe/device/delete
echo 1 > /sys/block/sdrm/device/delete
echo 1 > /sys/block/sdli/device/delete
echo 1 > /sys/block/sdxq/device/delete
echo 1 > /sys/block/sdcc/device/delete
echo 1 > /sys/block/sdok/device/delete
echo 1 > /sys/block/sdig/device/delete
echo 1 > /sys/block/sduo/device/delete
multipath -f netapp-lunvd10

echo 1 > /sys/block/sdxt/device/delete
echo 1 > /sys/block/sdyb/device/delete
echo 1 > /sys/block/sdxx/device/delete
echo 1 > /sys/block/sdyf/device/delete
echo 1 > /sys/block/sdxr/device/delete
echo 1 > /sys/block/sdxz/device/delete
echo 1 > /sys/block/sdxv/device/delete
echo 1 > /sys/block/sdyd/device/delete
multipath -f netapp-lunvd11

echo 1 > /sys/block/sdxu/device/delete
echo 1 > /sys/block/sdyc/device/delete
echo 1 > /sys/block/sdxy/device/delete
echo 1 > /sys/block/sdyg/device/delete
echo 1 > /sys/block/sdxs/device/delete
echo 1 > /sys/block/sdya/device/delete
echo 1 > /sys/block/sdxw/device/delete
echo 1 > /sys/block/sdye/device/delete
multipath -f netapp-lunvd12

附4:网卡配置中添加 hotplug=no 参数,避免start_udev命令导致Oracle RAC 的vip漂移问题

start_udev 命令导致网卡重启

从而导致vip漂移

解决办法:网卡配置中添加 hotplug=no 参数
注意:如果是使用网卡绑定,比如绑定后的网卡为bond0,则要在bond0的配置文件里添加hotplug配置信息,在eth0里添加不起作用。

参考:

Network interface going down when dynamically adding disks to storage using udev in RHEL 6 (文档 ID 1569028.1)

udev重新加载配置

RAC ASM磁盘扩容执行start_udev命令导致vip漂移问题分析及解决办法

dbtan21 测试节点

[root@dbtan21: /etc/sysconfig/network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.18.21
NETMASK=255.255.255.0
GATEWAY=192.168.18.1
HOTPLUG="no"
[root@dbtan21: /etc/sysconfig/network-scripts]# cat ifcfg-bond1
DEVICE=bond1
BOOTPROTO=static
ONBOOT=yes
IPADDR=192.168.118.21
NETMASK=255.255.255.0
HOTPLUG="no"
[root@dbtan21: /etc/sysconfig/network-scripts]# cat ifcfg-bond2
DEVICE=bond2
BOOTPROTO=static
ONBOOT=yes
IPADDR=172.16.18.21
NETMASK=255.255.254.0
HOTPLUG="no"
[root@dbtan21: /etc/sysconfig/network-scripts]# 

网卡绑定信息

[root@dbtan21: /etc/modprobe.d]# cat modprobe.conf 
alias bond0 bonding
options bond0 miimon=100 mode=4
alias bond1 bonding
options bond1 miimon=100 mode=1
alias bond2 bonding
options bond2 miimon=100 mode=4
[root@dbtan21: /etc/modprobe.d]# 
[root@dbtan21: /etc/modprobe.d]# cat /proc/net/bonding/bond0 
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 41
        Number of ports: 2
        Actor Key: 11
        Partner Key: 32816
        Partner Mac Address: 00:23:04:ee:be:64

Slave Interface: eth6
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:c2:d1:f4:5f:dc
Aggregator ID: 41
Slave queue ID: 0

Slave Interface: eth8
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:c2:d1:f4:5f:de
Aggregator ID: 41
Slave queue ID: 0
[root@dbtan21: /etc/modprobe.d]# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 41
        Number of ports: 1
        Actor Key: 11
        Partner Key: 1
        Partner Mac Address: 00:00:00:00:00:00

Slave Interface: eth10
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 44:6a:2e:ee:18:f0
Aggregator ID: 41
Slave queue ID: 0

Slave Interface: eth11
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 44:6a:2e:ee:18:f1
Aggregator ID: 42
Slave queue ID: 0
[root@dbtan21: /etc/modprobe.d]# cat /proc/net/bonding/bond2
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 41
        Number of ports: 2
        Actor Key: 11
        Partner Key: 833
        Partner Mac Address: 70:79:90:a8:40:91

Slave Interface: eth7
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:c2:d1:f4:5f:dd
Aggregator ID: 41
Slave queue ID: 0

Slave Interface: eth9
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:c2:d1:f4:5f:df
Aggregator ID: 41
Slave queue ID: 0
[root@dbtan21: /etc/modprobe.d]# ethtool bond0
Settings for bond0:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 20000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes
[root@dbtan21: /etc/modprobe.d]# ethtool bond1
Settings for bond1:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes
[root@dbtan21: /etc/modprobe.d]# ethtool bond2
Settings for bond2:
        Supported ports: [ ]
        Supported link modes:   Not reported
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 20000Mb/s
        Duplex: Full
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Link detected: yes
[root@dbtan21: /etc/modprobe.d]# 

Reload and Restart the udev rules

[root@dbtan21: ~]# udevadm control --reload-rules
[root@dbtan21: ~]# udevadm trigger
[root@dbtan21: ~]# udevadm trigger  subsystem-match=block
[root@dbtan21: ~]# udevadm trigger  subsystem-nomatch=net
[root@dbtan21: ~]# start_udev 
Starting udev:                                             [  OK  ]
[root@dbtan21: ~]# 

udevadm trigger [options]
接收内核发送来的设备事件。主要用于重放coldplug事件信息
(译者补充:内核在启动时已经检测到了系统的硬件设备,并把硬件设备信息通过sysfs内核虚拟文件系统导出。udev扫描sysfs文件系统,根据硬件设备信息生成热插拔(hotplug)事件,udev再读取这些事件,生成对应的硬件设备文件。由于没有实际的硬件插拔动作,所以这一过程被称为coldplug。)

--verbose 输出将要被触发的设备列表。
--dry-run 不真的触发事件
--type=type 触发一个特殊的设备。合法的类型:devices,subsystem,failed.默认是devices
--action=action
被触发的事件,默认是change
--subsystem-match=subsystem
触发匹配子系统的设备事件。这个选项可以被多次指定,并且支持shell模式匹配。
--attr-match=attribute=value
触发匹配sysfs属性的设备事件。如果属性值和属性一起指定,属性的值可以使用shell模式匹配。如果没有指定值,会重新确认现有属性。这个选项可以被多次指定。
--attr-nomatch=attribute=value
不要触发匹配属性的设备事件。如果可以使用模式匹配。也可以多次指定
--property-match=property=value
匹配属性吻合的设备。可以多次指定支持模式匹配
--tag-match=property
匹配标签吻合的设备。可以多次指定。
--sysname-match=name

-- The End --