SRP (SCSI Remote protocol or SCSI RDMA protocol) is a protocol that allows the use of SCSI devices across infiniband. If you have infiniband storage, use can use SRP to acess the devices.
Ensure that your infiniband storage is presented to the host in question. Check your storage controller documentation. Ensure that the ib_srp kernel module is loaded and that the srptools package is installed.
modprobe ib_srp
srp_daemon is responsible for discovering and connecting to SRP targets. The default configuration shipped with srp_daemon is to ignore all presented devices; this is a failsafe to prevent devices from being mounted by accident on the wrong hosts.
The srp_daemon config file /etc/srp_daemon.conf has a simply syntax, and is described in the srp_daemon(1) manpage. Each line in this file is a rule which can be either to allow connection or to disallow connection according to the first character in the line (a or d accordingly) and ID of the storage device.
You can determine the IDs of SRP devices presented to your hosts by running the ibsrpdm -c command.
# ibsrpdm -c
id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,dgid=fe8000000000000050001ff10005052a,pkey=ffff,service_id=2a050500f11f0050
Once we have the IDs of the devices, we can add them to /etc/srp_daemon.conf. You can also specify other srp related options for the target, such as max_cmd_per_lun and Max_sect. These are storage specific; check your vendor documentation for reccomended values.
# This rule allows connection to our target
a id_ext=50001ff10005052a,ioc_guid=50001ff10005052a,max_cmd_per_lun=32,max_sect=65535
# This rule disallows everything else
d
/etc/init.d/srptools restart
In the example kernel log output the disk has been descovered as scsi device sdb.
scsi 3:0:0:1: Direct-Access IBM DCS9900 5.03 PQ: 0 ANSI: 5
sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
sd 3:0:0:1: [sdb] Write Protect is off
sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
sd 3:0:0:1: [sdb] 1953458176 4096-byte hardware sectors (8001365 MB)
sd 3:0:0:1: [sdb] Write Protect is off
sd 3:0:0:1: [sdb] Mode Sense: 97 00 10 08
sd 3:0:0:1: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
sdb:<6>scsi4 : SRP.T10:50001FF10005052A
unknown partition table
sd 3:0:0:1: [sdb] Attached SCSI disk
sd 3:0:0:1: Attached scsi generic sg5 type 0
The newly detected SRP device can be treated as an other scsi device. If you have multiple infiniband adapters you can use multipath-tools on top of the SRP devices to protects against a network failure. If you are not using multipathed IO you can simply format the device as normal.