On Ebay I got hold of two Infiniband cards: infiniband Set 2 X Voltaire HCA 410Ex + 5m Kabel 10Gbit Netzwerkverbindung

My basic setup looks like.

  • ESXi Server (Dell T20) with an Infiniband card. CentOS7 is running as VM on the ESXi host. Infiniband card via pass-thru to the VM.
  • Old Compaq/HP Tower, running CentOS7 no VM.

The DELL T20 acts as iSCSI Initiator (Client) and the Compaq tower as iSCSI Target (Server)

Disabling firewall/iptables

As I’m already behind a pfSense firewall, no need for another firewall, which causes some headaches before.

[root@serv]# systemctl stop firewalld
[root@serv]# systemctl disable firewalld 

Disabling NetworkManager

Another service I got rid of is:NetworkManager. As this service tried to allocate network interfaces. Very nasty. If I did not disable NetworkManager, the interface for ib0 never got up after start/boot. [https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/3/html/Installation_and_Configuration_Guide/Disabling_Network_Manager.html]

[root@serv]# chkconfig --list NetworkManager
[root@serv]# service NetworkManager stop
[root@serv]# chkconfig NetworkManager off
[root@serv]# service network start
[root@serv]# chkconfig network on

First install the driver for Inifiniband on both machines.

[root@serv]# yum groupinstall "Infiniband Support"
[root@serv]# yum install infiniband-diags perftest qperf opensm
[root@serv]# chkconfig rdma on
[root@serv]# chkconfig opensm on
[root@serv]# shutdown -r now
[root@serv]# for i in `ls /sys/class/infiniband/*/ports/*/state`; do echo $i; cat $i; done
[root@serv]# ibhosts
[root@serv]# iblinkinfo

Setting static IP Address on the adapters

Edit /etc/sysconfig/network-scripts/ifcfg-ib0 and change the addresses according to your set up. I used for the client 10.0.0.51 and for the server 10.0.0.50

DEVICE=ib0
BOOTPROTO=static
NM_CONTROLLED=no
IPADDR=10.0.0.51
NETMASK=255.255.255.0
NETWORK=10.0.0.0
BROADCAST=10.0.0.255
ONBOOT=yes

Install iSCSI Initiator (Client)

[root@centnas init.d]# yum install iscsi-initiator-utils.x86_64
[root@centnas init.d]# more /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:3a2fba187ab3
InitiatorAlias=initiator01

[http://www.tecmint.com/iscsi-initiator-client-setup/]

Install iSCSI Target (Server)

[https://www.thomas-krenn.com/de/wiki/Linux-IO_Target_%28LIO%29_unter_Ubuntu_14.04] Bei ACL Anlegen den InitiatorName von oben eintragen.

Connect iSCSI Initiator

Finding target

[root@centnas init.d]# iscsiadm -m discovery -t sendtargets -p 10.0.0.50

Logging out iSCSI Targets

If you already once logged in and something changed, you have to logout first.

[root@centnas init.d]# iscsiadm -m node -u 

Connect

[root@centnas init.d]# iscsiadm --mode node --targetname iqn.2003-01.org.linux-iscsi.centsan.x8664:sn.6724a814bb24 --portal 10.0.0.50:3260 --login

Test

[root@centnas init.d]# iscsiadm --mode node --targetname iqn.2003-01.org.linux-iscsi.centsan.x8664:sn.6724a814bb24 --portal 10.0.0.50:3260

Create Filesystem

Now create a file

[root@centnas init.d]# parted /dev/sdf
(parted) mklabel gpt
(parted) mkpart primary ext4 0% 100%
(parted) quit
[root@centnas init.d]# mkfs.ext4 /dev/sdf1
[root@centnas init.d]# mkdir /mnt/iscsi_share
[root@centnas init.d]# mount /dev/sdf1 /mnt/iscsi_share/

Mount at startup. Add the following line to /etc/fstab

/dev/sdf1 /mnt/iscsi_share ext4 defaults,_netdev 0 0

INSTALL Samba/Cifs

[http://sharadchhetri.com/2014/10/09/centos-7-rhel-7-install-and-setup-samba-server-file-sharing/]

INSTALL crashplan service

[root@centnas system]# more /usr/lib/systemd/system/crashplan.service

[Unit]
Description=CrashPlan Backup Engine
After=network.target
[Service]
Type=forking
PIDFile=/usr/local/crashplan/CrashPlanEngine.pid
EnvironmentFile=/usr/local/crashplan/bin/run.conf
WorkingDirectory=/usr/local/crashplan
ExecStart=/usr/local/crashplan/bin/CrashPlanEngine start
ExecStop=/usr/local/crashplan/bin/CrashPlanEngine stop
[Install]
WantedBy=multi-user.target