Summary of the content on the page No. 1
Cisco SFS InfiniBand Host Drivers
User Guide for Linux
Release 3.2.0
June 2007
Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
Text Part Number: OL-12309-01
Summary of the content on the page No. 2
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE IN
Summary of the content on the page No. 3
CONTENTS Preface vii Audience vii Organization vii Conventions viii Root and Non-root Conventions in Examples ix Related Documentation ix Obtaining Documentation, Obtaining Support, and Security Guidelines ix CHAPTER 1 About Host Drivers 1-1 Introduction 1-1 Architecture 1-2 Supported Protocols 1-3 IPoIB 1-3 SRP 1-3 SDP 1-3 Supported APIs 1-4 MVAPICH MPI 1-4 uDAPL 1-4 Intel MPI 1-4 HP MPI 1-4 HCA Utilities and Diagnostics 1-4 CHAPTER 2 Installing Host Drivers 2-1 Introduction 2-1 Contents of ISO
Summary of the content on the page No. 4
Contents Subinterfaces 3-2 Creating a Subinterface Associated with a Specific IB Partition 3-3 Removing a Subinterface Associated with a Specific IB Partition 3-4 Verifying IPoIB Functionality 3-5 IPoIB Performance 3-6 Sample Startup Configuration File 3-8 IPoIB High Availability 3-8 Merging Physical Ports 3-8 Unmerging Physical Ports 3-9 CHAPTER 4 SCSI RDMA Protocol 4-1 Introduction 4-1 Configuring SRP 4-1 Configuring ITLs when Using Fibre Channel Gateway 4-2 Configuring ITLs with Element Manag
Summary of the content on the page No. 5
Contents CHAPTER 7 MVAPICH MPI 7-1 Introduction 7-1 Initial Setup 7-2 Configuring SSH 7-2 Editing Environment Variables 7-5 Setting Environment Variables in System-Wide Startup Files 7-6 Editing Environment Variables in the Users Shell Startup Files 7-6 Editing Environment Variables Manually 7-7 MPI Bandwidth Test Performance 7-7 MPI Latency Test Performance 7-8 Intel MPI Benchmarks (IMB) Test Performance 7-9 Compiling MPI Programs 7-12 CHAPTER 8 HCA Utilities and Diagnostics 8-1 Introduction 8-
Summary of the content on the page No. 6
Contents Cisco SFS InfiniBand Host Drivers User Guide for Linux vi OL-12309-01
Summary of the content on the page No. 7
Preface This preface describes who should read the Cisco SFS InfiniBand Host Drivers User Guide for Linux, how it is organized, and its document conventions. It includes the following sections: • Audience, page vii Organization, page vii Conventions, page viii Root and Non-root Conventions in Examples, page ix Related Documentation, page ix Obtaining Documentation, Obtaining Support, and Security Guidelines, page ix Audience The intended audience is the administrator responsible for i
Summary of the content on the page No. 8
Preface Conventions Chapter Title Description Chapter 8 HCA Utilities and Diagnostics Describes the fundamental HCA utilities and diagnostics. Appendix A Acronyms and Abbreviations Defines the acronyms and abbreviations that are used in this publication. Conventions This document uses the following conventions: Convention Description boldface font Commands, command options, and keywords are in boldface. Bold text indicates Chassis Manager elements or text that you must enter as-is. italic fo
Summary of the content on the page No. 9
Preface Root and Non-root Conventions in Examples Notes use the following convention: Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual. Cautions use the following convention: Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data. Root and Non-root Conventions in Examples This document uses the following conventions to signify root and non-root accounts: C
Summary of the content on the page No. 10
Preface Obtaining Documentation, Obtaining Support, and Security Guidelines Cisco SFS InfiniBand Host Drivers User Guide for Linux x OL-12309-01
Summary of the content on the page No. 11
CH A P T E R1 About Host Drivers This chapter describes host drivers and includes the following sections: Introduction, page 1-1 Architecture, page 1-2 Supported Protocols, page 1-3 Supported APIs, page 1-4 HCA Utilities and Diagnostics, page 1-4 Note For expansions of acronyms and abbreviations used in this publication, see Appendix A, “Acronyms and Abbreviations.” Introduction The Cisco IB HCA offers high-performance 10-Gbps and 20-Gbps IB connectivity to PCI-X and PCI-Express-base
Summary of the content on the page No. 12
Chapter 1 About Host Drivers Architecture Note See the “Root and Non-root Conventions in Examples” section on page ix for details about the significance of prompts used in the examples in this chapter. Architecture Figure 1-1 displays the software architecture of the protocols and APIs that HCAs support. The figure displays ULPs and APIs in relation to other IB software elements. Figure 1-1 HCA Supported Protocols and API Architecture IP Based Block Application MPI Based App Storage Lev
Summary of the content on the page No. 13
Chapter 1 About Host Drivers Supported Protocols Supported Protocols This section describes the supported protocols and includes the following topics: IPoIB SRP SDP Protocol here refers to software in the networking layer in kernel space. IPoIB The IPoIB protocol passes IP traffic over the IB network. Configuring IPoIB requires similar steps to configuring IP on an Ethernet network. SDP relies on IPoIB to resolve IP addresses. (See the “SDP” section on page 1-3.) To configure IPoIB
Summary of the content on the page No. 14
Chapter 1 About Host Drivers Supported APIs Supported APIs This section describes the supported APIs and includes the following topics: MVAPICH MPI uDAPL Intel MPI HP MPI API refers to software in the networking layer in user space. MVAPICH MPI MPI is a standard library functionality in C, C++, and Fortran that can be used to implement a message-passing program. MPI allows the coordination of a program running as multiple processes in a distributed memory environment. This documen
Summary of the content on the page No. 15
CH A P T E R2 Installing Host Drivers The chapter includes the following sections: Introduction, page 2-1 Contents of ISO Image, page 2-2 Installing Host Drivers from an ISO Image, page 2-2 Uninstalling Host Drivers from an ISO Image, page 2-3 Note See the “Root and Non-root Conventions in Examples” section on page ix for details about the significance of prompts used in the examples in this chapter. Introduction The Cisco Linux IB driver is delivered as an ISO image. The ISO image cont
Summary of the content on the page No. 16
Chapter 2 Installing Host Drivers Contents of ISO Image Contents of ISO Image The ISO image contains the following directories and files: docs/ This directory contains the related documents. tsinstall This is the installation script. redhat/ This directory contains the binary RPMs for Red Hat Enterprise Linux. suse/ This directory contains the binary RPMs for SUSE Linux Enterprise Server. Installing Host Drivers from an ISO Image See the Cisco InfiniBand Host Channel Adapter Hardw
Summary of the content on the page No. 17
Chapter 2 Installing Host Drivers Uninstalling Host Drivers from an ISO Image topspin-ib-mpi-rhel4-3.2.0-136.x86_64 (MPI libraries, source code, docs, etc) topspin-ib-mod-rhel4-2.6.9-34.ELsmp-3.2.0-136.x86_64 (kernel modules) installing 100% ############################################################### Upgrading HCA 0 HCA.LionMini.A0 to firmware build 3.2.0.136 New Node GUID = 0005ad0000200848 New Port1 GUID = 0005ad0000200849 New Port2 GUID = 0005ad000020084a Programming HCA fir
Summary of the content on the page No. 18
Chapter 2 Installing Host Drivers Uninstalling Host Drivers from an ISO Image Cisco SFS InfiniBand Host Drivers User Guide for Linux 2-4 OL-12309-01
Summary of the content on the page No. 19
CH A P T E R3 IP over IB Protocol This chapter describes IP over IB protocol and includes the following sections: Introduction, page 3-1 Manually Configuring IPoIB for Default IB Partition, page 3-2 Subinterfaces, page 3-2 Verifying IPoIB Functionality, page 3-5 IPoIB Performance, page 3-6 Sample Startup Configuration File, page 3-8 IPoIB High Availability, page 3-8 Note See the “Root and Non-root Conventions in Examples” section on page ix for details about the significance of pr
Summary of the content on the page No. 20
Chapter 3 IP over IB Protocol Manually Configuring IPoIB for Default IB Partition Manually Configuring IPoIB for Default IB Partition To manually configure IPoIB for the default IB partition, perform the following steps: Step 1 Log in to your Linux host. Step 2 To configure the interface, enter the ifconfig command with the following items: The appropriate IB interface (ib0 or ib1 on a host with one HCA) The IP address that you want to assign to the interface The netmask keyword Th
