Summary of the content on the page No. 1
Sim p lif y
Q
InfiniPath User Guide
Version 2.0
IB6054601-00 D Page i
Summary of the content on the page No. 2
InfiniPath User Guide Version 2.0 Q Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corpor
Summary of the content on the page No. 3
InfiniPath User Guide Version 2.0 Q Added info about using MPI over uDAPL. Need to load modules 3.7 rdma_cm and rdma_ucm. Added section: Error messages generated by mpirun. This explains C.8.12 more about the types of errors found in the sub-sections. Also added error messages related to failed connections between nodes Added mpirun error message about stray processes to error message C.8.12.2 section Added driver and link error messages reported by MPI programs C.8.12.3 Added section about err
Summary of the content on the page No. 4
InfiniPath User Guide Version 2.0 Q © 2006, 2007 QLogic Corporation. All rights reserved worldwide. © PathScale 2004, 2005, 2006. All rights reserved. First Published: August 2005 Printed in U.S.A. Page iv IB6054601-00 D
Summary of the content on the page No. 5
Table of Contents Section 1 Introduction 1.1 Who Should Read this Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 1.2 How this Guide is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1 1.3 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 1.4 Switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2 1.5 Interoperabilit
Summary of the content on the page No. 6
InfiniPath User Guide Version 2.0 Q 2.10 Performance and Management Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17 2.10.1 Remove Unneeded Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-17 2.10.2 Disable Powersaving Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-18 2.10.3 Balanced Processor Power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-19 2.10.4 SDP Module Parameters for Best Performance . . . .
Summary of the content on the page No. 7
InfiniPath User Guide Version 2.0 Q InfiniPath User Guide 3.11 Debugging MPI Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 3.11.1 MPI Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 3.11.2 Using Debuggers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-20 3.12 InfiniPath MPI Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-21
Summary of the content on the page No. 8
InfiniPath User Guide Version 2.0 Q C.4.5 OpenFabrics Load Errors If ib_ipath Driver Load Fails . . . . . . . . . . C-10 C.4.6 InfiniPath ib_ipath Initialization Failure . . . . . . . . . . . . . . . . . . . . . . C-11 C.4.7 MPI Job Failures Due to Initialization Problems . . . . . . . . . . . . . . . . . C-11 C.5 OpenFabrics Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-12 C.5.1 Stop OpenSM Before Stopping/Restarting InfiniPath . . . . . . . . .
Summary of the content on the page No. 9
InfiniPath User Guide Version 2.0 Q InfiniPath User Guide C.9.11 ipath_pkt_test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-35 C.9.12 ipathstats . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-35 C.9.13 lsmod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-36 C.9.14 mpirun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary of the content on the page No. 10
InfiniPath User Guide Version 2.0 Q Notes Page x IB6054601-00 D
Summary of the content on the page No. 11
Section 1 Introduction This chapter describes the objectives, intended audience, and organization of the InfiniPath User Guide. The InfiniPath User Guide is intended to give the end users of an InifiniPath cluster what they need to know to use it. In this case, end users are understood to include both the cluster administrator and the MPI application programmers, who have different but overlapping interests in the details of the technology. For specific instructions about installing th
Summary of the content on the page No. 12
1 – Introduction Interoperability Q ■ Appendix E Glossary of technical terms ■ Index In addition, the InfiniPath Install Guide contains information on InfiniPath hardware and software installation. 1.3 Overview The material in this documentation pertains to an InfiniPath cluster. This is defined as a collection of nodes, each attached to an InfiniBand™-based fabric through the InfiniPath Interconnect. The nodes are Linux-based computers, each having up to eight processors. The InfiniPath int
Summary of the content on the page No. 13
1 – Introduction What’s New in this Release Q NOTE: OpenFabrics was known as OpenIB until March 2006. All relevant references to OpenIB in this documentation have been updated to reflect this change. See the OpenFabrics website at http://www.openfabrics.org for more information on the OpenFabrics Alliance. 1.6 What’s New in this Release QLogic Corp. acquired PathScale in April 2006. In this 2.0 release, product names, internal program and output message names now refer to QLogic rather than
Summary of the content on the page No. 14
1 – Introduction Supported Distributions and Kernels Q Support for multiple versions of MPI has been added. You can use a different version of MPI and achieve the high-bandwidth and low-latency performance that is standard with InfiniPath MPI. Also included is expanded operating system support, and support for the latest OpenFabrics software stack. Multiple InfiniPath cards per node are supported. A single software installation works for all the cards. Additional up-to-date information ca
Summary of the content on the page No. 15
1 – Introduction Software Components Q 1.8 Software Components The software provided with the InfiniPath Interconnect product consists of: ■ InfiniPath driver (including OpenFabrics) ■ InfiniPath ethernet emulation ■ InfiniPath libraries ■ InfiniPath utilities, configuration, and support tools ■ InfiniPath MPI ■ InfiniPath MPI benchmarks ■ OpenFabrics protocols, including Subnet Management Agent ■ OpenFabrics libraries and utilities OpenFabrics kernel module support is now built and installed as
Summary of the content on the page No. 16
1 – Introduction Documentation and Technical Support Q NOTE: 32 bit OpenFabrics programs using the verb interfaces are not supported in this InfiniPath release, but will be supported in a future release. 1.9 Conventions Used in this Document This Guide uses these typographical conventions: Table 1-3. Typographical Conventions Convention Meaning command Fixed-space font is used for literal items such as commands, functions, programs, files and pathnames, and program output; variable Italic fix
Summary of the content on the page No. 17
1 – Introduction Documentation and Technical Support Q ■ Readme file The Troubleshooting Appendix for installation, InfiniPath and OpenFabrics administration, and MPI issues is located in the InfiniPath User Guide. Visit the QLogic support Web site for documentation and the latest software updates. http://www.qlogic.com IB6054601-00 D 1-7
Summary of the content on the page No. 18
1 – Introduction Documentation and Technical Support Q Notes 1-8 IB6054601-00 D
Summary of the content on the page No. 19
Section 2 InfiniPath Cluster Administration This chapter describes what the cluster administrator needs to know about the InfiniPath software and system administration. 2.1 Introduction The InfiniPath driver ib_ipath, layered Ethernet driver ipath_ether, OpenSM, and other modules and the protocol and MPI support libraries are the components of the InfiniPath software providing the foundation that supports the MPI implementation. Figure 2-1, below, shows these relationships. MPI Applic
Summary of the content on the page No. 20
2 – InfiniPath Cluster Administration Memory Footprint Q MPI include files are in: /usr/include MPI programming examples and source for several MPI benchmarks are in: /usr/share/mpich/examples InfiniPath utility programs, as well as MPI utilities and benchmarks are installed in: /usr/bin The InfiniPath kernel modules are installed in the standard module locations in: /lib/modules (version dependent) They are compiled and installed when the infinipath-kernel RPM is installed. They must be rebu
