3.5.6. Applications and Libraries on PMIx v7

For developers of applications, libraries, and programming models - MPI implementations, for example - that call PMIx_Init.

3.5.6.1. Binary and source compatibility

A program built against PMIx v5.0 runs against v7 without being rebuilt, and builds against v7 unchanged. See Version Numbers and Binary Compatibility.

There are two exceptions.

The provisional resource-usage structures are gone. In v5.0, pmix_proc_stats_t, pmix_node_stats_t, pmix_disk_stats_t, pmix_net_stats_t, their PMIx_*_stats_* functions, and their macros were provisional, in the PMIx Standard’s sense: open to change until the Standard’s committee decided. The committee chose arrays of attributes instead, and v7 replaces the provisional definitions with that final one. No v5.0 API returned them, so only a program that built them itself is affected. For resource usage, call PMIx_Process_monitor with PMIX_MONITOR_PROC_RESOURCE_USAGE (or the NODE, DISK, or NET form); the results are arrays of attributes, such as PMIX_PROC_RSS.

Three attribute strings changed, to fix a bug. In v5.0, three attributes had the same string as another attribute, so the two could not be told apart. Each now has its own string:

  • PMIX_FABRIC_DEVICE_COORDINATES (shared with PMIX_FABRIC_COORDINATES);

  • PMIX_JOB_CTRL_CHECKPOINT_TIMEOUT (shared with PMIX_JOB_CTRL_CHECKPOINT_SIGNAL);

  • PMIX_QUERY_SUPPORTED_QUALIFIERS (shared with PMIX_QUERY_QUALIFIERS).

A program built against v5.0 that uses one of the three still sends the old string, which v7 reads as the other attribute of the pair. Rebuild it to use the new one.

  • Newly deprecated, still present: PMIX_CRYPTO_KEY, PMIX_COMPRESSED_STRING, PMIX_REGEX, PMIX_QUERY_PARTIAL_SUCCESS, PMIX_GROUP_CONTEXT_ID_ASSIGNED, PMIx_generate_regex, and PMIx_generate_ppn.

  • Capability flags (PMIX_CAP_* in pmix_version.h) say at build time which features the library has. Test for a flag rather than a version number.

3.5.6.2. Initialization

Failures are more precise.

  • PMIX_ERR_UNREACH - run as a singleton - now means only that no server was described to the process.

  • A server that was described but cannot be reached gives PMIX_ERR_COMM_FAILURE. v5.0 became a singleton silently.

  • New codes: PMIX_ERR_EXISTS (the rank is already connected), PMIX_ERR_OUTDATED (the server is older than v3.2), PMIX_ERR_LOCAL_URI_REQUIRED (the process was given a TCP address for a v7 server), and PMIX_ERR_INVALID_CRED (the namespace, rank, or connection token does not match what the host registered).

The identity variables leave the environment at init. PMIX_NAMESPACE, PMIX_RANK, the server’s address, the connection token, and the others that describe the process to PMIx are removed from the environment by PMIx_Init.

  • If you need them, read them before PMIx_Init, or ask PMIx with PMIx_Get afterwards.

  • Copies a language runtime made earlier - Python’s os.environ, say - are not changed, except by PMIx’s own bindings.

One connection per rank.

  • A process started after PMIx_Init does not inherit the identity, so if it calls PMIx_Init it is a singleton.

  • A process started before PMIx_Init that also calls it, as the same rank, is refused.

  • A process that execs loses its connection. The new program is a new process to the server.

A library that forks helper processes and expects them to share the parent’s PMIx identity needs to change: have the parent do the PMIx work, or have the host start the helpers.

3.5.6.3. Running under a server of a different release

  • A v7 library connects to servers from v3.2 on only when it was configured --enable-legacy-servers. That is what lets a newer container run on an older system. Without it - the default - the library refuses an earlier server and PMIx_Init fails.

  • A library older than the server - an application in a container built with PMIx v5, say - needs a server that takes legacy clients. Otherwise every process starts as a singleton: PMIx_Init reports one error, and the process runs alone. Sites choose whether to take them; see Installing and Administering PMIx v7.

  • Open MPI singletons that spawn. A singleton that calls MPI_Comm_spawn starts a prte and reads its address from a pipe. A v7 prte that takes legacy clients writes two addresses there, TCP first, and Open MPI built against PMIx v7 must take the second. That needs an Open MPI release with the change.

3.5.6.4. Data access

  • Another job’s data, output, or control follows that job’s access list. A process of a user the list does not admit gets PMIX_ERR_NO_PERMISSIONS.

  • PMIx_Put refuses reserved pmix keys (PMIX_ERR_BAD_PARAM), and a commit carrying one is refused whole.

  • Shared-memory job data (gds/shmem3, which replaces shmem2) is read-only. A process the file permissions do not admit fetches from the server instead. A client of an earlier release falls back to the hash store.

  • PMIx_Get on the caller’s thread with shmem3 is now the default.

  • PMIx_Get qualified by PMIX_SESSION_ID answers about the session named.

3.5.6.5. Callbacks and threads

A blocking PMIx call made from a PMIx callback or event handler returns PMIX_ERR_WOULD_BLOCK instead of hanging. Use the non-blocking form there, or hand the work to another thread.

3.5.6.6. Groups and events

  • A group ID must be unique while the group exists, non-empty, and may not be a job’s namespace (PMIX_ERR_EXISTS).

  • A process that is not a participant in a collective or group operation gets PMIX_ERR_NOT_A_MEMBER. The server checks this too, so clients of earlier releases see it.

  • Events a process raises reach the host with its verified identity. An event at PMIX_RANGE_NAMESPACE with no targets goes only to the source’s own job; more than one PMIX_EVENT_CUSTOM_RANGE is refused.

  • A rank that dies or is deregistered leaves its job’s whole-job collectives, so the survivors’ next PMIx_Fence completes. A rank the host restarts is counted back in.

3.5.6.7. Spawning

  • When the host provides no spawn, PMIx starts the processes itself only for a requester of the server’s own user.

  • GPU visibility is set for you. New in v7: when a process is mapped to GPUs, PMIx sets CUDA_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, or ZE_AFFINITY_MASK in its environment at launch, naming those devices. A value already set is replaced. A process not mapped to a GPU is left alone.

  • NIC selection is set for you too. When a process is mapped to network devices, PMIx names them to the libraries that use them, each in the form that library matches, and only for the fabrics whose devices the process was given: NCCL_IB_HCA and UCX_NET_DEVICES for InfiniBand and RoCE adapters (NVIDIA, Broadcom, and Intel’s in RoCE mode), with FI_VERBS_DEVICE_NAME as well unless that name would also match another device on the node; FI_EFA_IFACE for EFA; FI_CXI_DEVICE_NAME for Slingshot; and FI_OPX_HFI_SELECT for a single Omni-Path adapter (OPA100, CN5000 and later), for libfabric’s OPX provider. A value already set is replaced. A variable is left unset where the device cannot be named exactly in it, rather than written with a name that would select a different device. A device is named by the name these libraries accept even when the process was mapped against another name for it - an adapter’s network interface (ib0, hsn0) rather than its RDMA device.

  • OPX gets a job key. On an Omni-Path node, a launch that asks PMIx for security keys also sets FI_OPX_UUID, the key libfabric’s OPX provider uses for its shared-memory segments and hardware contexts. A job and the jobs it spawns share one key, so they reach each other through shared memory; jobs launched separately get different keys, even when they later connect, since OPX reads its key only when it opens its device. Give such jobs the same FI_OPX_UUID to share memory between them: a value already in the process’s environment is kept.