10.3.6. Operation by Operation

This page shows, for each operation, what PMIx does before it calls the host, and what it then expects the host to do. Each up-call is described in full in pmix_server_module_t(5).

Some things are true of every up-call made for a client or tool, and are not repeated below:

  • PMIx has already authenticated the requester’s connection.

  • PMIx has checked the request’s arguments - counts, array types, the form of names.

  • PMIx adds the requester’s identity (see Deciding What a Requester May Do).

  • The host decides whether the requester may have the operation, and reports the result through the callback.

“If NULL” says what happens when the host leaves the up-call out.

10.3.6.1. Collective operations

A collective involves processes on many nodes. PMIx assembles the part on its own node. The host completes it across nodes.

For fence_nb, connect and disconnect, PMIx:

  • replaces any group name among the participants with the group’s members;

  • checks that each local caller is one of the participants, and fails it with PMIX_ERR_NOT_A_MEMBER if not;

  • sorts the participants, and says so with PMIX_SORTED_PROC_ARRAY;

  • waits until every local participant has called, enforcing any PMIX_TIMEOUT meanwhile;

  • calls the host once for the whole node, with PMIX_LOCAL_COLLECTIVE_STATUS reporting how the local part went. If that status is an error, the host should fail the collective rather than complete it.

The host must use the job information it registered to know how many local participants to expect; that is why every job taking part in a collective with a local process must be registered, even one with no process on this node.

10.3.6.1.1. fence_nb

Up-call: fence_nb.

Services PMIx_Fence(3).

PMIx: collects the data the local participants posted into one blob and passes it to the host. If every participant is on this node, PMIx completes the fence itself and does not call the host (unless pmix_server_fence_localonly_opt is set to false).

The host:

  • runs the collective across every server hosting a participant;

  • if data is being collected (PMIX_COLLECT_DATA), returns the combined blobs of all those servers through the callback;

  • honors the directives, and fails the fence if it cannot honor one marked required;

  • free()s the blob it was given when it is done with it.

If NULL: a fence that spans nodes fails with PMIX_ERR_NOT_SUPPORTED.

10.3.6.1.2. connect / disconnect

Up-calls: connect, disconnect.

Service PMIx_Connect(3) and PMIx_Disconnect(3).

PMIx: for connect, adds to the request, for each local participant, the data it posted for remote access (PMIX_PROC_INFO_ARRAY), and passes on the job information each participant supplied (PMIX_JOB_INFO_ARRAY). When the host completes the operation, PMIx gives each local participant the job-level information of every participating job, from what it holds.

The host:

  • runs the collective across every server hosting a participant. For connect this is an exchange of data, not just a barrier, even though the request carries no PMIX_COLLECT_DATA: the PMIX_PROC_INFO_ARRAY and PMIX_JOB_INFO_ARRAY entries from every server must reach every other;

  • before completing the operation on a server, gives that server those entries and the job-level information of every participating job. Two ways to do it:

    • register each entry as an update to its job with PMIx_server_register_nspace(3), passing a negative number of local processes. A PMIX_PROC_INFO_ARRAY updates the process it describes; a PMIX_JOB_INFO_ARRAY updates the job. This is what PRRTE does. The job must already be registered with that server - PRRTE registers every job with every server;

    • or deliver them with PMIx_server_register_resources(3), as for group below;

  • from then on, treats the connected processes as one: when one of them terminates without disconnecting, it generates PMIX_ERR_PROC_TERM_WO_SYNC for the others;

  • for disconnect, fails the operation if the processes were not connected.

If NULL: an operation whose participants are all on this node completes locally; any other fails with PMIX_ERR_NOT_SUPPORTED.

10.3.6.1.3. group

Up-call: group.

Services PMIx_Group_construct(3), PMIx_Group_destruct(3), and their related operations.

PMIx:

  • replaces any group name among the participants with its members, and checks that each local caller is a participant (except for processes joining as added members or through bootstrap, which are not named in advance);

  • builds, for each local participant, the data it posted for remote access, and attaches it to the request;

  • combines the local participants into one request, except for bootstrap and invited members, which reach the host one call at a time;

  • when the operation completes, stores what the host returned and releases the participants.

The host:

  • runs the collective across every server hosting a participant, and declares it complete only when every member has taken part - for a bootstrap, the stated number of leaders plus each added member;

  • returns through the callback the final membership (PMIX_GROUP_MEMBERSHIP), a context ID if one was requested (PMIX_GROUP_CONTEXT_ID), and the group information participants contributed (PMIX_GROUP_INFO);

  • before completing the operation on a server, gives that server:

    • the job-level information of every job with a participant. A host that registers every job with every server, as PRRTE does, has done this already. Otherwise, collect it where it is held with PMIx_server_collect_job_info(3) and deliver it with PMIx_server_register_resources(3) as PMIX_GROUP_JOB_INFO;

    • the data every participant posted for remote access - the contributions PMIx attached to the request - delivered with PMIx_server_register_resources as PMIX_GROUP_ENDPT_DATA;

  • stops the operation when PMIx sends PMIX_GROUP_CANCEL;

  • keeps operations that share a group ID apart. A group ID names at most one operation of each kind (construct, destruct) at a time, across every server. PMIx may pass up more than one request for the same ID and operation: bootstrap leaders and added members each reach the host on their own call. When the host answers any one of those requests, PMIx gives that answer to every request with the same ID and operation that the host still holds. It does this because a host that combines the requests answers only once (PRRTE combines them by group ID and operation). So when a request arrives for an ID that is already in use, and its processes are not part of the operation under way, the host must refuse it rather than run it alongside. Run separately, whichever operation completed first would be delivered to the participants of both.

Each participant then has the job-level information of every participating job, and every participant’s posted data.

If NULL: PMIX_ERR_NOT_SUPPORTED.

For the construction methods and their data flow, see Group Construction, Destruction, and Fault Tolerance and Group Operations: Theory and Implementation.

10.3.6.2. Data

10.3.6.2.1. direct_modex

Up-call: direct_modex.

Fetches data a process posted, when it is held on another node.

PMIx: applies the access rule for the requester, then asks the host for the target process’s data. Requesters with the same identity share one request.

The host: asks the server holding the data with PMIx_server_dmodex_request2(3), passing the identity from the up-call. That server applies the rule again before answering. The host returns the data through the callback; it may not answer PMIX_OPERATION_SUCCEEDED.

If NULL: the request fails with PMIX_ERR_NOT_FOUND.

10.3.6.2.2. publish / lookup / unpublish

Up-calls: publish, lookup, unpublish.

Service PMIx_Publish(3), PMIx_Lookup(3) and PMIx_Unpublish(3).

PMIx: passes the request up with the requester’s identity.

The host: keeps the published data, applies its range and persistence, records who published each item, and decides who may see or remove it - honoring any PMIX_ACCESS_PERMISSIONS the publisher gave. lookup may not answer PMIX_OPERATION_SUCCEEDED.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.2.3. query

Up-call: query.

Services PMIx_Query_info(3), and the queries behind PMIx_Resolve_peers(3) and PMIx_Resolve_nodes(3).

PMIx: answers every query it can from what it holds, applying the access rule. Only the rest go to the host.

The host: answers the remaining queries, deciding what the requester may learn.

If NULL: the requester receives what PMIx could answer, as PMIX_ERR_PARTIAL_SUCCESS. If PMIx could answer none of it, the request fails.

10.3.6.3. Jobs and resources

10.3.6.3.1. spawn

Up-call: spawn.

Services PMIx_Spawn(3).

PMIx: resolves user and group names in the job’s directives, and sets up output forwarding and inheritance for the new job.

The host: decides whether the requester may launch the job, then launches all of it or none of it. It records any PMIX_ACCESS_PERMISSIONS given, to register with the job. When it registers the job, it includes PMIX_PARENT_ID and the output formatting directives the request carried, so the job’s output can be forwarded and formatted where its parent’s is (see PMIx_server_register_nspace(3)). The new job’s namespace is returned through the callback; PMIX_OPERATION_SUCCEEDED is not allowed.

If NULL: PMIX_ERR_NOT_SUPPORTED - except in a launcher that runs its own processes, which starts the job itself for its own user.

10.3.6.3.2. abort2 / abort

Up-calls: abort2, abort.

Services PMIx_Abort(3).

PMIx: checks the request’s form only. It does not check who the targets are.

The host: decides whether the requester may terminate the processes named (all of the requester’s job, if none are named), then terminates them.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.3.3. job_control

Up-call: job_control.

Services PMIx_Job_control(3).

PMIx: handles cleanup directives itself (PMIX_REGISTER_CLEANUP and related), applying the access rule to each job they name. A request containing nothing else is completed without calling the host.

The host: decides whether the requester may act on the processes named, and performs every other action - signal, kill, pause, resume, terminate.

If NULL: PMIX_ERR_NOT_SUPPORTED, including for requests that contain only cleanup directives.

10.3.6.3.4. monitor

Up-call: monitor.

Services PMIx_Process_monitor(3).

PMIx: monitors processes on its own node itself, applying the access rule. It calls the host only when the request names processes on other nodes.

The host: decides whether the requester may monitor those processes, and arranges it. A host that wants its processes monitored must record their pids (PMIX_PROC_PID) when it registers them. See Process & Resource Statistics (Monitoring).

If NULL: requests for other nodes fail with PMIX_ERR_NOT_SUPPORTED.

10.3.6.3.5. allocate / session_control / resource_block

Up-calls: allocate, session_control, resource_block.

Service PMIx_Allocation_request(3), PMIx_Session_control(3) and PMIx_Resource_block(3).

PMIx: passes the request up with the requester’s identity.

The host: decides whether the requester may obtain, change, or release the resources or session concerned, and does so - often by relaying the request to the scheduler. See Scheduler Integration.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.4. Input and output

10.3.6.4.1. iof_pull

Up-call: iof_pull.

Services PMIx_IOF_pull(3).

PMIx: records the request but forwards nothing until the host approves it. Output that arrives meanwhile is held, and delivered once approved.

The host: decides whether the requester may receive output from the named processes, which may belong to any job. To refuse, report an error through the callback.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.4.2. push_stdin

Up-call: push_stdin.

Services PMIx_IOF_push(3).

PMIx: passes the input up, naming its source. It does not check the targets.

The host: decides whether the requester may send input to the targets, and delivers it.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.5. Events

10.3.6.5.1. register_events / deregister_events

Up-calls: register_events, deregister_events.

PMIx: counts registrations itself. It calls register_events only for system event codes gaining their first registrant, and deregister_events when one loses its last.

The host: forwards those system events while they are registered. It reports events concerning its jobs whether or not anything is registered.

If NULL: registering for a system event fails with PMIX_ERR_NOT_SUPPORTED. Other registrations are handled locally.

10.3.6.5.2. notify_event

Up-call: notify_event.

Services PMIx_Notify_event(3).

PMIx: sets the event’s source to the process that raised it, checks the few events a client may not raise, delivers the event on this node, and calls the host only if the event’s range extends beyond the node.

The host: delivers the event to every server where its targets run.

If NULL: the event is delivered on this node only.

10.3.6.6. Other operations

10.3.6.6.1. log2 / log

Up-calls: log2, log.

Services PMIx_Log(3).

PMIx: adds the source and, if requested, a timestamp. A server acting as a logging gateway logs the data itself and does not call the host.

The host: sends the data to the channels requested, or to a gateway. It must not call PMIx_Log from this up-call.

If NULL: PMIX_ERR_NOT_AVAILABLE.

10.3.6.6.2. fabric

Up-call: fabric.

Services PMIx_Fabric_register(3) and PMIx_Fabric_update(3).

PMIx: answers from its own network support if it can.

The host: obtains the fabric information, normally from the server that supports the system scheduler.

If NULL: PMIX_ERR_NOT_SUPPORTED.

10.3.6.6.3. get_credential / validate_credential

Up-calls: get_credential, validate_credential.

See Accepting Connections. The result travels only through the callback, so PMIX_OPERATION_SUCCEEDED is not allowed.

10.3.6.6.4. client_connected2 / tool_connected2 / client_finalized

Up-calls: client_connected2, tool_connected2, client_finalized.

See Accepting Connections and What a Host Must Do.