10.3.6. Operation by Operation
This page shows, for each operation, what PMIx does before it calls the host, and what it then expects the host to do. Each up-call is described in full in pmix_server_module_t(5).
Some things are true of every up-call made for a client or tool, and are not repeated below:
PMIx has already authenticated the requester’s connection.
PMIx has checked the request’s arguments - counts, array types, the form of names.
PMIx adds the requester’s identity (see Deciding What a Requester May Do).
The host decides whether the requester may have the operation, and reports the result through the callback.
“If NULL” says what happens when the host leaves the up-call out.
10.3.6.1. Collective operations
A collective involves processes on many nodes. PMIx assembles the part on its own node. The host completes it across nodes.
For fence_nb, connect and disconnect, PMIx:
replaces any group name among the participants with the group’s members;
checks that each local caller is one of the participants, and fails it with
PMIX_ERR_NOT_A_MEMBERif not;sorts the participants, and says so with
PMIX_SORTED_PROC_ARRAY;waits until every local participant has called, enforcing any
PMIX_TIMEOUTmeanwhile;calls the host once for the whole node, with
PMIX_LOCAL_COLLECTIVE_STATUSreporting how the local part went. If that status is an error, the host should fail the collective rather than complete it.
The host must use the job information it registered to know how many local participants to expect; that is why every job taking part in a collective with a local process must be registered, even one with no process on this node.
10.3.6.1.1. fence_nb
Up-call: fence_nb.
Services PMIx_Fence(3).
PMIx: collects the data the local participants posted into one
blob and passes it to the host. If every participant is on this node,
PMIx completes the fence itself and does not call the host (unless
pmix_server_fence_localonly_opt is set to false).
The host:
runs the collective across every server hosting a participant;
if data is being collected (
PMIX_COLLECT_DATA), returns the combined blobs of all those servers through the callback;honors the directives, and fails the fence if it cannot honor one marked required;
free()s the blob it was given when it is done with it.
If NULL: a fence that spans nodes fails with
PMIX_ERR_NOT_SUPPORTED.
10.3.6.1.2. connect / disconnect
Up-calls: connect, disconnect.
Service PMIx_Connect(3) and PMIx_Disconnect(3).
PMIx: for connect, adds to the request, for each local
participant, the data it posted for remote access
(PMIX_PROC_INFO_ARRAY), and passes on the job information each
participant supplied (PMIX_JOB_INFO_ARRAY). When the host completes
the operation, PMIx gives each local participant the job-level
information of every participating job, from what it holds.
The host:
runs the collective across every server hosting a participant. For
connectthis is an exchange of data, not just a barrier, even though the request carries noPMIX_COLLECT_DATA: thePMIX_PROC_INFO_ARRAYandPMIX_JOB_INFO_ARRAYentries from every server must reach every other;before completing the operation on a server, gives that server those entries and the job-level information of every participating job. Two ways to do it:
register each entry as an update to its job with PMIx_server_register_nspace(3), passing a negative number of local processes. A
PMIX_PROC_INFO_ARRAYupdates the process it describes; aPMIX_JOB_INFO_ARRAYupdates the job. This is what PRRTE does. The job must already be registered with that server - PRRTE registers every job with every server;or deliver them with PMIx_server_register_resources(3), as for
groupbelow;
from then on, treats the connected processes as one: when one of them terminates without disconnecting, it generates
PMIX_ERR_PROC_TERM_WO_SYNCfor the others;for
disconnect, fails the operation if the processes were not connected.
If NULL: an operation whose participants are all on this node
completes locally; any other fails with PMIX_ERR_NOT_SUPPORTED.
10.3.6.1.3. group
Up-call: group.
Services PMIx_Group_construct(3), PMIx_Group_destruct(3), and their related operations.
PMIx:
replaces any group name among the participants with its members, and checks that each local caller is a participant (except for processes joining as added members or through bootstrap, which are not named in advance);
builds, for each local participant, the data it posted for remote access, and attaches it to the request;
combines the local participants into one request, except for bootstrap and invited members, which reach the host one call at a time;
when the operation completes, stores what the host returned and releases the participants.
The host:
runs the collective across every server hosting a participant, and declares it complete only when every member has taken part - for a bootstrap, the stated number of leaders plus each added member;
returns through the callback the final membership (
PMIX_GROUP_MEMBERSHIP), a context ID if one was requested (PMIX_GROUP_CONTEXT_ID), and the group information participants contributed (PMIX_GROUP_INFO);before completing the operation on a server, gives that server:
the job-level information of every job with a participant. A host that registers every job with every server, as PRRTE does, has done this already. Otherwise, collect it where it is held with PMIx_server_collect_job_info(3) and deliver it with PMIx_server_register_resources(3) as
PMIX_GROUP_JOB_INFO;the data every participant posted for remote access - the contributions PMIx attached to the request - delivered with
PMIx_server_register_resourcesasPMIX_GROUP_ENDPT_DATA;
stops the operation when PMIx sends
PMIX_GROUP_CANCEL;keeps operations that share a group ID apart. A group ID names at most one operation of each kind (construct, destruct) at a time, across every server. PMIx may pass up more than one request for the same ID and operation: bootstrap leaders and added members each reach the host on their own call. When the host answers any one of those requests, PMIx gives that answer to every request with the same ID and operation that the host still holds. It does this because a host that combines the requests answers only once (PRRTE combines them by group ID and operation). So when a request arrives for an ID that is already in use, and its processes are not part of the operation under way, the host must refuse it rather than run it alongside. Run separately, whichever operation completed first would be delivered to the participants of both.
Each participant then has the job-level information of every participating job, and every participant’s posted data.
If NULL: PMIX_ERR_NOT_SUPPORTED.
For the construction methods and their data flow, see Group Construction, Destruction, and Fault Tolerance and Group Operations: Theory and Implementation.
10.3.6.2. Data
10.3.6.2.1. direct_modex
Up-call: direct_modex.
Fetches data a process posted, when it is held on another node.
PMIx: applies the access rule for the requester, then asks the host for the target process’s data. Requesters with the same identity share one request.
The host: asks the server holding the data with
PMIx_server_dmodex_request2(3),
passing the identity from the up-call. That server applies the rule
again before answering. The host returns the data through the callback;
it may not answer PMIX_OPERATION_SUCCEEDED.
If NULL: the request fails with PMIX_ERR_NOT_FOUND.
10.3.6.2.2. publish / lookup / unpublish
Up-calls: publish, lookup, unpublish.
Service PMIx_Publish(3), PMIx_Lookup(3) and PMIx_Unpublish(3).
PMIx: passes the request up with the requester’s identity.
The host: keeps the published data, applies its range and
persistence, records who published each item, and decides who may see
or remove it - honoring any PMIX_ACCESS_PERMISSIONS the publisher
gave. lookup may not answer PMIX_OPERATION_SUCCEEDED.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.2.3. query
Up-call: query.
Services PMIx_Query_info(3), and the queries behind PMIx_Resolve_peers(3) and PMIx_Resolve_nodes(3).
PMIx: answers every query it can from what it holds, applying the access rule. Only the rest go to the host.
The host: answers the remaining queries, deciding what the requester may learn.
If NULL: the requester receives what PMIx could answer, as
PMIX_ERR_PARTIAL_SUCCESS. If PMIx could answer none of it, the
request fails.
10.3.6.3. Jobs and resources
10.3.6.3.1. spawn
Up-call: spawn.
Services PMIx_Spawn(3).
PMIx: resolves user and group names in the job’s directives, and sets up output forwarding and inheritance for the new job.
The host: decides whether the requester may launch the job, then
launches all of it or none of it. It records any
PMIX_ACCESS_PERMISSIONS given, to register with the job. When it
registers the job, it includes PMIX_PARENT_ID and the output
formatting directives the request carried, so the job’s output can be
forwarded and formatted where its parent’s is (see
PMIx_server_register_nspace(3)).
The new job’s namespace is returned through the callback;
PMIX_OPERATION_SUCCEEDED is not allowed.
If NULL: PMIX_ERR_NOT_SUPPORTED - except in a launcher that
runs its own processes, which starts the job itself for its own user.
10.3.6.3.2. abort2 / abort
Services PMIx_Abort(3).
PMIx: checks the request’s form only. It does not check who the targets are.
The host: decides whether the requester may terminate the processes named (all of the requester’s job, if none are named), then terminates them.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.3.3. job_control
Up-call: job_control.
Services PMIx_Job_control(3).
PMIx: handles cleanup directives itself (PMIX_REGISTER_CLEANUP
and related), applying the access rule to each job they name. A request
containing nothing else is completed without calling the host.
The host: decides whether the requester may act on the processes named, and performs every other action - signal, kill, pause, resume, terminate.
If NULL: PMIX_ERR_NOT_SUPPORTED, including for requests that
contain only cleanup directives.
10.3.6.3.4. monitor
Up-call: monitor.
Services PMIx_Process_monitor(3).
PMIx: monitors processes on its own node itself, applying the access rule. It calls the host only when the request names processes on other nodes.
The host: decides whether the requester may monitor those
processes, and arranges it. A host that wants its processes monitored
must record their pids (PMIX_PROC_PID) when it registers them. See
Process & Resource Statistics (Monitoring).
If NULL: requests for other nodes fail with
PMIX_ERR_NOT_SUPPORTED.
10.3.6.3.5. allocate / session_control / resource_block
Up-calls: allocate, session_control, resource_block.
Service PMIx_Allocation_request(3), PMIx_Session_control(3) and PMIx_Resource_block(3).
PMIx: passes the request up with the requester’s identity.
The host: decides whether the requester may obtain, change, or release the resources or session concerned, and does so - often by relaying the request to the scheduler. See Scheduler Integration.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.4. Input and output
10.3.6.4.1. iof_pull
Up-call: iof_pull.
Services PMIx_IOF_pull(3).
PMIx: records the request but forwards nothing until the host approves it. Output that arrives meanwhile is held, and delivered once approved.
The host: decides whether the requester may receive output from the named processes, which may belong to any job. To refuse, report an error through the callback.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.4.2. push_stdin
Up-call: push_stdin.
Services PMIx_IOF_push(3).
PMIx: passes the input up, naming its source. It does not check the targets.
The host: decides whether the requester may send input to the targets, and delivers it.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.5. Events
10.3.6.5.1. register_events / deregister_events
Up-calls: register_events, deregister_events.
PMIx: counts registrations itself. It calls register_events
only for system event codes gaining their first registrant, and
deregister_events when one loses its last.
The host: forwards those system events while they are registered. It reports events concerning its jobs whether or not anything is registered.
If NULL: registering for a system event fails with
PMIX_ERR_NOT_SUPPORTED. Other registrations are handled locally.
10.3.6.5.2. notify_event
Up-call: notify_event.
Services PMIx_Notify_event(3).
PMIx: sets the event’s source to the process that raised it, checks the few events a client may not raise, delivers the event on this node, and calls the host only if the event’s range extends beyond the node.
The host: delivers the event to every server where its targets run.
If NULL: the event is delivered on this node only.
10.3.6.6. Other operations
10.3.6.6.1. log2 / log
Services PMIx_Log(3).
PMIx: adds the source and, if requested, a timestamp. A server acting as a logging gateway logs the data itself and does not call the host.
The host: sends the data to the channels requested, or to a
gateway. It must not call PMIx_Log from this up-call.
If NULL: PMIX_ERR_NOT_AVAILABLE.
10.3.6.6.2. fabric
Up-call: fabric.
Services PMIx_Fabric_register(3) and PMIx_Fabric_update(3).
PMIx: answers from its own network support if it can.
The host: obtains the fabric information, normally from the server that supports the system scheduler.
If NULL: PMIX_ERR_NOT_SUPPORTED.
10.3.6.6.3. get_credential / validate_credential
Up-calls: get_credential, validate_credential.
See Accepting Connections. The result travels only through
the callback, so PMIX_OPERATION_SUCCEEDED is not allowed.
10.3.6.6.4. client_connected2 / tool_connected2 / client_finalized
Up-calls: client_connected2, tool_connected2, client_finalized.
See Accepting Connections and What a Host Must Do.