25.2.21. PMIx_Session_control

PMIx_Session_control — Request a control action be applied to a session.

25.2.21.1. SYNOPSIS

#include <pmix.h>

pmix_status_t PMIx_Session_control(uint32_t sessionID,
                                   const pmix_info_t directives[], size_t ndirs,
                                   pmix_info_cbfunc_t cbfunc, void *cbdata);

25.2.21.1.1. Python Syntax

from pmix import *

# session control is bound on PMIxServer and inherited by PMIxTool and
# PMIxScheduler
foo = PMIxServer()
# ... after a successful foo.init() ...
# the directives is a list of Python ``pmix_info_t`` dictionaries
pydirs = [{'key': PMIX_SESSION_APP,
           'value': "myapp", 'val_type': PMIX_STRING}]
rc = foo.session_control(1, pydirs)

25.2.21.2. INPUT PARAMETERS

  • sessionID: A uint32_t identifying the session to which the control action is to be applied. A value of UINT32_MAX indicates that the action applies to all sessions under the caller’s control.

  • directives: Pointer to an array of pmix_info_t(5) structures describing the control action(s) to be taken (see DIRECTIVES). A NULL value is supported when no directives are desired.

  • ndirs: Number of elements in the directives array.

  • cbfunc: Callback function of type pmix_info_cbfunc_t. If NULL, the call is treated as a blocking operation; otherwise it is non-blocking and the callback is invoked once the request has been processed.

  • cbdata: Opaque pointer that is passed, unmodified, to cbfunc.

25.2.21.3. DESCRIPTION

Request that a control action be applied to the session identified by sessionID. The specific action(s) to be taken are conveyed as attributes in the directives array.

PMIx_Session_control supports both blocking and non-blocking operation through a single entry point, selected by the cbfunc argument:

  • If cbfunc is NULL, the call is blocking: it does not return until the request has been processed, and the return status reflects the overall outcome of the operation.

  • If cbfunc is non-NULL, the call is non-blocking: it returns immediately, and cbfunc is invoked with the final status and any returned information once the request has been processed. The callback reports whether the request was granted and may supply additional information — for example, the reason for any denial — in its array of pmix_info_t(5) structures.

As with all non-blocking PMIx APIs, when a cbfunc is supplied the caller must keep the directives array valid until cbfunc is invoked.

25.2.21.4. DIRECTIVES

The requested action is expressed entirely through the directives array. The PMIx library is not required to interpret these directives itself — it passes them to the host environment (typically the scheduler) for processing. Actual support for any given directive depends on the host environment.

Identifying the request:

  • PMIX_SESSION_CTRL_ID (char*) — a string identifier for this request, allowing its status to later be queried or the operation to be cancelled.

  • PMIX_REQUESTOR (pmix_proc_t*) — identifier of the process on whose behalf the request is being made, used when one process relays the operation to the PMIx library on behalf of another (the API itself takes no requestor parameter).

Instantiating a session (typically issued by a scheduler):

  • PMIX_SESSION_INSTANTIATE (bool) — instantiate the specified session.

  • PMIX_ALLOC_UID (uint32_t) — the user id the session belongs to.

  • PMIX_ALLOC_GID (uint32_t) — the group id the session belongs to. A scheduler instantiates a session on behalf of a user, and these say which one. PMIX_USERID and PMIX_GRPID cannot: the library sets them to the calling process’s own user and group, which are the scheduler’s.

  • PMIX_SESSION_RESOURCES (pmix_data_array_t*) — array of pmix_info_t(5) structures describing the resources to be included in the session (e.g., a regular expression of node names, an array of resource blocks, etc.).

  • PMIX_SESSION_APP (pmix_data_array_t*) — array of pmix_app_t structures to be executed in the assigned session upon instantiation.

  • PMIX_SESSION_JOB (pmix_data_array_t*) — array of pmix_info_t(5) structures describing the job information to be applied in conjunction with the specified PMIX_SESSION_APP definitions. It is an error to provide this without a PMIX_SESSION_APP description.

  • PMIX_SESSION_PROVISION (pmix_data_array_t*) — description of the nodes to be provisioned with a specified image.

  • PMIX_SESSION_PROVISION_NODES (char*) — regular expression identifying nodes that are to be provisioned.

  • PMIX_SESSION_PROVISION_IMAGE (char*) — name of the image that is to be provisioned.

Controlling a session’s operation:

  • PMIX_SESSION_PAUSE (bool) — pause all jobs in the specified session.

  • PMIX_SESSION_RESUME (bool) — resume (“un-pause”) all jobs in the specified session.

  • PMIX_SESSION_PREEMPT (bool) — preempt the indicated jobs (named via a PMIX_NSPACE attribute in the accompanying array) in the specified session and recover all their resources. If no PMIX_NSPACE is given, all jobs in the session are preempted.

  • PMIX_SESSION_RESTORE (bool) — restore the indicated jobs (named via a PMIX_NSPACE attribute in the accompanying array) in the specified session, including all their resources. If no PMIX_NSPACE is given, all jobs in the session are restored.

  • PMIX_SESSION_SIGNAL (int) — send the given signal to all processes of every job in the session.

  • PMIX_SESSION_EXTEND (bool) — extend the specified session, either in time or in resources, based on additional provided attributes.

  • PMIX_SESSION_SEP (bool) — separate the specified session from its parent — i.e., allow the two to terminate independently.

  • PMIX_SESSION_TERMINATE (bool) — terminate all jobs in the specified session and recover all resources included in the session.

  • PMIX_SESSION_COMPLETE (bool) — indicate that the specified session has completed and all of its resources have been recovered and are available for scheduling. The accompanying array must include pmix_info_t(5) structures reporting the returned status of any jobs executed in the session.

25.2.21.5. RETURN VALUE

For the blocking form (cbfunc == NULL), a successful request returns PMIX_OPERATION_SUCCEEDED. For the non-blocking form, a return of PMIX_SUCCESS indicates only that the request was accepted for processing; the final status and any data are delivered to cbfunc.

  • PMIX_SUCCESS — (non-blocking form) the request was accepted for processing.

  • PMIX_OPERATION_SUCCEEDED — (blocking form) the request was processed successfully.

  • PMIX_ERR_NOT_SUPPORTED — the operation is not supported in the caller’s role or environment — for example, the caller is a system controller that is not connected to the scheduler.

  • PMIX_ERR_UNREACH — the caller is not connected to a server capable of servicing the request.

  • PMIX_ERR_COMM_FAILURE — the connection to the server was lost before a result could be returned.

  • PMIX_ERR_NOT_AVAILABLE — the operation cannot be serviced because the library’s progress engine has been stopped.

  • PMIX_ERR_INIT — the PMIx library has not been initialized.

Any other negative value indicates an appropriate error condition. PMIx error constants are defined in pmix_common.h.

25.2.21.6. NOTES

Unlike most PMIx operations, PMIx_Session_control does not provide a separate _nb entry point; the single API selects blocking versus non-blocking behavior based on whether cbfunc is NULL.

PMIx_Session_control is a PMIx library extension and is not part of the published PMIx Standard.

25.2.21.7. PROGRESS THREAD RESTRICTION

A blocking PMIx call must not be made from within the PMIx progress thread. Any code the library itself invokes runs on that thread: an event handler registered through PMIx_Register_event_handler(3), a callback passed to a non-blocking PMIx API, and — in a server or tool — the completion of a host-module up-call. A blocking call waits for work that the progress thread has to perform, so making one from that thread waits for itself and never returns. The PMIx Standard disallows it, and there is no way for an implementation to service such a request.

Where this call has a blocking form — including the blocking behavior a non-blocking entry point adopts when it is passed a NULL cbfunc — that form detects the situation and returns PMIX_ERR_WOULD_BLOCK immediately, accompanied by a diagnostic naming the call. Nothing is done and no callback is invoked.

PMIX_ERR_WOULD_BLOCK here is not a transient condition to retry: it reports a call that cannot be serviced from where it was made. Reissue it as the non-blocking form with a callback, or from a thread of your own.

25.2.21.8. SESSION CONTROL AND ALLOCATION REQUESTS

PMIx_Session_control issues directives to a session. Most of them — pause, resume, signal, terminate — act on a session that already exists, and may come from the user the session belongs to as well as from the scheduler.

PMIX_SESSION_INSTANTIATE and PMIX_SESSION_EXTEND are different. They do not ask for resources: a user asks for resources with PMIx_Allocation_request(3), and the scheduler decides. These two directives are how the scheduler then tells the runtime managing the session — a distributed virtual machine (DVM), for example — what it decided: start a session over these resources for this user, or give this session more time or more resources. A runtime would typically accept them only from the scheduler (or, where there is none, from whoever acts with the scheduler’s authority) — not from the user the session is for, who asks with an allocation request instead.

The scheduler’s answer to an allocation request goes to the requester, and may carry much the same information as the directive, but the directive is still needed:

  • The runtime may never see the answer. The host environment that passed the request on is not required to watch for the scheduler’s reply, and the scheduler cannot know whether it will intercept the answer and act on it. A runtime that is told directly needs no such arrangement.

  • The runtime may not know a request was made. A request can come from a tool connected to the scheduler, or to some other server, rather than from one of the runtime’s own processes. The runtime then has no record of the request at all, and only a directive from the scheduler tells it what to do.

  • Not every change starts with a request. A scheduler sometimes needs a runtime to change a session for its own reasons — for example, to modify a job’s resources to keep the whole system within an overall energy limit. There is no allocation request to answer in that case; there is only the directive.

25.2.21.9. EXAMPLES

A workflow tool, connected to the scheduler rather than to any runtime, asks for four nodes for two hours on behalf of the user running it:

pmix_info_t info[2];
pmix_info_t *results = NULL;
size_t nresults = 0;
uint64_t nnodes = 4;
pmix_status_t rc;

PMIx_Info_load(&info[0], PMIX_ALLOC_NUM_NODES, &nnodes, PMIX_UINT64);
PMIx_Info_load(&info[1], PMIX_ALLOC_TIME, "2:00:00", PMIX_STRING);
rc = PMIx_Allocation_request(PMIX_ALLOC_NEW, info, 2, &results, &nresults);
/* on success, results include the PMIX_ALLOC_ID of the new allocation */
PMIx_Info_destruct(&info[0]);
PMIx_Info_destruct(&info[1]);
if (NULL != results) {
    PMIx_Info_free(results, nresults);
}

The runtime that will host the work saw none of this. The scheduler chooses the nodes, answers the tool with the allocation’s identifier, and directs the runtime to instantiate a session over them for the user who asked — identified by the PMIX_USERID and PMIX_GRPID the library attached to the tool’s request (here, 1001):

pmix_info_t dirs[4], *res;
pmix_data_array_t resources;
uint32_t uid = 1001, gid = 1001;
bool flag = true;
pmix_status_t rc;
size_t n;

PMIx_Data_array_construct(&resources, 3, PMIX_INFO);
res = (pmix_info_t *) resources.array;
PMIx_Info_load(&res[0], PMIX_ALLOC_NODE_LIST,
               "node01,node02,node03,node04", PMIX_STRING);
PMIx_Info_load(&res[1], PMIX_ALLOC_ID, "alloc-57", PMIX_STRING);
PMIx_Info_load(&res[2], PMIX_ALLOC_TIME, "2:00:00", PMIX_STRING);

PMIx_Info_load(&dirs[0], PMIX_SESSION_INSTANTIATE, &flag, PMIX_BOOL);
PMIx_Info_load(&dirs[1], PMIX_ALLOC_UID, &uid, PMIX_UINT32);
PMIx_Info_load(&dirs[2], PMIX_ALLOC_GID, &gid, PMIX_UINT32);
/* the info takes its own copy of the array */
PMIx_Info_load(&dirs[3], PMIX_SESSION_RESOURCES, &resources, PMIX_DATA_ARRAY);
rc = PMIx_Session_control(12, dirs, 4, NULL, NULL);
/* PMIX_OPERATION_SUCCEEDED once the runtime has created session 12 */
for (n = 0; n < 4; n++) {
    PMIx_Info_destruct(&dirs[n]);
}
PMIx_Data_array_destruct(&resources);

The tool can now refer to the allocation by its PMIX_ALLOC_ID — for example, to spawn a job into it. See PMIx_Allocation_request(3) for an extension requested by a process already running in a session.