3.5.5. Schedulers and PMIx v7

For developers of system schedulers (workload managers) that use PMIx: a scheduler that embeds the PMIx server library and declares itself the system scheduler (PMIX_SERVER_SCHEDULER), so that runtime environments - a DVM such as PRRTE’s prte, say - attach to it and pass on their users’ requests.

A scheduler is a host environment, so everything on Host Environments and PMIx v7 applies to it. This page collects what matters most to a scheduler, and adds the parts only a scheduler meets: being found by the runtimes that attach to it, accepting them, and judging the requests they relay. How the scheduler integration works is in Scheduler Integration.

3.5.5.1. How a scheduler fits

A runtime environment attaches to the scheduler as a tool, with PMIX_CONNECT_TO_SCHEDULER. It then uses that connection for two things:

  • its own business with the scheduler - for example, telling it that a session has ended;

  • its users’ requests, which it relays: an allocation request (PMIx_Allocation_request, the allocate up-call), a session control request (PMIx_Session_control, session_control), a resource block (PMIx_Resource_block, resource_block), a query, or a fabric request.

The second is where v7 changes the most: what a request tells the scheduler about who made it. See Who is asking.

3.5.5.2. Runtimes connect only on the scheduler’s node

What changed. PMIx messaging is node-local. A tool - and an attaching runtime is a tool to the scheduler - connects only to a PMIx server on its own node. Connections from other nodes are gone, and a server that asks for them (PMIX_SERVER_REMOTE_CONNECTIONS) fails to start.

What it means. A runtime attaches to a scheduler server on the node it runs on. It looks for pmix.sched.<hostname>, named for its own node. A scheduler that serves runtimes on other nodes runs a PMIx server on each of those nodes - an agent - and carries requests between its agents and its core over its own transport, as resource managers always have for their clients.

3.5.5.3. Being found

Ask for tool support (PMIX_SERVER_TOOL_SUPPORT) at PMIx_server_init. Without it, no runtime can attach.

Where the scheduler’s rendezvous file goes. The scheduler writes pmix.sched.<hostname>:

  • in the system temporary directory, if the host names one (PMIX_SYSTEM_TMPDIR);

  • else in /run/pmix, when the scheduler runs as root;

  • else in its own user’s directory, <tmpdir>/pmix-user.<uid>/.

Where a runtime looks. A runtime looking for the scheduler tries, in order, the system temporary directory it was given, /run/pmix, /var/run/pmix, root’s and its own user’s directories, and the directories of the users its ptl_base_server_access parameter names. With legacy servers on (ptl_base_legacy_servers), it also looks in the system temporary directory, where a scheduler of an earlier release writes its file.

What a runtime believes. It uses a file only if it was written by root, by its own user, or by a user ptl_base_server_access names - and only if the kernel confirms that the server behind the socket runs as the user who wrote the file.

So:

  • a scheduler running as root, or as each runtime’s own user, is found with no configuration;

  • a scheduler running as a service account - sched, say - is found only by runtimes whose PMIx parameters name that account: ptl_base_server_access = users=sched in the parameter file of each account a runtime runs as.

Handing out the address yourself. A TCP address for a v7 server is refused (PMIX_ERR_LOCAL_URI_REQUIRED). PMIX_TCP_REPORT_URI set to - or + now reports the local socket’s address, which is the one to give a runtime; tcp- and tcp+ report the TCP address, for runtimes of earlier releases.

Use a short temporary directory. The local socket lives under it, and a socket path is limited to about 100 characters.

3.5.5.4. Accepting runtimes

The local socket is owner-only. The server takes, without being told to, tools of root, of the scheduler’s own user, and of the user the host names as PMIX_USERID at PMIx_server_init. Runtimes usually run as other accounts - a user’s own DVM, or a privileged “backbone” DVM. To take them:

  • pass PMIX_SERVER_ALLOW_FOREIGN_TOOLS at PMIx_server_init;

  • and widen the socket to them: PMIX_SOCKET_MODE, with PMIX_USERID and PMIX_GRPID naming the socket’s owner and group - for example, a group the runtimes’ accounts belong to, with mode 0660. Connecting to a socket takes write permission on it.

Decide in tool_connected2. It receives the runtime’s verified user and group, the authentication method, the kernel’s pid for the process, and PMIX_TIMEOUT. Check:

  • that this user may use the scheduler at all;

  • that what the runtime asks to be is something that user may be. The roles a tool declares at connection (PMIX_LAUNCHER, for instance) are its own claim.

Answer once, within the timeout. A runtime may arrive with no namespace: assign one, and keep the names you assign unique. See Accepting Connections.

Handle PMIX_ERR_LOST_CONNECTION. It is raised for a runtime that leaves without finalizing, so the scheduler can act on the sessions and requests it held.

Runtimes of earlier releases. A runtime linked to a PMIx release before v7 connects over TCP, on loopback only, and only to a server set up to take legacy clients: PMIx configured with --enable-legacy-clients, and either the host’s PMIX_SERVER_LEGACY_PEERS or the site’s ptl_base_legacy_clients. Without both, it is refused.

3.5.5.5. Who is asking

What changed.

  • PMIX_USERID and PMIX_GRPID in an up-call always describe the process connected to the scheduler - the runtime - as the kernel reported them when it connected. PMIX_REQUESTOR_PID comes with them, on the local socket. Nothing the runtime sends replaces them. PMIX_REQUESTOR_IS_TOOL and PMIX_REQUESTOR_IS_CLIENT likewise describe the runtime.

  • The process that first made the request, as the runtime states it, arrives beside them: PMIX_ORIGIN_USERID, PMIX_ORIGIN_GRPID, PMIX_ORIGIN_PID, PMIX_ORIGIN_IS_TOOL and PMIX_ORIGIN_IS_CLIENT, with PMIX_REQUESTOR for its process ID. PMIx passes these through untouched, and refuses a request that carries any of them more than once.

  • A group the requester asks the work to be charged to arrives as PMIX_REQUESTED_GRPID. A requester’s own PMIX_GRPID is turned into one; a group name is resolved to a number; two are refused. PMIx does not check whether the requester belongs to the group.

  • PMIX_INFO_RELAYED and PMIx_Info_relayed() are gone.

  • PMIX_CAP_ORIGIN_ID in pmix_version.h says the library behaves this way.

What it means. A scheduler that has not been changed judges every relayed request as the runtime’s own. It never acts as a user a runtime merely names - but it may refuse, or charge to the runtime’s account, work its users asked for.

What to do.

  1. Authorize on PMIX_USERID and PMIX_GRPID. They are the runtime you are talking to.

  2. Decide which runtimes may speak for others, from those verified ids, and from the pid if you want to be exact. Which accounts qualify is the scheduler’s policy - in a three-level setup of user DVM, backbone DVM and scheduler, typically the backbone’s account. PRRTE’s rule, as an example: it believes a claim when the claimed user is the relay’s own, when the relay runs as root or as the DVM’s user, or when the relay’s uid is listed in its prte_pmix_relay_uids parameter.

  3. For a runtime you trust, attribute the work to the PMIX_ORIGIN_* identity. For any other, treat the request as the runtime’s own.

  4. Check PMIX_REQUESTED_GRPID against the user you attributed the work to.

  5. If the scheduler passes a request on to another server, copy the identity it attributed the work to into PMIX_ORIGIN_* - at the first hop, the up-call’s own PMIX_USERID, PMIX_GRPID and PMIX_REQUESTOR_PID - and carry a requested group on as PMIX_REQUESTED_GRPID.

See “Requests that pass through more than one server” in Deciding What a Requester May Do.

3.5.5.6. Users’ groups

PMIx never looks groups up. A process’s groups come from the kernel for a connection on the local socket; for anything else, PMIx uses the groups the host gave, else the single gid the request is made under.

So a scheduler that checks PMIX_REQUESTED_GRPID needs the user’s groups from its own source. Look them up on the scheduler’s own thread, not in an up-call it must answer quickly.

A scheduler that applies PMIx’s access rule to its own objects (pmix_server_access_check()) gives the groups of users who own no job registered with it through PMIx_server_register_resources, with PMIX_USERID and PMIX_USER_GRPIDS. See Access control by user and group.

3.5.5.7. The up-calls

  • Every up-call made for a runtime carries PMIX_USERID, PMIX_GRPID and PMIX_REQUESTOR_PID, and the PMIX_ORIGIN_* claim when the runtime sends one.

  • allocate, session_control and resource_block: PMIx passes the request up with that identity. The scheduler decides whether the requester may obtain, change or release the resources or session concerned. A NULL entry answers PMIX_ERR_NOT_SUPPORTED.

  • Returning PMIX_OPERATION_SUCCEEDED from spawn, lookup, direct_modex, get_credential or validate_credential is treated as an error: each has a result that only the callback can deliver.

  • The library no longer relays on its own. A spawn or query it cannot serve goes to your up-call, and is refused if there is none.

See Operation by Operation.

3.5.5.8. Building

  • Build against the PMIx major release the scheduler runs with. The library is given only a pointer to the host’s pmix_server_module_t, and cannot tell which release’s definition of it the host was built with.

  • Test the capability flags in pmix_version.h from configure - PMIX_CAP_ORIGIN_ID and PMIX_CAP_REQUESTER_ID in particular - if the scheduler must also build against earlier releases.