10.3.5. Access control by user and group

Note

This is a plan. It describes the access model PMIx is moving to, and the order in which the pieces are being implemented. The Status section at the end says which parts are in place today. Anything not yet implemented behaves as it did before this work began.

This page is for two audiences:

  • Users and administrators, who need to know who can see a job’s data and act on a job, and how to widen that.

  • Developers of PMIx and of host environments, who need to know where each check is made and what a host has to do.

10.3.5.1. Summary

  • Access to a job - reading its data, pulling its output, monitoring or signalling its processes - is decided by who is asking (their user ID and groups), not by which job the requester belongs to.

  • By default, only the job’s owner can access it. The owner can widen that when the job is started, by naming other users and/or groups with PMIX_ACCESS_PERMISSIONS.

  • Crossing jobs is normal and supported. A debugger or tool that asks about another job succeeds whenever its user is allowed.

  • These rules govern requests that clients and tools make of a PMIx server. They do not restrict the host environment itself - see below.

10.3.5.2. Who is governed: the host versus clients and tools

A PMIx server lives inside a host environment: a resource manager, launcher, or other system daemon that calls PMIx_server_init and supplies the pmix_server_module_t up-calls. Many hosts run as a privileged service account that is not root.

  • The host is not restricted. The library does not limit what the host can do through the server APIs (registering jobs, delivering output, answering data requests, and so on), and does not second-guess what the host does with a request passed up to it. The host is the authority for its own operations.

  • Clients and tools are governed. A request a client or tool sends to a server is checked against the target job’s permissions:

    • by the server library, for data and operations it handles itself;

    • by the host, for operations the library passes up to it. For these the library tells the host exactly who is asking - the requester’s process ID, PMIX_USERID and PMIX_GRPID (see pmix_server_module_t(5)) - and the host applies the same rule.

The requester’s identity is the one established when it connected to the server (the connection handshake verifies it against the host’s registration). It is not something the request itself can claim.

A request can pass through more than one server: a host relays what one of its clients or tools asked for to another server - a workflow manager to the resource manager, a launcher to a scheduler. The next server is told who is connected to it - the relaying host - and, beside that, what the relay claims about who first asked (PMIX_ORIGIN_USERID and the other PMIX_ORIGIN_* attributes). Whether to believe that claim is the receiving host’s decision, made from the relay’s own verified identity; the library never substitutes the claim for it. See “Requests that pass through more than one server” in Deciding What a Requester May Do.

10.3.5.3. Terms

Owner

The user (and primary group) a job belongs to - normally the user who started it.

Requester

The client or tool making a request, identified by the user ID and group ID recorded when it connected and, on the server’s local socket, by the groups the kernel reported for its process then.

Access list

Additional users and groups the owner has allowed, given with PMIX_ACCESS_PERMISSIONS.

10.3.5.4. The rule

A requester may access a job if any of the following holds:

  1. the requester’s user ID is 0 (root);

  2. the requester’s user ID is the server’s own effective user ID - the account the host runs as, which can already inspect the server process itself;

  3. the requester’s user ID is the job owner’s;

  4. the requester’s user ID is in the job’s access list;

  5. the requester belongs to a group in the job’s access list - one of the groups described under Group membership below.

Otherwise the request fails with PMIX_ERR_NO_PERMISSIONS.

10.3.5.5. Establishing a job’s permissions

The owner. The host gives the job’s owner when it registers the job, as PMIX_USERID and PMIX_GRPID in the info passed to PMIx_server_register_nspace. Either may be a number or a user or group name; names are resolved when the job is registered. If the host does not give them:

  • the owner is the user and group the job’s clients were registered with (PMIx_server_register_client);

  • failing that, the owner is the server’s own user.

A tool’s owner is the user and group it presented when it connected.

The access list. A user widens access when starting a job, by passing PMIX_ACCESS_PERMISSIONS in the job-level directives of PMIx_Spawn. Its value is a data array of pmix_info_t holding either or both of:

  • PMIX_ACCESS_USERIDS - a data array of user IDs (numbers or names);

  • PMIX_ACCESS_GRPIDS - a data array of group IDs (numbers or names).

The host passes the list on when it registers the job, on every node that hosts a part of it. The same attributes already govern access to published data (PMIx_Publish), so users meet one vocabulary.

The list widens access and names nobody’s ownership. Only the two attributes above are read inside it: a PMIX_USERID or PMIX_GRPID there is not the job’s owner, and an access list nested inside another is not read. PMIx_Spawn is refused with PMIX_ERR_BAD_PARAM when its access list - or a PMIX_JOB_INFO_ARRAY among its job-level directives - carries an identity attribute (PMIX_USERID, PMIX_GRPID, PMIX_REQUESTOR_PID, PMIX_REQUESTED_GRPID or one of the PMIX_ORIGIN_* attributes), or when one access list is nested inside another. The owner is the host’s to name, at the top level of its registration.

Group membership is the set of groups the requester’s process belongs to - its effective group and its supplementary groups. The set belongs to the process, not to its user: two processes of one user may hold different groups. The library never looks a user’s groups up: no directory lookup runs on its behalf, now or later. The set is taken from the first of these that gives one:

  1. The kernel, for a client or tool connected to the server’s local socket: the groups of the connecting process, which the server reads from the connection when it is made - on Linux with SO_PEERGROUPS (kernel 4.13 or later), and on macOS and FreeBSD with LOCAL_PEERCRED, which reports at most 16 groups. On Linux, a group not mapped into the server’s user namespace - which the kernel reports as the overflow gid - is left out. The effective gid the kernel reports for the process is always one of them. The set is the one the process had when it connected, and is kept for as long as the connection lasts.

  2. The host, for the requester’s user: the groups it gives with PMIX_USER_GRPIDS (a data array of group IDs, numbers or names):

    • for a job’s owner, in that job’s PMIx_server_register_nspace;

    • for any other user - a tool’s user, say - in PMIx_server_register_resources with PMIX_USERID. A single PMIX_GRPID there says the user belongs to that group alone.

    These are used for a process on the local socket only where the kernel reports no list - on a platform without either option, or when reading it fails - and are what a requester connected over TCP, or a request the host relays from another node, is judged by.

  3. The one group the request is made under, where the host gave no groups for the user: for a client or tool on the local socket, the effective gid the kernel reports for it; for one connected over TCP, the gid it connected with; for a request the host relays from another node, the PMIX_GRPID carried with it. A group check then passes only if the job’s access list names exactly that group.

Providing the host’s groups is the host’s responsibility. The library uses exactly the groups it was given until the host gives them again; a host whose users’ groups change registers them again (with the user’s next job, for example). A user the host gave no groups for has no record. Root and the server’s own user are never recorded - they need no groups. A record is dropped when the host deregisters that user (PMIx_server_deregister_resources with PMIX_USERID), and at finalize.

A host that keeps its own records and wants the account’s groups may call pmix_server_user_refresh() from its own thread; the library itself never does.

10.3.5.6. What is covered

Area

Examples

Checked by

Job-level data

PMIx_Get of another job’s data (rank PMIX_RANK_WILDCARD)

server library

Process data on this node

PMIx_Get of a process’s committed (modex) data

server library

Process data on another node

PMIx_Get answered by direct modex

the server that holds the data (see below)

Query and resolve

process tables, namespace information, PMIx_Resolve_peers

server library for what it answers; host for the rest

Output forwarding

PMIx_IOF_pull, and a pull following a job into the jobs it spawns

host approves each pull; library for inheritance

Output files

PMIX_IOF_OUTPUT_TO_FILE / _TO_DIRECTORY

file ownership and mode (see below)

Monitoring

PMIx_Process_monitor targets, process statistics

server library for processes on its node; host for others

Job control

signal, kill, terminate, abort, stdin

host; server library for cleanup directives it handles

Sessions and allocations

extending, releasing, spawning into an allocation

host (by user, not by namespace)

10.3.5.7. What the server library checks

Beyond the rule itself, a few things hold for every check the server library makes:

  • A job’s own processes always access their own job.

  • The server’s own namespace describes the server, not anyone’s job, and anyone may read it.

  • A job the host has not registered with this server - one known here only because data for it arrived from elsewhere - has no permissions here to judge by. Requests for its data are checked by the server that holds it (see Data held on another node). Acting on such a job - cleanup, monitoring, following its output - is allowed only for root and the server’s own user.

Where the checks are made:

Request

Check

PMIx_Get, and a refresh of the client’s cached copy

The requester must be allowed the target job. A request waiting for a job to be registered is checked again once it is.

PMIx_Query_info naming a namespace (PMIX_NSPACE)

The requester must be allowed that job.

PMIx_Resolve_peers, PMIx_Resolve_nodes

Naming a job, the requester must be allowed it. Naming none, the answer includes only the jobs the requester may access.

PMIx_Process_monitor of processes on this node

Processes named by PMIX_MONITOR_TARGET_PROCS must all belong to jobs the requester may access, or the request is refused; each is read at the process ID its host recorded (PMIX_PROC_PID). Processes named by PMIX_MONITOR_TARGET_PIDS must all be owned by the requester, and -1 means every process it owns. A request for every local process leaves out those the requester may not access. Requests from the host are not restricted, except one it relays from another node (PMIX_MONITOR_PROXY), which must name the requester’s PMIX_USERID and is checked for that user, and is refused without it. Each process must still belong to the expected user when it is read.

Cleanup directives (PMIX_REGISTER_CLEANUP and its family) on PMIx_Job_control

The requester must be allowed every job it names, and the cleanup runs as the requester - each entry records who asked - never as the job’s user or the server. A job the server does not know is refused.

Output forwarding inherited by a spawned job

A subscription to a job’s output follows into the jobs it spawns only if its requester may access each of them.

10.3.5.8. Data held on another node

When a requester asks for data the local server does not have, the server asks its host, which fetches it from the server that holds it (direct modex). The check is made by the server that holds the data, because that server has the job’s permissions:

  • The requesting server names the requester in its direct_modex up-call - PMIX_REQUESTOR, PMIX_USERID and PMIX_GRPID, the user and group being those established when the requester connected. Requesters of different identities are sent up separately; requesters of one identity share a fetch.

  • The host passes that identity to the holding server through PMIx_server_dmodex_request2, a new form of PMIx_server_dmodex_request that takes an info array (every PMIx API carries one).

  • The holding server applies the rule and answers, or refuses with PMIX_ERR_NO_PERMISSIONS.

  • The requesting server keeps a copy of what it receives, and remembers which identities the holding server approved. From that copy it answers only those, the job’s own processes, root, and its own user; any other requester is asked about as though the copy were not there, and gets its own fetch. This applies to PMIx_Get, to a refresh of a client’s cache, and to query and resolve answered from the copy.

A host that still uses PMIx_server_dmodex_request passes no requester identity, and the holding server answers as it does today. The capability flag PMIX_CAP_DMODEX_REQUEST2 tells a host that the library provides PMIx_server_dmodex_request2.

10.3.5.9. Output files and shared memory

Output files (PMIX_IOF_OUTPUT_TO_FILE, PMIX_IOF_OUTPUT_TO_DIRECTORY). The paths a job names are the job owner’s, and everything about its output files is done as the owner: the directories are created, and the files opened, checked, emptied and given their mode, by the owner - in place when the server runs as the owner, and otherwise in a short-lived child of the server that has dropped to the owner’s uid, gid and the groups the host registered for it (PMIX_USER_GRPIDS; with none, the owner’s gid alone). A system takes only so many groups for a process (sysconf(_SC_NGROUPS_MAX): 16 on macOS); the child has the owner’s gid and as many of the others, in the order the host gave them, as fit. On Linux the child also gives up any capability it still holds once it is the owner - a server given CAP_SETUID as a file capability keeps the rest of its capabilities across the change of user - and does nothing if one remains. So the kernel applies the owner’s own permissions to every path the job named: a privileged server creates nothing, and writes over nothing, that the owner could not.

  • A file and its directories belong to the owner from creation: 0600 and 0700.

  • If the access list names groups, the files are made readable by the first listed group (0640, directories 0750), where the owner can set that group. File modes can express only an owner and one group, so further users and groups in the list are not applied to files. Access control lists are not used, as they are not portable.

  • An output file that is not a regular file with a single name, owned by the job’s owner, is not written - an existing file of someone else’s is refused and left as it was, even where the owner could write it.

  • The launcher writes them, when it can. A tool that spawns a job and is sent the job’s output - prun, say - names itself the job’s file writer (PMIX_IOF_FILE_WRITER). The server that takes the spawn accepts that only from the tool it names, and passes it on to jobs that the job’s processes spawn with output files of their own. No server then writes that job’s files: the one sending the job’s output to the tool sends the file directives with the first of it, and the tool writes the files as itself. So a server running as a service account other than root can run another user’s job with output files. The files are written where the tool runs, not where each process runs.

  • Otherwise a server that can neither run as the owner nor become it - one running as a service account other than root - writes no output files for that owner’s jobs, and says so once per job, when the job is registered - before any of its processes run.

  • The owner’s child is given a deadline. It works on paths the owner chose, which may be on a filesystem that does not respond. A server waits for it no longer than the runas_timeout MCA parameter (10 seconds by default), then kills it and does not write that file. None of the job’s other output files is tried after that, since each would cost the same wait. The files themselves are then written by a thread of their own rather than by the one that serves every job on the node, so a file that stops answering later holds only that thread. What is held for such a file meanwhile is limited by the iof_writer_backlog MCA parameter, and output beyond it is not written to the file. The same deadline bounds the cleanup a job registers (PMIX_REGISTER_CLEANUP, PMIX_REGISTER_CLEANUP_DIR), which is done as the requester in the same way. Setting the parameter to 0 waits indefinitely.

Shared-memory job data (gds/shmem3). Only the server writes a segment: clients open its backing file read-only and map it read-only. The file belongs to the job’s owner, with mode 0400 - 0440 for the first group the access list names - and never grants write access. A client the mode does not admit - of another user’s job, or of a job whose file the server could not give to its owner - receives the job’s data from the server instead, where the rule is applied.

10.3.5.10. Not covered

  • Event notification. The process raising an event chooses who receives it through the event’s range. The one exception is PMIX_DEBUGGER_RELEASE, which lets a process held for a debugger continue: a server accepts it from a client or tool only if the rule above permits that requester for the job of every process the event can reach. A job the server has not had registered is left to the host, which relays the event.

  • Published data (PMIx_Publish / PMIx_Lookup). The host’s data server already governs it per published item with PMIX_ACCESS_PERMISSIONS.

10.3.5.11. Compatibility

  • A host that registers no owner gets the fallback above: the user its clients were registered with, then the server’s own user. On a node that hosts none of a job’s processes there are no clients, so there only root and the server’s own user can access that job. A host running as the same user as its jobs sees no change; a host running as a service account should register each job’s owner.

  • A host that does not pass ``PMIX_ACCESS_PERMISSIONS`` at registration gets owner-only access for every job, which is the default.

  • Clients and tools of any version are identified by their connection, so nothing changes for them.

  • Hosts can detect the library’s support through capability flags in pmix_version.h: PMIX_CAP_REQUESTER_ID (the requester’s identity on up-calls), PMIX_CAP_ORIGIN_ID (an up-call’s identity is always the connected peer’s, with PMIX_REQUESTOR_PID, PMIX_REQUESTED_GRPID and a relay’s PMIX_ORIGIN_* claims beside it), PMIX_CAP_DMODEX_REQUEST2 (PMIx_server_dmodex_request2) and PMIX_CAP_ACCESS_CHECK (pmix_server_access_check()).

10.3.5.12. What a host needs to do

  • Register each job’s owner (PMIX_USERID, PMIX_GRPID) and access list (PMIX_ACCESS_PERMISSIONS) with PMIx_server_register_nspace on every node that hosts part of the job.

  • Accept PMIX_ACCESS_PERMISSIONS in spawn requests and carry it with the job.

  • Apply the rule to the operations it performs for clients and tools - job control, abort, stdin, IOF pull approval, queries it answers, monitoring of other nodes, sessions and allocations - using the requester identity the library passes in each up-call. A host linked with the library applies exactly the library’s rule by keeping its own copies and calling pmix_server_access_check() (declared in the installed src/server/pmix_server_ops.h) from its own thread: a pmix_access_t for each job, loaded with pmix_server_access_load() from the info it registers the job with, and a pmix_user_t for each requester. None of these touch the library’s state. When the host drops a user it deregisters it from the library too, so the two stay in step. The PMIX_GRPID in an up-call is the group the requester chose to charge the work to and is not verified: it is not an input to the rule.

  • Carry the requester’s identity when it relays a request to another of its daemons.

  • Answer direct-modex requests with PMIx_server_dmodex_request2, passing the requester’s identity.

PRRTE is the reference host, and implements these alongside the library changes.

10.3.5.13. Status

Phase

Work

State

0

The requester’s process ID, PMIX_USERID and PMIX_GRPID are passed on every up-call made for a client or tool; PMIX_REQUESTOR where an up-call has no process argument; user and group names accepted; IOF pulls forwarded only after the host approves. PRRTE approves an IOF pull only for root, the DVM’s user, or the job’s owner.

Done

1

Each job’s owner and access list stored by the server library, from the host’s registration; one check implementing the rule; group membership cached per user.

Done

2

The server library applies the rule to what it answers itself: job and process data, query and resolve, monitoring of local processes, cleanup directives, and IOF inheritance.

Done

3

PMIx_server_dmodex_request2; the requester’s identity on the direct_modex up-call; the check at the holding server; the requesting server answers from its copy only the requesters the holder approved. PRRTE registers each job’s owner (PMIX_USERID, PMIX_GRPID) with PMIx_server_register_nspace, carries the requester’s identity on its direct-modex relay, and uses PMIx_server_dmodex_request2.

Done

4

Output file ownership and modes; server-mediated job data for requesters the shared-memory file mode does not admit, with shared-memory segments read-only to their readers.

Done

5

pmix_server_access_check(), the rule as a function a host calls with its own copies of a job’s access list and the requester’s groups; users registered and deregistered by the host, their groups as the host gives them (PMIX_USER_GRPIDS); a monitor request relayed from another node checked for the requester it names. PRRTE carries each job’s access list (--rtos users=,groups=, and PMIX_ACCESS_PERMISSIONS at spawn) and applies the rule to the operations it performs.

Done