10.3.3. Accepting Connections
Every client and tool reaches a PMIx server over a connection. This page says what PMIx checks when a connection arrives, what it tells the host, and what the host is expected to check in turn. The host always makes the final decision to accept or reject a connection.
10.3.3.1. What PMIx checks
PMIx’s checks are a first pass. They reject a connection that is plainly invalid as early as possible, before the host spends any work on it.
For every connection:
The opening message must arrive within a time limit (
ptl_base_connect_ack_timeout) and stay within a size limit.The peer must say what it is: a client, a tool, or a launcher.
The peer’s PMIx version must be one the server supports.
The peer’s credential is checked by the security (
psec) component the peer named, which must be one the server has loaded. That check settles the peer’s user ID (uid) and group ID (gid). How it does so depends on the component - see Authentication methods.
For a client, in addition:
Its namespace and rank must already be registered with this server (PMIx_server_register_client(3)).
The uid and gid the credential establishes must be the ones the host registered for that client.
A client of this release must present its rank’s connection token, the one PMIx_server_setup_fork(3) put in its environment. A connection with no token, or another one, is refused with
PMIX_ERR_INVALID_CRED, and the host is not asked about it. The token is bound to the first client process that connects with it. The first client connection accepted with a rank’s token on the local socket binds the token to the pid the kernel reports for that connection, and from then on a client presenting the token is taken only from that pid: the process connects again after it finalizes or loses its connection, and so does a program it execs, but a client connection presenting the token from any other pid is refused withPMIX_ERR_INVALID_CRED, before the host is asked. The binding stays when the connection ends. It ends when the host replaces the rank: preparing the rank again withPMIx_server_setup_forkissues a new token, not yet bound to any pid, and the old token is refused from then on; deregistering the rank or its namespace drops its token; and registering the rank again with PMIx_server_register_client(3) keeps its token but ends the binding, so the next client process to connect with it binds it. Where the kernel reports no pid for a connection on the local socket, that connection is checked for its token alone and binds nothing. A tool that names a client’s rank presents the rank’s token too. Once the token is bound, a tool on the local socket is taken on it only from the pid it is bound to: a tool from any other pid is refused withPMIX_ERR_EXISTS, before the host is asked, as it is while the rank has a connection, and one that took the rank from its environment then connects with an identity the host gives it (see PMIx_tool_init(3)). A token not yet bound is checked alone, from any pid. A tool’s connection never binds the token, so the tools a job script starts, one after another, each take the script’s rank while no client process has connected with it.A rank takes one connection at a time. A second connection for a rank that already has one - live, or waiting on the host’s answer - is refused with
PMIX_ERR_EXISTS, and the host is not asked about it - a tool’s as a client’s. A client connection from a pid other than the one the rank’s token is bound to is refused for its token first, withPMIX_ERR_INVALID_CRED, whether or not the rank has a connection. A process the client started before it calledPMIx_Initinherits its identity, and makes such a connection if it callsPMIx_Inittoo. One it starts afterwards inherits none: the client library takes the variablesPMIx_server_setup_forkset out of the environment as it initializes (see PMIx_Init(3)). A rank whose earlier connection finalized, or was lost, can connect again.A client of this release that execs is lost: its connection is close-on-exec, so it closes in the exec, and the host is sent
PMIX_ERR_LOST_CONNECTIONat once, whether the process runs on or not (see When a connection ends). The program it execs is a new process to the server. A client of an earlier release leaves its connection open across an exec: the rank stays connected until the program it exec’d exits, and that program cannot connect for the rank meanwhile.
A PMIX_USERID or PMIX_GRPID in a client’s connection data is
ignored. The client’s identity is the one the credential establishes.
For a tool, in addition:
The uid the credential establishes replaces whatever uid the tool claimed. On the local socket, a gid the tool claims stands only if it is one of the tool’s process’s groups - those the kernel reports for the connection, or where it reports none, those the host registered for the tool’s user (
PMIX_USER_GRPIDS), or where it registered none, the effective gid the kernel reports; otherwise that effective gid replaces it. Over TCP, a gid the tool claims stands only if its user belongs to that group (see below); otherwise the tool is refused.A tool whose uid differs from the server’s is refused unless the host started the server with
PMIX_SERVER_ALLOW_FOREIGN_TOOLS- except a tool of root’s, and a tool of the user the host named to PMIx_server_init(3) asPMIX_USERID, if it named one. A host that runs the server as root for one job’s user, and names that user asPMIX_USERID, therefore takes that user’s tools without allowing tools of other users.A tool that says it is a registered client, or that joins a namespace the server already knows, must have that process’s uid and gid.
A tool that names its own namespace never joins a job of the host’s. One naming a namespace the host registered, whose rank the host did not register for it, is refused with
PMIX_ERR_EXISTS- and so is one whose namespace the host registers while it decides on the tool, even with no processes in it yet: the namespace is taken to be a job of the host’s that took the name meanwhile. So do not register a self-named tool’s namespace while deciding on it; the tool library sets the well-known job values (job size, local rank, hostname) for itself. A host that registers the name in full after accepting the tool takes the name back: the tool’s namespace and data go, its connections are closed, and the host’s job is built fresh. A tool the host names may join the namespace the host registers for it.A tool joining a rank that already has a connection is refused with
PMIX_ERR_EXISTS, as a client is. A tool that execs is lost, as a client is - the host is sentPMIX_ERR_LOST_CONNECTION- and if its new program initializes, it connects as a new tool, which the host is asked about, under the same name if it names one.A tool that took its namespace and rank from its environment - one started from a job script, which inherits the script’s - and is refused that way connects again as a tool that needs an identity. The host must then give it one: such a tool arrives at
tool_connected2with noPMIX_NSPACE. A tool of an earlier release does not connect again; itsPMIx_tool_initfails withPMIX_ERR_EXISTS.A tool of an earlier release with the server’s address in its environment but no namespace and rank also arrives at
tool_connected2with noPMIX_NSPACE, and must be given an identity in the same way. The tool is not told the name it was given.While tools wait for the host’s decision, at most
ptl_base_max_pending_connectionsof them may wait at once, and a tool of a uid other than the server’s, root’s and thePMIX_USERIDthe host gavePMIx_server_initmay have at most a quarter of that number waiting (at least one). A tool beyond either limit is refused withPMIX_ERR_OUT_OF_RESOURCE, and the host is not asked about it.The host decides on one tool of a name at a time. A tool that brings its own namespace and rank waits while the host decides on a tool of that name, so the host never decides on two tools for one name at once. Once it accepts a tool, any waiting tool of the same name is refused without the host being asked. Tools of other names, and tools that need an identity, are put to the host together: the host must keep the names it gives unique, and a tool it gives a name that a tool has taken for itself meanwhile is refused with
PMIX_ERR_EXISTSonce the host accepts it - the host is then sent the notice described below.
When a tool connects over TCP, for which the kernel reports no gid -
unless it is the server’s own user claiming the server’s own gid - the
server checks whether the tool’s user belongs to the gid it claims in the
system’s user and group database (getpwuid_r and getgrgid_r);
root belongs to every group without a lookup. A tool on the local socket
is never checked this way: its gid is decided from its process’s groups,
as above. The check runs on
the server’s progress thread, before the host is asked about the tool,
and waits for the database to answer - which, for a directory service,
may mean a network round trip. Each answer, whether the user belongs to
the group or not, is kept for 60 seconds for that (uid, gid) pair, for
up to 256 pairs; when all 256 are in use, the least recently used is
dropped. A lookup that fails, rather than answering, is not kept.
Repeated connections for one pair therefore consult the database at
most once a minute.
A connection that fails any of these checks is sent a status saying why, and closed. The host is not told about it.
10.3.3.2. What the host is told
A connection that passes reaches the host through one up-call. Until the host answers, the process waits.
A client arrives at client_connected2. The host receives:
the client’s process ID (namespace and rank), as its
procargument;PMIX_USERIDandPMIX_GRPID- the uid and gid PMIx established for the connection, which it has checked against the registration;PMIX_CRED_TYPE- the name of the security component that established them;PMIX_PROC_PID- the client’s process ID on the node. On the local socket it is the pid the kernel reports for the connection, in the host’s own process-ID namespace; where the kernel reports none, the host is given none. Over TCP - a client of an earlier release - it is the one the client’s PMIx library obtains withgetpid()and sends, which for a client in a container with its own process-ID namespace is its pid within that namespace, not the pid the host sees;PMIX_REALUIDandPMIX_REALGID, if the client’s library sent them.
A tool arrives at
tool_connected2. The host receives everything
the tool sent, with these replaced by values PMIx established:
PMIX_USERID and PMIX_GRPID, PMIX_CRED_TYPE, and
PMIX_VERSION_INFO; and PMIX_PROC_PID, as for a client. The
rest is passed on as the tool sent it: PMIX_REALUID and
PMIX_REALGID, which the tool’s PMIx library obtains from the
operating system; PMIX_CMD_LINE; and the roles the tool asked
for.
Both are also given PMIX_TIMEOUT: the number of seconds the host is
asked to answer within. It is what the process has left of the time
it waits for the reply - the server’s ptl_base_handshake_wait_time,
less any time it has already spent waiting for this reply, behind a
tool of its name, say - and is absent when that wait is unbounded. A
process with none left is refused with PMIX_ERR_TIMEOUT, and the
host is not asked.
The deprecated forms, client_connected and tool_connected, are
called only when the newer form is NULL. client_connected
receives no information about the client beyond its process ID.
10.3.3.3. What the host is expected to check
PMIx cannot know which processes the host meant to start, or which tools the site allows. The host does. Before accepting a connection, a host should check at least:
For a client:
that it started a process with this namespace and rank;
that
PMIX_PROC_PIDis the pid of the process it started with that identity. The host knows the pids of the processes it launched; PMIx does not.
For a tool:
that the tool’s user may use this server at all;
that what the tool asks to be - a launcher, a scheduler, a debugger attaching to a job - is something that user is allowed to be.
PRRTE, the reference host, makes all of these checks.
To reject a connection:
return an error from
client_connected2ortool_connected2; orreturn
PMIX_SUCCESSand later call the callback with an error.
tool_connected2 must return PMIX_SUCCESS to accept: the tool’s
assigned identity can only come back through the callback.
Answer within PMIX_TIMEOUT. A host that cannot decide in time
should answer no. The server waits a little longer, then refuses the
process with PMIX_ERR_TIMEOUT. The host still owes its answer -
the callback’s cbdata stays valid until it gives it - and until it
does, the process’s name stays held. A slow answer for a tool also
delays every tool behind it.
A yes does not always stand. The server may be unable to complete a connection the host accepted:
the yes came after the server stopped waiting;
the process went away while the host decided;
the host deregistered the client, its job, or the tool’s rank while it decided.
The host is then sent a PMIX_ERR_LOST_CONNECTION event naming the
process, as for any process that leaves without finalizing (see
When a connection ends), and should release what it recorded for it.
A host that provides neither connection up-call accepts every client PMIx accepts. Tools that need an identity assigned are then refused.
10.3.3.4. When a connection ends
A client or tool that finalizes reaches the host through client_finalized. For a tool,
server_objectisNULL.A connection that closes without a finalize - a process that died, a tool that disconnected, or a tool’s connection to a server other than the one it finalized with - raises
PMIX_ERR_LOST_CONNECTIONnaming the process. It goes only to the event handlers the host registered in its own process with PMIx_Register_event_handler(3), not to clients or tools.A process the host accepted that never became connected raises the same event (see What the host is expected to check). It is not raised when another connection has the process’s name by then, since it would name that connection.
A process of this release that execs raises the same event, at once: its connection closes in the exec.
A client is the host’s child, so the host sees it end anyway. A tool
the host did not start is not, and the lost-connection event is the
only notice that it left. A host that accepts such tools should
register a handler for PMIX_ERR_LOST_CONNECTION and release what
it holds for the tool when the event names it. PRRTE does.
A tool that disconnects is treated as gone: neither PMIx nor the host can tell when it later ends, so its state is released at once. If it connects again, it arrives as a new tool.
See Connection Lifecycle: Clients and Tools for everything the server does when a connection ends.
10.3.3.5. Authentication methods
The psec framework does the authentication. Two components exist:
Component |
How it establishes the peer’s uid and gid |
Availability |
|---|---|---|
|
Takes the uid and gid the kernel reports for a connection on the local socket. Over TCP, asks the server’s own kernel who owns the peer’s end of the connection. |
Always built. The default. |
|
Decodes a credential issued by the MUNGE daemon. |
Built only when configured |
A server loads every component that is available and permitted, and uses for each peer the one that peer names. A server with no usable component fails to start; it never accepts unauthenticated connections.
From v7.0, native takes the peer’s uid from the kernel rather than
from the peer - except over TCP on a platform whose kernel keeps no table
of TCP connections, where a server that takes legacy clients takes the uid
and gid the peer claims. While a server takes legacy clients, a connection
over TCP that declares a release earlier than v7.0 is not checked for a
connection token, so a process of a user can connect over TCP as any
rank registered for that user without its token. For how it works, and what a container needs
for it, see The native Mechanism. For a survey of
methods for authenticating a peer in a container, see
Authenticating a Containerized Peer over TCP.
What a system administrator can control is on Controlling Authentication.
10.3.3.6. Credentials requested by applications
PMIx_Get_credential(3) and
PMIx_Validate_credential(3) let
an application obtain and check credentials of its own - for example,
to authenticate to another service. For a client connected to a server,
these are the host’s job. PMIx passes the request to the host’s
get_credential or validate_credential up-call, with the
requester’s identity. If the host does not provide the up-call, the
client receives PMIX_ERR_NOT_SUPPORTED.