admin.tex

\section{Device groups}\label{sec:Basic Facilities of a Virtio Device / Device groups}

It is occasionally useful to have a device control a group of
other devices. Terminology used in such cases:

\begin{description}
\item[Device group]
        or just group, includes zero or more devices.
\item[Owner device]
        or owner, the device controlling the group.
\item[Member device]
        a device within a group. The owner device itself is not
	a member of the group.
\item[Member identifier]
        each member has this identifier, unique within the group
	and used to address it through the owner device.
\item[Group type identifier]
	specifies what kind of member devices there are in a
	group, how the member identifier is interpreted
	and what kind of control the owner has.
	A given owner can control multiple groups
	of different types but only a single group of a given type,
	thus the type and the owner together identify the group.
	\footnote{Even though some group types only support
			specific transports, group type identifiers
			are global rather than transport-specific -
			a flood of new group types is not expected.}
\end{description}

\begin{note}
Each device only has a single driver, thus for the purposes of
this section, "the driver" is usually unambiguous and refers to
the driver of the owner device.  When there's ambiguity, "owner
driver" refers to the driver of the owner device, while "member
driver" refers to the driver of a member device.
\end{note}

The following group types, and their identifiers, are currently specified:
\begin{description}
\item[SR-IOV group type (0x1)]
This device group has a PCI Single Root I/O Virtualization
(SR-IOV) physical function (PF) device as the owner and includes
all its SR-IOV virtual functions (VFs) as members (see
\hyperref[intro:PCIe]{[PCIe]}).

The PF device itself is not a member of the group.

The group type identifier for this group is 0x1.

A member identifier for this group can have a value from 0x1 to
\field{NumVFs} as specified in the
SR-IOV Extended Capability of the owner device
and equals the SR-IOV VF number of the member device;
the group only exists when the \field{VF Enable} bit
in the SR-IOV Control Register within the
SR-IOV Extended Capability of the owner device is set
(see \hyperref[intro:PCIe]{[PCIe]}).

Both owner and member devices for this group type use the Virtio
PCI transport (see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus}).
\end{description}

\subsection{Group administration commands}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group administration commands}

The driver sends group administration commands to the owner device of
a group to control member devices of the group.
This mechanism can
be used, for example, to configure a member device before it is
initialized by its driver.
\footnote{The term "administration" is intended to be interpreted
widely to include any kind of control. See specific commands
for detail.}

All the group administration commands are of the following form:

\begin{lstlisting}
struct virtio_admin_cmd {
        /* Device-readable part */
        le16 opcode;
        /*
         * 1       - SR-IOV
         * 2-65535 - reserved
         */
        le16 group_type;
        /* unused, reserved for future extensions */
        u8 reserved1[12];
        le64 group_member_id;
        le64 command_specific_data[];

        /* Device-writable part */
        le16 status;
        le16 status_qualifier;
        /* unused, reserved for future extensions */
        u8 reserved2[4];
        u8 command_specific_result[];
};
\end{lstlisting}

For all commands, \field{opcode}, \field{group_type} and if
necessary \field{group_member_id} and \field{command_specific_data} are
set by the driver, and the owner device sets \field{status} and if
needed \field{status_qualifier} and
\field{command_specific_result}.

Generally, any unused device-readable fields are set to zero by the driver
and ignored by the device.  Any unused device-writeable fields are set to zero
by the device and ignored by the driver.

\field{opcode} specifies the command. The valid
values for \field{opcode} can be found in the following table:

\begin{tabularx}{\textwidth}{ |l||l|X| }
\hline
opcode & Name & Command Description \\
\hline \hline
0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
\hline
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
\hline
0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Writes into the legacy common configuration structure \\
\hline
0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Reads from the legacy common configuration structure  \\
\hline
0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Writes into the legacy device configuration structure \\
\hline
0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Reads into the legacy device configuration structure \\
\hline
0x0006 & VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_INFO & Query the notification region information \\
\hline
0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
\hline
\end{tabularx}

The \field{group_type} specifies the group type identifier.
The \field{group_member_id} specifies the member identifier within the group.
See section \ref{sec:Basic Facilities of a Virtio Device / Device groups}
for the definition of the group type identifier and group member
identifier.

The \field{status} describes the command result and possibly
failure reason at an abstract level, this is appropriate for
forwarding to applications. The \field{status_qualifier} describes
failures at a low virtio specific level, as appropriate for debugging.
The following table describes possible \field{status} values;
to simplify common implementations, they are intentionally
matching common \hyperref[intro:errno]{Linux error names and numbers}:

\begin{tabular}{|l|l|l|}
\hline
Status (decimal) & Name & Description \\
\hline \hline
00   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
\hline
11   & VIRTIO_ADMIN_STATUS_EAGAIN    & try again \\
\hline
12   & VIRTIO_ADMIN_STATUS_ENOMEM    & insufficient resources \\
\hline
22   & VIRTIO_ADMIN_STATUS_EINVAL    & invalid command \\
\hline
other   & -    & group administration command error  \\
\hline
\end{tabular}

When \field{status} is VIRTIO_ADMIN_STATUS_OK, \field{status_qualifier}
is reserved and set to zero by the device.

The following table describes possible \field{status_qualifier} values:

\begin{tabularx}{\textwidth}{ |l||l|X| }
\hline
Status & Name & Description \\
\hline \hline
0x00   & VIRTIO_ADMIN_STATUS_Q_OK               & used with VIRTIO_ADMIN_STATUS_OK  \\
\hline
0x01   & VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND  & command error: no additional information  \\
\hline
0x02   & VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE   & unsupported or invalid \field{opcode}  \\
\hline
0x03   & VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD    & unsupported or invalid field within \field{command_specific_data}  \\
\hline
0x04   & VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP    & unsupported or invalid \field{group_type} \\
\hline
0x05   & VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER   & unsupported or invalid \field{group_member_id} \\
\hline
0x06   & VIRTIO_ADMIN_STATUS_Q_NORESOURCE       & out of internal resources: ok to retry \\
\hline
0x07   & VIRTIO_ADMIN_STATUS_Q_TRYAGAIN         & command blocks for too long: should retry \\
\hline
0x08-0xFFFF   & -    & reserved for future use \\
\hline
\end{tabularx}

Each command uses a different \field{command_specific_data} and
\field{command_specific_result} structures and the length of
\field{command_specific_data} and \field{command_specific_result}
depends on these structures and is described separately or is
implicit in the structure description.

Before sending any group administration commands to the device, the driver
needs to communicate to the device which commands it is going to
use. Initially (after reset), only two commands are assumed to be used:
VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.

Before sending any other commands for any member of a specific group to
the device, the driver queries the supported commands via
VIRTIO_ADMIN_CMD_LIST_QUERY and sends the commands it is
capable of using via VIRTIO_ADMIN_CMD_LIST_USE.

Commands VIRTIO_ADMIN_CMD_LIST_QUERY and
VIRTIO_ADMIN_CMD_LIST_USE
both use the following structure describing the
command opcodes:

\begin{lstlisting}
struct virtio_admin_cmd_list {
       /* Indicates which of the below fields were returned
       le64 device_admin_cmd_opcodes[];
};
\end{lstlisting}

This structure is an array of 64 bit values in little-endian byte
order, in which a bit is set if the specific command opcode
is supported. Thus, \field{device_admin_cmd_opcodes[0]} refers to the
first 64-bit value in this array corresponding to opcodes 0 to
63, \field{device_admin_cmd_opcodes[1]} is the second 64-bit value
corresponding to opcodes 64 to 127, etc.
For example, the array of size 2 including
the values 0x3 in \field{device_admin_cmd_opcodes[0]}
and 0x1 in \field{device_admin_cmd_opcodes[1]} indicates that only
opcodes 0, 1 and 64 are supported.
The length of the array depends on the supported opcodes - it is
large enough to include bits set for all supported opcodes,
that is the length can be calculated by starting with the largest
supported opcode adding one, dividing by 64 and rounding up.
In other words, for
VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE the
length of \field{command_specific_result} and
\field{command_specific_data} respectively will be
$DIV_ROUND_UP(max_cmd, 64) * 8$ where DIV_ROUND_UP is integer division
with round up and \field{max_cmd} is the largest available command opcode.

The array is also allowed to be larger and to additionally
include an arbitrary number of all-zero entries.

Accordingly, bits 0 and 1 corresponding to opcode 0
(VIRTIO_ADMIN_CMD_LIST_QUERY) and 1
(VIRTIO_ADMIN_CMD_LIST_USE) are
always set in \field{device_admin_cmd_opcodes[0]} returned by VIRTIO_ADMIN_CMD_LIST_QUERY.

For the command VIRTIO_ADMIN_CMD_LIST_QUERY, \field{opcode} is set to 0x0.
The \field{group_member_id} is unused. It is set to zero by driver.
This command has no command specific data.
The device, upon success, returns a result in
\field{command_specific_result} in the format
\field{struct virtio_admin_cmd_list} describing the
list of group administration commands supported for the group type
specified by \field{group_type}.

For the command VIRTIO_ADMIN_CMD_LIST_USE, \field{opcode}
is set to 0x1.
The \field{group_member_id} is unused. It is set to zero by driver.
The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_list} describing the
list of group administration commands used by the driver
with the group type specified by \field{group_type}.

This command has no command specific result.

The driver issues the command VIRTIO_ADMIN_CMD_LIST_QUERY to
query the list of commands valid for this group and before sending
any commands for any member of a group.

The driver then enables use of some of the opcodes by sending to
the device the command VIRTIO_ADMIN_CMD_LIST_USE with a subset
of the list returned by VIRTIO_ADMIN_CMD_LIST_QUERY that is
both understood and used by the driver.

If the device supports the command list used by the driver, the
device completes the command with status VIRTIO_ADMIN_STATUS_OK.
If the device does not support the command list
(for example, if the driver is not capable to use
some required commands), the device
completes the command with status
VIRTIO_ADMIN_STATUS_INVALID_FIELD.

Note: the driver is assumed not to set bits in
device_admin_cmd_opcodes
if it is not familiar with how the command opcode
is used, since the device could have dependencies between
command opcodes.

It is assumed that all members in a group support and are used
with the same list of commands. However, for owner devices
supporting multiple group types, the list of supported commands
might differ between different group types.

\input{admin-cmds-legacy-interface.tex}

\devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}

The device MUST validate \field{opcode}, \field{group_type} and
\field{group_member_id}, and if any of these has an invalid or
unsupported value, set \field{status} to
VIRTIO_ADMIN_STATUS_EINVAL and set \field{status_qualifier}
accordingly:
\begin{itemize}
\item if \field{group_type} is invalid, \field{status_qualifier}
	MUST be set to VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
\item otherwise, if \field{opcode} is invalid,
	\field{status_qualifier} MUST be set to
	VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE;
\item otherwise, if \field{group_member_id} is used by the
	specific command and is invalid, \field{status_qualifier} MUST be
	set to VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER.
\end{itemize}

If a command completes successfully, the device MUST set
\field{status} to VIRTIO_ADMIN_STATUS_OK.

If a command fails, the device MUST set
\field{status} to a value different from VIRTIO_ADMIN_STATUS_OK.

If \field{status} is set to VIRTIO_ADMIN_STATUS_EINVAL, the
device state MUST NOT change, that is the command MUST NOT have
any side effects on the device, in particular the device MUST NOT
enter an error state as a result of this command.

If a command fails, the device state generally SHOULD NOT change,
as far as possible.

The device MAY enforce additional restrictions and dependencies on
opcodes used by the driver and MAY fail the command
VIRTIO_ADMIN_CMD_LIST_USE with \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
and \field{status_qualifier} set to VIRTIO_ADMIN_STATUS_Q_INVALID_FIELD
if the list of commands used violate internal device dependencies.

If the device supports multiple group types, commands for each group
type MUST operate independently of each other, in particular,
the device MAY return different results for VIRTIO_ADMIN_CMD_LIST_QUERY
for different group types.

After reset, if the device supports a given group type
and before receiving VIRTIO_ADMIN_CMD_LIST_USE for this group type
the device MUST assume
that the list of legal commands used by the driver consists of
the two commands VIRTIO_ADMIN_CMD_LIST_QUERY and VIRTIO_ADMIN_CMD_LIST_USE.

After completing VIRTIO_ADMIN_CMD_LIST_USE successfully,
the device MUST set the list of legal commands used by the driver
to the one supplied in \field{command_specific_data}.

The device MUST validate commands against the list used by
the driver and MUST fail any commands not in the list with
\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
and \field{status_qualifier} set to
VIRTIO_ADMIN_STATUS_Q_INVALID_OPCODE.

The list of supported commands reported by the device MUST NOT
shrink (but MAY expand): after reporting a given command as
supported through VIRTIO_ADMIN_CMD_LIST_QUERY the device MUST NOT
later report it as unsupported.  Further, after a given set of
commands has been used (via a successful
VIRTIO_ADMIN_CMD_LIST_USE), then after a device or system reset
the device SHOULD complete successfully any following calls to
VIRTIO_ADMIN_CMD_LIST_USE with the same list of commands; if this
command VIRTIO_ADMIN_CMD_LIST_USE fails after a device or system
reset, the device MUST not fail it solely because of the command
list used.  Failure to do so would interfere with resuming from
suspend and error recovery. Exceptions MAY apply if the system
configuration assures, in some way, that the driver does not
cache the previous value of VIRTIO_ADMIN_CMD_LIST_USE,
such as in the case of a firmware upgrade or downgrade.

When processing a command with the SR-IOV group type,
if the device does not have an SR-IOV Extended Capability or
if \field{VF Enable} is clear
then the device MUST fail all commands with
\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
\field{status_qualifier} set to
VIRTIO_ADMIN_STATUS_Q_INVALID_GROUP;
otherwise, if \field{group_member_id} is not
between $1$ and \field{NumVFs} inclusive,
the device MUST fail all commands with
\field{status} set to VIRTIO_ADMIN_STATUS_EINVAL and
\field{status_qualifier} set to
VIRTIO_ADMIN_STATUS_Q_INVALID_MEMBER;
\field{NumVFs}, \field{VF Migration Capable}  and
\field{VF Enable} refer to registers within the SR-IOV Extended
Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

\drivernormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}

The driver MAY discover whether device supports a specific group type
by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
\field{group_type}.

The driver MUST issue VIRTIO_ADMIN_CMD_LIST_USE
and wait for it to be completed with status
VIRTIO_ADMIN_STATUS_OK before issuing any commands
(except for the initial VIRTIO_ADMIN_CMD_LIST_QUERY
and VIRTIO_ADMIN_CMD_LIST_USE).

The driver MAY issue VIRTIO_ADMIN_CMD_LIST_USE any number
of times but MUST NOT issue VIRTIO_ADMIN_CMD_LIST_USE commands
if any other command has been submitted to the
device and has not yet completed processing by the device.

The driver SHOULD NOT set bits in device_admin_cmd_opcodes
if it is not familiar with how the command opcode
is used, as dependencies between command opcodes might exist.

The driver MUST NOT request (via VIRTIO_ADMIN_CMD_LIST_USE)
the use of any commands not previously reported as
supported for the same group type by VIRTIO_ADMIN_CMD_LIST_QUERY.

The driver MUST NOT use any commands for a given group type
before sending VIRTIO_ADMIN_CMD_LIST_USE with the correct
list of command opcodes and group type.

The driver MAY block use of VIRTIO_ADMIN_CMD_LIST_QUERY and
VIRTIO_ADMIN_CMD_LIST_USE by issuing VIRTIO_ADMIN_CMD_LIST_USE
with respective bits cleared in \field{command_specific_data}.

The driver MUST handle a command error with a reserved \field{status}
value in the same way as \field{status} set to VIRTIO_ADMIN_STATUS_EINVAL
(except possibly for different error reporting/diagnostic messages).

The driver MUST handle a command error with a reserved
\field{status_qualifier} value in the same way as
\field{status_qualifier} set to
VIRTIO_ADMIN_STATUS_Q_INVALID_COMMAND (except possibly for
different error reporting/diagnostic messages).

When sending commands with the SR-IOV group type,
the driver specify a value for \field{group_member_id}
between $1$ and \field{NumVFs} inclusive,
the driver MUST also make sure that as long as any such command
is outstanding, \field{VF Migration Capable} is clear and
\field{VF Enable} is set;
\field{NumVFs}, \field{VF Migration Capable}  and
\field{VF Enable} refer to registers within the SR-IOV Extended
Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.

\section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}

An administration virtqueue of an owner device is used to submit
group administration commands. An owner device can have more
than one administration virtqueue.

If VIRTIO_F_ADMIN_VQ has been negotiated, an owner device exposes one
or more adminstration virtqueues. The number and locations of the
administration virtqueues are exposed by the owner device in a transport
specific manner.

The driver enqueues requests to an arbitrary administration
virtqueue, and they are used by the device on that same
virtqueue. It is the responsibility of the driver to ensure
strict request ordering for commands, because they will be
consumed with no order constraints.  For example, if consistency
is required then the driver can wait for the processing of a
first command by the device to be completed before submitting
another command depending on the first one.

Administration virtqueues are used as follows:
\begin{itemize}
\item The driver submits the command using the \field{struct virtio_admin_cmd}
structure using a buffer consisting of two parts: a device-readable one followed by a
device-writable one.
\item the device-readable part includes fields from \field{opcode}
through \field{command_specific_data}.
\item the device-writeable buffer includes fields from \field{status}
through \field{command_specific_result} inclusive.
\end{itemize}

For each command, this specification describes a distinct
format structure used for \field{command_specific_data} and
\field{command_specific_result}, the length of these fields
depends on the command.

However, to ensure forward compatibility
\begin{itemize}
\item drivers are allowed to submit buffers that are longer
than the device expects
(that is, longer than the length of
\field{opcode} through \field{command_specific_data}).
This allows the driver to maintain
a single format structure even if some structure fields are
unused by the device.
\item drivers are allowed to submit buffers that are shorter
than what the device expects
(that is, shorter than the length of \field{status} through
\field{command_specific_result}). This allows the device to maintain
a single format structure even if some structure fields are
unused by the driver.
\end{itemize}

The device compares the length of each part (device-readable and
device-writeable) of the buffer as submitted by driver to what it
expects and then silently truncates the structures to either the
length submitted by the driver, or the length described in this
specification, whichever is shorter.  The device silently ignores
any data falling outside the shorter of the two lengths. Any
missing fields are interpreted as set to zero.

Similarly, the driver compares the used buffer length
of the buffer to what it expects and then silently
truncates the structure to the used buffer length.
The driver silently ignores any data falling outside
the used buffer length reported by the device.  Any missing
fields are interpreted as set to zero.

This simplifies driver and device implementations since the
driver/device can simply maintain a single large structure (such
as a C structure) for a command and its result. As new versions
of the specification are designed, new fields can be added to the
tail of a structure, with the driver/device using the full
structure without concern for versioning.

\devicenormative{\subsection}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}

The device MUST support device-readable and device-writeable buffers
shorter than described in this specification, by
\begin{enumerate}
\item acting as if any data that would be read outside the
device-readable buffers is set to zero, and
\item discarding data that would be written outside the
specified device-writeable buffers.
\end{enumerate}

The device MUST support device-readable and device-writeable buffers
longer than described in this specification, by
\begin{enumerate}
\item ignoring any data in device-readable buffers outside
the expected length, and
\item only writing the expected structure to the device-writeable
buffers, ignoring any extra buffers, and reporting the
actual length of data written, in bytes,
as buffer used length.
\end{enumerate}

The device SHOULD initialize the device-writeable buffer
up to the length of the structure described by this specification or
the length of the buffer supplied by the driver (even if the buffer is
all set to zero), whichever is shorter.

The device MUST NOT fail a command solely because the buffers
provided are shorter or longer than described in this
specification.

The device MUST initialize the device-writeable part of
\field{struct virtio_admin_cmd} that is a multiple of 64 bit in
size.

The device MUST initialize \field{status} and
\field{status_qualifier} in \field{struct virtio_admin_cmd}.

The device MUST process commands on a given administration virtqueue
in the order in which they are queued.

If multiple administration virtqueues have been configured,
device MAY process commands on distinct virtqueues with
no order constraints.

If the device sets \field{status} to either VIRTIO_ADMIN_STATUS_EAGAIN
or VIRTIO_ADMIN_STATUS_ENOMEM, then the command MUST NOT
have any side effects, making it safe to retry.

\drivernormative{\subsection}{Group administration commands}{Basic Facilities of a Virtio Device / Administration virtqueues}

The driver MAY supply device-readable or device-writeable parts
of \field{struct virtio_admin_cmd} that are longer than described in
this specification.

The driver SHOULD supply device-readable part of
\field{struct virtio_admin_cmd} that is at least as
large as the structure described by this specification
(even if the structure is all set to zero).

The driver MUST supply both device-readable or device-writeable parts
of \field{struct virtio_admin_cmd} that are a multiple of 64 bit
in length.

The device MUST supply both device-readable or device-writeable parts
of \field{struct virtio_admin_cmd} that are larger than zero in
length. However, \field{command_specific_data} and
\field{command_specific_result} MAY be zero in length, unless
specified otherwise for the command.

The driver MUST NOT assume that the device will initialize the whole
device-writeable part of \field{struct virtio_admin_cmd} as described in the specification; instead,
the driver MUST act as if the structure
outside the part of the buffer used by the device
is set to zero.

If multiple administration virtqueues have been configured,
the driver MUST ensure ordering for commands
placed on different administration virtqueues.

The driver SHOULD retry a command that completed with
\field{status} VIRTIO_ADMIN_STATUS_EAGAIN.