Repository navigation
Add support for P4Runtime RegisterEntry reads and writes - #692
Lori-Ringhio97 wants to merge 9 commits into
Conversation
|
What reason did you have for closing this? It is still a feature that would be nice to have implemented in PI and behavioral-model repos, if you are interested and believe you have a working approach. |
DeviceMgr used to return UNIMPLEMENTED for every RegisterEntry. This adds register support at each layer of PI: * p4info: new register resource type (PI_REGISTER_ID, id prefix 0x16) storing the size of the array and the bitwidth of one cell. It is imported from P4Info, from the bmv2 JSON (register_arrays) and from the native JSON format, and exported back to P4Info. Registers whose type_spec is not a bitstring are still imported, with a bitwidth of 0, so that valid P4Info messages which were accepted before are still accepted; such registers cannot be read or written. P4Info messages with a register id which does not have the register prefix are rejected, since p4info stores resources according to that prefix. bmv2 JSON files without register_arrays are still accepted. * PI C API: pi_register_read and pi_register_write, which exchange one cell as a byte array in network byte order, like match keys and action data. The library checks the id, the type and the index before calling the target. * Target API: _pi_register_read and _pi_register_write in the new pi_register_imp.h. PI_ABI_VERSION is incremented from 3 to 4, as required by CONTRIBUTING.md. The dummy target, the rpc target (which returns PI_STATUS_RPC_NOT_IMPLEMENTED) and the Thrift-based bmv2 target implement the new functions. The Thrift target supports cells of up to 63 bits: the bmv2 Thrift API carries register values as signed 64-bit integers, and bmv2 saturates the values which do not fit when reading them. * DeviceMgr: RegisterEntry writes and reads, following the P4Runtime spec and what DeviceMgr already does for CounterEntry: only MODIFY is accepted, a negative index returns INVALID_ARGUMENT and an index beyond the size of the array returns OUT_OF_RANGE, data must be a bitstring which fits in the cell. A read with an unset index returns all the cells of the array, and register_id 0 reads all the registers (with the index, if set, selecting the same cell in each one, as for counters). Read responses use canonical bytestrings. Writes with an unset index return UNIMPLEMENTED: the target API writes one cell at a time, so such a write would not be atomic with respect to the data plane. Tests: the RegisterTest placeholder in test_proto_fe.cpp is replaced by tests for reads and writes (single index, wildcard reads, wide cells, bytestring formats, invalid index, id, update type and data, target errors, unsupported types), the mock switch stores register cells, and unittest.p4 / unittest.p4info.txt gain a bit<12> and a bit<160> register. There are also new tests for the P4Info conversion and for the p4info C API. Targets outside this repository which implement the PI target API, such as the bmv2 PI implementation in p4lang/behavioral-model, need to provide _pi_register_read and _pi_register_write to link against this version of PI. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
The first RegisterEntry implementation only supported registers whose cells have type bit<W>. This adds the other types which p4c accepts for bmv2 registers, following the order suggested in p4lang#376: * int<W>: the data is a bitstring in 2's complement. As required by the spec, sign-extended bytestrings are accepted as long as the value fits in W bits, and reads return the shortest representation. In PI, the value is the 2's complement on W bits, without sign extension. * serializable enums: the data is an enum_value, the value of the underlying bit<W> type; values without a name are valid, as in P4. * enums without an underlying type: the data is an enum (the name of the member). P4Info has no numeric value for these members, so PI uses the representation of p4c, which numbers the members 0, 1, ... in declaration order (the order of P4Info) and stores them as bit<32>. A cell which does not hold a valid member returns INTERNAL when read. * user-defined types (P4 type declarations) whose original_type is one of the above. Types with a @p4runtime_translation annotation are still UNIMPLEMENTED, since PI does not support translation. p4info stores the bitwidth of the cells of these registers, so the C API and the target API are unchanged: a cell is still a bitstring of that bitwidth. As for digests, the cells are exported back to P4Info as bit<W>. DeviceMgr computes how the cells of each register are represented in P4Runtime when the P4Info is set. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
A RegisterEntry write with an unset index writes the data to every cell of the register array. The spec requires writes to be atomic with respect to the data plane, which DeviceMgr cannot guarantee by writing one cell at a time, so this was returning UNIMPLEMENTED. This adds pi_register_write_all to the PI C API and _pi_register_write_all to the target API: the target writes the same value to all the cells of the array, atomically with respect to the data plane, or returns PI_STATUS_NOT_IMPLEMENTED_BY_TARGET (mapped to UNIMPLEMENTED) if it cannot. The dummy and rpc targets implement it as the other register functions, and the Thrift-based bmv2 target uses bm_register_write_range, which holds the lock of the register array while writing. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
To preserve the forwarding state, RECONCILE_AND_COMMIT reads it before updating the device and replays it afterwards, using INSERT updates for every entity. The state includes all the cells of indirect counters and meters, for which DeviceMgr only accepts MODIFY updates, so the replay failed with "Error when reconciling config" for any P4 program with an indirect counter or meter. Counter and meter entities are now replayed with MODIFY updates. The new ReconcileCountersMeters test uses unittest.p4info.txt, which has both, and checks that a counter value and a meter config are preserved. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
Registers are now part of the forwarding state saved before updating the device and replayed afterwards, with MODIFY updates as for counters and meters. Registers which cannot be read are skipped, so that they do not prevent the reconciliation: those whose type is not supported, as in any read of all the registers, and those of targets which do not support registers (UNIMPLEMENTED). Their state is not preserved, which used to be the case for all registers. The reconcile test also checks that a register cell is preserved, and the new ReconcileTargetWithoutRegisters test checks a target which does not support registers. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
The p4c frontend accepts registers whose cells are structs, headers, header unions, header stacks, tuples, bools, errors or varbits, and generates their type in P4Info, but PI only supported bit<W>, int<W>, enums and type declarations of these. (p4c-bm2-ss rejects these types, so they cannot be used with bmv2.) A PI register cell is a bitstring of the bitwidth stored in p4info, so the values of the other types are laid out in that bitstring, as documented in pi_register.h, starting from the most significant bit: the members of structs and tuples in order, the validity bit of headers followed by their fields, the headers of header unions and the elements of stacks in order, 1 bit for bools, the index of the member on 32 bits for errors (as for enums), and the dynamic bitwidth on 32 bits followed by the value for varbits. Targets which support registers of these types must use the same layout. p4info computes the bitwidth of the cells from P4Info, following type declarations and the types in type_info. The depth of the types and the number of types visited are bounded, as an invalid P4Info could have recursive types, or types reused so many times that the value would have too many members. DeviceMgr converts P4Data to and from the cells with the new RegisterType class, which replaces the previous per-type helpers and checks that it agrees with p4info on the bitwidth. Registers whose type cannot be represented are still imported with a bitwidth of 0 and return UNIMPLEMENTED, with a message giving the reason: types with a @p4runtime_translation annotation (PI does not support translation), headers with a varbit field (P4Header cannot represent the dynamic bitwidth of a field), types whose values have 0 bits (e.g. empty structs) and types missing from type_info. A wildcard read of all the registers still skips them. Tests: new RegisterTypeTest tests for each type check the layout of the cells, round trips and invalid data (wrong members, invalid headers and header unions, wrong number of stack entries, values which do not fit). P4InfoProtoConvertRegistersTest.Types checks the bitwidth computed for each type. UnsupportedType now uses a header with a varbit field, since structs are supported. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
The CLI gains register_read, register_write, register_write_all and register_reset, which writes 0 to all the cells as in the bmv2 CLI. Values are hexadecimal strings, as for action parameters, and must fit in the bitwidth of the cells. The CLI uses the rpc target, which returned PI_STATUS_RPC_NOT_IMPLEMENTED for registers. It now sends PI_RPC_REGISTER_READ, PI_RPC_REGISTER_WRITE and PI_RPC_REGISTER_WRITE_ALL requests, which carry the size of the cell, and pi_rpc_server checks that size against its p4info before calling the target functions. The new tests/CLI/register.test reads and writes RegisterA of stats.json on bmv2, through pi_rpc_server and the Thrift-based bmv2 target. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
a554a3a to
234f148
Compare
|
Sorry, I closed it because the first version only handled |
|
Sorry for the long delay in responding to this and the related bmv2 PR. I will try building them both locally today and test them out, then start looking at the code changes. |
| // - struct, tuple: the members, in order. | ||
| // - header: the validity bit (set if the header is valid), followed by the | ||
| // fields, in order. The fields of an invalid header are written as 0. | ||
| // - header union: the headers, in order. |
There was a problem hiding this comment.
All of the headers in the union take up separate space? Even though at most one of them can be valid? I agree this can be implemented correctly, but just wanted to point out it is a waste of memory.
I will check whether the storage in BMv2 is the same as this format, or something more compact, e.g. with a storage that is close to the maximum size of any of the members of the union.
There was a problem hiding this comment.
It appears that the BMv2 implementation simply copies the pi representation to/from the storage of the data plane's register array element that is being read/written, if I understand correctly.
Today p4c's bmv2 back end only supports type bit<W> as register array elements (I think -- it definitely does not support structs now, for example). It appears that if we want to support other types in the future, either the p4c bmv2 back end needs to know about this storage format, or bmv2's PI code for reading/writing registers would need to change to convert between the pi format and whatever bmv2 format that the p4c bmv2 back end uses. Is that correct?
There was a problem hiding this comment.
Note: Given that p4c bmv2 back end only supports a smaller set of types as register array elements than this PR does, please do not take my questions above to mean "we must change this PR before it can be merged". They are mainly clarifying questions to ensure I understand these changes.
I would prefer to have working support for register read/write, even if all the projects only work together to support register array element types of bit<W>, and perhaps a few other similar ones. That is much better than the situation we have right now (i.e. no P4Runtime API register read/write support at all, for any type).
There was a problem hiding this comment.
Yes, your understanding is correct. The layout in pi_register.h is the format of the values passed between PI and the target (_pi_register_read / _pi_register_write), like the PI formats of match keys and action data; it does not have to be how the target stores the cells. Today a bmv2 register cell is a single bm::Data of the bitwidth in the JSON, and p4c-bm2-ss only accepts bit<W>, int<W>, enums and type declarations as register cells, so the bmv2 PI code just copies the bytes. If the bmv2 back end supports other types one day, either it uses this layout and the bmv2 PI code stays as it is, or the bmv2 PI code converts between the two. I added a sentence about this to pi_register.h.
You were right about the memory, and since no target uses this layout yet, I changed it in bfcb503: a header union is now the index of the valid header, on the smallest number of bits that can hold it, followed by that header, padded to the widest one. A union of two 13-bit headers takes 14 bits instead of 26, and a cell can no longer hold more than one valid header. If you prefer the previous layout, I can revert that commit.
The comments of pi_p4info_register_get_bitwidth and of the target API still said that only bitstring types are supported, as in the first version of this change. The PI library calls the target for every register whose type is supported, i.e. whose bitwidth is not 0, and the values of the cells are laid out as described in PI/pi_register.h. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
A header union holds at most one valid header, but the cells of a register of header unions held all the headers, one after the other. They now hold the index of the valid header, on the smallest number of bits which can hold the index of the last header, followed by that header, padded with 0s to the bitwidth of the widest header. For example, a union of two headers of 13 bits takes 14 bits instead of 26. A cell can no longer hold more than one valid header; if the validity bit is not set, no header is valid and the index is ignored. pi_register.h also says now that this is the format of the values passed to and from the target, which may store the cells in another format and convert the values in its implementation of the target functions. Signed-off-by: Lori-Ringhio97 <lorenzo.rinieri97@libero.it>
| // have 14 bits: the index of the valid header on 1 bit (0 for h1, 1 for h2), | ||
| // followed by that header. Types with a @p4runtime_translation annotation, | ||
| // headers with a varbit field and types whose values have 0 bits are not | ||
| // supported (bitwidth is 0). |
There was a problem hiding this comment.
If I understand correctly, it appears that two different values of a header union are ambiguously represented using all 0 bits:
(1) the first member is valid, and all of its field values equal 0
(2) all members are invalid.
If you agree that is the case, I can see at least 2 main alternatives to correct this problem.
(a) revert to the previous representation with all N members having separate storage
(b) change the compact encoding a little bit, perhaps one of the following:
(b1) use a member index of 0 to mean "all members invalid", and use 1 through N to encode that a member is valid, with numbering 1 through N instead of 0 through N-1.
(b2) use a member index of N to mean "all members invalid", and 0 through N-1 to encode that one member is valid.
I do not have a strong preference which of the above to use.
Fixes #376.
Until now DeviceMgr returned
UNIMPLEMENTEDfor everyRegisterEntry. This PR adds register support at every layer of PI, for registers of any P4 type that P4Runtime can represent. The companion bmv2 PR, p4lang/behavioral-model#1473, implements the new target functions forsimple_switch_grpc.The PR has 9 commits, which can be reviewed one at a time; the last two address the first review. Two others fix existing problems and can be split into a separate PR if you prefer: "Fix RECONCILE_AND_COMMIT for programs with counters or meters" and, on top of it, "Preserve registers on RECONCILE_AND_COMMIT".
What changes
p4info: new register resource type (
PI_REGISTER_ID, prefix0x16as inP4Ids), with the size of the array and the bitwidth of one cell. It is imported from P4Info, from the bmv2 JSON (register_arrays) and from the native JSON format, and exported back to P4Info (asbit<W>, as for digests). Registers whose type cannot be represented (see "Not supported") are still imported, with a bitwidth of 0, so every valid P4Info accepted before is still accepted, and so is a bmv2 JSON withoutregister_arrays. A P4Info with a register id that does not have the register prefix is rejected: p4info stores each resource according to that prefix, so it would otherwise corrupt memory.PI C API (
PI/pi_register.h):pi_register_read,pi_register_writeandpi_register_write_all. A cell is a byte array in network byte order, the same format as match keys and action data. The library checks the id, the type and the index before calling the target.Target API (
PI/target/pi_register_imp.h):_pi_register_read,_pi_register_writeand_pi_register_write_all. The last one writes the same value to every cell, atomically with respect to the data plane (an action that accesses the array sees either the old value or the new one in all the cells), or returnsPI_STATUS_NOT_IMPLEMENTED_BY_TARGET.PI_ABI_VERSIONgoes from 3 to 4, asCONTRIBUTING.mdrequires. The dummy, rpc and Thrift-based bmv2 targets implement the new functions. The Thrift target handles cells of up to 63 bits: the bmv2 Thrift API carries register values asi64, and bmv2 saturates the values that do not fit when reading them (a 64-bit cell holding 2^63 is read as 2^63 - 1, which I checked onsimple_switch).DeviceMgr:
RegisterEntrywrites and reads, following the spec and what DeviceMgr already does forCounterEntry:MODIFYwithindexMODIFYwithoutindex_pi_register_write_all(atomic)INSERT,DELETEINVALID_ARGUMENTindexINVALID_ARGUMENTindex≥ sizeOUT_OF_RANGEdataof the wrong kind, or which does not fit in the typeINVALID_ARGUMENTUNIMPLEMENTED, with the reasonindexindexregister_id0index, that cell of every register, as for counters (registers of unsupported types are skipped)Reads return canonical bytestrings.
PI_STATUS_NOT_IMPLEMENTED_BY_TARGETmaps toUNIMPLEMENTED, and other target errors map toUNKNOWN, as for counters.RECONCILE_AND_COMMIT: registers are now part of the forwarding state that is saved and replayed. While testing this I found that the replay used
INSERTfor every entity, so reconciling failed with "Error when reconciling config" for any program with an indirect counter or meter. Counters, meters and registers are now replayed withMODIFY. Registers that cannot be read (unsupported type, or a target that returnsPI_STATUS_NOT_IMPLEMENTED_BY_TARGET) are skipped, so they do not make the reconciliation fail; their state is not preserved, as for all registers until now. As for tables, counters and meters, the reconciliation fails if the saved state does not fit the new config (e.g. a register made smaller).PI CLI:
register_read,register_write,register_write_allandregister_reset, through new messages of the internal RPC (PI_RPC_REGISTER_READ,PI_RPC_REGISTER_WRITE,PI_RPC_REGISTER_WRITE_ALL) handled bypi_rpc_server, which checks the size of the cell against its p4info.Register types
PI exchanges a cell with the target as a bitstring of the bitwidth stored in p4info. For types other than
bit<W>, the value is laid out in that bitstring as documented inpi_register.h, from the most significant bit:P4Databit<W>bitstringint<W>bitstring(sign-extended input accepted, shortest 2's complement returned)boolboolenum_value(any value of the underlying type, even without a name)bit<W>enumbit<32>errorerrorbit<32>varbit<M>varbitbit<32>, then M bitsstruct,tupleheaderheader_unionheader_stack,header_union_stacktypedeclarationTwo choices you may want to look at:
error. PI uses the representation of p4c, which numbers the members in declaration order (the order of P4Info) and stores them asbit<32>. A cell that does not hold a valid member returnsINTERNALwhen read.p4c-bm2-ssrejects register types other thanbit<W>andint<W>(and enums), but other backends generate them in P4Info.Not supported
@p4runtime_translationannotation, since PI does not implement translation for any entity.varbitfield:P4Headerhas no way to carry the dynamic bitwidth of a field.These registers are imported with a bitwidth of 0 and return
UNIMPLEMENTEDwith a message giving the reason.Compatibility
The target API gains three functions, so targets maintained outside this repository have to provide them before they can link against this version. For bmv2, p4lang/behavioral-model#1473 implements them with
register_read,register_writeandregister_write_rangeof the bmv2 runtime, with no 64-bit limit. bmv2 CI builds against PI main, so the two PRs should be merged close together: this PR first, then the bmv2 one.Testing
proto/tests/test_proto_fe.cpp: theRegisterTestplaceholder is replaced by tests for single-index and wildcard writes and reads, abit<160>register, bytestring formats, invalid index, id, update type and data, target errors and unsupported types, andRegisterTypeTestchecks every type in the table above: the exact layout of the cell, the round trip, and invalid data.unittest.p4/unittest.p4info.txtgain abit<12>and abit<160>register.proto/tests/test_proto_fe_set_pipeline_config.cpp: a counter, a meter and a register are preserved byRECONCILE_AND_COMMIT(this test fails without the reconcile fix), and a target that does not support registers does not prevent the reconciliation.proto/tests/test_p4info_convert.cpp: P4Info → p4info → P4Info for registers, and the bitwidth computed for each type.tests/test_p4info.c: the p4info register API, the native JSON round trip, and configs without registers.tests/CLI/register.test: the new CLI commands on bmv2, throughpi_rpc_serverand the Thrift target.Run locally, as in the workflows:
test.yml, with gcc and clang:Dockerfile.bmv2test image (--enable-Werror),proto/ptf/base_test.pyandmake check: all tests pass.bazel-build.yml: buildifier andbazel test //proto/tests:pi_proto_tests //proto/tests:pi_proto_server_tests: both pass.make checkon its own.tools/cpplint.pyon the changed C++ files andreuse lint(outside the submodules): clean. On the new code, clang-format only reports the include order, which follows the existing files (check_style.shis disabled intest.yml).ctest(53 tests), thesimple_switch_grpctests with CMake (12) and autotools (33 gtest tests, 9 of them for registers): all pass. The bmv2 Bazel build and tests, with this branch as the PI module, pass too.