Skip to content

[Nexthop][fboss2-dev] Add config switch resource-limit - #1981

Open
hillol-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:switch-resource-limit-config
Open

hillol-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:switch-resource-limit-config

Conversation

@hillol-nexthop

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

Adds fboss2-dev config switch resource-limit <attr> <value> to configure the SW-agent ResourceAccountant limits that guard against FDB/neighbor hardware-table overflow.

Attributes map onto AgentConfig.defaultCommandLineArgs gflags:

attr gflag type
l2-max max_l2_entries non-negative int
arp-max max_arp_entries non-negative int
ndp-max max_ndp_entries non-negative int
hw-protection enable_hw_update_protection true/false

Motivation. When the hardware FDB/neighbor table fills, the SAI INSUFFICIENT_RESOURCES/table-full status is fatal and the hw agent aborts (crash loop, BGP flaps). ResourceAccountant already guards against this, but it is gated on enable_hw_update_protection (off by default) and there is no CLI to turn it on or to set the per-platform limits.

The *-max limits require hw-protection. ResourceAccountant::isValidUpdate() / stateChanged() only run the L2/neighbor accounting under FLAGS_enable_hw_update_protection, so a limit written without it would be committed, coldbooted and then ignored. The three *-max attributes are therefore refused unless the flag is true in the session config (same idea as acl_utils::requireAclTableGroupMode). hw-protection true can be staged in the same session, so it is still a single commit:

fboss2-dev config switch resource-limit hw-protection true
fboss2-dev config switch resource-limit l2-max 7000
fboss2-dev config session commit

hw-protection is more than a gate for the limits. enable_hw_update_protection also routes L2 learning (MacTableManager) and neighbor programming (NeighborCacheImpl) through the hardware-failure-protected state update path on transaction-capable switches, so turning it on changes how those updates are applied, not only whether they are counted. Turning it off with limits configured is allowed; the command then notes that the limits stay in place but are not enforced. With enforce_resource_hw_limits (the default) the agent also caps arp-max/ndp-max at the ASIC neighbor table size, so a larger value is accepted but not effective.

Action level: coldboot (DISRUPTIVE_SERVICE_RESTART). The flags are consumed once at agent start by initFlagDefaults and are not re-read by the running agent. The handler read-modify-writes a single defaultCommandLineArgs entry, preserving all other flags, and follows the existing config switch admin-distance coldboot pattern.

Test harness. The integration test snapshots the config in SetUp and restores it in TearDown, and asserts on the agent's effective gflags. That needs three small Fboss2IntegrationTest helpers, added here: snapshotConfig(), restoreConfig() and getAgentOption() (fb303 getOption).

Test Plan

Unit tests, built and run with the OSS cmake build (fboss2_cmd_config_test, whole suite green):

CmdConfigResourceLimitTestFixture.argValidation_validInt
CmdConfigResourceLimitTestFixture.argValidation_validBool
CmdConfigResourceLimitTestFixture.argValidation_badArity
CmdConfigResourceLimitTestFixture.argValidation_unknownAttr
CmdConfigResourceLimitTestFixture.argValidation_nonInteger
CmdConfigResourceLimitTestFixture.argValidation_negative
CmdConfigResourceLimitTestFixture.argValidation_nonBool
CmdConfigResourceLimitTestFixture.setL2Max
CmdConfigResourceLimitTestFixture.setArpMax
CmdConfigResourceLimitTestFixture.setNdpMax
CmdConfigResourceLimitTestFixture.setHwProtection
CmdConfigResourceLimitTestFixture.idempotentSameValue
CmdConfigResourceLimitTestFixture.limitRefusedWithoutHwProtection
CmdConfigResourceLimitTestFixture.limitRefusedWhenHwProtectionFalse
CmdConfigResourceLimitTestFixture.hwProtectionOffNotesInertLimits
CmdConfigResourceLimitTestFixture.hwProtectionOffWithoutLimits
100% tests passed

They cover arg validation per attribute, the gflag each attribute writes with other flags preserved, the idempotent path not re-saving the session, the *-max refusal (message and nothing written) with hw-protection unset or false, and the note printed when hw-protection false leaves limits inert.

Integration tests (ConfigResourceLimitTest) against an agent running on fake SAI:

[ RUN      ] ConfigResourceLimitTest.SetLimitsWithHwProtection
[       OK ] ConfigResourceLimitTest.SetLimitsWithHwProtection (5110 ms)
[ RUN      ] ConfigResourceLimitTest.ToggleHwProtection
[       OK ] ConfigResourceLimitTest.ToggleHwProtection (5020 ms)
[ RUN      ] ConfigResourceLimitTest.LimitRefusedWithoutHwProtection
[       OK ] ConfigResourceLimitTest.LimitRefusedWithoutHwProtection (88 ms)
  • SetLimitsWithHwProtection stages hw-protection true plus the three limits, commits once, then checks the running config's defaultCommandLineArgs, the agent's effective gflags (fb303 getOption), and that every other defaultCommandLineArgs entry survived.
  • ToggleHwProtection flips the flag from whatever the agent runs with and checks the effective value.
  • LimitRefusedWithoutHwProtection checks each *-max exits non-zero with hw-protection off; nothing is committed.

Not covered here: the integration tests have not yet been run on hardware with this exact revision, and no test drives a table past its limit, so enforcement by the accountant itself is not exercised by this change's tests.

Review Findings

Pre-publication review with fboss-review (5 generic reviewers + Testing reviewer, threshold 0.7). The findings at or above the threshold were fixed before pushing: hw-protection false silently leaving configured limits inert (now noted in the command output), the ASIC cap on arp-max/ndp-max being undocumented, a hand-rolled bool parser (now folly::toLowerAscii), a staged session left behind when an integration test fails before committing, and the summary understating what enable_hw_update_protection changes. Nothing above the threshold remains open.

Add `fboss2-dev config switch resource-limit <attr> <value>` to configure
the SW-agent ResourceAccountant limits that guard against FDB/neighbor
hardware-table overflow.

Attributes map onto AgentConfig.defaultCommandLineArgs gflags:
  l2-max        -> max_l2_entries              (non-negative int)
  arp-max       -> max_arp_entries             (non-negative int)
  ndp-max       -> max_ndp_entries             (non-negative int)
  hw-protection -> enable_hw_update_protection (true/false)

ResourceAccountant only runs its L2/neighbor checks under
enable_hw_update_protection (off by default), so a limit written without
it would be committed and then ignored. The three *-max attributes are
therefore refused unless the flag is true in the session config;
`hw-protection true` can be staged in the same session, so enabling
protection and setting the limits is still one commit.

enable_hw_update_protection is more than a gate for the limits: it also
routes L2 learning (MacTableManager) and neighbor programming
(NeighborCacheImpl) through the hardware-failure-protected state update
path. Turning it off with limits configured is allowed; the command then
notes that the limits stay in place but are not enforced. With
enforce_resource_hw_limits (the default) the agent also caps arp-max and
ndp-max at the ASIC neighbor table size.

These flags are consumed once at agent start by initFlagDefaults and are
not re-read by the running agent, so the action level is coldboot
(DISRUPTIVE_SERVICE_RESTART). The handler read-modify-writes a single
defaultCommandLineArgs entry, preserving all other flags, and follows the
existing `config switch admin-distance` coldboot pattern.

With hw-update-protection enabled and the limits set to a platform's real
table sizes, a flood of unique MAC/neighbor entries is rejected by the
accountant instead of aborting the hw agent on SAI INSUFFICIENT_RESOURCES.

The integration test snapshots the config in SetUp and restores it in
TearDown, and asserts on the agent's effective gflags. That needs three
Fboss2IntegrationTest helpers this change adds: snapshotConfig(),
restoreConfig() and getAgentOption() (fb303 getOption).
@hillol-nexthop
hillol-nexthop requested review from a team as code owners November 16, 2026 05:00
@github-actions github-actions Bot added nexthop project:fboss2-dev FBOSS project: fboss2-dev labels Nov 16, 2026
@meta-cla meta-cla Bot added the CLA Signed label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants