Repository navigation
[Nexthop][fboss2-dev] Add config switch resource-limit - #1981
Open
hillol-nexthop wants to merge 1 commit into
Open
hillol-nexthop wants to merge 1 commit into
hillol-nexthop wants to merge 1 commit into
Conversation
Add `fboss2-dev config switch resource-limit <attr> <value>` to configure the SW-agent ResourceAccountant limits that guard against FDB/neighbor hardware-table overflow. Attributes map onto AgentConfig.defaultCommandLineArgs gflags: l2-max -> max_l2_entries (non-negative int) arp-max -> max_arp_entries (non-negative int) ndp-max -> max_ndp_entries (non-negative int) hw-protection -> enable_hw_update_protection (true/false) ResourceAccountant only runs its L2/neighbor checks under enable_hw_update_protection (off by default), so a limit written without it would be committed and then ignored. The three *-max attributes are therefore refused unless the flag is true in the session config; `hw-protection true` can be staged in the same session, so enabling protection and setting the limits is still one commit. enable_hw_update_protection is more than a gate for the limits: it also routes L2 learning (MacTableManager) and neighbor programming (NeighborCacheImpl) through the hardware-failure-protected state update path. Turning it off with limits configured is allowed; the command then notes that the limits stay in place but are not enforced. With enforce_resource_hw_limits (the default) the agent also caps arp-max and ndp-max at the ASIC neighbor table size. These flags are consumed once at agent start by initFlagDefaults and are not re-read by the running agent, so the action level is coldboot (DISRUPTIVE_SERVICE_RESTART). The handler read-modify-writes a single defaultCommandLineArgs entry, preserving all other flags, and follows the existing `config switch admin-distance` coldboot pattern. With hw-update-protection enabled and the limits set to a platform's real table sizes, a flood of unique MAC/neighbor entries is rejected by the accountant instead of aborting the hw agent on SAI INSUFFICIENT_RESOURCES. The integration test snapshots the config in SetUp and restores it in TearDown, and asserts on the agent's effective gflags. That needs three Fboss2IntegrationTest helpers this change adds: snapshotConfig(), restoreConfig() and getAgentOption() (fb303 getOption).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
Summary
Adds
fboss2-dev config switch resource-limit <attr> <value>to configure the SW-agentResourceAccountantlimits that guard against FDB/neighbor hardware-table overflow.Attributes map onto
AgentConfig.defaultCommandLineArgsgflags:l2-maxmax_l2_entriesarp-maxmax_arp_entriesndp-maxmax_ndp_entrieshw-protectionenable_hw_update_protectionMotivation. When the hardware FDB/neighbor table fills, the SAI
INSUFFICIENT_RESOURCES/table-full status is fatal and the hw agent aborts (crash loop, BGP flaps).ResourceAccountantalready guards against this, but it is gated onenable_hw_update_protection(off by default) and there is no CLI to turn it on or to set the per-platform limits.The
*-maxlimits requirehw-protection.ResourceAccountant::isValidUpdate()/stateChanged()only run the L2/neighbor accounting underFLAGS_enable_hw_update_protection, so a limit written without it would be committed, coldbooted and then ignored. The three*-maxattributes are therefore refused unless the flag is true in the session config (same idea asacl_utils::requireAclTableGroupMode).hw-protection truecan be staged in the same session, so it is still a single commit:hw-protectionis more than a gate for the limits.enable_hw_update_protectionalso routes L2 learning (MacTableManager) and neighbor programming (NeighborCacheImpl) through the hardware-failure-protected state update path on transaction-capable switches, so turning it on changes how those updates are applied, not only whether they are counted. Turning it off with limits configured is allowed; the command then notes that the limits stay in place but are not enforced. Withenforce_resource_hw_limits(the default) the agent also capsarp-max/ndp-maxat the ASIC neighbor table size, so a larger value is accepted but not effective.Action level: coldboot (
DISRUPTIVE_SERVICE_RESTART). The flags are consumed once at agent start byinitFlagDefaultsand are not re-read by the running agent. The handler read-modify-writes a singledefaultCommandLineArgsentry, preserving all other flags, and follows the existingconfig switch admin-distancecoldboot pattern.Test harness. The integration test snapshots the config in
SetUpand restores it inTearDown, and asserts on the agent's effective gflags. That needs three smallFboss2IntegrationTesthelpers, added here:snapshotConfig(),restoreConfig()andgetAgentOption()(fb303getOption).Test Plan
Unit tests, built and run with the OSS cmake build (
fboss2_cmd_config_test, whole suite green):They cover arg validation per attribute, the gflag each attribute writes with other flags preserved, the idempotent path not re-saving the session, the
*-maxrefusal (message and nothing written) withhw-protectionunset or false, and the note printed whenhw-protection falseleaves limits inert.Integration tests (
ConfigResourceLimitTest) against an agent running on fake SAI:SetLimitsWithHwProtectionstageshw-protection trueplus the three limits, commits once, then checks the running config'sdefaultCommandLineArgs, the agent's effective gflags (fb303getOption), and that every otherdefaultCommandLineArgsentry survived.ToggleHwProtectionflips the flag from whatever the agent runs with and checks the effective value.LimitRefusedWithoutHwProtectionchecks each*-maxexits non-zero withhw-protectionoff; nothing is committed.Not covered here: the integration tests have not yet been run on hardware with this exact revision, and no test drives a table past its limit, so enforcement by the accountant itself is not exercised by this change's tests.
Review Findings
Pre-publication review with fboss-review (5 generic reviewers + Testing reviewer, threshold 0.7). The findings at or above the threshold were fixed before pushing:
hw-protection falsesilently leaving configured limits inert (now noted in the command output), the ASIC cap onarp-max/ndp-maxbeing undocumented, a hand-rolled bool parser (nowfolly::toLowerAscii), a staged session left behind when an integration test fails before committing, and the summary understating whatenable_hw_update_protectionchanges. Nothing above the threshold remains open.