Repository navigation
[Nexthop][fboss2-dev] Refuse config rollback when live config drifted from HEAD - #1829
Open
vybhav-nexthop wants to merge 1 commit into
Open
vybhav-nexthop wants to merge 1 commit into
vybhav-nexthop wants to merge 1 commit into
Conversation
Contributor
Author
|
/autorebase |
rabbit-nexthop
force-pushed
the
fboss2-refuse-config-rollback-when-live
branch
from
November 10, 2026 12:02
d01ac50 to
47e14ca
Compare
Contributor
|
@vybhav-nexthop : Can you resolve the conflicts so I can import this PR? |
Signed-off-by: vybhav-nexthop <vybhav@nexthop.ai>
2 tasks done
rabbit-nexthop
force-pushed
the
fboss2-refuse-config-rollback-when-live
branch
from
November 16, 2026 05:01
47e14ca to
04e52d5
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
pip install -r requirements-dev.txt && pre-commit installpre-commit runSummary
What
fboss2-dev config rollbacknow refuses to run when the config the daemon is running differs from the committed git HEAD. A new global--forceflag commits that running config as a snapshot first, then rolls back.Why: the problem
The live config can get ahead of git HEAD: someone hand-edits
/etc/coop/agent.conf(or BGP'sbgpcpp.conf), runsconfig reload, and never commits. Nothing in the CLI notices this state, and rollback destroys it.How rollback worked before this PR (
ConfigSession::rollback()):git log -- cli/cli_metadata.json).rolledBackActionLevels(), and commitRollback to <sha>.No step compares the file on disk with HEAD. Uncommitted live content is overwritten in step 3 and was never in git, so it is gone with no warning and no way back.
config session diffcompares session vs live, so there was also no way to see this happening.On a test switch this happened for real: live config and HEAD differed in VLAN membership, interfaces, port state and
l2LearningMode. Rolling back to HEAD droppedeth1/33/1fromvlanPorts, which led to an agent crash loop, an ASIC left with only VLANs 1 and 4095, a single broadcast domain across all ports, and dark optics whileshow hw-agentstill reportedCONFIGURED.How: the fix
The check is in
ConfigSession::rollback(), so the CLI and the DUT test runner both go through it. After resolving the target revision and before touching any file:ConfigFileManager::currentPath(): the service's reported--configpath, normally a symlink onto the git-trackedcli/agent.conf) and compare it to that domain's file at HEAD. Reading the daemon-facing path catches a hand edit that replaced the symlink with a regular file. A domain whose tracked file does not exist on disk is skipped: nothing is running from it.--force. If any domain drifted, throw before writing anything. The message names the drifted domain(s), the HEAD sha, the command to inspect it (config session diff <head> current), and warns that--forcewill restart disruptively.--force(snapshotLiveDrift()):cli/cli_metadata.jsonwithbase = HEADandaction = DISRUPTIVE_SERVICE_RESTARTfor each drifted domain.Snapshot of uncommitted live config before rollback by <user>. If the commit fails, the touched files are restored and the error propagates; the running config is not changed.Why the snapshot also commits metadata: no-arg rollback and
rolledBackActionLevels()both walkgit log -- cli/cli_metadata.json, which only lists commits that touch that file. Without it, a plainconfig rollbackwould skip the snapshot, and the level scan would never stop at it. With it,--force config rollback Xfollowed byconfig rollbacklands back on the snapshot.Why the snapshot records a coldboot: a hand edit bypassed the CLI, so the restart level needed to undo it is unknown.
l2LearningMode, part of that drift, needs a coldboot (CmdConfigL2LearningMode), and undoing it with a hitless reload is what went wrong there. So the forced rollback, and any later rollback across the snapshot, coldboots the agent. BGP still gets its normal bgpd restart.Other changes:
--forceis a global flag (fboss2-dev --force config rollback ...), like--detail.rollback()overloads are merged intorollback(hostInfo, commitSha = "", force = false), with emptycommitShameaning the previous commit, so the check has one home.fboss_test_runner/services/config_baseline.pypasses--forceon soft restore. That path runs exactly when the coop tree is dirty from test debris, so without the flag every soft restore would refuse. A dirty-tree soft restore now coldboots the agent, as the hard restore path already does.fboss/cli/fboss2/session/ConfigSession.{h,cpp}snapshotLiveDrift(), singlerollback()fboss/cli/fboss2/CmdGlobalOptions.{h,cpp}--forceflagfboss/cli/fboss2/commands/config/rollback/CmdConfigRollback.cpp--forcethroughfboss/cli/fboss2/test/config/CmdConfigSessionTest.cppfboss_test_runner/services/config_baseline.py,unittests/test_config_baseline.py--forceon soft restoreNot in this PR:
config commitoverwrites a hand-edited live file the same way. The same check belongs there, but it changes commit behavior for every CLI user and needs its own DUT pass.Test Plan
fboss2_cmd_config_test(CmdConfigSessionTest.cpp): 1186 tests passed, including all 59 ConfigSession tests. Forced-rollback tests use aStrictMockthat expects exactly one agentrestartService(DISRUPTIVE_SERVICE_RESTART)and no reload. Drift tests:python3 -m unittest fboss_test_runner.unittests.test_config_baseline:Ran 15 tests ... OKmain: passed.config reload:show hw-agentCONFIGURED and in sync.