Skip to content

[Nexthop][fboss2-dev] Refuse config rollback when live config drifted from HEAD - #1829

Open
vybhav-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:fboss2-refuse-config-rollback-when-live
Open

vybhav-nexthop wants to merge 1 commit into
meta:mainfrom
nexthop-ai:fboss2-refuse-config-rollback-when-live

Conversation

@vybhav-nexthop

@vybhav-nexthop vybhav-nexthop commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Summary

What

fboss2-dev config rollback now refuses to run when the config the daemon is running differs from the committed git HEAD. A new global --force flag commits that running config as a snapshot first, then rolls back.

fboss2-dev config rollback [<sha>]            # refuses if live config != HEAD
fboss2-dev --force config rollback [<sha>]    # snapshots live config, then rolls back

Why: the problem

The live config can get ahead of git HEAD: someone hand-edits /etc/coop/agent.conf (or BGP's bgpcpp.conf), runs config reload, and never commits. Nothing in the CLI notices this state, and rollback destroys it.

How rollback worked before this PR (ConfigSession::rollback()):

  1. Resolve the target revision (no-arg form: the second entry of git log -- cli/cli_metadata.json).
  2. For each domain (Agent, BGP), compare the target revision's content with the file on disk. Skip the domain if they match.
  3. Write the target content to disk, reload/restart the service at the level from rolledBackActionLevels(), and commit Rollback to <sha>.

No step compares the file on disk with HEAD. Uncommitted live content is overwritten in step 3 and was never in git, so it is gone with no warning and no way back. config session diff compares session vs live, so there was also no way to see this happening.

On a test switch this happened for real: live config and HEAD differed in VLAN membership, interfaces, port state and l2LearningMode. Rolling back to HEAD dropped eth1/33/1 from vlanPorts, which led to an agent crash loop, an ASIC left with only VLANs 1 and 4095, a single broadcast domain across all ports, and dark optics while show hw-agent still reported CONFIGURED.

How: the fix

The check is in ConfigSession::rollback(), so the CLI and the DUT test runner both go through it. After resolving the target revision and before touching any file:

  1. Detect drift. For each domain, read the file the daemon actually reads (ConfigFileManager::currentPath(): the service's reported --config path, normally a symlink onto the git-tracked cli/agent.conf) and compare it to that domain's file at HEAD. Reading the daemon-facing path catches a hand edit that replaced the symlink with a regular file. A domain whose tracked file does not exist on disk is skipped: nothing is running from it.
  2. Refuse without --force. If any domain drifted, throw before writing anything. The message names the drifted domain(s), the HEAD sha, the command to inspect it (config session diff <head> current), and warns that --force will restart disruptively.
  3. Snapshot with --force (snapshotLiveDrift()):
    • Copy each drifted domain's live content onto its git-tracked path.
    • Write cli/cli_metadata.json with base = HEAD and action = DISRUPTIVE_SERVICE_RESTART for each drifted domain.
    • Commit both as Snapshot of uncommitted live config before rollback by <user>. If the commit fails, the touched files are restored and the error propagates; the running config is not changed.
  4. Roll back as before. A drifted domain is always re-applied, even if the target equals what is now on disk. The snapshot is HEAD now, so a failure later in the rollback restores to the snapshot.

Why the snapshot also commits metadata: no-arg rollback and rolledBackActionLevels() both walk git log -- cli/cli_metadata.json, which only lists commits that touch that file. Without it, a plain config rollback would skip the snapshot, and the level scan would never stop at it. With it, --force config rollback X followed by config rollback lands back on the snapshot.

Why the snapshot records a coldboot: a hand edit bypassed the CLI, so the restart level needed to undo it is unknown. l2LearningMode, part of that drift, needs a coldboot (CmdConfigL2LearningMode), and undoing it with a hitless reload is what went wrong there. So the forced rollback, and any later rollback across the snapshot, coldboots the agent. BGP still gets its normal bgpd restart.

Other changes:

  • --force is a global flag (fboss2-dev --force config rollback ...), like --detail.
  • The two rollback() overloads are merged into rollback(hostInfo, commitSha = "", force = false), with empty commitSha meaning the previous commit, so the check has one home.
  • fboss_test_runner/services/config_baseline.py passes --force on soft restore. That path runs exactly when the coop tree is dirty from test debris, so without the flag every soft restore would refuse. A dirty-tree soft restore now coldboots the agent, as the hard restore path already does.
File Change
fboss/cli/fboss2/session/ConfigSession.{h,cpp} Drift check, snapshotLiveDrift(), single rollback()
fboss/cli/fboss2/CmdGlobalOptions.{h,cpp} --force flag
fboss/cli/fboss2/commands/config/rollback/CmdConfigRollback.cpp Pass --force through
fboss/cli/fboss2/test/config/CmdConfigSessionTest.cpp 8 drift tests
fboss_test_runner/services/config_baseline.py, unittests/test_config_baseline.py --force on soft restore

Not in this PR: config commit overwrites a hand-edited live file the same way. The same check belongs there, but it changes commit behavior for every CLI user and needs its own DUT pass.

Test Plan

  • fboss2_cmd_config_test (CmdConfigSessionTest.cpp): 1186 tests passed, including all 59 ConfigSession tests. Forced-rollback tests use a StrictMock that expects exactly one agent restartService(DISRUPTIVE_SERVICE_RESTART) and no reload. Drift tests:
    [       OK ] ConfigSessionTestFixture.rollbackRefusesUncommittedLiveDrift
    [       OK ] ConfigSessionTestFixture.rollbackForcedSnapshotsUncommittedLiveDrift
    [       OK ] ConfigSessionTestFixture.rollbackNamesEveryDriftedDomain
    [       OK ] ConfigSessionTestFixture.rollbackDetectsDriftWhenSymlinkReplaced
    [       OK ] ConfigSessionTestFixture.rollbackNoArgRefusesUncommittedLiveDrift
    [       OK ] ConfigSessionTestFixture.rollbackNoArgReachesForcedSnapshot
    [       OK ] ConfigSessionTestFixture.rollbackRefusesUncommittedBgpDrift
    [       OK ] ConfigSessionTestFixture.rollbackIgnoresMissingPromotedFile
    
  • python3 -m unittest fboss_test_runner.unittests.test_config_baseline: Ran 15 tests ... OK
  • Full CMake build and unit tests of this branch rebased on current main: passed.
  • NH-4010-F, first revision of this PR (no snapshot metadata, no coldboot level). Reproduced the precondition with a CLI commit, a hand edit of the live file, and config reload:
    $ fboss2-dev config rollback 248806cc ; echo rc=$?
    Failed to rollback config: Live Agent config differs from the committed HEAD (248806cc); rolling back would discard uncommitted running config. Review the difference with 'config session diff 248806cc current', then rerun with --force (before the command: 'fboss2-dev --force config rollback ...') to commit it as a snapshot and proceed.
    rc=1
    
    $ fboss2-dev --force config rollback 248806cc ; echo rc=$?
    Warning: live Agent config differed from HEAD; committed it as snapshot ba3d0f3a before rolling back.
    Successfully rolled back. New commit: c0dcc422. Config reloaded.
    rc=0
    
    $ git log --oneline -3
    c0dcc42 Rollback to 248806cc by root
    ba3d0f3 Snapshot of uncommitted live config before rollback by root
    248806c Config commit by root
    
    Agents stayed active; show hw-agent CONFIGURED and in sync.

@vybhav-nexthop
vybhav-nexthop requested review from a team as code owners October 23, 2026 04:40
@meta-cla meta-cla Bot added the CLA Signed label Oct 23, 2026
@github-actions github-actions Bot added nexthop project:fboss2-dev FBOSS project: fboss2-dev labels Oct 23, 2026
@vybhav-nexthop

Copy link
Copy Markdown
Contributor Author

/autorebase

@rabbit-nexthop
rabbit-nexthop force-pushed the fboss2-refuse-config-rollback-when-live branch from d01ac50 to 47e14ca Compare November 10, 2026 12:02
@joseph5wu

Copy link
Copy Markdown
Contributor

@vybhav-nexthop : Can you resolve the conflicts so I can import this PR?

Signed-off-by: vybhav-nexthop <vybhav@nexthop.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants