Skip to content

[Nexthop][fboss2-dev] Add config init to start the config history from every live config - #1980

Open
vybhav-nexthop wants to merge 3 commits into
meta:mainfrom
nexthop-ai:fboss2-config-init
Open

vybhav-nexthop wants to merge 3 commits into
meta:mainfrom
nexthop-ai:fboss2-config-init

Conversation

@vybhav-nexthop

@vybhav-nexthop vybhav-nexthop commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • pre-commit run

Important

This PR is stacked on two open PRs. Please review only the last commit.

# Commit From
1 [fboss2] Refuse config rollback when live config drifted from HEAD #1829 (adds ConfigSession::snapshotLiveDrift(), which this PR refactors into the shared commitLiveConfigs())
2 Commit the bootstrap configs as the first fboss2 config history commit #1897 (adds commit_bootstrap_config() to fboss_init.sh, which this PR replaces)
3 [fboss2] Add config init to start the config history from every live config this PR

Commits 1 and 2 are here only so this branch builds; they are the same changes as in #1829 and #1897. Once #1829 and #1897 land, I will rebase this PR down to commit 3 only.

Note on #1897: this PR replaces the shell git commands that #1897 adds with a call to fboss2-dev config init. If reviewers prefer, #1897 and this PR can be reviewed together, since the final fboss_init.sh is what this PR leaves.

Summary

fboss2-dev config rollback could delete bgpd's config file, after which bgpd crash-looped. This happened because the first commit in fboss2's config history was missing the BGP config.

This PR adds a command, fboss2-dev config init, that saves every config fboss2 manages into the history. The boot script runs it on every boot, so the history always holds every config, and a rollback can no longer delete one.

Background: what the config history is

fboss2 keeps a history of the switch's configs, so that an operator can undo a change.

  • The history is a git repository in /etc/coop.

  • fboss2 currently manages two configs: the agent config and the BGP config. Each exists twice:

    Service File the service reads Copy saved in the history
    agent (fboss_sw_agent) /etc/coop/agent.conf /etc/coop/cli/agent.conf
    BGP (bgpd) /etc/coop/bgpcpp.conf /etc/coop/bgpcpp/bgpcpp.conf
  • When an operator changes config with fboss2 (fboss2-dev config ... then fboss2-dev config session commit), fboss2 applies the change to the service and adds a git commit with the new files.

  • fboss2-dev config rollback <commit> makes each config look exactly as it did in that commit, then reloads or restarts the services whose config changed.

The important rule: if the commit you roll back to does not contain a config at all, rollback takes that to mean "this config did not exist then" and removes it. That is intended: it is how you undo adding a config. But it means a config missing from the history by mistake gets deleted by a rollback.

The bug

The first commit in the history is made at boot by the boot script, fboss_init.sh (added in #1897). That script has its own git commands, and its own list of files to commit. Before the BGP config was added to that list, this happened:

  1. The switch boots. The first commit holds the agent config, but not the BGP config.
  2. An operator changes a BGP setting with fboss2. That makes a second commit, which does hold the BGP config.
  3. The operator rolls back to the first commit to undo the change.
  4. The first commit has no BGP config, so rollback deletes /etc/coop/bgpcpp.conf and restarts bgpd.
  5. bgpd cannot start without a config file (Could not read config file) and crash-loops.

#1897 includes the BGP config in its list, which fixes this case. But the list still has to be kept in sync with fboss2 by hand: every config fboss2 learns to manage in the future would need the same boot script change, or the bug comes back for that config.

The fix

Move the job of making the first commit out of the boot script and into fboss2 itself, which already knows the full list of configs it manages.

The new command fboss2-dev config init compares each config on the box with the latest commit, and saves whatever the history does not have yet:

Situation What config init does
No history yet (first boot) Makes the first commit, containing every config on the box.
A config the history has never seen (e.g. a service installed later) Commits it.
A config that was edited by hand, outside fboss2 Commits the edit.
Everything already matches Nothing.

It only saves files. It never restarts or reloads a service, so it is safe to run at boot and to run again at any time.

The boot script's commit_bootstrap_config now just runs fboss2-dev config init, on every boot, instead of its own git commands.

Details for reviewers

Where the first commit is made. The "no history yet" case lives in ConfigSession::initializeGit(), which runs whenever any fboss2 config command finds no history. It now copies each service's config from the file the service reads (column 2 above) into the history (column 3) before making the first commit. So the first commit is complete whichever fboss2 command runs first, not only config init. No service is contacted in this step.

The other cases are in the new ConfigSession::initializeHistory(). When it finds differences, it saves them all in a single commit.

Restart level stored with each commit. Each commit also updates cli/cli_metadata.json, which records for each service how disruptive applying that change was: HITLESS (config reload), SERVICE_RESTART (agent warmboot, or bgpd restart) or DISRUPTIVE_SERVICE_RESTART (agent coldboot). Rollback reads these to decide how to apply the undo: it uses the most disruptive level among the commits it undoes. config init records:

Why config init commits always touch the metadata file. Rollback finds the commits it undoes by walking the git log of cli/cli_metadata.json. A commit that did not change that file would be invisible to it. config init sets the metadata's base field (the commit it was made on top of) to the previous commit, so the file always changes.

Finding the file a service reads. config init first asks the running service for its --config path. If the service is not running (as at boot, before the agents start), it uses the path recorded in the metadata. If no path is recorded either, it uses the path where the image installs the config. This new fallback is a field on each config entry, ConfigDomain::defaultCurrentPath.

Shared code. The "write these configs into the history and commit" step is now one function, ConfigSession::commitLiveConfigs(), used by both config init and the existing rollback --force snapshot (snapshotLiveDrift(), from #1829).

Boot script.

  • [Nexthop][distro] Commit the bootstrap configs as the first fboss2 config history commit #1897 makes the commit only when that boot has just created agent.conf (the AGENT_CONF_CREATED check). That check is gone, and so is the variable, which had no other reader. Because config init does nothing when the history already matches, running it every boot is safe, and it also picks up a service installed after the first boot.
  • A failure is logged and boot continues; fboss2 still starts the history itself the first time an operator uses it.
  • It sets HOME and USER if unset, because fboss2 needs them and a boot-time service may not have them.

Test Plan

Unit tests: cmd_config_test: all pass on our internal tree. New tests:

  • configInitFirstCommitHoldsEveryLiveConfig: with no history, the first commit holds both the agent and the BGP config; running init afterwards makes no commit; the files the services read are left as plain files.
  • configInitAdoptsUntrackedConfig: a BGP config that appears after the history started is committed once with no restart level; running init again makes no commit.
  • configInitCommitsHandEditAsDisruptive: a hand edit is committed with DISRUPTIVE_SERVICE_RESTART, on top of the previous commit.

This branch (on top of current main) was also built and its unit tests run with the OSS CMake build.

On a switch: a Minipack3, freshly imaged with an FBOSS main build, with this change's fboss2-dev installed at /usr/local/bin/fboss2-dev.

  1. History from the image, holding only the agent config: config init committed the BGP config (md5 identical to the file the bgpd package installs), with no restart level. Running it again made no commit.
  2. First boot, using this PR's commit_bootstrap_config: after deleting the history (.git, cli/, bgpcpp/), ran the function with no HOME/USER set (env -i). Result: one Initial commit holding cli/agent.conf, bgpcpp/bgpcpp.conf and cli/cli_metadata.json, byte-identical to the files the services read; the repository is group-shared and its git identity is fboss-cli. As with every fboss2 commit, the commit author name is the user fboss2 runs as ($USER, which the boot function defaults to root); before this PR the boot script's commit used fboss-cli for both.
  3. Same, with fboss_sw_agent and bgpd stopped, as they are at real first boot: one complete commit in 0.1 s, no waiting on the services, and running it again made no commit.
  4. The bgpd crash-loop scenario: set BGP graceful-restart-time 222, session commit, then --force config rollback to the first commit. bgpd stayed up (NRestarts=0, no Could not read config file), and graceful-restart went back to the package default of 120.
  5. Hand edit of /etc/coop/agent.conf: committed with AGENT: DISRUPTIVE_SERVICE_RESTART. Running it again made no commit.
  6. Full fboss2 integration suite (fboss2_integration_test): 256 passed, 11 skipped, 1 failed. The failure, ConfigPfcTest.BufferPoolPriorityGroupAndPortPfc (hw_agent SIGABRT in SAI setAttribute), is a known-bad test on that platform and SDK. bgpd did not abort during the run.

Not tested on a switch:

  • Three changes made after the switch run, covered by unit tests only: moving the install path into ConfigDomain::defaultCurrentPath, removing the AGENT_CONF_CREATED check, and the commit message text (Config init: commit live <Domain> config by <user>).
  • A full image build containing this fboss_init.sh. Step 2 ran the PR's boot function by hand on the switch, not as part of a real boot of a new image.

Not in this PR

  • Rollback still removes a config that the target commit does not contain. That is intended (see Background); this PR makes sure the history never misses a config.
  • A config restored by rollback loses its trailing newline (because git show output is trimmed). Unchanged here.

Review Findings

Pre-publication review of the last commit: no issues found above the 0.7 confidence threshold.

vybhav-nexthop and others added 3 commits November 16, 2026 04:32
Signed-off-by: vybhav-nexthop <vybhav@nexthop.ai>
fboss2 starts its config history in /etc/coop at the first `fboss2-dev
config` session, from whatever agent.conf holds then. Anything written
before that (provisioning, test tooling, hand edits) is never recorded,
so the first commit is not reliably the agent.conf the image installed.

On the boot that creates agent.conf, fboss_init.sh now starts the
history as ConfigSession::initializeGit would and commits the bootstrap
as cli/agent.conf with cli/cli_metadata.json ("Initial commit: bootstrap
agent.conf"). bgpd's packaged bgpcpp.conf, when it is a plain file, goes
into the same commit as bgpcpp/bgpcpp.conf: fboss2's rollback removes a
domain absent at the target commit, so a rollback to an agent-only
bootstrap would delete bgpd's config and leave bgpd aborting. The commit
is skipped when agent.conf already existed or the history has commits,
and a failure is logged without stopping boot, since fboss2's own lazy
init still covers that case. fboss2 then skips its init and commits on
top, so the first commit stays the bootstrap.

The fboss2 integration test runner records the history's HEAD as each
suite's baseline and rolls back to it afterwards, but it seeded the
tracked configs from the live files only when it created the repository
itself. With the repository now created at boot, capture() copies any
daemon config that is still a plain file and differs from its tracked
copy into the repository and commits it as the baseline snapshot, so the
baseline is what the daemons run (including the test agent config the
runner copies over the plain agent.conf) and a rollback to it keeps
bgpd's config.

Co-authored-by: Lotus Fenn <lotus@nexthop.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Earnest Phyo <earnest@nexthop.ai>
…config

fboss_init.sh started the /etc/coop config history with hand-written git
commands and its own list of files to commit. Every new config domain would
need the same shell change, and a domain missing from the first commit gets
deleted by a rollback to it (rollback removes a config the target commit does
not contain; bgpd then crash-loops without its config file).

Add `fboss2-dev config init`, which uses ConfigSession's own domain list:
- On an empty repo, initializeGit() now seeds each domain's desired file from
  the service's config at its default path (/etc/coop/agent.conf,
  /etc/coop/bgpcpp.conf) before the first commit, so that commit holds every
  config on the box. No service is contacted.
- With history, initializeHistory() commits, in one commit, every domain whose
  live config differs from HEAD. A config committed for the first time records
  no action level; a hand-edited tracked config records
  DISRUPTIVE_SERVICE_RESTART, as a --force rollback snapshot does. Rerunning
  with nothing changed makes no commit.
- It never restarts or reloads a service and never stages a session.

The default path of each service's config is a new ConfigDomain field,
defaultCurrentPath. The commit path is shared with snapshotLiveDrift() via
commitLiveConfigs().

fboss_init.sh now runs `fboss2-dev config init` on every boot instead of
running git, and no longer gates it on agent.conf having been created on this
boot: config init is idempotent, so running it every boot also commits a
config installed after the history started (e.g. a new service), and does
nothing otherwise. AGENT_CONF_CREATED had no other reader and is removed.

Signed-off-by: Vybhav <vybhav@nexthop.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants