Repository navigation
[Nexthop][fboss2-dev] Add config init to start the config history from every live config - #1980
Open
vybhav-nexthop wants to merge 3 commits into
Open
vybhav-nexthop wants to merge 3 commits into
vybhav-nexthop wants to merge 3 commits into
Conversation
Signed-off-by: vybhav-nexthop <vybhav@nexthop.ai>
fboss2 starts its config history in /etc/coop at the first `fboss2-dev
config` session, from whatever agent.conf holds then. Anything written
before that (provisioning, test tooling, hand edits) is never recorded,
so the first commit is not reliably the agent.conf the image installed.
On the boot that creates agent.conf, fboss_init.sh now starts the
history as ConfigSession::initializeGit would and commits the bootstrap
as cli/agent.conf with cli/cli_metadata.json ("Initial commit: bootstrap
agent.conf"). bgpd's packaged bgpcpp.conf, when it is a plain file, goes
into the same commit as bgpcpp/bgpcpp.conf: fboss2's rollback removes a
domain absent at the target commit, so a rollback to an agent-only
bootstrap would delete bgpd's config and leave bgpd aborting. The commit
is skipped when agent.conf already existed or the history has commits,
and a failure is logged without stopping boot, since fboss2's own lazy
init still covers that case. fboss2 then skips its init and commits on
top, so the first commit stays the bootstrap.
The fboss2 integration test runner records the history's HEAD as each
suite's baseline and rolls back to it afterwards, but it seeded the
tracked configs from the live files only when it created the repository
itself. With the repository now created at boot, capture() copies any
daemon config that is still a plain file and differs from its tracked
copy into the repository and commits it as the baseline snapshot, so the
baseline is what the daemons run (including the test agent config the
runner copies over the plain agent.conf) and a rollback to it keeps
bgpd's config.
Co-authored-by: Lotus Fenn <lotus@nexthop.ai>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Earnest Phyo <earnest@nexthop.ai>
…config fboss_init.sh started the /etc/coop config history with hand-written git commands and its own list of files to commit. Every new config domain would need the same shell change, and a domain missing from the first commit gets deleted by a rollback to it (rollback removes a config the target commit does not contain; bgpd then crash-loops without its config file). Add `fboss2-dev config init`, which uses ConfigSession's own domain list: - On an empty repo, initializeGit() now seeds each domain's desired file from the service's config at its default path (/etc/coop/agent.conf, /etc/coop/bgpcpp.conf) before the first commit, so that commit holds every config on the box. No service is contacted. - With history, initializeHistory() commits, in one commit, every domain whose live config differs from HEAD. A config committed for the first time records no action level; a hand-edited tracked config records DISRUPTIVE_SERVICE_RESTART, as a --force rollback snapshot does. Rerunning with nothing changed makes no commit. - It never restarts or reloads a service and never stages a session. The default path of each service's config is a new ConfigDomain field, defaultCurrentPath. The commit path is shared with snapshotLiveDrift() via commitLiveConfigs(). fboss_init.sh now runs `fboss2-dev config init` on every boot instead of running git, and no longer gates it on agent.conf having been created on this boot: config init is idempotent, so running it every boot also commits a config installed after the history started (e.g. a new service), and does nothing otherwise. AGENT_CONF_CREATED had no other reader and is removed. Signed-off-by: Vybhav <vybhav@nexthop.ai>
Open
2 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pre-submission checklist
pip install -r requirements-dev.txt && pre-commit installpre-commit runImportant
This PR is stacked on two open PRs. Please review only the last commit.
[fboss2] Refuse config rollback when live config drifted from HEADConfigSession::snapshotLiveDrift(), which this PR refactors into the sharedcommitLiveConfigs())Commit the bootstrap configs as the first fboss2 config history commitcommit_bootstrap_config()tofboss_init.sh, which this PR replaces)[fboss2] Add config init to start the config history from every live configCommits 1 and 2 are here only so this branch builds; they are the same changes as in #1829 and #1897. Once #1829 and #1897 land, I will rebase this PR down to commit 3 only.
Note on #1897: this PR replaces the shell git commands that #1897 adds with a call to
fboss2-dev config init. If reviewers prefer, #1897 and this PR can be reviewed together, since the finalfboss_init.shis what this PR leaves.Summary
fboss2-dev config rollbackcould delete bgpd's config file, after which bgpd crash-looped. This happened because the first commit in fboss2's config history was missing the BGP config.This PR adds a command,
fboss2-dev config init, that saves every config fboss2 manages into the history. The boot script runs it on every boot, so the history always holds every config, and a rollback can no longer delete one.Background: what the config history is
fboss2 keeps a history of the switch's configs, so that an operator can undo a change.
The history is a git repository in
/etc/coop.fboss2 currently manages two configs: the agent config and the BGP config. Each exists twice:
fboss_sw_agent)/etc/coop/agent.conf/etc/coop/cli/agent.confbgpd)/etc/coop/bgpcpp.conf/etc/coop/bgpcpp/bgpcpp.confWhen an operator changes config with fboss2 (
fboss2-dev config ...thenfboss2-dev config session commit), fboss2 applies the change to the service and adds a git commit with the new files.fboss2-dev config rollback <commit>makes each config look exactly as it did in that commit, then reloads or restarts the services whose config changed.The important rule: if the commit you roll back to does not contain a config at all, rollback takes that to mean "this config did not exist then" and removes it. That is intended: it is how you undo adding a config. But it means a config missing from the history by mistake gets deleted by a rollback.
The bug
The first commit in the history is made at boot by the boot script,
fboss_init.sh(added in #1897). That script has its own git commands, and its own list of files to commit. Before the BGP config was added to that list, this happened:/etc/coop/bgpcpp.confand restarts bgpd.Could not read config file) and crash-loops.#1897 includes the BGP config in its list, which fixes this case. But the list still has to be kept in sync with fboss2 by hand: every config fboss2 learns to manage in the future would need the same boot script change, or the bug comes back for that config.
The fix
Move the job of making the first commit out of the boot script and into fboss2 itself, which already knows the full list of configs it manages.
The new command
fboss2-dev config initcompares each config on the box with the latest commit, and saves whatever the history does not have yet:config initdoesIt only saves files. It never restarts or reloads a service, so it is safe to run at boot and to run again at any time.
The boot script's
commit_bootstrap_confignow just runsfboss2-dev config init, on every boot, instead of its own git commands.Details for reviewers
Where the first commit is made. The "no history yet" case lives in
ConfigSession::initializeGit(), which runs whenever any fboss2 config command finds no history. It now copies each service's config from the file the service reads (column 2 above) into the history (column 3) before making the first commit. So the first commit is complete whichever fboss2 command runs first, not onlyconfig init. No service is contacted in this step.The other cases are in the new
ConfigSession::initializeHistory(). When it finds differences, it saves them all in a single commit.Restart level stored with each commit. Each commit also updates
cli/cli_metadata.json, which records for each service how disruptive applying that change was:HITLESS(config reload),SERVICE_RESTART(agent warmboot, or bgpd restart) orDISRUPTIVE_SERVICE_RESTART(agent coldboot). Rollback reads these to decide how to apply the undo: it uses the most disruptive level among the commits it undoes.config initrecords:DISRUPTIVE_SERVICE_RESTARTfor a hand edit, since nobody knows what undoing it needs. This matches whatconfig rollback --force([Nexthop][fboss2-dev] Refuse config rollback when live config drifted from HEAD #1829) already records when it saves a hand edit before rolling back.Why
config initcommits always touch the metadata file. Rollback finds the commits it undoes by walking the git log ofcli/cli_metadata.json. A commit that did not change that file would be invisible to it.config initsets the metadata'sbasefield (the commit it was made on top of) to the previous commit, so the file always changes.Finding the file a service reads.
config initfirst asks the running service for its--configpath. If the service is not running (as at boot, before the agents start), it uses the path recorded in the metadata. If no path is recorded either, it uses the path where the image installs the config. This new fallback is a field on each config entry,ConfigDomain::defaultCurrentPath.Shared code. The "write these configs into the history and commit" step is now one function,
ConfigSession::commitLiveConfigs(), used by bothconfig initand the existingrollback --forcesnapshot (snapshotLiveDrift(), from #1829).Boot script.
agent.conf(theAGENT_CONF_CREATEDcheck). That check is gone, and so is the variable, which had no other reader. Becauseconfig initdoes nothing when the history already matches, running it every boot is safe, and it also picks up a service installed after the first boot.HOMEandUSERif unset, because fboss2 needs them and a boot-time service may not have them.Test Plan
Unit tests:
cmd_config_test: all pass on our internal tree. New tests:configInitFirstCommitHoldsEveryLiveConfig: with no history, the first commit holds both the agent and the BGP config; runninginitafterwards makes no commit; the files the services read are left as plain files.configInitAdoptsUntrackedConfig: a BGP config that appears after the history started is committed once with no restart level; runninginitagain makes no commit.configInitCommitsHandEditAsDisruptive: a hand edit is committed withDISRUPTIVE_SERVICE_RESTART, on top of the previous commit.This branch (on top of current
main) was also built and its unit tests run with the OSS CMake build.On a switch: a Minipack3, freshly imaged with an FBOSS
mainbuild, with this change'sfboss2-devinstalled at/usr/local/bin/fboss2-dev.config initcommitted the BGP config (md5 identical to the file the bgpd package installs), with no restart level. Running it again made no commit.commit_bootstrap_config: after deleting the history (.git,cli/,bgpcpp/), ran the function with noHOME/USERset (env -i). Result: oneInitial commitholdingcli/agent.conf,bgpcpp/bgpcpp.confandcli/cli_metadata.json, byte-identical to the files the services read; the repository is group-shared and its git identity isfboss-cli. As with every fboss2 commit, the commit author name is the user fboss2 runs as ($USER, which the boot function defaults toroot); before this PR the boot script's commit usedfboss-clifor both.fboss_sw_agentandbgpdstopped, as they are at real first boot: one complete commit in 0.1 s, no waiting on the services, and running it again made no commit.graceful-restart-time 222,session commit, then--force config rollbackto the first commit. bgpd stayed up (NRestarts=0, noCould not read config file), and graceful-restart went back to the package default of 120./etc/coop/agent.conf: committed withAGENT: DISRUPTIVE_SERVICE_RESTART. Running it again made no commit.fboss2_integration_test): 256 passed, 11 skipped, 1 failed. The failure,ConfigPfcTest.BufferPoolPriorityGroupAndPortPfc(hw_agent SIGABRT in SAIsetAttribute), is a known-bad test on that platform and SDK. bgpd did not abort during the run.Not tested on a switch:
ConfigDomain::defaultCurrentPath, removing theAGENT_CONF_CREATEDcheck, and the commit message text (Config init: commit live <Domain> config by <user>).fboss_init.sh. Step 2 ran the PR's boot function by hand on the switch, not as part of a real boot of a new image.Not in this PR
git showoutput is trimmed). Unchanged here.Review Findings
Pre-publication review of the last commit: no issues found above the 0.7 confidence threshold.