[mjcarroll]

The .dsv descriptors are colcon's own representation of an environment change. They arrived in 2019 (cbee991) together with a pure-Python evaluator generated into every install prefix (bd55f3f); sh and bat were moved onto that evaluator for their prefix scripts within days (bccec1d, 8fc78e3), and #244 made the descriptor authoritative even where no shell-specific script exists. The per-shell hooks are renderings of the same data.

generate_command_environment is the one consumer that never moved. It still writes a script which dot-sources each dependency's per-shell rendering plus all of its hooks and spawns a shell to dump the result. This implements it on DsvShell instead, at a priority above the shell extensions, and falls back to them whenever a descriptor references a script that has no descriptor of its own, or a dependency provides no package.dsv.

Not spawning a shell is also considerably cheaper. On a 363 package ROS 2 workspace on Windows the shell path costs roughly 438 + 60.5 * n_deps ms per package and the median package has 95 dependencies; evaluating the descriptors instead costs about 390 ms per package, measured over all 32664 dependency pairs of that workspace.

package	dependencies	shell	descriptors
rcutils	53	11.2 s	4.8 s
rviz_default_plugins	182	22.7 s	5.5 s

[codecov]

Codecov Report

❌ Patch coverage is 87.17949% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.35%. Comparing base (6bfb100) to head (30925d9).

Files with missing lines	Patch %	Lines
colcon_core/shell/dsv.py	87.17%	13 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #748      +/-   ##
==========================================
+ Coverage   87.25%   87.35%   +0.09%     
==========================================
  Files          75       75              
  Lines        4692     4808     +116     
  Branches      817      844      +27     
==========================================
+ Hits         4094     4200     +106     
- Misses        468      476       +8     
- Partials      130      132       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[cottsay]

Thanks for the work here.

In addition to promoting DsvShell to be a fully-featured and higher priority shell than the "native" primary shells, this change introduces some aggressive caching that I'm a bit concerned about (namely _flattened_files). While this mechanism should properly detect when the .dsv itself changes, DSV effectively supports chaining that wouldn't be caught by this invalidation strategy.

Given the impact of spawning subprocesses in Windows, I'm skeptical that this aggressive caching is even worth the risk of invalid caches causing cryptic and difficult to reproduce bugs. Please try simplifying the change to focus only on the high-impact DSV capabilities and priority changes so we have the numbers behind that specific part of the change. If we decide that there is an opportunity for some package-level or hook-level caching, I'd rather see that explored in a way that would provide the benefit to all shells, not just .dsv.

It's probably worth noting here that the type of workspace plays a huge role in describing the impact of BatShell's poor performance. In many of our CI builds, we intentionally use isolated workspaces to help us uncover dependency issues. Merged workspaces, like the ones used to distribute binaries like the comprehensive .zip archive, would not see even close to the same penalty.

[mjcarroll]

Thanks for the early look, wasn't quite ready for another review yet, but this is good feedback, I will incorporate.

[mjcarroll]

Okay, I dropped the caching bits. The bugs are too subtle to be worth it.

Also, agreed on workspace type. My profiling is done with merge-install, which means all the scripts should be effectively no-op, right? An isolated workspace would have the same number of scripts, but they would actually be doing things in each script.

[cottsay]

Thanks for slimming down the change.

Looking pretty good, just a couple of comments and some manual testing to do.

[mjcarroll]

Followed up on the workspace-type question with actual data instead of just reasoning about it.

Built a full rolling workspace from ros2.repos (367 packages, default isolated-install) and timed source install/setup.bash under stock colcon-core vs. this branch, 5 runs each:

• stock colcon-core: ~0.3s (0.27–0.48s)
• this branch: ~0.3s (0.26–0.43s)

No measurable difference on Linux.

[cottsay]

'Lotta em-dash in there...

[mikepurvis]

I'm surprised it didn't move the needle at all for you. On Nix, I did something similar with building the environment using the dsvs, in part because sourcing hundreds of local_install.sh files got to be very slow (like, multiple seconds), as briefly discussed here:

https://github.com/clearpathrobotics/nix-ros-base/tree/devel/lib/setup#colcon-env-setup

I think the big time eater was the envvar dedup, as each new local_setup would iterate longer and longer path lists, checking each entry to make sure it wasn't adding something already present. That's an N^2 operation so it really adds up past a certain point.

[mjcarroll]

'Lotta em-dash in there...

Yeah, moved to a new computer so 👷 has gotten a little trigger happy.

I'm going to retest this on Windows though, because I think it will be more obvious there.

[mjcarroll]

Win11, colcon-core 0.17.1 vs this branch (537b783)

	stock	this PR
run 1	3764 s	1567 s
run 2	2349 s	1710 s
mean	51 min	27 min

generate_command_environment alone, 140 deps:

stock ps1 10.8 s (merge) / 7.4 s (isolated)
bat 22.5 s / 17.9 s
dsv 241 ms / 125 ms.

[mjcarroll]

running this through a Windows test in CI:

plus a baseline

[mjcarroll]

• colcon build phase: 9302.7s (PR) vs 10562.9s (baseline): 11.9% faster
• colcon test phase: 2330.3s (PR) vs 3313.7s (baseline): 29.7% faster