Indirect agent connection improvements by sureshanaparti · Pull Request #13345 · apache/cloudstack

sureshanaparti · 2026-06-04T06:23:25Z

Description

Continued from #13028

This PR improves the Indirect agent connection handling, has the following improvements.

Enhances the Host connecting logic to avoid connecting storm (where Agent opens multiple sockets against Management Server).
Implements HostConnectProcess task where Host upon connection checks whether lock is available, traces Host connecting progress, status and timeout.
Introduces AgentConnectStatusCommand, where Host checks whether lock for the Host is available (i.e. "previous" connect process is finished).
Implementes logic to check whether Management Server has lock against Host (exposed MySQL DB lock presence via API)
Removes synchronization on Host disconnect process, double-disconnect logic in clustered Management Server environment, added early removal from ping map (in case of combination ping timeout delay + synchronized disconnect process the Agent Manager submits more disconnect requests)
Introduces parameterized connection and status check timeouts
Implements backoff algorithm abstraction - can be used either constant backoff timeout or exponential with jitter to wait between connection Host attempts to Management Server
Implements ServerAttache to be used on the Agent side of communication (similar to Attache on Management Server side)
Enhances/Adds logs significantly to Host Agent and Agent Manager logic to trace Host connecting and disconnecting process, including ids, names, context UUIDs and timings (how much time took overall initialization/deinitialization)
Adds logs to communication between Management Servers (PDU requests)
Adds DB indexes to improve search performance, uses IDEMPOTENT_ADD_INDEX for safer DB schema updates
Dedupes agent connection requests. Observed scenarios where on agent with old agent code send a storm of connection requests. In such a case, to ensure other agents are not impacted, deduping the connection requests.
Remove IP address hard dependency for management server communication. It allows the management.server.address config to be a list of hostnames instead of static IPs. Rebalancing, cluster propagation, alerts, and stats collection all detect the format at runtime via ManagementServerAddressUtil and use the matching lookup (listNonUpStateMsHostnames vs listNonUpStateMsIPs) when computing the "avoid" list sent to agents. Hostnames survive MS restarts and IP reassignments where static IPs do not, which is required in deployment environments where management server instances can be rescheduled or replaced.
Propagates instance (vm, volume, etc) uuid to all operations. The uuid is propagated from MS to agent communications and to all async jobs.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Checked the indirect agents connections (and re-connections) with KVM hosts, SSVM & CPVM.

How did you try to break this feature and the system with this change?

sureshanaparti · 2026-06-04T06:25:27Z

@blueorangutan package

blueorangutan · 2026-06-04T06:26:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-06-04T06:30:59Z

Codecov Report

❌ Patch coverage is 27.11592% with 1817 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.75%. Comparing base (4dae995) to head (92a0c48).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
agent/src/main/java/com/cloud/agent/Agent.java	15.48%	309 Missing and 2 partials ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	42.11%	270 Missing and 31 partials ⚠️
...t/src/main/java/com/cloud/agent/ServerAttache.java	8.19%	277 Missing and 3 partials ⚠️
.../main/java/com/cloud/agent/HostConnectProcess.java	25.23%	159 Missing and 1 partial ⚠️
...c/main/java/com/cloud/utils/nio/NioConnection.java	25.92%	87 Missing and 13 partials ⚠️
...s/src/main/java/com/cloud/utils/nio/NioClient.java	18.36%	79 Missing and 1 partial ⚠️
...b/src/main/java/com/cloud/utils/db/GlobalLock.java	15.66%	51 Missing and 19 partials ⚠️
...main/java/com/cloud/agent/SynchronousListener.java	0.00%	47 Missing ⚠️
...he/cloudstack/threadcontext/ThreadContextUtil.java	9.52%	36 Missing and 2 partials ⚠️
...cloud/agent/manager/ClusteredAgentManagerImpl.java	18.60%	35 Missing ⚠️
... and 33 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13345      +/-   ##
============================================
+ Coverage     20.67%   20.75%   +0.07%     
- Complexity    21727    21884     +157     
============================================
  Files          6492     6505      +13     
  Lines        590737   592595    +1858     
  Branches      72448    72665     +217     
============================================
+ Hits         122129   122976     +847     
- Misses       455335   456259     +924     
- Partials      13273    13360      +87

Flag	Coverage Δ
uitests	`3.88% <ø> (ø)`
unittests	`22.05% <27.11%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR is a broad refactor/enhancement of indirect agent connection handling across both Management Server and Agent sides. It introduces a host-side connect/status-check process, adds configurable backoff and host-status heuristics, and expands logging/lock-check mechanisms to reduce reconnect storms and improve observability.

Changes:

Adds agent-side connection orchestration (status polling + startup submission) and a server-side AgentConnectStatus* command/answer pair to coordinate connect progress.
Introduces configurable backoff (factory + exponential-with-jitter implementation) and propagates configuration from Management Server to Agent during startup.
Refactors NIO connection lifecycle and adds supporting utilities/logging, plus DB lock availability checks and DB indexes for performance.

Reviewed changes

Copilot reviewed 47 out of 47 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
utils/src/test/java/com/cloud/utils/testcase/NioTest.java	Test logging cleanup (exception logging).
utils/src/test/java/com/cloud/utils/backoff/impl/ConstantTimeBackoffTest.java	Updates config key name for constant backoff.
utils/src/main/java/com/cloud/utils/nio/NioServer.java	NIO server init tweaks and docstring.
utils/src/main/java/com/cloud/utils/nio/NioConnection.java	Connection lifecycle refactor, selector loop/logging, reject logic.
utils/src/main/java/com/cloud/utils/nio/NioClient.java	Client connect/handshake logging and expanded cleanup.
utils/src/main/java/com/cloud/utils/nio/Link.java	Adds local-port tracking, richer toString, termination helpers/logging changes.
utils/src/main/java/com/cloud/utils/nio/HandlerFactory.java	Changes new-connection registration API to `InetSocketAddress`.
utils/src/main/java/com/cloud/utils/net/NetUtils.java	Null guard in hostname-to-IP resolution helper.
utils/src/main/java/com/cloud/utils/LogUtils.java	Adds host logging helper with optional reverse lookup.
utils/src/main/java/com/cloud/utils/exception/CSExceptionErrorCode.java	Adds error-code mapping for `ConnectionException`.
utils/src/main/java/com/cloud/utils/DateUtil.java	Adds `formatMillis` duration formatter.
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoffMBean.java	New MBean interface for exponential-jitter backoff.
utils/src/main/java/com/cloud/utils/backoff/impl/ExponentialWithJitterBackoff.java	New exponential-with-jitter backoff implementation.
utils/src/main/java/com/cloud/utils/backoff/impl/ConstantTimeBackoff.java	Migrates to namespaced config keys + exposes configuration.
utils/src/main/java/com/cloud/utils/backoff/BackoffFactory.java	New backoff factory for selecting/configuring algorithms.
utils/src/main/java/com/cloud/utils/backoff/BackoffAlgorithm.java	Adds `getConfiguration()` API for propagating settings.
server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java	Adds notes/timeout tweaks around agent migration command dispatch.
framework/db/src/main/java/com/cloud/utils/db/GlobalLock.java	Refactors lock bookkeeping/logging and adds lock-availability query.
framework/db/src/main/java/com/cloud/utils/db/DbUtil.java	Adds `IS_FREE_LOCK` helper and improves logging around lock DB connections.
framework/config/src/main/java/org/apache/cloudstack/framework/config/ConfigKeyUtil.java	New utility to parse `key=value;...` configuration strings.
framework/cluster/src/main/java/com/cloud/cluster/dao/ManagementServerHostDaoImpl.java	Adds DAO method to list all MS hosts including removed.
framework/cluster/src/main/java/com/cloud/cluster/dao/ManagementServerHostDao.java	Adds DAO API for “including removed” listing.
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletImpl.java	Adds request logging and more detailed RemoteException messages.
framework/cluster/src/main/java/com/cloud/cluster/ClusterServiceServletHttpHandler.java	Adds debug logging of inbound request line/body.
engine/schema/src/main/resources/META-INF/db/schema-42210to42300.sql	Adds indexes via idempotent add-index procedure calls.
engine/schema/src/main/resources/META-INF/db/procedures/cloud.idempotent_add_index.sql	Adds/defines `IDEMPOTENT_ADD_INDEX` procedure.
engine/orchestration/src/test/java/com/cloud/agent/manager/ClusteredAgentManagerImplTest.java	Expands tests for disconnect broadcast behavior and lock usage.
engine/orchestration/src/test/java/com/cloud/agent/manager/AgentManagerImplTest.java	Adds tests for config keys and deregister/disconnect behaviors.
engine/orchestration/src/main/java/com/cloud/agent/manager/ClusteredAgentManagerImpl.java	Refactors attache creation/removal and event handling behavior.
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentManagerImpl.java	Major connect/disconnect refactor, new config keys, status checks, backoff propagation.
engine/orchestration/src/main/java/com/cloud/agent/manager/AgentAttache.java	Cancels listener alarm futures on unregister/cancel paths.
core/src/main/java/org/apache/cloudstack/threadcontext/ThreadContextUtil.java	New helper for propagating Log4j ThreadContext across threads.
core/src/main/java/com/cloud/resource/ServerResource.java	Changes default `isExitOnFailures()` behavior.
core/src/main/java/com/cloud/agent/transport/Request.java	Improves JSON deserialization error logging.
core/src/main/java/com/cloud/agent/api/StartupAnswer.java	Adds params + agent-side status-check delay transport fields.
core/src/main/java/com/cloud/agent/api/AgentConnectStatusCommand.java	New command for host connect-status checks.
core/src/main/java/com/cloud/agent/api/AgentConnectStatusAnswer.java	New answer carrying lock availability and host status.
agent/src/test/java/com/cloud/agent/HostConnectProcessTest.java	New test around scheduling the agent connect process.
agent/src/test/java/com/cloud/agent/AgentTest.java	Updates tests for new link/logging and reconnect helpers.
agent/src/main/java/com/cloud/agent/SynchronousListener.java	New blocking listener for synchronous waits on answers.
agent/src/main/java/com/cloud/agent/ServerListener.java	New listener interface for agent-side ServerAttache callbacks.
agent/src/main/java/com/cloud/agent/ServerAttache.java	New agent-side counterpart to MS Attache for command/answer flow.
agent/src/main/java/com/cloud/agent/properties/AgentProperties.java	Adds new agent properties for async timeouts and status-check delay.
agent/src/main/java/com/cloud/agent/IAgentShell.java	Adds setter for dynamically updated backoff algorithm.
agent/src/main/java/com/cloud/agent/HostConnectProcess.java	New connect/status-check orchestration loop on the agent side.
agent/src/main/java/com/cloud/agent/AgentShell.java	Uses backoff factory + persists configuration; relaxes version handling.
agent/src/main/java/com/cloud/agent/Agent.java	Major reconnect/startup refactor; integrates HostConnectProcess and ServerAttache.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

blueorangutan · 2026-06-04T07:20:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18146

sureshanaparti · 2026-06-04T08:28:33Z

@blueorangutan package

blueorangutan · 2026-06-04T08:30:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated 7 comments.

blueorangutan · 2026-06-04T09:37:24Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18149

sureshanaparti · 2026-06-04T19:01:41Z

@blueorangutan package

blueorangutan · 2026-06-04T19:04:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-06-04T19:57:33Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18154

bernardodemarco · 2026-06-05T22:06:10Z

+/**
+ * Command to check status of {@link StartupCommand} from the Agent.
+ *
+ * @author mprokopchuk
+ */


These @author fields are not necessary, right?

Suggested change

/**

* Command to check status of {@link StartupCommand} from the Agent.

*

* @author mprokopchuk

*/

/**

* Command to check status of {@link StartupCommand} from the Agent.

*

*/

@bernardodemarco I think, it's ok to have it. we've author mentioned in few other classes as well. (check with '@author' in the code base).

github-actions · 2026-06-08T07:10:24Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

nvazquez · 2026-10-24T18:45:20Z

@blueorangutan package

blueorangutan · 2026-10-24T18:46:05Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

🟡 Changes recommended

Unresolved critical and moderate issues remain in connection handling, concurrent backoff state, hostname resolution, and context propagation.

14 open issues

_{Review effort: Lite}

+        } catch (UnknownHostException e) {
+            logger.debug("Unable to resolve configured management server address {}", hostname);
+        }
+        resolutionCache.put(hostname, new ResolvedAddress(ip, now));


blueorangutan · 2026-10-24T20:07:10Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19714

nvazquez · 2026-10-24T20:35:14Z

@blueorangutan test

blueorangutan · 2026-10-24T20:36:04Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

sureshanaparti · 2026-10-28T08:41:13Z

@blueorangutan package

blueorangutan · 2026-10-28T08:42:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

Copilot

🟡 Changes recommended

Several active correctness issues remain, including stale trace-context propagation, removal of the wrong agent during replacement, clustered peer failures for IP-configured deployments, and incomplete failure tracking for null responses.

13 open issues

4 resolved since last review

_{Review effort: Lite}

Copilot

🟡 Changes recommended

Unresolved critical lifecycle, forwarding-registration, and public-interface compatibility defects, along with an unsafe empty-response dereference, can cause resource leaks, failed lookups, runtime incompatibility, and exceptions.

13 open issues

3 resolved since last review

_{Review effort: Lite}

+    private ScheduledExecutorService hostStatusExecutor;
+
+    public HostConnectProcess(Agent agent) {
+        this.agent = agent;
+        initExecutors();
+    }
+
+    private void initExecutors() {
+        stop();
+        var threadFactory = new NamedThreadFactory("Agent-" + HostStatusTask.class.getSimpleName());
+        hostStatusExecutor = Executors.newScheduledThreadPool(1, threadFactory);


Copilot

🟡 Changes recommended

Critical concurrency and agent-replacement defects remain that can trigger duplicate startup commands and remove the newly connected host from the agent map.

12 open issues

2 resolved since last review

_{Review effort: Lite}

+    public void scheduleConnectProcess(Link link, boolean connectionTransfer) {
+        logger.debug("Scheduling connect process for {}", link);
+        initExecutors();
+
+        var task = new HostStatusTask(link, connectionTransfer, agent);
+        var future = hostStatusExecutor.scheduleWithFixedDelay(ThreadContextUtil.wrapThreadContext(task),
+                HOST_STATUS_CHECK_INITIAL_DELAY_SEC,
+                hostStatusCheckDelaySec, TimeUnit.SECONDS);
+        task.setFuture(future);
+        hostStatusFutureRef.set(future);


sureshanaparti · 2026-10-28T10:29:58Z

@blueorangutan package

blueorangutan · 2026-10-28T10:32:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-10-28T12:04:24Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19798

sureshanaparti · 2026-10-28T12:19:05Z

@blueorangutan test keepEnv

blueorangutan · 2026-10-28T12:20:04Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-10-29T06:23:02Z

[SF] Trillian test result (tid-17310)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 59401 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13345-t17310-kvm-ol8.zip
Smoke tests completed. 155 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_uservm_host_control_state	`Failure`	17.11	test_host_control_state.py
test_hostha_enable_ha_when_host_disabled	`Error`	3.09	test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	303.23	test_hostha_kvm.py

boring-cyborg Bot added component:agent component:api component:database component:orchestration labels Jun 4, 2026

sureshanaparti mentioned this pull request Jun 4, 2026

Indirect agent connection improvements #13028

Closed

2 of 14 tasks

sureshanaparti added this to the 4.23.0 milestone Jun 4, 2026

sureshanaparti requested a review from nvazquez June 4, 2026 06:29

sureshanaparti requested a lite review from Copilot June 4, 2026 06:30

Copilot started reviewing on behalf of sureshanaparti June 4, 2026 06:31

View session

sureshanaparti added status:needs-review status:needs-testing labels Jun 4, 2026

Copilot AI reviewed Jun 4, 2026

View reviewed changes

sureshanaparti requested a review from Copilot June 4, 2026 08:28

Copilot started reviewing on behalf of sureshanaparti June 4, 2026 08:28

View session

Copilot AI reviewed Jun 4, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jun 4, 2026

[repo-status] Daily Status Report – June 4, 2026 #13350

Closed

bernardodemarco reviewed Jun 5, 2026

View reviewed changes

github-actions Bot added the status:has-conflicts label Jun 8, 2026

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from d5a6d20 to 22e832c Compare June 8, 2026 13:18

Copilot started reviewing on behalf of sureshanaparti October 24, 2026 18:44

View session

Copilot AI reviewed Oct 24, 2026

View reviewed changes

Merge branch 'main' into indirect-agent-connection-improvements

d9b60bc

Copilot AI lite review requested due to automatic review settings October 28, 2026 08:40

Copilot started reviewing on behalf of sureshanaparti October 28, 2026 08:43

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from debdd10 to 64df4e3 Compare October 28, 2026 09:14

Copilot AI lite review requested due to automatic review settings October 28, 2026 09:14

Copilot started reviewing on behalf of sureshanaparti October 28, 2026 09:29

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from 64df4e3 to 5a5b922 Compare October 28, 2026 09:55

Copilot AI lite review requested due to automatic review settings October 28, 2026 09:55

Copilot started reviewing on behalf of sureshanaparti October 28, 2026 10:07

View session

Copilot AI reviewed Oct 28, 2026

View reviewed changes

sureshanaparti marked this pull request as draft October 28, 2026 10:24

review changes

92a0c48

sureshanaparti force-pushed the indirect-agent-connection-improvements branch from 5a5b922 to 92a0c48 Compare October 28, 2026 10:29

Conversation

sureshanaparti commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Jun 4, 2026

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

codecov Bot commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

sureshanaparti commented Jun 4, 2026

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

sureshanaparti commented Jun 4, 2026

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

blueorangutan commented Jun 4, 2026

Uh oh!

bernardodemarco Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

sureshanaparti Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 8, 2026

Uh oh!

nvazquez commented Oct 24, 2026

Uh oh!

blueorangutan commented Oct 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 4, 2026 •

edited

Loading

codecov Bot commented Jun 4, 2026 •

edited

Loading

sureshanaparti Jun 8, 2026 •

edited

Loading