[Caideyipi]

Description

Add integration coverage for recovery windows that can lose replicated or loaded data, leave stale metadata, or retain staging resources indefinitely.

Pipe recovery

• Exercise receiver outages, history/realtime handoff, multiple actual DataRegions and writers, rejected requests, lost successful responses, and queue/backpressure isolation.
• Cover runtime STOP/START, failed and successful ALTER, DROP and same-name recreation, role revocation, interrupted TsFile transfer, schema changes, Region redistribution, processor restart, forwarding loops, and ConfigNode restart during DROP.

Metadata recovery

• Cover live-node lease fencing/resynchronization, DDL procedure rollback or continuation across ConfigNode failover, cross-SchemaRegion deletion, asynchronous DROP and same-name recreation, delayed attribute batches, and SchemaRegion snapshot installation followed by log-tail recovery.
• Retain schema-log tag/alias recovery, database TTL and view-cache recovery, and ConfigNode catalog snapshot plus subsequent DDL recovery.
• Schema-log recovery uses a nonempty durable mlog and a forced restart. Catalog recovery observes snapshot loading on the restarted peer, then requires that peer to become leader before checking the complete catalog, data, and follow-up DDL.

LOAD TsFile recovery

• P0-04: keep the original committed task's staged TsFile/mods/progress and WAL payload references across the IoTConsensus writer's process restart while a real replica is behind. Observe the actual cleaner scan and retain the same file set and bytes until durable peer checkpoints permit cleanup, without additional writes. Check physical data on every replica, per-file mods, duplicate timestamps, and complete query values/types.
• P0-06: require original async Tree and Table pending handoffs to complete after a witnessed temporary failure is removed, including process restart, without resubmission. Check same-named Tree files and their distinct mods, source ownership, queue cleanup, explicit Table databases, null TAG/FIELD values, and complete ordered schema.
• Keep the Tree schema conflict until an original identity handoff is rejected for missing WRITE_DATA and moved to failed; then release the other handoffs. Stop owned cluster processes before directory cleanup.

The LOAD additions are limited to three IT files and add no production hooks or interfaces. Existing fault-injection agents remain in test sources; production metadata hooks removed earlier are not restored. Compatible scenarios share class-scoped clusters.

Validation

• All newly added LOAD scenarios have been run locally on Apache IoTDB / JDK 17 with ClusterIT and TableClusterIT. A combined selection passed 4 tests, 0 failures, 0 errors, 0 skipped. After strengthening the ordered Table schema assertion, the latest focused run passed 1 test, 0 failures, 0 errors, 0 skipped, with Maven exit code 0.
• Those LOAD runs used source revision 28c5028bb462 plus the working changes. The exact reviewed three-file contents were transferred into this PR. Existing source-built dependencies and node templates were reused under recorded provenance; this was not a fresh uncached full build.
• Earlier metadata P0 verification at 030e5c543f9 passed 17 tests, 0 failures, 0 errors, 0 skipped. The retained adapted schema-log and catalog recovery cases subsequently passed individually. These historical results are not a full-PR-head validation.
• Spotless passed for the LOAD source changes; whitespace and exact file-content checks passed for this update. No executable test was rerun while transferring the unchanged files to the PR branch.
• Existing Pipe and metadata suites were not rerun in this update. The combined PR head, full IT suite, other editions/profiles, and RAT remain unverified. Directory-cleanup failures were not injected.
• Precise LOAD PIECE/PREPARE faults, lost COMMIT responses, and coordinator-orphan task recovery (P0-01/02/03/05) remain outside this addition. Ordinary restart/throttling and the empty cleaner scan probe are not counted as that coverage.

Review status

Keep this PR as a draft pending an independent review of the complete updated PR diff and context. The LOAD workspace review does not approve the existing Pipe/metadata changes or the combined PR head.

The Pipe-focused workflow-dispatch input pipe_p0_only continues to select IoTDBPipeRecoveryIT under MultiClusterIT2DualTreeAutoEnhanced.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 48.05%. Comparing base (d452ddd) to head (ed60f11).
⚠️ Report is 20 commits behind head on master.

Files with missing lines	Patch %	Lines
...e/impl/subscription/topic/AlterTopicProcedure.java	0.00%	6 Missing ⚠️
...re/impl/subscription/topic/DropTopicProcedure.java	0.00%	6 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19051      +/-   ##
============================================
+ Coverage     48.04%   48.05%   +0.01%     
  Complexity      604      604              
============================================
  Files          5570     5572       +2     
  Lines        409051   410194    +1143     
  Branches      53848    54112     +264     
============================================
+ Hits         196512   197123     +611     
- Misses       212539   213071     +532     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Caideyipi]

I reviewed the P1 metadata recovery test additions in commit c5ebc9b1c15d. I found no concrete correctness issues in the covered scenarios; the partial items below are coverage gaps, not code defects.

• P1-1: Partial — concurrent same/conflicting auto-creation and ConfigNode leader failover are checked; SchemaRegion leader failover is not covered.
• P1-2: Partial — template extension, interrupted deactivation, and recovered results are checked, but the missed-broadcast DataNode is stopped and restarted, so recovery of a still-running node with stale cache is not covered.
• P1-3: Covered — schema-log recovery checks tag/alias indexes, complete SHOW/count results, and follow-up reads and writes.
• P1-4: Partial — database TTL inheritance, an overridden table TTL, expired-read filtering, and the recovered-node write boundary are checked; dynamic per-table or Tree parent/child TTL changes are not covered.
• P1-5: Partial — Table tree-to-table auto-sync is checked after rename/replace/disable and node recovery; Tree logical-view recovery is not covered.
• P1-6: Partial — a real SchemaRegion migration is verified with metadata changes and post-migration reads/writes; other migration/rebuild failure stages are not covered.
• P1-7: Covered — quota exhaustion, failed activation, deletion/restart, bounded quota refresh, retry, and subsequent writes are checked.
• P1-8: Covered — catalog snapshot, subsequent DDL, ConfigNode recovery and leadership transfer, catalog state, reads/writes, and continued DDL are checked.

Validation: the IT-enabled reactor compilation passed across 37 modules in the source worktree. The Failsafe report confirms only IoTDBSchemaLogRecoveryIT executed (1 test passed, 0 failures); the other new ITs were not executed. The PR remains a draft.

[Caideyipi]

Metadata P0 review/fix round for 030e5c543f9efe3b378e6e01c77fde35a4b875bb.

This round found and fixed test-logic and fixture precondition issues:

• Use supported equality predicates and complete device keys for cross-Region DELETE DEVICES; retain full metadata/data assertions and the unaffected-device checks.
• Make snapshot installation real by reducing the SchemaRegion log purge gap only inside the test agent, checking actual log truncation and a newer installed snapshot, selecting an actual follower distinct from the SQL coordinator, and addressing leader-transfer RPCs to the intended new leader.
• Give the PBTree fixture enough schema memory for its pinned Region roots; prepare template catalog extensions while all nodes are online; assert the alias query's actual column header.
• Wait for all actual IoTConsensus DataRegion replicas before the DROP fixture's flush/stop. Recreate only after the exact SUCCESS / written=1 / remaining=0 result while the stale DataNode remains offline.
• Wait for a direct RPC response from the restarted ConfigNode before reusing the shared cluster.

Final validation: Apache IoTDB on JDK 17, ClusterIT + TableClusterIT, 3 ConfigNodes/3 DataNodes. 17 tests passed, with 0 failures, 0 errors and 0 skips; Maven exited 0. Tree ran in Memory/PBTree and Table in Memory. The Failsafe class XML reports confirm the counts. Spotless and git diff --check passed. Required IoTDB dependencies were built from clean source before the provenance-matched retries.

The new P0 fault agent and controls stay under integration-test/src/test; no new production hook was added. Existing P1 production hooks are unchanged and remain a follow-up/full-PR review concern. The existing Pipe/P1 suites were not rerun in this round.

This round does not approve the full PR. It remains a draft pending a fresh independent review of the latest complete diff and context.

[sonarqubecloud]

Quality Gate failed

Failed conditions
C Security Rating on New Code (required ≥ A)
E Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE