Repository navigation
Conversation
…tervals
During first boot initialization, the official MySQL and PostgreSQL
container entrypoints start temporary bootstrap servers listening only
on Unix domain sockets (--skip-networking / listen_addresses='').
Probing with `mysqladmin ping -h localhost` and `pg_isready -U postgres`
uses Unix domain sockets and reports healthy prematurely against the
temporary servers before they are shut down and restarted with TCP networking.
1. Probe over TCP explicitly:
- MySQL: mysqladmin ping -h 127.0.0.1 --protocol=TCP --silent
- PostgreSQL: pg_isready -U postgres -h 127.0.0.1
(MSSQL already probes over TCP via sqlcmd with a SELECT 1 query).
2. Configure startup grace period and cadence:
- start_period: 60s absorbs slow initialization (such as MSSQL creating
system databases) without penalizing fast boots or consuming retries.
- start_interval: 2s polls frequently during bootstrap so container
readiness is detected promptly once TCP accepts connections.
3. Relax steady-state health checks and unify retries:
- interval: 120s minimizes background process forks and connection overhead
during test execution, since the test suite itself actively queries the
databases and detects connection drops immediately.
- retries: 1 marks a container unhealthy immediately if a scheduled probe
fails outside bootstrap, without waiting multiple 120s interval cycles.
- Unify healthcheck parameters identically across PostgreSQL, MySQL, and MSSQL.
In CI, integration test shards use ExUnit's `--only database:<service>`. If a test declares a `:database` tag with a typo (e.g. `:postgres` instead of `:postgresql`), ExUnit silently skips the test across all shards: the `none` shard excludes it because a `:database` tag key exists, and the database shards skip it because the value does not match the filter. Unlike tests that omit the tag entirely (which run on the `none` shard and fail loudly if they attempt to connect to an unstarted database container), tests with invalid tags are completely omitted from execution and summary reporting without warning. Validate all `@tag database: ...`, `@describetag database: ...`, and `@moduletag database: ...` declarations in `test_helper.exs` before tests run.
The initial prototype for summarizing and aggregating integration test
results was quickly drafted by an AI coding agent to validate the
sharding concept. While it functioned, it was filled with prototype
"slop": unnecessarily defensive guards, redundant fallback chains (like
runtime checks for Jason vs JSON), mixed concerns across parsing and
presentation, and formatting output meant solely for multi-shard CI
runs.
This commit refactors both the ExUnit formatter and the summary
aggregator with a clearer, layered architecture and human-friendly
observability.
Summary Formatter:
- Fail early if missing required environment variables.
- Output deterministic, pretty-printed, human-readable JSON that is
easier to inspect and debug.
- Rather than naively accumulating tests one by one via
`handle_cast({:test_finished, ...})`, rely on ExUnit's lifecycle
guarantees by reading completed tests directly in `:module_finished`.
Aggregator Script:
- Separate raw JSON ingestion, domain transformation, and Markdown
presentation. This eliminates ad-hoc dictionary navigation and
fallback checks scattered across report rendering.
- Use Elixir's native `JSON.decode!/1` only.
- Unified concept from "Shard" to "Job": Abstract terminology across the
pipeline to "Job". This establishes a consistent naming model that
works cleanly whether aggregating multiple matrix jobs in CI or
inspecting a single local run.
- Stop mutating `$GITHUB_STEP_SUMMARY` directly inside the Elixir
script. The script now expects a directory argument, formats Markdown,
and emits it to stdout, making it equally usable locally or in the CI
workflow.
- Visual timeline & metrics: Reconstruct concurrent module execution
into Mermaid Gantt charts and provide aggregated top-slowest tables
across versions without cluttering the summary.
CI Workflow:
- Isolate intermediate summary artifacts in
`${{ runner.temp }}/summaries` to keep checkout trees clean.
- Add `continue-on-error: true` so reporting issues never mask or fail
actual test suite results.
Previous work overhauled CI integration tests to use explicit workflow steps rather than the monolithic `integration_test/test.sh` script, leaving `test.sh` obsolete and the local container testing experience outdated. Meanwhile, local test execution lacked parity with CI, relied on both Docker Compose and `docker run`, and suffered from database container restart delays. This change brings the architectural improvements made in CI to local development by unifying everything under Docker Compose. It establishes an isolated container environment that minimizes developer machine exposure while optimizing for fast, iterative local test cycles. Docker Compose Architecture & Isolation: - Replace separate `docker run` and compose commands by consolidating the test runner and databases into `docker-compose.yml`. This keeps all containers on a shared, managed Docker network without needing loosely configured host network bindings or exposed ports on the host. - Using a custom profile, `docker compose up` continues to only start database services unless the `runner` profile is specifically invoked, keeping database lifecycle decoupled from test runs. - Wait until databases are fully initialized before the test container runs, avoiding race conditions during startup. - Run tests as a non-root user in an Ubuntu-based image matching the CI environment. - Mount named volumes to isolate build artifacts from the host. This prevents native C compilation artifacts (e.g. `bcrypt_elixir`, `argon2_elixir`) from conflicting across host and container OS/arch, while preserving fast incremental compilation across runs. - Environment & version sync: Define `integration_test/.env` to parameterize `ELIXIR_IMAGE_TAG` against the latest CI matrix version, making multi-version testing trivial via environment overrides. Local Developer Workflow: - Database services remain running in the background between test invocations. Developers can re-run tests immediately without paying database shutdown/boot latency on every run. - Explicit teardown: Add `./docker.sh down` to easily clean up resources when finished. - Automatically calculate compilation concurrency based on available CPU cores, bringing test execution times down to ~3m30s on modern hardware. - Forward all arguments and flags directly to `mix test`, allowing developers to target specific files or database tags (e.g. `./docker.sh --include database:postgresql test/...`). - Remove the obsolete `integration_test/test.sh` script now that local runs are handled by `docker.sh`.
In local development environments, publishing database ports to all host interfaces (0.0.0.0) by default risks port collisions with locally running database instances or other projects, and unnecessarily exposes services on the local network. 1. Remove default host `ports:` bindings from `docker-compose.yml`. Containerized test runs (`docker.sh`) connect to databases via internal Docker bridge networking without requiring host port exposure. 2. Introduce `docker-compose.ports.yml` with ports bound strictly to loopback (`127.0.0.1`). 3. Update `ci.yml` to pass `-f docker-compose.ports.yml` because CI runs `mix test` directly on the host runner. 4. Update `README.md` to document using `-f docker-compose.ports.yml` for host-native testing.
The official Microsoft SQL Server image only supports the `amd64` architecture. On `arm64` machines (e.g., Apple Silicon), running the `amd64` image is either not possible or likely to crash under emulation. This makes MSSQL optional in the local Docker workflow: - Assign the `mssql` Compose profile to the `mssql` service and mark it as not required under the `runner` service dependencies. - In `docker.sh`, detect host architecture via `uname -m` and only enable the `mssql` profile on `amd64`. - In `CodeGeneratorCase`, provide `reason_if_mssql_unsupported/0` to skip tests tagged `database: :mssql` on unsupported architectures, ensuring `mix test --include database` skips MSSQL instead of failing with connection errors while non-DB tests in the same files continue to run. - In `entrypoint.sh`, probe hostname resolution before starting the background `socat` bridge. CI runs on `amd64` (`ubuntu-24.04`) and starts database containers on demand by service name. In Docker Compose, explicitly naming a service bypasses profile filters, ensuring all MSSQL integration tests continue to run in CI.
Generated Phoenix applications configure their test repository pool size
based on available CPU cores:
pool_size: System.schedulers_online() * 2
On developer workstations with higher core counts (e.g. 16+ cores),
running multiple async integration test modules in parallel causes total
client connections to exceed the default limits of official container
images (PostgreSQL's default of 100 and MySQL's default of 151), leading
to connection exhaustion failures during test runs.
Raising the ceiling to 300 adds negligible shared memory overhead while
providing sufficient headroom under high concurrency.
MSSQL Server defaults to dynamic user connection management (up to
32,767 connections), therefore requires no adjustment.
Enable test duration and module timeline tracking for local
containerized integration test runs by writing `summary.json` and
`summary.md` to a timestamped output directory
(`tmp/summary_${TIMESTAMP}/`).
Normal usage prints innocuous error messages interleaved with `mix test`
output like:
2026/09/28 19:29:03 socat[7147] E write(6, 0xaaaad4508000, 55): Broken pipe
If the socat bridge would not be working, tests would fail loudly anyway.
Collaborator
Author
|
It caught my attention that some tests are running way faster depending on scheduling (Elixir 1.20/OTP 29 is generally faster but that is not the sole explanation): https://github.com/phoenixframework/phoenix/actions/runs/37909833492/attempts/1#summary-113753209311 Top 10 Slowest Tests
I think this has to do with running with the default I tried some local runs with |
While generally useful, the default `--max-cases` being two times the number of schedulers/cores can lead to CPU oversubscription when running integration tests which spend significant time doing CPU work. This also changes CI to compute the number of cores dynamically instead of using hardcoded values based on the current runner capabilities.
Collaborator
Author
A few more runs during lunch break with a bespoke Benchee runner and I confirmed that I also hit a flaky failure on one of the SQLite tests, database busy when the integration test was running the tests of a generated app. The integration tests were running with Pushed a commit to see how |
Collaborator
Author
|
Comparing the top 10 slowest test tables:
So confirmed that 4 is better than 8 in CI 🎉 |
Collaborator
Author
Hmm looking at the overall wall time, they were similar,
Part of the explanation might be that Elixir 1.18 doesn't support |
SteffenDE
approved these changes
Oct 9, 2026
Member
|
🙌🏻 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Building on prior changes to the CI integration test workflow that reduced CI time and improved observability (e.g. #6817, #6825, and #6836), this PR brings time and workflow improvements to local integration test runs.
It also refines the test summary reporting pipeline to support both local and CI runs (single vs multi-job) with a cleaner code structure.
On my machine I get about ~3m30s for cold runs (downloading and building images, fresh compilations) and ~2m10s for subsequent runs (less than 10s overhead, 2m3s actually running the tests).
Some commits are kept separately both as a means to facilitate review/transparency or to make it easier to drop them if so desired. Maybe some could have been separate PRs, but I'm bundling them here to hopefully be the last integration test-related PR of the season :)
The commit messages have the context on the changes. One additional note is that the commit "Harden database container healthchecks" is inspired by @SteffenDE's fixes in #6834 and #6835 that ended up not being merged.
Example local run summary
Phoenix Integration Tests Summary
2m 03s2m 03sAppWithScopesHtmlTest: phx.gen.auth generates scope for phx.gen.html (1m 16s)2m 03sAppWithScopesHtmlTest: phx.gen.auth generates scope for phx.gen.html (1m 16s)Top 10 Slowest Tests
1m 16sAppWithScopesHtmlTesttest/code_generation/app_with_scopes_html_test.exs:61m 06sAppWithMySQLAdapterScopesTesttest/code_generation/app_with_mysql_adapter_scopes_test.exs:61m 02sAppWithMySQLAdapterAuthHtmlTesttest/code_generation/app_with_mysql_adapter_auth_html_test.exs:2156sAppWithScopesJsonTesttest/code_generation/app_with_scopes_json_test.exs:4255sAppWithSQLite3AdapterAuthLiveTesttest/code_generation/app_with_sqlite3_adapter_auth_live_test.exs:1855sAppWithScopesCustomRoutesTesttest/code_generation/app_with_scopes_custom_routes_test.exs:654sUmbrellaAppWithPostgresAdapterJsonTesttest/code_generation/umbrella_app_with_postgres_adapter_json_test.exs:3254sAppWithPostgresAdapterAuthHtmlTesttest/code_generation/app_with_postgres_adapter_auth_html_test.exs:1753sUmbrellaAppWithPostgresAdapterHtmlTesttest/code_generation/umbrella_app_with_postgres_adapter_html_test.exs:3253sAppWithMySQLAdapterHtmlTesttest/code_generation/app_with_mysql_adapter_html_test.exs:6Job Details: Passed — 48 tests (2m 03s in 1.20.4)
Module Execution Timelines
1.20.4/29
--- displayMode: compact --- gantt title Module Execution Timeline dateFormat mm:ss axisFormat %M:%S todayMarker off section Lane 1 PostgresJson :active, 00:00, 01:09 UmbrellaPostgresAuthLive :active, 01:09, 02:02 section Lane 2 SQLite3AuthLive :active, 00:00, 01:39 section Lane 3 SQLite3Json :active, 00:00, 00:33 Postgres :active, 00:33, 01:39 section Lane 4 MSSQLAuthHtml :active, 00:00, 00:49 MySQLAuthLive :active, 00:49, 01:51 section Lane 5 MySQLJson :active, 00:00, 00:49 UmbrellaPostgres :active, 00:49, 01:48 section Lane 6 SQLite3AuthHtml :active, 00:00, 01:36 section Lane 7 MSSQLAuthLive :active, 00:00, 00:46 PostgresAuthLive :active, 00:46, 01:49 section Lane 8 UmbrellaPostgresLive :active, 00:00, 01:31 section Lane 9 UmbrellaPostgresJson :active, 00:00, 01:37 section Lane 10 SQLite3Live :active, 00:00, 00:47 ScopesLive :active, 00:47, 01:35 section Lane 11 MySQLScopes :active, 00:00, 01:05 MySQLLive :active, 01:05, 01:44 section Lane 12 ScopesHtml :active, 00:00, 01:16 SQLite3Html :active, 01:16, 01:47 section Lane 13 PostgresHtml :active, 00:00, 01:20 MinimalApp :active, 01:20, 01:52 section Lane 14 MySQLAuthHtml :active, 00:00, 01:40 section Lane 15 PostgresAuthHtml :active, 00:00, 01:38 section Lane 16 UmbrellaPostgresHtml :active, 00:00, 01:38 section Lane 17 UmbrellaPostgresAuthHtml :active, 00:00, 01:38 section Lane 18 PostgresLive :active, 00:00, 01:27 section Lane 19 MySQLHtml :active, 00:00, 00:52 ScopesCustomRoutes :active, 00:52, 01:47 section Lane 20 ScopesJson :crit, active, 00:00, 01:45Module Durations
1m 45sAppWithScopesJsonTestTimeline:
ScopesJson56s1m 40sAppWithMySQLAdapterAuthHtmlTestTimeline:
MySQLAuthHtml1m 02s1m 39sAppWithSQLite3AdapterAuthLiveTestTimeline:
SQLite3AuthLive55s1m 39sUmbrellaAppWithPostgresAdapterAuthHtmlTestTimeline:
UmbrellaPostgresAuthHtml52s1m 38sAppWithPostgresAdapterAuthHtmlTestTimeline:
PostgresAuthHtml54s1m 38sUmbrellaAppWithPostgresAdapterHtmlTestTimeline:
UmbrellaPostgresHtml53s1m 38sUmbrellaAppWithPostgresAdapterJsonTestTimeline:
UmbrellaPostgresJson54s1m 36sAppWithSQLite3AdapterAuthHtmlTestTimeline:
SQLite3AuthHtml51s1m 31sUmbrellaAppWithPostgresAdapterLiveTestTimeline:
UmbrellaPostgresLive51s1m 28sAppWithPostgresAdapterLiveTestTimeline:
PostgresLive51s1m 20sAppWithPostgresAdapterHtmlTestTimeline:
PostgresHtml44s1m 16sAppWithScopesHtmlTestTimeline:
ScopesHtml1m 16s1m 09sAppWithPostgresAdapterJsonTestTimeline:
PostgresJson38s1m 07sAppWithPostgresAdapterTestTimeline:
Postgres38s1m 06sAppWithMySQLAdapterScopesTestTimeline:
MySQLScopes1m 06s1m 03sAppWithPostgresAdapterAuthLiveTestTimeline:
PostgresAuthLive39s1m 02sAppWithMySQLAdapterAuthLiveTestTimeline:
MySQLAuthLive51s59sUmbrellaAppWithPostgresAdapterTestTimeline:
UmbrellaPostgres37s55sAppWithScopesCustomRoutesTestTimeline:
ScopesCustomRoutes55s54sUmbrellaAppWithPostgresAdapterAuthLiveTestTimeline:
UmbrellaPostgresAuthLive49s53sAppWithMySQLAdapterHtmlTestTimeline:
MySQLHtml53s50sAppWithMySQLAdapterJsonTestTimeline:
MySQLJson50s50sAppWithMSSQLAdapterAuthHtmlTestTimeline:
MSSQLAuthHtml50s48sAppWithSQLite3AdapterLiveTestTimeline:
SQLite3Live48s48sAppWithScopesLiveTestTimeline:
ScopesLive48s46sAppWithMSSQLAdapterAuthLiveTestTimeline:
MSSQLAuthLive46s38sAppWithMySQLAdapterLiveTestTimeline:
MySQLLive38s33sAppWithSQLite3AdapterJsonTestTimeline:
SQLite3Json33s33sMinimalAppTestTimeline:
MinimalApp27s31sAppWithSQLite3AdapterHtmlTestTimeline:
SQLite3Html31sTest Durations
1m 16sAppWithScopesHtmlTesttest/code_generation/app_with_scopes_html_test.exs:61m 06sAppWithMySQLAdapterScopesTesttest/code_generation/app_with_mysql_adapter_scopes_test.exs:61m 02sAppWithMySQLAdapterAuthHtmlTesttest/code_generation/app_with_mysql_adapter_auth_html_test.exs:2156sAppWithScopesJsonTesttest/code_generation/app_with_scopes_json_test.exs:4255sAppWithSQLite3AdapterAuthLiveTesttest/code_generation/app_with_sqlite3_adapter_auth_live_test.exs:1855sAppWithScopesCustomRoutesTesttest/code_generation/app_with_scopes_custom_routes_test.exs:654sUmbrellaAppWithPostgresAdapterJsonTesttest/code_generation/umbrella_app_with_postgres_adapter_json_test.exs:3254sAppWithPostgresAdapterAuthHtmlTesttest/code_generation/app_with_postgres_adapter_auth_html_test.exs:1753sUmbrellaAppWithPostgresAdapterHtmlTesttest/code_generation/umbrella_app_with_postgres_adapter_html_test.exs:3253sAppWithMySQLAdapterHtmlTesttest/code_generation/app_with_mysql_adapter_html_test.exs:652sUmbrellaAppWithPostgresAdapterAuthHtmlTesttest/code_generation/umbrella_app_with_postgres_adapter_auth_html_test.exs:1851sAppWithSQLite3AdapterAuthHtmlTesttest/code_generation/app_with_sqlite3_adapter_auth_html_test.exs:1851sUmbrellaAppWithPostgresAdapterLiveTesttest/code_generation/umbrella_app_with_postgres_adapter_live_test.exs:3551sAppWithMySQLAdapterAuthLiveTesttest/code_generation/app_with_mysql_adapter_auth_live_test.exs:1851sAppWithPostgresAdapterLiveTesttest/code_generation/app_with_postgres_adapter_live_test.exs:3450sAppWithMySQLAdapterJsonTesttest/code_generation/app_with_mysql_adapter_json_test.exs:650sAppWithMSSQLAdapterAuthHtmlTesttest/code_generation/app_with_mssql_adapter_auth_html_test.exs:549sUmbrellaAppWithPostgresAdapterAuthLiveTesttest/code_generation/umbrella_app_with_postgres_adapter_auth_live_test.exs:1849sAppWithScopesJsonTesttest/code_generation/app_with_scopes_json_test.exs:648sAppWithSQLite3AdapterLiveTesttest/code_generation/app_with_sqlite3_adapter_live_test.exs:648sAppWithScopesLiveTesttest/code_generation/app_with_scopes_live_test.exs:646sAppWithMSSQLAdapterAuthLiveTesttest/code_generation/app_with_mssql_adapter_auth_live_test.exs:546sUmbrellaAppWithPostgresAdapterAuthHtmlTesttest/code_generation/umbrella_app_with_postgres_adapter_auth_html_test.exs:546sUmbrellaAppWithPostgresAdapterHtmlTesttest/code_generation/umbrella_app_with_postgres_adapter_html_test.exs:545sAppWithPostgresAdapterAuthHtmlTesttest/code_generation/app_with_postgres_adapter_auth_html_test.exs:545sAppWithSQLite3AdapterAuthHtmlTesttest/code_generation/app_with_sqlite3_adapter_auth_html_test.exs:545sAppWithSQLite3AdapterAuthLiveTesttest/code_generation/app_with_sqlite3_adapter_auth_live_test.exs:544sUmbrellaAppWithPostgresAdapterJsonTesttest/code_generation/umbrella_app_with_postgres_adapter_json_test.exs:544sAppWithPostgresAdapterHtmlTesttest/code_generation/app_with_postgres_adapter_html_test.exs:3141sUmbrellaAppWithPostgresAdapterLiveTesttest/code_generation/umbrella_app_with_postgres_adapter_live_test.exs:539sAppWithPostgresAdapterAuthLiveTesttest/code_generation/app_with_postgres_adapter_auth_live_test.exs:1738sAppWithMySQLAdapterLiveTesttest/code_generation/app_with_mysql_adapter_live_test.exs:638sAppWithPostgresAdapterJsonTesttest/code_generation/app_with_postgres_adapter_json_test.exs:3138sAppWithMySQLAdapterAuthHtmlTesttest/code_generation/app_with_mysql_adapter_auth_html_test.exs:538sAppWithPostgresAdapterTesttest/code_generation/app_with_postgres_adapter_test.exs:537sUmbrellaAppWithPostgresAdapterTesttest/code_generation/umbrella_app_with_postgres_adapter_test.exs:1537sAppWithPostgresAdapterLiveTesttest/code_generation/app_with_postgres_adapter_live_test.exs:536sAppWithPostgresAdapterHtmlTesttest/code_generation/app_with_postgres_adapter_html_test.exs:533sAppWithSQLite3AdapterJsonTesttest/code_generation/app_with_sqlite3_adapter_json_test.exs:631sAppWithSQLite3AdapterHtmlTesttest/code_generation/app_with_sqlite3_adapter_html_test.exs:631sAppWithPostgresAdapterJsonTesttest/code_generation/app_with_postgres_adapter_json_test.exs:529sAppWithPostgresAdapterTesttest/code_generation/app_with_postgres_adapter_test.exs:1527sMinimalAppTesttest/code_generation/minimal_app_test.exs:2224sAppWithPostgresAdapterAuthLiveTesttest/code_generation/app_with_postgres_adapter_auth_live_test.exs:522sUmbrellaAppWithPostgresAdapterTesttest/code_generation/umbrella_app_with_postgres_adapter_test.exs:511sAppWithMySQLAdapterAuthLiveTesttest/code_generation/app_with_mysql_adapter_auth_live_test.exs:56sMinimalAppTesttest/code_generation/minimal_app_test.exs:45sUmbrellaAppWithPostgresAdapterAuthLiveTesttest/code_generation/umbrella_app_with_postgres_adapter_auth_live_test.exs:5