Repository navigation
Use better statistics for device_lists_changes_in_room in postgres - #20495
erikjohnston wants to merge 2 commits into
Conversation
The default statistics for `device_lists_changes_in_room` were often wrong, underestimating the number of distinct values of `room_id` and `stream_id` columns. This meant in Postgres 18 that it used the wrong index, leading to large amounts of IO. In Postgres 18 the query planner now supports "skip scans", which means more of the indexes are available to be used, and thus makes it more important to have accurate statistics. The problematic query was operating on an array of room IDs and a range of stream IDs. Previously, it would use the index on `(room_id, stream_id)`. After upgrade, it started using `(stream_id, room_id)` on the assumption that we were matching the majority of all rooms and thus may as well pull out all rows matching the stream ID range.
Schema DiffPlease check that this looks as expected! common (sqlite)Unchanged common (postgres)Unchanged main (sqlite)Unchanged main (postgres)--- /tmp/schema_diff_80tbk03e/before/main/full_schemas/9999/full.sql.postgres 2026-11-10 12:02:32.001148763 +0000
+++ /tmp/schema_diff_80tbk03e/after/main/full_schemas/9999/full.sql.postgres 2026-11-10 12:02:20.583055168 +0000
@@ -275,20 +275,22 @@
CREATE TABLE device_lists_changes_in_room (
user_id text NOT NULL,
device_id text NOT NULL,
room_id text NOT NULL,
stream_id bigint NOT NULL,
converted_to_destinations boolean NOT NULL,
opentracing_context text,
instance_name text,
inserted_ts bigint
);
+ALTER TABLE ONLY device_lists_changes_in_room ALTER COLUMN room_id SET STATISTICS 1000;
+ALTER TABLE ONLY device_lists_changes_in_room ALTER COLUMN stream_id SET (n_distinct=-0.05);
CREATE TABLE device_lists_changes_in_room_max_pruned_stream_id (
lock character(1) DEFAULT 'X'::bpchar NOT NULL,
stream_id bigint NOT NULL
);
CREATE TABLE device_lists_outbound_last_success (
destination text NOT NULL,
user_id text NOT NULL,
stream_id bigint NOT NULL
);
CREATE TABLE device_lists_outbound_pokes (
state (sqlite)Unchanged state (postgres)Unchanged |
| @@ -0,0 +1 @@ | |||
| Mitigate potential database performance issues with Postgres 18 by updating the statistics for `device_lists_changes_in_room`. | |||
There was a problem hiding this comment.
Spawning from flare up in #matrix-ops:matrix.org: incident doc
| -- We need to increase the number of rows to sample for `room_id` column, | ||
| -- otherwise the stats will underestimate the number of distinct rooms and make | ||
| -- the query planner use the wrong index. | ||
| ALTER TABLE device_lists_changes_in_room ALTER COLUMN room_id SET STATISTICS 1000; |
There was a problem hiding this comment.
What's the default? Why is 1000 good enough?
There was a problem hiding this comment.
What's the default?
The default_statistics_target is 100 (linked from docs)
There was a problem hiding this comment.
Yeah, it's mostly a number picked that seems to help. We need something big enough to allow Postgres to better understand the behaviour vs. the bigger it is the slower ANALYZE is.
| @@ -0,0 +1,22 @@ | |||
| -- | |||
There was a problem hiding this comment.
To restate for myself, the problem is that Postgres 18 now supports "skip scans" which means the database might decide to use certain indexes that it previously didn't consider.
For get_device_list_changes_in_rooms(...), Postgres decided to use the (stream_id, room_id) index because it looked more attractive (according to it's inaccurate estimates) than the better (room_id, stream_id) index.
This PR is updating the STATISTICS to get more accurate estimates so that it chooses (room_id, stream_id) appropriately.
|
|
||
| -- Manually tell the query planner that we expect the number of distinct values | ||
| -- in the `stream_id` column to be proportional to the number of rows. | ||
| ALTER TABLE device_lists_changes_in_room ALTER COLUMN stream_id SET (n_distinct = -0.05); |
There was a problem hiding this comment.
Docs:
This form sets or resets per-attribute options. Currently, the only defined per-attribute options are
n_distinctandn_distinct_inherited, which override the number-of-distinct-values estimates made by subsequent ANALYZE operations.n_distinctaffects the statistics for the table itself, whilen_distinct_inheritedaffects the statistics gathered for the table plus its inheritance children, and for the statistics gathered for partitioned tables. When the value specified is a positive value, the query planner will assume that the column contains exactly the specified number of distinct nonnull values. Fractional values may also be specified by using values below 0 and above or equal to -1. This instructs the query planner to estimate the number of distinct values by multiplying the absolute value of the specified number by the estimated number of rows in the table. For example, a value of -1 implies that all values in the column are distinct, while a value of -0.5 implies that each value appears twice on average. This can be useful when the size of the table changes over time. For more information on the use of statistics by the PostgreSQL query planner, refer to Section 14.2.-- https://www.postgresql.org/docs/current/sql-altertable.html#SQL-ALTERTABLE-DESC-SET-ATTRIBUTE-OPTION
|
|
||
| -- Manually tell the query planner that we expect the number of distinct values | ||
| -- in the `stream_id` column to be proportional to the number of rows. | ||
| ALTER TABLE device_lists_changes_in_room ALTER COLUMN stream_id SET (n_distinct = -0.05); |
There was a problem hiding this comment.
How did you decide on this value? It would be good to use a measured value from the actual data for matrix.org.
n_distinct = -0.05 means we're saying that there is ~1 distinct stream_id per 20 rows in device_lists_changes_in_room.
There was a problem hiding this comment.
Good q, -0.05 was taken from matrix.org, though it fluctuates quite a lot. I don't think it massively matters so long as its roughly right and negative.
The default statistics for
device_lists_changes_in_roomwere often wrong, underestimating the number of distinct values ofroom_idandstream_idcolumns. This meant in Postgres 18 that it used the wrong index, leading to large amounts of IO.In Postgres 18 the query planner now supports "skip scans", which means more of the indexes are available to be used, and thus makes it more important to have accurate statistics.
The problematic query was operating on an array of room IDs and a range of stream IDs. Previously, it would use the index on
(room_id, stream_id). After upgrade, it started using(stream_id, room_id)on the assumption that we were matching the majority of all rooms and thus may as well pull out all rows matching the stream ID range.