Skip to content

group_vfold_cv() the same group ends up in both the analysis and the assessment set #618

Open

Description

@AlexRyabov

When a stratum makes up less than pool of the rows, group_vfold_cv() assigns
rows of the same group to several folds, although the strata are constant within
groups. As a result, rows appear in more than one assessment set, and the same
group ends up in both the analysis and the assessment set of a resample.

library(rsample)

# 12 groups; the stratum is constant within each group.
# Stratum "d" (groups g11 and g12) has 15 of 570 rows, less than pool = 0.1.
sizes <- c(100, 90, 80, 70, 60, 50, 40, 30, 20, 15, 10, 5)
dat <- data.frame(
  group = rep(paste0("g", seq_along(sizes)), sizes),
  stratum = rep(c("a", "b", "c", "a", "b", "c", "a", "b", "c", "a", "d", "d"), sizes)
)

set.seed(2)
folds <- group_vfold_cv(dat, group = group, v = 4, strata = stratum)

# Each row should be held out exactly once:
held_out <- unlist(lapply(folds$splits, complement))
c(n_rows = nrow(dat), n_held_out = length(held_out))
#>     n_rows n_held_out
#>        570        585

# Groups held out in more than one fold:
fold_groups <- lapply(folds$splits, function(s) unique(assessment(s)$group))
names(which(table(unlist(fold_groups)) > 1))
#> [1] "g11" "g12"

# With pool = 0 the partition is correct:
set.seed(2)
folds_no_pool <- group_vfold_cv(dat, group = group, v = 4, strata = stratum, pool = 0)
length(unlist(lapply(folds_no_pool$splits, complement)))
#> [1] 570

Cause. check_grouped_strata() runs before pooling. make_strata() then sets
rows of rare strata to NA and assigns them to existing strata row by row
(out[is.na(x)] <- sample(levels(out), size = num_miss, replace = TRUE)), so
one group gets several strata. balance_groups_strata() assigns a fold to every
unique (group, stratum) pair, and the full_join(..., by = "..group") in
make_groups() duplicates the rows of such groups.

Related. group_vfold_splits() calls as.character(strata) before
make_strata(), so numeric strata are not binned into quartiles as documented
for strata; every value becomes its own stratum and is pooled in the same way.

rsample 1.3.2, R 4.5.1.

Activity

  1. algoria-contributions commented on Nov 10, 2026

    @algoria-contributions

    Hi, we’d be happy to work on this at Algoria.

    We would start by reproducing the reported behavior on the current main branch, then prepare a focused fix that preserves group integrity when pooling rare strata. Regression tests would check that analysis and assessment groups remain disjoint and that each observation appears in exactly one assessment set per repeat.

    Would this approach be welcome, and is anyone already working on it?

    Thanks!
    Algoria

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions