Subsetting data in R: a balance between safety and speed

Work in Progress
Decision modelling
R
Author

Edifofon Akpan

Published

27 July 2026

Within the week, I was testing a function for use in a combined microsimulation model of of three smoking-related cancers. My objective was to make the code run as safe as possible, such that I am able to detect when something isn’t working well. One of the columns in the R data table of the model population (named pop_model), was smok_amt_yrs . I had renamed this column from smoke_amt just hours ago. I needed to extract this vector from the data table for further calculations, so I mistakenly did

amount <- pop_model$smoke_amt

Previously, I had changed another column from smoke_dur to smoke_yrs , because I felt the suffix yrs better indicated the number of years that an individual had smoked. Again, I mistakenly did

duration <- pop_model[["smoke_dur"]]

The final output of the function was intended to be a column in pop_model that expresses the risk of onset of cancer based on the amount and number of years an individual had smoked. I got the following error

Error in [.data.table:
! RHS of assignment to existing column 'prob_inc_cancer' is zero length but not NULL. If you intend to delete the column use NULL....

On investigation, I realized that pop_model$smoke_amt returned the right vector, even though it was meant to be pop_model$smoke_amt_yrs; however, pop_model[["smoke_dur"]] silently returned NULL. When I changed the latter to pop_model[, smoke_dur] – it returned an error, but the function was notoriously slow. To understand what happened, we need to start from the basics, and the most basic of basics is this:

There are three subsetting operators, [[, [, and $.

Three subsetting operators - safety first

To explain this further, consider we have the following data table.

if (!requireNamespace("data.table", quietly = TRUE)) {
  install.packages("data.table")
}

n <- 1e+06
set.seed(12345)

age <- sample(40:60, n, TRUE)
sex <- sample(c("Male", "Female"), n, TRUE, c(0.65, 0.35))
smok <- ifelse(
  sex == "Male",
  sample(x = c("Never", "Former", "Current"), size = n, replace = TRUE, prob = c(0.2, 0.5, 0.3)),
  sample(x = c("Never", "Former", "Current"), size = n, replace = TRUE, prob = c(0.3, 0.6, 0.1))
)

dt <- data.table::data.table(age, sex, smok)

head(dt)
##      age    sex   smok
##    <int> <char> <char>
## 1:    53 Female Former
## 2:    58   Male Former
## 3:    55 Female Former
## 4:    50   Male  Never
## 5:    41   Male Former
## 6:    50 Female  Never

Actually, i have decided that age_diag is a better descriptive term than just age. For illustration, assume that these changes are not made from the original source

n <- 1e+06
set.seed(12345)

age_diag <- sample(40:60, n, TRUE)
sex <- sample(c("Male", "Female"), n, TRUE, c(0.65, 0.35))
smok <- ifelse(
  sex == "Male",
  sample(x = c("Never", "Former", "Current"), size = n, replace = TRUE, prob = c(0.2, 0.5, 0.3)),
  sample(x = c("Never", "Former", "Current"), size = n, replace = TRUE, prob = c(0.3, 0.6, 0.1))
)

dt <- data.table::data.table(age_diag, sex, smok)

Suppose I want to subset the age_diag column as a vector. And then I run the following code

dt20 <- dt[1:20]
dt20$age
##  [1] 53 58 55 50 41 50 45 46 49 56 47 46 45 40 51 59 47 51 42 48

Even though I have changed the column name to age_diag, the above code will return the vector of ages. This is because the $ operator does not throw an error when a column name is not found. Instead, R uses partial matching to find the closest match. In this case, it finds age_diag and returns the corresponding vector.Suppose I now decide to use the [ operator to subset the age column. I run the following code

dt20[, age]
## Error in `[.data.table`:
## ! j (the 2nd argument inside [...]) is a single symbol but column name 'age' is not found. If you intended to select columns using a variable in calling scope, please try DT[, ..age]. The .. prefix conveys one-level-up similar to a file system path.

This time, it returns an error, because the column name age does not exist in the data table. The [ operator does not use partial matching, and it requires an exact match for the column name. Suppose I now decide to use the [[ operator to subset the age column. I run the following code

dt20[["age"]]
## NULL

This time, it returns NULL, without throwing an error. This can lead to silent errors in your code, as you may not realize that you are working with a NULL value instead of the expected data. For instance, assume I want to create a new column in the data table that is five years after the age of diagnosis. I run the following code, and get the following result, but it is not what I expected. The code runs without error, but the result is of length zero, because dt20[["age"]] returned NULL. This is a silent error that can be difficult to detect.

dt20[["age"]] + 5
## numeric(0)

What have we learned from this?

The [[ operator is the least safe – while it requires an exact match for the column name, but it can still return NULL without throwing an error. The $ operator is the safer, because it uses partial matching and can lead to silent errors. The [ operator is the safest, because it requires an exact match for the column name and throws an error if the column name does not exist.

Three subsetting operators - need for speed

Now let’s test the speed of the three subsetting operators. We will use the microbenchmark package to measure the time taken by each operator to subset the smok column from the dt data table with 1 million rows. bracket2, dollar, and dt_j formats respectively represent the [ , $ , and [[ operators. We will run each operator 10000 times and measure the average time taken for each operation.

if (!requireNamespace("microbenchmark", quietly = TRUE)) {
  install.packages("microbenchmark")
}
microbenchmark::microbenchmark(
  bracket2 = dt[["smok"]],
  dollar   = dt$smok,
  dt_j     = dt[, smok],
  times = 1000,
  unit = "milliseconds"
)
## Warning in microbenchmark::microbenchmark(bracket2 = dt[["smok"]], dollar = dt$smok, : less accurate nanosecond times to avoid potential integer overflows
## Unit: milliseconds
##      expr      min       lq    mean   median      uq     max neval cld
##  bracket2 0.001681 0.002030 0.00455 0.003198 0.00521  0.0918  1000  a 
##    dollar 0.000246 0.000369 0.00124 0.000574 0.00123  0.0909  1000  a 
##      dt_j 0.410984 0.811738 1.60199 0.894764 1.13025 38.2157  1000   b

The results indicate that the $ operator runs in 0.002255 milliseconds, the [[ operator runs in 0.000451 ms, and the [ operator runs in 0.378635 ms. The $ operator is the fastest, followed by the [[ operator, and then the [ operator. The $ operator is the fastest for a single named extraction, since it goes straight to the column with almost no overhead. The [[ operator is a bit slower, since it goes through an extra dispatch step first1. The [ operator is by far the slowest on a data.table, since it has to parse the whole call (i.e., checking for row filters, grouping, and joins) before it even gets to pulling out the column.

1 a “low-level extractor” typically means a function or operator that pulls a value out of a data structure with minimal extra layers of logic – zero method dispatch, validation, or other operations that a “high-level” version would include.

This may not be a big deal for one-off operations, but in a microsimulation model with millions of operations, the difference in speed can add up to a significant amount of time. For instance, if you are running a microsimulation model with 1 million individuals and you need to extract columns from the data table 100 times throughout the model, the $ operator would take approximately 0.3 milliseconds, while the [ operator would take approximately 37.9 milliseconds. still not too bad. What if you ran a probabilistic sensitivity analysis (PSA) on that model with 10000 iterations, 5 sub-populations, comparing 4 interventions – a total of \(100*10000*5*4 = 20,000,000\) operations on the 1 million rows. The $ operator would take approximately 45 seconds to complete these 20 million operations, while the [ operator would take approximately 126 minutes.

Okay, 45 seconds is great, but subsetting is not the only operation in a microsimulation model. Can we do better? The answer is yes, we can do better. It turns out there is a low-level extractor2 that behaves like [[ or $, but is much faster, called .subset2. Let’s test the speed of the .subset2 function against the other three subsetting operators.

2 a “low-level extractor” typically means a function or operator that pulls a value out of a data structure with minimal extra layers of logic – zero method dispatch, validation, or other operations that a “high-level” version would include.

if (!requireNamespace("microbenchmark", quietly = TRUE)) {
  install.packages("microbenchmark")
}
microbenchmark::microbenchmark(
  bracket2 = dt[["smok"]],
  dollar   = dt$smok,
  dt_j     = dt[, smok],
  subset2  = .subset2(dt, "smok"),
  times = 1000,
  unit = "milliseconds"
)
## Unit: milliseconds
##      expr      min       lq     mean   median       uq      max neval cld
##  bracket2 0.001681 0.002214 0.005404 0.003752 0.005945  0.10016  1000  a 
##    dollar 0.000246 0.000369 0.001695 0.000738 0.001804  0.11013  1000  a 
##      dt_j 0.386343 0.841484 1.802951 0.940601 1.202243 66.13796  1000   b
##   subset2 0.000000 0.000041 0.000305 0.000123 0.000328  0.00623  1000  a

Crazy! The .subset2 function runs in 0.000082 milliseconds, which is 6 times faster than the $ operator, 30 times faster than the [[ operator, and 4634 times faster than the [ operator. For a microsimulation model with 1 million individuals and 20 million operations, the .subset2 function would take approximately 1.6 seconds to complete these 20 million operations, while the $ operator would take approximately 9 seconds, the [[ operator would take approximately45 seconds, and the [ operator would take approximately 126 minutes! Thus .subset2 function is the fastest of all the subsetting operators

In terms of safety, the .subset2 function behaves like the [[ operator, in that it requires an exact match for the column name and returns NULL if the column name does not exist. However, it is much faster than the [[ operator, and it is also faster than the $ operator. For example, back to the age_diag example, if I run the following code

.subset2(dt20, "age")
## NULL

It returns NULL, without throwing an error. This is the same behavior as the [[ operator, and it can lead to silent errors in your code, as you may not realize that you are working with a NULL value instead of the expected data. So what’s the best option for subsetting data in R? It depends on your priorities.

If you prioritize safety over speed, and you just want to carry out a few operations, then the [ operator is the best option. If you prioritize speed over safety, then the .subset2 function is the best option.

What about readability? The $ operator is the most readable, followed by the [[ operator, and then the [ operator. The .subset2 function is the least readable; it is not as commonly used as the other three operators.

.subset2(dt20, "age")
dt20[, age]
dt20[["age"]]
dt20$age

However, if you are working with a large data table with multiple operations and you need to extract columns quickly, then the .subset2 function is probably the best option.

Back to top