Parquet and fastreg

Point fastreg at your registers, convert SAS to parquet, and read the ones it cannot find by name

Published

September 9, 2026

Everything about getting registers into parquet and reading them with fastreg, in the order most people need it:

  1. Point fastreg at your registers - one options() call at the top of each script. Every read_register() example in this guide assumes you have done this. If a colleague already converted the registers, this is the only section you need.
  2. Convert SAS files to Parquet - when your project received .sas7bdat and nothing has been converted yet. Done once per register, usually by one person for the whole project.
  3. When read_register() cannot find the register - reading parquet that is not in fastreg’s layout.

Which file types you meet on the server, and what opens each one, is Phase 4 - File types and loading.

Point fastreg at your registers

If someone else already converted the registers, you do not convert anything. You tell fastreg where they are, once per script, and then read them by name. Ask whoever set the project up for the exact paths:

# Point fastreg at the converted registers - done once per script
library(fastreg)

options(
  fastreg.project_workdata_dir = "E:/workdata/[projectnumber]/cleaned-data/",
  fastreg.project_rawdata_dir = "E:/rawdata/[projectnumber]/"
)

After that, read_register("bef") works by name, with no conversion and no path in each call.

Set it once, at the top of the script, next to the library() calls. Every read_register() example in this guide assumes it has been set, and none of them repeat it: two lines of someone else’s project paths in every code block would be noise, and it would suggest the call belongs next to each read rather than once at the top.

Two things that otherwise cost you time:

  • Write E:/ with the colon: the DST server is Windows, also when you log in from a Mac. fastreg’s own guide shows E without the colon, but only because the example writes to a temporary folder with fs::path_temp().
  • The folder does not have to be the innermost one: you can set the path to cleaned-data/ even though it holds several subfolders (parquet-registers/, parquet-external/). read_register() searches further down for the register name.

Convert SAS files to Parquet

Most projects on DST receive registers as SAS files (.sas7bdat). Before you can use them with open_dataset() and lazy evaluation, they must be converted to Parquet once. After that you use them exactly like any other register.

Relevant for most projects outside DARTER. If you are working on a project where the registers have not already been converted to parquet, this step is necessary before you can run extractions. Done once per register - after that the normal extraction pattern applies.

Why Parquet is worth it (SAS vs Parquet)
SAS (.sas7bdat) Parquet
Read time (1M rows) ~30–120 sec ~1–3 sec
Disk space Large 50–75% smaller
Requires package haven arrow
Lazy evaluation No - all into RAM Yes - filter BEFORE collect

When read_register() cannot find the register

This is the direct consequence of how the parquet was laid out, which is why it sits on the same page as the conversion. read_register() can only read registers that sit in fastreg’s own format:

<register>/year=YYYY/part-XXX.parquet

That year= level is what produces the year column you filter on later. It is a partition marker, not a DST variable and not a date - see Phase 4 - reading a register split by year.

Both halves count: the year= subfolder and the part- filename. This is not a bug but a deliberate choice. The function is built for data converted with fastreg’s convert(), and is not meant to guess every way parquet files might otherwise be named. If the register is laid out differently - say as one flat folder of parquet files, or with filenames without part- - read_register() cannot see it, no matter how correctly you set the path.

In that case you use one of fastreg’s two other reading functions:

Function You write Used for
read_register("name") the register name registers converted with fastreg
read_parquet_dataset("path/to/folder") the full path to the folder a folder of parquet files
read_parquet_file("path/to/file.parquet") the full path to the file a single parquet file

All three hand you a DuckDB table, so the rest of your code is exactly the same whichever one you use.

read_parquet_dataset() does not use your options() path. Only read_register() looks names up under the folders you set. The two read_parquet_*() functions need the full path, including any subfolders below the folder you set. So if you set the path to cleaned-data/, you still have to write the whole way down: "E:/workdata/[projectnumber]/cleaned-data/parquet-external/lmdb".

# Find the register when read_register() cannot look the name up
library(fastreg)
library(dplyr)

# 1. Which names does read_register() know? It lists the datasets it can find
#    under the paths you set with options() - always start here.
list_parquet_datasets()

# 2. If the register is not on the list, point at the folder with the full path
lmdb <- read_parquet_dataset(
  "E:/workdata/[projectnumber]/cleaned-data/parquet-external/lmdb"
)

# 3. If it is a single file rather than a folder, use read_parquet_file()
table <- read_parquet_file(
  "E:/workdata/[projectnumber]/cleaned-data/parquet-external/my_table.parquet"
)

# From there the code is the same as after read_register()
lmdb %>%
  rename_with(tolower) %>%
  filter(eksd >= as.Date("2010-01-01")) %>%
  select(pnr, atc, eksd) %>%
  collect()

On DARTER (project 708421) the remaining registers are being converted to fastreg format during August 2026. Until then a few registers may need read_parquet_dataset() even though they are listed with read_register() in DARTER - Register paths and datastores. After that, the name lookup works for all of them. (Noted July 2026.)

See also

Back to top