File types and how to open them

What you encounter in your workspace - and what opens it

When you open File Explorer on the DST server, you encounter files with different extensions. This page is your reference tool: what file type is it, which function opens it, and does the data enter memory immediately or not?

The last point - lazy vs. full loading - is the most important distinction here.

What a project folder looks like

A typical project on the DST server is organised roughly like this. The registers live under cleaned-data/; your own scripts and results sit under workspace/[your-name]/:

E:/rawdata/[projectnumber]/
├── Grunddata/                     # raw register data from DST (often SAS) - read-only
│   └── bef/  lpr_adm/  lpr_diag/  …
└── Eksterne data/
│   └── project-specific extracts etc/  …

E:/workdata/[projectnumber]/
├── cleaned-data/
│   └── parquet-registers/       # registers converted to parquet
│       └── bef/  lpr_adm/  lpr_a_kontakt/  …
└── workspaces/
    └── [your-name]/             # your own working area
        ├── R/                   # your analysis scripts (01_, 02_, …)
        ├── datasets/            # your own intermediate results (.rds)
        └── output/              # tables, figures and logs

The paths in the code examples follow this structure, but your own folder names may well differ - check with File Explorer. Two things are worth knowing:

• rawdata/ is read-only. You cannot edit or write to the raw register data DST delivers. You work in workspace/, where each project member typically has their own subfolder ([your-name]/) for scripts, intermediate results and output.
• Almost all the code in this guide assumes the registers are in parquet. If your project only has raw SAS files, converting them to parquet - and setting up a sensible folder structure for the project - is the first job. See Convert SAS files to Parquet.

How to name the scripts in R/ is covered in Good code practice.

Lazy vs. full loading - which file types behave which way

Every file type below loads one of two ways.

Full loading (readRDS, read_sas, read_csv, read_xlsx) reads the entire file into RAM straight away. Fine for your own intermediate results; it would crash your session on a whole register.

Lazy loading (parquet) opens a connection instead, and fetches rows only when you call collect(). That is what lets you work with registers of millions of rows.

For this page that is all you need: parquet is lazy, everything else is not. Why it works, and how to write a query against a lazy connection, is Phase 5 - Extracting data step by step.

SAS files are also large - and are shared with everyone on the server. On DST all users share the server’s RAM. read_sas() on a large SAS file burdens the server for everyone at the same time. DST automatically kills processes (RStudio sessions, jobs, etc.) when RAM is close to full - so an oversized extraction can cost you your unsaved work. If you use a SAS file repeatedly, it is worth converting it to parquet once - this saves significant RAM and makes loading much faster. See Convert SAS to parquet for the procedure, and Mind your RAM in the shared environment for the practical habits. DST’s official advice is collected in DST guide: Reducing RAM use in the shared environment (PDF, Danish).

Overview - file type, package, function

File type	Package	Function	Loading
.parquet / parquet folder	arrow, fastreg or duckplyr	open_dataset("path/"), read_register("name") or read_parquet_duckdb("path/")	Lazy - nothing in RAM until collect()
.rds	base R	readRDS("path/to/file.rds")	Full - entire file into RAM
.sas7bdat	haven	read_sas("path/to/file.sas7bdat")	Full - entire file into RAM
.csv	readr	read_csv("path/to/file.csv")	Full - entire file into RAM
.xlsx	readxl	read_xlsx("path/to/file.xlsx")	Full - entire file into RAM

The last column is the lazy/full distinction from above: parquet is the only lazy format, everything else loads fully into RAM.

What do you write in the parentheses?

• With open_dataset() (arrow) you write the path to the parquet folder - e.g. open_dataset("path/to/bef/"). It works on any parquet, on any project.
• With read_register() (fastreg) you write just the register name - e.g. read_register("bef") - because fastreg already knows where your parquet lives (you set that once during conversion). It also hands you a DuckDB connection, so more dplyr functions work without an extra to_duckdb() step. It requires that the registers were set up with fastreg.
• With read_parquet_duckdb() (duckplyr) you write the path, like open_dataset(), but you get a DuckDB-backed table straight away. It is reported to be the fastest of the three and to keep memory use lowest, and to avoid the rapi errors arrow sometimes throws on very large registers. It needs an up-to-date duckplyr and duckdb - run install.packages() for both, and repeat after a server reset.

Whichever you pick, the rest of your code is the same: filter(), select() and collect() behave identically. You do not need to rewrite existing scripts to use a different one.

The exact path depends on your project and server. Column names for each register are in Overview of registers.

NoteWhen the years do not have the same columns - union_by_name

A converted register folder holds one parquet file per year (see Reading a register that is split by year), and they are read as one table. That works as long as every year has the same columns. It stops working when a register changes structure partway through - a column that appears in 2022 but not in 2021, for instance.

With read_parquet_duckdb() you handle that with an option:

library(duckplyr) # read_parquet_duckdb()

bef <- read_parquet_duckdb(
  "E:/workdata/[projectnumber]/cleaned-data/parquet-registers/bef",
  options = list(union_by_name = TRUE)
)

union_by_name = TRUE matches the files’ columns by name instead of by position, and fills in NA for the years where a column does not exist. Without it, DuckDB expects every file to have the same columns and stops with an error when they do not.

Use it deliberately, not by default. Without it you get an error, and that error is information: it tells you the register changed structure, which is something you want to know before you analyse it. With it you get a column that is silently NA for some years, and an NA you did not expect is much harder to spot than an error. Reach for it once you know what changed and have decided to read everything anyway.

You only need it for a folder of several files with differing columns. A single file, or a register whose years all match, does not need it.

Before any read_register() works, point fastreg at your registers once. It is a single options() call at the top of each script, and every read_register() example in this guide assumes you have made it: Point fastreg at your registers. Using open_dataset() instead? Then you write the path in each call and there is nothing to set up.

Reading a register that is split by year

DST delivers registers as SAS files, often one per year. Converting them to parquet is a job the project does once, and fastreg writes the result partitioned by year - <register>/year=YYYY/part-XXX.parquet - so a converted register is a folder of yearly files. See Parquet and fastreg.

You do not open those one at a time. Both open_dataset("path/to/bef/") and read_register("bef") read the whole folder, including the per-year subfolders, as a single combined dataset. So you pick years by filtering on the year column in the data:

bef %>% filter(year == 2015)

That column exists because of how the register was partitioned at conversion, not because DST supplies it - so its name can differ between projects. Confirm it against your own files rather than assuming.

year says which folder a row came from, nothing more. It is not a date and it is not a DST variable. Because the files are split by year, filtering on it means the other years are never opened at all, which is what makes a query against a register with a billion rows finish in seconds. That is what it is for: limiting how much gets read.

It is not for deciding when something happened. year == 2020 in BEF does not tell you which of the four quarterly snapshots you have, only which folder the row sat in. Whenever the timing matters to your research question, use the register’s own date column - hf_vfra in UDDA, d_inddto in lpr_adm, referencetid in BEF - and keep year for speed.

The two uses are easy to mix up because they look identical in the code, which is why it is also listed in DST pitfalls.

RDS is the format you write yourself most. It is R’s own format - fast, compact, and it preserves types, factor levels and column names exactly. You save an intermediate result in one script and reload it in the next, so script 2 does not have to re-run script 1:

saveRDS(cohort, "path/to/full_cohort.rds") # save an R object to disk
cohort <- readRDS("path/to/full_cohort.rds") # read it back in the next script

Rarer formats (Stata, SPSS, Feather, RData)

You rarely encounter these in a typical DST cohort study, but here they are for completeness:

File type	Package	Function
.dta (Stata)	haven	read_dta()
.sav (SPSS)	haven	read_sav()
.feather	arrow	read_feather()
.rdata / .rda	base R	load()

.rdata/.rda differs from .rds in that it can save multiple objects at once - but .rds is preferred because you control what the object is called when you read it back in.

When to use each format (Parquet, RDS, SAS, CSV)

The three formats you work with day to day:

File type	Used for
Parquet	The large registers (BEF, LPR, LMDB …). You load them lazily and filter before fetching data.
RDS	Your own intermediate results - datasets you save from one script and reload in the next.
SAS	Format tables and raw register data not yet converted to parquet.

SAS - for format tables and unconverted register data:

library(haven)
df <- read_sas("E:/rawdata/[projectnumber]/lpr_adm2018.sas7bdat")

Loading large SAS files is very slow - which is exactly why data on DST is converted to parquet. Only use SAS for format tables and files without a parquet version.

CSV - for exporting finished tables (e.g. at repatriation):

library(readr)
write_csv(my_table, "output/table1.csv")

Never save raw register data as CSV - only aggregated results. See Phase 14 - Export and repatriation for the rules.

Next step

Why lazy loading works, and how collect() functions, is the topic of the next phase.

→ Phase 5 - Extracting data step by step

Not got parquet yet, or read_register() cannot see a register you know is there? Both are on Parquet and fastreg.

TipFurther information

Further depth (in English):

• Import and export in The Epidemiologist R Handbook.
• Arrow in R for Data Science: reading parquet with open_dataset() and using dplyr directly on arrow data - exactly the loading pattern this guide builds on.