Getting started

What does this package do?

The osdc (Open Source Diabetes Classifier) package classifies individuals as having type 1 diabetes (T1D) or type 2 diabetes (T2D) based on Danish register data. You provide the register data, and the package returns a data table of individuals identified as having diabetes, along with their diabetes type (T1D or T2D) and the date of the classification.

This package serves two overarching purposes:

1. To provide an open-source, code-based algorithm to classify type 1 and type 2 diabetes using Danish registers as data sources.
2. To inspire discussions within the Danish register-based research space on the openness and ease of use on the existing tooling and registers, and on the need for an official process for updating or contributing to existing data sources.

For a detailed description of the algorithm, see vignette("algorithm"). For the motivations and rationale behind this package, see vignette("rationale"). For a full list of required register variables, see vignette("data-sources").

Step-by-step usage

This section walks through the full workflow: loading the package, preparing data, running the classification, as well as understanding and saving the output. We use simulated data here, but the same or similar steps apply to real register data. For more information on using osdc on real register data, see Section 3.

Step 1: Install and load the package

First, you need to install and load the package:

# install.packages("osdc")
library(osdc)

Step 2: Check which registers are needed

The algorithm requires data from multiple Danish registers. The names of these registers can be seen below:

Register abbreviation	Register name
bef	CPR-registerets befolkningstabel
lmdb	Laegemiddelstatistikregisteret
lpr_adm	Landspatientregisterets administrationstabel (LPR2)
lpr_diag	Landspatientregisterets diagnosetabel (LPR2)
lpr3a_kontakt	Landspatientregisterets kontakttabel (LPR3A)
lpr3a_diagnose	Landspatientregisterets diagnosetabel (LPR3A)
lpr3f_kontakter	Landspatientregisterets kontakttabel (LPR3F)
lpr3f_diagnoser	Landspatientregisterets diagnosetabel (LPR3F)
sysi	Sygesikringsregisteret
sssy	Sygesikringsregisteret
lab_forsker	Laboratoriedatabasens forskertabel

For a table showing the specific variables needed from each register, see vignette("data-sources").

Important

To use the osdc algorithm, you need to have all the variables described in vignette("data-sources"), so please ensure that you have the required variables before continuing.

Step 3: Prepare the data

The package requires data to be in DuckDB format, which is a high-performance database format that can handle the large volumes of data without loading everything into memory at once. See vignette("design") for more on why we use DuckDB.

For this example, we generate simulated register data using simulate_registers() and then convert it to DuckDB format:

register_data <- registers() |>
  names() |>
  simulate_registers() |>
  purrr::map(duckplyr::as_duckdb_tibble) |>
  # Convert to a DuckDB connection, as duckplyr is still
  # in early development, while the DBI-DuckDB connection
  # is more stable.
  purrr::map(duckplyr::as_tbl)
#> duckdb keeps downloaded extensions and secrets in a temporary directory:
#> ℹ /tmp/RtmpWT4Vp2/duckdb
#> This is removed when the R session ends.
#> • Extensions are re-downloaded each session.
#> • Secrets are lost.
#> ℹ Run duckdb(shared_home = TRUE) (or create ~/.duckdb) to keep them (suitable for most users).
#> ℹ Run duckdb(shared_home = FALSE) to accept the temporary directory (and silence this message).
#> ℹ See ?duckdb_storage for details and alternatives.

The result is a named list where each element is one register as a DuckDB table. The LPR data spans multiple versions (LPR2 and LPR3) that need to be prepared and joined into a single table before being used as input for classify_diabetes(). The same goes for the health service registers (SSSY, and SYSI). osdc provides helper functions for this:

lpr <- list(
  prepare_lpr2(register_data$lpr_adm, register_data$lpr_diag),
  prepare_lpr3f(register_data$lpr3f_kontakter, register_data$lpr3f_diagnoser),
  prepare_lpr3a(register_data$lpr3a_kontakt, register_data$lpr3a_diagnose)
) |>
  join_registers()

hsr <- list(register_data$sssy, register_data$sysi) |> join_registers()

The remaining registers are extracted from the register_data list, so they are ready to be passed as arguments to classify_diabetes():

bef <- register_data$bef
lmdb <- register_data$lmdb
lab_forsker <- register_data$lab_forsker

Step 4: Run the classification

Now, we’re ready to run the classification algorithm. Pass each register to classify_diabetes():

classified_diabetes <- classify_diabetes(
  lpr = lpr,
  hsr = hsr,
  lab_forsker = lab_forsker,
  bef = bef,
  lmdb = lmdb
)

classified_diabetes
#> # A query:  ?? x 5
#> # Database: DuckDB 1.5.5 [unknown@Linux 6.17.0-1022-azure:R 4.6.1//tmp/RtmpWT4Vp2/duckplyr/duckplyr1c465937d48a.duckdb]
#>    pnr          stable_inclusion_date raw_inclusion_date has_t1d has_t2d
#>    <chr>        <date>                <date>             <lgl>   <lgl>  
#>  1 732715981647 2016-12-19            2016-12-19         FALSE   TRUE   
#>  2 238357358504 2015-11-14            2015-11-14         FALSE   TRUE   
#>  3 476020884782 2011-11-19            2011-11-19         FALSE   TRUE   
#>  4 963036718466 2016-09-25            2016-09-25         FALSE   TRUE   
#>  5 070597786658 2025-04-23            2025-04-23         FALSE   TRUE   
#>  6 409442575549 2017-08-21            2017-08-21         FALSE   TRUE   
#>  7 742356346597 2011-06-04            2011-06-04         FALSE   TRUE   
#>  8 706974528463 2010-10-11            2010-10-11         FALSE   TRUE   
#>  9 240771768588 2008-03-31            2008-03-31         FALSE   TRUE   
#> 10 298944792608 2012-04-30            2012-04-30         FALSE   TRUE   
#> 11 498989088479 2007-04-09            2007-04-09         FALSE   TRUE

As seen above, this returns a DuckDB table with the individuals classified as having either T1D or T2D along with the date of the classification. Each of the columns in the output is explained in Section 2.5.1 below.

Step 5 (optional): Collect the results into R

Because the data is stored in DuckDB, the result above is a lazy reference to a database query, i.e., the data has not been loaded into R’s memory. To bring the results into R as a regular data frame, use dplyr::collect():

classified_diabetes <- classified_diabetes |>
  dplyr::collect()

classified_diabetes
#> # A tibble: 11 × 5
#>    pnr          stable_inclusion_date raw_inclusion_date has_t1d has_t2d
#>    <chr>        <date>                <date>             <lgl>   <lgl>  
#>  1 706974528463 2010-10-11            2010-10-11         FALSE   TRUE   
#>  2 409442575549 2017-08-21            2017-08-21         FALSE   TRUE   
#>  3 742356346597 2011-06-04            2011-06-04         FALSE   TRUE   
#>  4 963036718466 2016-09-25            2016-09-25         FALSE   TRUE   
#>  5 070597786658 2025-04-23            2025-04-23         FALSE   TRUE   
#>  6 240771768588 2008-03-31            2008-03-31         FALSE   TRUE   
#>  7 732715981647 2016-12-19            2016-12-19         FALSE   TRUE   
#>  8 298944792608 2012-04-30            2012-04-30         FALSE   TRUE   
#>  9 498989088479 2007-04-09            2007-04-09         FALSE   TRUE   
#> 10 238357358504 2015-11-14            2015-11-14         FALSE   TRUE   
#> 11 476020884782 2011-11-19            2011-11-19         FALSE   TRUE

Now, we can see that with the simulated data, 11 individuals are classified as having diabetes.

Understanding the output

The output is a table with one row per classified individual and five columns:

Column	Description
pnr	The pseudonymised personal identification number.
stable_inclusion_date	The date of the second inclusion event, if on or after stable_inclusion_start_date (default: 1998). NA for earlier dates.
raw_inclusion_date	The date of the second inclusion event without setting NA for date earlier than the stable_inclusion_start_date.
has_t1d	TRUE if classified as type 1 diabetes, FALSE otherwise.
has_t2d	TRUE if classified as type 2 diabetes, FALSE otherwise.

About stable_inclusion_date vs raw_inclusion_date

The raw_inclusion_date is simply the date of the second qualifying event. However, for events before 1998, the register data may not have sufficient coverage to reliably distinguish new (incident) cases from existing (prevalent) ones. The stable_inclusion_date column is set to NA for these earlier dates to flag this uncertainty.

The classify_diabetes() function includes a stable_inclusion_start_date parameter that is 01-01-1998 by default. This means that you can change the date for when the classification is considered stable

For more information about the output, see the Interface section under vignette("design").

Step 6: Saving the results

Once you have the classification results, you can save them as a Parquet file for yourself or your collaborators on your DST project:

classified_diabetes |>
  duckplyr::as_duckdb_tibble() |>
  duckplyr::compute_parquet(
    "classified_diabetes.parquet"
  )

Working with real register data

In a real-world scenario, the register data is too large to read into memory all at once. We recommend converting your register files into Parquet format on disk, with each register in its own folder (e.g., all lmdb files in one folder, all lab_forsker files in another, etc.).

Tip

To convert SAS (.sas7bdat) files to Parquet, you can use the fastreg package.

If you’re working on Statistics Denmark’s (DST) server, be aware that the pre-installed R packages can be quite old (for example, at the time of writing, library(duckplyr) loads version 0.4 of the duckplyr package from 2024, whereas the latest version on CRAN is >1.1 from 2026). These old versions don’t support the dplyr operations necessary for osdc to work.

Installing osdc with install.packages("osdc") should force the necessary updates of package dependencies. Otherwise, the latest version of any given package can be installed from DST’s local CRAN mirror by using install.packages(). The downside of this approach is that you have to repeat the package installations frequently (whenever you log on to a new virtual machine, or after the servers’ weekly reset).

After making sure that you have the newest version of osdc installed (which should install/update any necessary dependencies), you can load each register directly from its Parquet folder and convert it to DuckDB, as shown below (note: read_parquet_duckdb() requires at least version 2.5.4 of the duckdb package to work).

lpr3f_diagnoser <- "path/to/lpr3f_diagnoser_parquet_folder" |>
  duckplyr::read_parquet_duckdb(options = list(union_by_name = TRUE)) |>
  duckplyr::as_tbl() |>
  # Optionally, insert dplyr functions for variable renaming/type-casting, selection or row filtering here, e.g.:
  # dplyr::select(...) |>
  # dplyr::mutate(...) |>
  # dplyr::filter(...)

If your data (or parts of it) is already in R (e.g., as a hypothetical data.frame, named your_dataset_in_r in the example below), you can convert it to a DuckDB table with:

your_dataset_in_duckdb <- your_dataset_in_r |>
  duckplyr::as_duckdb_tibble() |>
  duckplyr::as_tbl()

Important

Important notes on using lpr3a data!

A: lpr3a contains duplicates of contacts in 2017 & 2018 that are also contained in lpr2. These rows must be filtered out before being input to osdc’s prepare_lpr3a() function.

B: The kont_starttidspunkt variable in lpr3a is a datetime type, and must be converted to a date before being input to prepare_lpr3a().

C: In our experience, lpr3a always contains all the data previously delivered in the lpr3_f format, and you should rarely (if ever) use lpr3f in addition to lpr3a in practice.

The duplicate rows in lpr3a_kontakt can be removed during processing by filtering to lprindberetningssystem == "LPR3" Similarly, the kont_starttidspunkt variable in lpr3a can be converted to a date in a single call to dplyr::mutate() during pre-processing. e.g.:

lpr3a_kontakt <- "path/to/lpr3a_kontakt_parquet_folder" |>
  duckplyr::read_parquet_duckdb(options = list(union_by_name = TRUE)) |>
  duckplyr::as_tbl() |>
  dplyr::mutate(kont_starttidspunkt = as.Date(kont_starttidspunkt)) |>
  dplyr::filter(lprindberetningssystem == "LPR3")

This can be used as input to prepare_lpr3a() (while this example shows how easy it is to use all three lpr formats with osdc, the lpr3f format is practically deprecated and should rarely be used in practice, as noted above):

lpr <- list(
  prepare_lpr2(lpr_adm, lpr_diag),
  prepare_lpr3f(lpr3f_kontakter, lpr3f_diagnoser),
  prepare_lpr3a(lpr3a_kontakt, lpr3a_diagnose)
) |>
  join_registers()

Most inputs will work in the Arrow/DuckDB equivalent of their type in the original SAS file. Unless the SAS-to-Arrow conversion introduced unexpected types, you should should not need to do any further type conversion.

Example of end-to-end classification pipeline from Parquet

The following shows an example pipeline for running on Statistics Denmark’s “Research Machine” server. The example uses the raw, unprocessed data as provided by Statistics Denmark and the Danish Health Data Authority after conversion to Parquet files. The register file names, variable names and types reflect the raw data structure as provided to the DARTER Project in Q2 2026, but these are likely to change in the future (for example, we experienced substantial changes in the LPR A data between a data update in Q4 2025 and one in Q2 2026). At the time of writing, some of the register sources (mainly LPR3) are undocumented, and users should carefully review the structure of their raw data and edit the pre-processing steps to account for any differences in their data.

A few notes on this example:

• On the DST server, the runtime from end to end was around 1h45m, with a maximum memory footprint around 130 GB.
• duckplyr::load_parquet_duckdb() requires an updated version of duckdb to work, hence the example starts with a code snippet to verify this.
• All the input registers cover the entire Danish population and contain data until the end of 2024 (with the exception of the laboratory results data, which include 2025 and part of 2026). As the project had received LPR A data, previous LPR F data was not used (it was presumed redundant).
• While the row filtering and variable selection performed on some of the input data during the pre-processing steps isn’t strictly necessary, it may lower the execution time and reduce the memory footprint.
• As a user, the main thing to pay attention to is the renaming/typing of variables to fit osdc’s expectations.
  • osdc is not case-sensitive, so you don’t need to worry about the casing of variable names.

Click here to show full example

Pre-process input data

Verify duckdb package version

if (packageVersion("duckdb") < "1.5.4") {
    stop(
      "For this workflow, duckplyr::load_parquet_duckdb() requires duckdb >= 1.5.4, but ",
      packageVersion("duckdb"), " is installed. Please update duckdb.",
      call. = FALSE
    )
  }

Background population data

bef_dir <- "E:/workdata/project-id/parquet-data/bef"

bef_ddb <- duckplyr::read_parquet_duckdb(
  bef_dir,
  options = list(union_by_name = TRUE)
  ) |>
     duckplyr::as_tbl() |>
     dplyr::mutate(koen = as.integer(KOEN)) |> # Cast to osdc's expected `integer` type (from `double`)
     dplyr::select(PNR, koen, foed_dato = FOED_DAG) # osdc expects the variable name `foed_dato`

Health Service Register: sssy & sysi

sssy_dir <- "E:/workdata/project-id/parquet-data/sssy"

sssy_ddb <- duckplyr::read_parquet_duckdb(
  sssy_dir,
  prudence = "stingy",
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::mutate(BARNMAK = as.integer(BARNMAK)) |>  # Cast to osdc's expected `integer` type (from `double`)
    dplyr::select(PNR, BARNMAK, HONUGE, SPECIALE) |>
    dplyr::filter(grepl("^54", SPECIALE)) # Filters to only the rows needed (diabetes-specific podiatrist services)


sysi_dir <- "E:/workdata/project-id/parquet-data/sysi"

sysi_ddb <- duckplyr::read_parquet_duckdb(
  sysi_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::mutate(BARNMAK = as.integer(BARNMAK)) |>  # Cast to osdc's expected `integer` type (from `double`)
    dplyr::select(PNR, BARNMAK, HONUGE, SPECIALE) |>
    dplyr::filter(grepl("^54", SPECIALE)) # Filters to only the rows needed (diabetes-specific podiatrist services)

Lab data: laboratorieproevesvar

lab_dir <- "E:/workdata/project-id/parquet-data/laboratorieproevesvar_"

lab_ddb <- duckplyr::read_parquet_duckdb(
  lab_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::filter(analysiscode %in% c("NPU27300", "NPU03835")) |> # Filter to only HbA1c tests
    dplyr::filter(samplingdate <= as.Date("2024-12-31")) |>  # Remove unusable data
    dplyr::filter(grepl("^[0-9]", samplevalue)) |> # Remove non-numeric values
    dplyr::mutate(value = as.numeric(samplevalue)) |> #  Convert to a `double` type (from `string`)
    dplyr::select(pnr = cprnummer, value, analysiscode, samplingdate) # osdc expects the variable name `pnr`

Patient Register: lpr_adm, lpr_diag, lpr_a_kontakt, lpr_a_diagnose

LPR2

lpr_adm_dir <- "E:/workdata/project-id/parquet-data/lpr_adm"

lpr_adm_ddb <- duckplyr::read_parquet_duckdb(
  lpr_adm_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::select(PNR, RECNUM, D_INDDTO, C_SPEC)


lpr_diag_dir <- "E:/workdata/project-id/parquet-data/lpr_diag"

lpr_diag_ddb <- duckplyr::read_parquet_duckdb(
  lpr_diag_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::select(RECNUM, C_DIAG, C_DIAGTYPE)

LPR_A

lpr_a_kontakt_dir <- "E:/workdata/project-id/parquet-data/lpr_a_kontakt"

lpr_a_kontakt_ddb <- duckplyr::read_parquet_duckdb(
  lpr_a_kontakt_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::filter(lprindberetningssystem == "LPR3") |> # Remove duplicates
    dplyr::mutate(kont_starttidspunkt = as.Date(kont_starttidspunkt)) |> # Recast to `date` from `datetime` type.
    dplyr::select(pnr, dw_ek_kontakt, kont_starttidspunkt, kont_ans_hovedspec)


lpr_a_diagnose_dir <- "E:/workdata/project-id/parquet-data/lpr_a_diagnose"

lpr_a_diagnose_ddb <- duckplyr::read_parquet_duckdb(
  lpr_a_diagnose_dir,
  options = list(union_by_name = TRUE)
  ) |>
     duckplyr::as_tbl() |>
     dplyr::filter(lprindberetningssystem == "LPR3") |> # Remove duplicates
     dplyr::select(dw_ek_kontakt, diag_kode, diag_type = diag_kode_type, senere_afkraeftet) # osdc expects the variable name `diag_type`

Prescription data: lmdb

lmdb_dir <- "E:/workdata/project-id/parquet-data/lmdb"


lmdb_ddb <- duckplyr::read_parquet_duckdb(
  lmdb_dir,
  options = list(union_by_name = TRUE)
  ) |>
    duckplyr::as_tbl() |>
    dplyr::filter(grepl("^A10", atc)) |>  # Filters to only the rows needed (glucose-lowering drugs)
    dplyr::select(pnr, eksd, atc, apk, volume, indo)

Execute osdc and save to disk

Join registers spread across multiple tables

hsr <- list(sssy_ddb, sysi_ddb) |> join_registers() # Join health service register tables to a single input

lpr2 <- prepare_lpr2(lpr_adm = lpr_adm_ddb, lpr_diag = lpr_diag_ddb) # Join lpr2 tables
lpr3_a <- prepare_lpr3a(lpr3a_kontakt = lpr_a_kontakt_ddb, lpr3a_diagnose = lpr_a_diagnose_ddb) # Join lpr3_a tables
lpr <- list(lpr2, lpr3_a) |> join_registers() # Join all patient register tables to a single input

Run the classification

osdc_population_202412 <- classify_diabetes(
  bef = bef_ddb,
  lpr = lpr,
  hsr = hsr,
  lab_forsker = lab_ddb,
  lmdb = lmdb_ddb)

osdc_population_202412_collected <- osdc_population_202412 |> dplyr::collect() # Execute the pipeline and collect results into R

# Post-processing

osdc_population_202412_clean <- osdc_population_202412_collected |>
  dplyr::filter(grepl("^[0-9]", pnr)) |> # Remove invalid pnr numbers from the population
  dplyr::mutate(data_coverage_limit = as.integer(202412)) # add metadata on data coverage

# Save to Parquet

osdc_population_202412_clean |>
  duckplyr::compute_parquet("osdc_population_202412.parquet")

Getting help

If you encounter a bug or have a question about how to use osdc, please open an issue on the GitHub repository. When reporting a bug, include a minimal example that reproduces the problem along to help us understand and investigate the problem.