Data analysis tips and help

Material and resources for best practices, learning, and support for doing data analysis.

Author

Affiliation

Omar Silverman

Warning

🚧 This website and most of its contents are often updated or modified. Many documents are at various stages of completion. 🚧

Some general practices doing data analysis

Save the raw data
Ensure that raw data are backed up in more than one location
Create the data you wish to see in the world
Create analysis-friendly data
Record all the steps used to process data
Aanticipate the need to use multiple tables, and use iunique identifier for every record
Submit data to a reptable DOI-issuing repository so that others can acces and cite it

Put each project in its own directory, which is named after the project
Put text documents associated with the project in the “doc” directory
Put raw data and metadata in a data directory and files generated during cleanup and analysis in a “result” directory
Put project source code in the “src” directory
Put external scripts or compiled programs in the “bin” directory
Name all files to reflect their content or function

Back up (almost) everything created by a human being as soon as it is created
Keep changes small
Share changes frequently
Create, maintain, and use a checklist for saving and sharing changes to the project
Store each project in a folder that is mirrored off the researchers’ working machine
Add a file called changelog.txt to the project’s “docs” subfolder
Copy the entire project whenever a significant change has made
Use a version control system

Write manuscripts using online tools with rich formatting, change tracking, and reference management
Write a manuscript in a plain text format that permits version control