Skip to content
BranchesTags

Repository files navigation

Data-driven Identification of Complex Disease Phenotypes

DOI License: GPL v3 Python 3.7+

This repository contains the computational analysis code for identifying complex disease phenotypes from electronic health records using multinomial na�ive Bayes models and itemset mining.

Publication

Strauss MJ, Niederkrotenthaler T, Thurner S, Klimek P (2021)
Data-driven identification of complex disease phenotypes
Journal of The Royal Society Interface 18(176): 20201040
DOI: 10.1098/rsif.2020.1040

⚠️ Important: Data Privacy Notice

This repository contains NO real patient data. The original analysis was performed on sensitive Austrian hospital records (1997-2008) that cannot be shared publicly due to privacy regulations.

This code repository uses synthetic mock data for demonstration purposes. The mock data:

  • Contains 10,000 simulated patients with synthetic diagnoses
  • Uses real ICD-10 code structure but purely random associations
  • Allows the code to run and demonstrate the methodology
  • Does NOT reproduce the results from the published paper

See DATA_PRIVACY.md for more information.

Overview

This project implements a novel approach to identify disease phenotypes - sets of co-occurring diagnoses that optimally predict future disease occurrences. The methodology combines:

  1. Itemset Mining (Apriori): Discover frequent diagnosis combinations
  2. Feature Mapping: Transform individual diagnoses to phenotype features
  3. Multinomial Naive Bayes: Predict future diagnoses from phenotypes
  4. Cross-validation: Hyperparameter tuning and model evaluation
  5. Network Analysis: Identify phenotype relationships and synergies

Key Features

  • Higher-order disease interactions (not just pairwise)
  • Synergistic phenotypes (diseases that amplify each other's risks)
  • Scalable to large patient populations
  • Interpretable model coefficients (weights of evidence)

Quick Start

Prerequisites

Installation

# Clone the repository
git clone https://github.com/YOUR_USERNAME/disease-phenotypes-public.git
cd disease-phenotypes-public

# Install Python dependencies with Poetry
poetry install

# OR with pip
pip install -r requirements.txt

# Install feature-mapper library
pip install feature-mapper

Generate Mock Data

# Generate synthetic patient data
python scripts/generate_mock_data.py

This creates CSV files in mock_data/:

  • patients.csv - 10,000 synthetic patients (demographics)
  • diagnoses_pre.csv - Pre-period diagnoses (2002-2004)
  • diagnoses_post.csv - Post-period diagnoses (2005-2008)
  • icd10_codes.csv - ~2,000 ICD-10 diagnosis codes

Run Analysis (Simplified)

# Run a quick example analysis
python scripts/run_example.py

Or for the full pipeline (takes longer):

# Load environment
poetry shell

# Generate mock data (if not done already)
python scripts/generate_mock_data.py

# Run the full analysis pipeline
python scripts/run_pipeline.py

Project Structure

disease-phenotypes-public/
├── README.md                   # This file
├── config.ini                  # Configuration (uses mock data)
├── pyproject.toml              # Python dependencies
├── CODE_OF_CONDUCT.md
├── LICENSE                     # GPL-3.0
├── CITATION.cff                # Citation information
│
├── mock_data/                  # Synthetic data (generated)
│   ├── README.md
│   ├── patients.csv
│   ├── diagnoses_pre.csv
│   ├── diagnoses_post.csv
│   └── icd10_codes.csv
│
├── scripts/                    # Entry point scripts
│   ├── generate_mock_data.py  # Generate synthetic data
│   ├── run_example.py          # Quick demonstration
│   └── run_pipeline.py         # Full analysis pipeline
│
├── code/                       # Main codebase
│   ├── pylib/                  # Python library
│   │   └── mt/                 # Main analysis modules
│   │       ├── dataset.py      # Dataset handling
│   │       ├── naive_bayes.py  # MNB classifier
│   │       ├── feature_mapper.py
│   │       ├── database/
│   │       │   ├── db.py       # Database wrapper
│   │       │   └── mock_db.py  # Mock data adapter
│   │       └── ...
│   ├── analysis/               # Analysis scripts
│   │   ├── main/               # Main analysis
│   │   ├── pre/                # Preprocessing
│   │   └── post/               # Post-processing
│   └── processing/             # Data processing utilities
│
├── docs/                       # Documentation
│   ├── DATA_PRIVACY.md         # Data privacy notice
│   ├── USAGE_EXAMPLES.md       # Detailed usage examples
│   └── LIMITATIONS.md          # Known limitations
│
└── notebooks/                  # Jupyter notebooks (optional)
    └── tutorial.ipynb

Methodology

1. Data Structure

The analysis requires patient data in two time periods:

  • Feature period (T1): 2002-2004 - diagnoses used as features
  • Target period (T2): 2005-2008 - diagnoses to predict

2. Phenotype Identification

Phenotypes are identified using the Apriori algorithm with constraints:

  • Minimum support: e.g., 50 patients must have the combination
  • minIDP: Minimum Information Difference to Prior (mutual information)
  • Model order: Maximum phenotype size (1-4 diagnoses)

Example phenotype: {I25p, E78s, E66s} = primary ischaemic heart disease + secondary lipid disorder + secondary obesity

3. Feature Mapping

The feature-mapper library (Rust + Python) transforms diagnosis matrices:

  • Input: Patient × Diagnosis matrix (sparse binary)
  • Mapping: Phenotype × Diagnosis matrix
  • Output: Patient × Phenotype matrix

Implemented as a greedy algorithm that applies phenotypes in order of:

  1. Size (larger phenotypes first)
  2. Information content (minIDP)
  3. Support (frequency)

4. Predictive Modeling

Multinomial Naive Bayes (MNB) model:

  • Features: Phenotypes in period T1
  • Targets: Individual diagnoses in period T2
  • Coefficients: Weights of evidence (log-odds in decibans)
  • Cross-validation: 5-fold with hyperparameter tuning

5. Synergy Detection

Lift metric quantifies synergistic interactions:

L(f, t) = C(f, t) - Ĉ(f, t)

Where:

  • C(f, t): Coefficient for phenotype f predicting target t
  • Ĉ(f, t): Expected coefficient from constituent diagnoses
  • Positive lift indicates synergy (whole > sum of parts)

Configuration

Edit config.ini to customize the analysis:

[database]
engine = mock  # Use mock data

[dataset]
observation-window = {(2002, 2004, 2005, 2009)}

[crossval]
min-model-order = 1
max-model-order = 4
reps = 2
folds = 5

[apriori]
min-support = 50
min-idp = 0.0

Usage with Your Own Data

To use this code with your own electronic health record data:

  1. Prepare your data in the same format as the mock data:

    • Patient demographics (ID, birth year, sex)
    • Diagnosis records (patient ID, ICD code, primary/secondary flag, year)

  2. Option A: Use CSV files

    • Place your CSV files in a directory
    • Set engine = mock:/path/to/your/data in config.ini

  3. Option B: Use PostgreSQL database

    • Set up a database following the schema in code/ext/goeg-data/db/schema.sql
    • Set engine = postgresql:///your_db_name in config.ini
    • Specify the appropriate adapter (see original code for GOEG/HVB adapters)

See docs/USAGE_EXAMPLES.md for detailed instructions.

Dependencies

Core Python Packages

  • numpy, pandas, scipy - Data manipulation
  • scikit-learn - Machine learning utilities
  • feature-mapper - Fast feature mapping (Rust backend)
  • tables (PyTables) - HDF5 file handling
  • networkx - Network analysis

External Tools

  • Apriori - Itemset mining (Borgelt's implementation)

Optional

  • matplotlib, seaborn - Visualization
  • jupyter - Interactive notebooks

See pyproject.toml for complete dependency list.

Citation

If you use this code in your research, please cite the paper:

@article{strauss2021data,
  title={Data-driven identification of complex disease phenotypes},
  author={Strauss, Markus J and Niederkrotenthaler, Thomas and Thurner, Stefan and Klimek, Peter},
  journal={Journal of The Royal Society Interface},
  volume={18},
  number={176},
  pages={20201040},
  year={2021},
  publisher={The Royal Society},
  doi={10.1098/rsif.2020.1040}
}

And consider citing the feature-mapper library:

@software{strauss2020feature,
  author = {Strauss, Markus J},
  title = {feature-mapper: Binary feature matrix transformation},
  year = {2020},
  url = {https://github.com/complexity-science-hub/feature-mapper}
}

License

This project is licensed under the GNU General Public License v3.0 - see the LICENSE file for details.

Contact & Contributions

Authors:

  • Markus J. Strauss - Complexity Science Hub Vienna
  • Thomas Niederkrotenthaler - Medical University of Vienna
  • Stefan Thurner - Complexity Science Hub Vienna
  • Peter Klimek - Complexity Science Hub Vienna

Issues and Questions: Please open an issue on GitHub if you:

  • Find bugs or problems with the code
  • Have questions about the methodology
  • Want to contribute improvements

Contributions: We welcome contributions! Please:

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes with tests
  4. Submit a pull request

Acknowledgments

This research was conducted at the Complexity Science Hub Vienna using data from the Austrian Federal Ministry of Health. We thank all collaborators and the institutions that made this work possible.

Note: The original data cannot be shared due to privacy regulations. This repository provides mock data for educational and reproducibility purposes only.

Related Resources

Last updated: February 2026

About

Data-driven identification of complex disease phenotypes

Resources

Readme
License
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages