Machine Learning Analysis

Predictive Modeling Analysis ServiceNested-CV Classifiers and Survival Models with Calibration Reporting

Predictive modeling trains supervised algorithms on omics or clinical matrices to estimate outcome probabilities or survival risk for unseen samples [1]. Pepkio delivers nested cross-validation, calibration plots, serialized models, full code ownership, and a Methods draft for academic, biotech, and pharma teams; custom outcomes and validation designs are scoped at kickoff. Sammut et al. [2] reported AUROC 0.87 on a 75-patient external cohort.

Key facts

Key facts about Predictive Modeling
FactValue
Supported platforms / instrumentsFeature matrices from bulk RNA-seq (DESeq2 VST or normalized counts), Olink NPX, MaxQuant or DIA-NN protein tables, metabolite peak tables, methylation beta values, clinical and laboratory covariates; vendor-normalized matrices accepted with documented upstream processing
Input requirementsSample-level feature matrix (samples × features) plus outcome labels; survival endpoints require event indicator and follow-up time; feasibility reviewed with pmsampsize when binary or time-to-event; p >> n cohorts flagged before modeling [5, 6]
Reference builds supportedHuman GRCh38 (GENCODE v44 / Ensembl 110); mouse GRCm39 (GENCODE vM33 / Ensembl 110); UniProt Swiss-Prot accessions for protein harmonization; HMDB 5.0 metabolite IDs when metabolomics features are included; not applicable for purely clinical tables
Primary tools (with versions)scikit-learn 1.6.1; XGBoost 2.1.3; scikit-survival 0.24.1; glmnet 4.1-8 (R); SHAP 0.46.0; pmsampsize (R); limma 3.60; sva/ComBat when batch correction is scoped; clusterProfiler 4.20.0 for pathway context when scoped
Typical turnaround time3–5 weeks (single cohort, one outcome, ≤500 features, internal validation); 5–8 weeks (survival models, multi-cohort external validation) — confirmed at kickoff
Deliverable formats.csv performance and feature tables; serialized models (.pkl, .joblib, or .rds); PDF/SVG ROC, calibration, and SHAP figures; DOME summary table; TRIPOD+AI checklist when clinical prediction reporting is scoped; commented R/Python scripts; Methods draft
Key cited best-practice referenceWalsh et al. [1], Nature Methods (DOME); Collins et al. [7], BMJ (TRIPOD+AI when scoped)
Custom / bespoke analysisNon-standard outcomes, fusion models, client-specified metrics or thresholds, bespoke validation schemes, or domain-specific model families scoped at kickoff

What is predictive modeling?

Predictive modeling fits supervised algorithms—logistic regression, gradient-boosted trees, or survival forests—to estimate probabilities or risk scores for a pre-specified endpoint from high-dimensional biological or clinical predictors [3]. Unlike biomarker discovery, which prioritizes compact feature lists and selection stability, predictive modeling evaluates whether those features generalize to unseen samples with reported discrimination and calibration [1]. Unlike unsupervised clustering, every model is trained against labeled outcomes. A 2023 systematic review screened 682 PubMed articles (2017–2023) and critically reviewed 30 cancer ML biomarker studies, noting data-leakage risks and inconsistent evaluation among common limitations [4]. See the predictive modeling glossary.

When should you use predictive modeling?

Predictive modeling fits when you have labeled outcomes and need out-of-sample performance estimates—not when the primary deliverable is a ranked feature list without outcome-model reporting, or when sample size cannot support even regularized models [5].

Comparison of predictive modeling, biomarker discovery, and classical multivariable regression
ApproachBest forLimitationsApproximate cost range
Predictive modeling (nested CV)Outcome prediction, treatment-response stratification, survival risk scoring with calibration reportingRequires labeled outcomes; small cohorts limit external validity; p >> n needs regularization and careful validationQuote-based; moderate bioinformatics effort [1]
Biomarker discoveryCompact signatures, feature stability across resamples, exploratory panel constructionDoes not by default deliver full outcome-model calibration or TRIPOD+AI reportingSimilar per-cohort cost; narrower deliverable scope
Classical multivariable regression (Cox, logistic)Interpretable coefficients when predictors are pre-specified and low-dimensionalNo nested hyperparameter tuning; overfitting risk when features >> samples without regularizationLower when predictors are fixed; similar when high-dimensional
  • Breast cancer therapy response: Sammut et al. [2] integrated multi-omic and digital pathology features from 168 pre-treatment biopsies and predicted pathological complete response with AUROC 0.87 on an external 75-patient cohort.
  • Cancer of unknown primary: Moon et al. [7] trained OncoNPC on 36,445 tumors across 22 cancer types; high-confidence predictions (≥0.9) achieved weighted F1 0.942 on held-out samples and identified CUP subgroups with distinct survival.
  • Targeted therapy response: Sinha et al. [8] developed PERCEPTION, using single-cell tumor transcriptomics to predict targeted-therapy response and resistance in multiple myeloma and breast cancer clinical cohorts.

How the analysis works — step by step

  1. 1. Scope outcome, cohort, and validation plan

    Pepkio confirms the prediction target, endpoint type, primary metrics, and whether a holdout or external cohort exists [1, 7]. Sample-size rationale uses pmsampsize when binary or time-to-event endpoints apply [6].

    Tools and outputs

    Tools used: pmsampsize (R); project scoping templates

    Output: Signed scope document with primary and secondary metrics

  2. 2. Validate inputs and harmonize metadata

    Sample IDs are matched between feature matrix and outcome table; missingness, duplicates, class balance, and batch structure are audited.

    Tools and outputs

    Tools used: Custom Python/R validation scripts; pandas 2.2 / data.table

    Output: sample_manifest.csv; feature_qc_summary.csv

  3. 3. Preprocess and normalize features

    Features are filtered by missing-value fraction and variance; continuous predictors are scaled within cross-validation pipelines. Batch correction with limma or ComBat is applied within training folds only when batch is not confounded with the outcome [5].

    Tools and outputs

    Tools used: scikit-learn 1.6.1 ColumnTransformer; limma 3.60; sva/ComBat when scoped

    Output: Analysis-ready feature matrix; preprocessing parameter log

  4. 4. Audit for data leakage

    Pepkio applies the seven guiding questions from Bernett et al. [9] before partitioning—checking whether preprocessing, feature selection, or repeated measures could leak test-fold information.

    Tools and outputs

    Tools used: Documented leakage checklist per Bernett et al. [9]

    Output: leakage_audit.md

  5. 5. Partition data with patient-level nested cross-validation

    Outer cross-validation estimates generalization; an inner loop tunes hyperparameters and embedded feature selection [1]. Group-aware splitters keep repeated measures from the same patient in one fold.

    Tools and outputs

    Tools used: scikit-learn 1.6.1 StratifiedKFold, GridSearchCV; scikit-survival 0.24.1 for censored outcomes

    Output: fold_assignments.csv (sample_id, outer_fold, inner_fold, split_role)

  6. 6. Train and tune candidate models

    Pepkio compares elastic-net logistic regression, XGBoost, and survival forests against simpler baselines [3], with feature selection embedded inside each outer fold. Deep learning is scoped separately when justified by cohort size.

    Tools and outputs

    Tools used: scikit-learn 1.6.1; XGBoost 2.1.3; glmnet 4.1-8; scikit-survival 0.24.1

    Output: Candidate model list with hyperparameter grids; serialized best models per fold

  7. 7. Evaluate discrimination and calibration

    Discrimination (AUROC, AUPRC, or C-index) and calibration (reliability diagrams, Brier score, calibration slope) are reported from outer-fold predictions [7]. External performance is reported separately when a holdout or external cohort is available [5].

    Tools and outputs

    Tools used: scikit-learn 1.6.1 calibration_curve, brier_score_loss; scikit-survival 0.24.1 concordance_index_censored

    Output: model_performance_summary.csv; calibration_metrics.csv

  8. 8. Select operating threshold and decision metrics

    Pepkio documents threshold selection (Youden index on outer-fold predictions or a clinical cutoff) and reports sensitivity, specificity, and PPV at that threshold. Decision-curve analysis is included when scoped.

    Tools and outputs

    Tools used: scikit-learn 1.6.1 roc_curve; custom threshold summary scripts

    Output: threshold_summary.csv (threshold, sensitivity, specificity, PPV, NPV)

  9. 9. Interpret features and biological context

    SHAP values and coefficient tables identify top predictors; GO/KEGG annotation is included when gene or protein lists are in scope. Interpretability does not support causal claims [10].

    Tools and outputs

    Tools used: SHAP 0.46.0; clusterProfiler 4.20.0 when scoped

    Output: SHAP summary plots; selected_features.csv; pathway enrichment tables

  10. 10. Package deliverables and draft Methods

    Pepkio assembles figures, serialized models, DOME summary table, TRIPOD+AI checklist when scoped, environment locks, README, and Methods draft.

    Tools and outputs

    Tools used: conda or renv lock files; DOME and TRIPOD+AI checklists

    Output: Final deliverable bundle

What Pepkio delivers

Processed data files

  • Analysis-ready feature matrix (.csv); fold_assignments.csv
  • Serialized models (.pkl, .joblib, or .rds)
  • Outer-fold predictions in predicted_scores.csv (sample_id, predicted_probability, true_label, outer_fold)

Figures (PDF/SVG)

  • ROC and precision-recall curves; calibration diagrams
  • SHAP summary and beeswarm plots
  • Kaplan–Meier stratification when survival models are scoped; decision-curve plots when scoped

Tables

  • model_performance_summary.csv; calibration_metrics.csv
  • selected_features.csv; threshold_summary.csv; DOME summary table

Code

  • Commented R/Python scripts with conda or renv locks
  • Delivery via private Git repository or agreed file transfer; you retain full ownership

Documentation

  • README; leakage audit; Methods draft; TRIPOD+AI checklist when clinical prediction reporting is scoped
  • Post-delivery support: Methods clarification and minor revisions within agreed scope (typically ≤20% of deliverables). New cohorts or endpoints are separate milestones

Technical decisions we make — and why

Validation design: patient-level nested CV vs single holdout split
Nested cross-validation avoids unstable estimates when hyperparameters are tuned on the same partition used for scoring [1]. A holdout or external cohort supplements nested CV when available.
Model family: elastic-net baseline before tree ensembles
Regularized logistic or Cox models are compared against XGBoost or survival forests; simpler models are retained unless complexity improves outer-fold performance [3, 5].
Calibration: isotonic or Platt scaling on outer-fold predictions
Scores are recalibrated so predicted probabilities match observed event rates [7]. Uncalibrated scores are reported alongside calibrated outputs.
Class imbalance: stratified folds and balanced class weights vs SMOTE
Pepkio uses stratified cross-validation and class_weight='balanced' rather than resampling the full dataset before partitioning, because pre-split oversampling can share information between training and test folds.
Batch correction: limma or ComBat within training folds when batch is not confounded with outcome
Technical batch is corrected within training folds when batch and condition are separable; fully confounded designs are flagged at kickoff because correction would remove biological signal [5].

Common questions

What is the minimum sample size for predictive modeling?

There is no universal minimum—Riley et al. [6] and van Smeden et al. [11] show the 10 events-per-variable rule is unreliable as a sole criterion. Pepkio reviews event counts and predictor count with pmsampsize at kickoff and flags p >> n designs [5]. Feasibility is confirmed before modeling begins.

Can you build a predictive model from poor-quality or sparse feature data?

Partially. Pepkio filters high-missingness features, documents exclusion counts, and may reduce dimensionality with embedded regularization inside nested CV. Severely underpowered cohorts or features with >50% missingness across samples are flagged before model training. Pepkio does not claim external validity without a holdout or external cohort [5].

Which omics platforms and data formats do you support?

Pepkio accepts sample-level matrices from bulk RNA-seq (DESeq2 VST or normalized counts), Olink NPX, MaxQuant or DIA-NN protein tables, metabolite peaks, methylation beta values, and clinical covariates. Vendor-normalized matrices require documented upstream processing and feature identifiers. Custom formats and MGI-derived matrices are scoped at kickoff.

How long does predictive modeling take at Pepkio?

Standard single-cohort projects with one binary outcome and ≤500 features typically complete in 3–5 weeks from data receipt. Survival models, multi-cohort external validation, or bespoke validation schemes may require 5–8 weeks. Exact timelines are confirmed at kickoff with milestone check-ins after QC and before delivery.

How do you handle batch effects in predictive modeling?

Batch is diagnosed with PCA and correlation checks on the feature matrix. When batch is not confounded with the outcome, Pepkio applies limma removeBatchEffect or ComBat within training folds only [5]. Confounded batch-by-condition designs are flagged at kickoff; correction proceeds only when a defensible contrast remains.

Do I own the code—and in what format is it delivered?

Yes—you retain full ownership of all code, scripts, and results. Pepkio delivers commented Python and/or R scripts with conda environment.yml or renv.lock files, organized by pipeline stage with a README. Models are serialized in standard .pkl, .joblib, or .rds formats. Delivery is via private Git repository or agreed file transfer.

Can I be involved during the predictive modeling analysis?

Yes. Checkpoint reviews occur after input QC, after the leakage audit, and before final delivery. You can review outcome definitions, model family choices, threshold selection, and figure priorities within agreed scope. A dedicated scientific contact leads the project and incorporates your feedback at each agreed milestone before finalizing all deliverables.

What does post-delivery reviewer support cover?

Support covers clarification of validation design, calibration methods, threshold rationale, and minor figure or table revisions within agreed scope (typically ≤20% of deliverables). Pepkio drafts Methods and Supplementary text for analyses we performed. Reviewer-requested re-analyses, new cohorts, or endpoints are scoped as separate milestones rather than included post hoc.

Is co-authorship required for predictive modeling work?

No. Pepkio operates as a fee-for-service provider and does not require co-authorship unless explicitly discussed in advance. Standard practice is acknowledgment of bioinformatics support in the Acknowledgments section. Co-authorship is considered only when Pepkio scientists make substantial intellectual contributions beyond routine predictive modeling analysis work itself.

Should I use a classification or survival model for my endpoint?

Use classification for binary or multiclass outcomes at a fixed time point (e.g., responder vs non-responder). Use survival models when follow-up is censored and time-to-event matters. Pepkio documents the endpoint in the scope document and reports C-index and Kaplan–Meier stratification for survival endpoints [7].

Can Pepkio validate my model on TCGA, GEO, or another public cohort?

When a matched external cohort exists and is scoped at kickoff, Pepkio harmonizes feature identifiers (GENCODE v44 gene IDs, UniProt accessions) and reports external performance separately from nested-CV estimates [5]. Public-cohort validation is not included by default and depends on outcome overlap and feature overlap.

How do you report AUROC, calibration, and C-index?

AUROC and AUPRC measure classification discrimination; C-index measures survival concordance—all on outer-fold or holdout predictions, not training data [1]. Calibration uses Brier score, calibration slope, and reliability diagrams [7]. Confidence intervals use outer-fold or bootstrap resampling where sample size permits.

Related services

  • Biomarker discovery — Compact signature construction and feature stability analysis when the primary deliverable is a ranked panel rather than a full outcome model.
  • Multi-omics integration — Fused feature matrices across transcriptomics, proteomics, and metabolomics as predictors for outcome models.
  • Bulk RNA-seq — Expression matrix generation and normalization when raw RNA-seq counts are the modeling input.
  • Statistical analysis — Experimental design, covariate planning, and sample-size rationale before committing to ML.
  • Bioinformatics consulting — Feasibility assessment and validation planning before model development.
References
  1. Walsh I, Fishman D, Garcia-Gasulla D, et al. DOME: recommendations for supervised machine learning validation in biology. Nature Methods. 2021;18(10):1122–1127. https://doi.org/10.1038/s41592-021-01205-4 (PMID: 34316068)
  2. Sammut S-J, Liu B, Ryu D, et al. Multi-omic machine learning predictor of breast cancer therapy response. Nature. 2022;601(7894):623–629. https://doi.org/10.1038/s41586-021-04278-5 (PMID: 34875674)
  3. Greener JG, Kandathil SM, Moffat L, Jones DT. A guide to machine learning for biologists. Nature Reviews Molecular Cell Biology. 2022;23(1):40–55. https://doi.org/10.1038/s41580-021-00407-0 (PMID: 34518686)
  4. Al-Tashi Q, Saad MB, Muneer A, et al. Machine learning models for the identification of prognostic and predictive cancer biomarkers: a systematic review. International Journal of Molecular Sciences. 2023;24(9):7781. https://doi.org/10.3390/ijms24097781 (PMID: 37175487)
  5. Whalen S, Schreiber J, Noble WS, Pollard KS. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics. 2022;23(3):169–181. https://doi.org/10.1038/s41576-021-00434-9 (PMID: 34837041)
  6. Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441. https://doi.org/10.1136/bmj.m441 (PMID: 32188600)
  7. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://doi.org/10.1136/bmj-2023-078378 (PMID: 38626948)
  8. Sinha S, Vegesna R, Mukherjee S, et al. PERCEPTION predicts patient response and resistance to treatment using single-cell transcriptomics of their tumors. Nature Cancer. 2024;5(6):938–952. https://doi.org/10.1038/s43018-024-00756-7 (PMID: 38637658)
  9. Bernett J, Blumenthal DB, Grimm DG, et al. Guiding questions to avoid data leakage in biological machine learning applications. Nature Methods. 2024;21(8):1444–1453. https://doi.org/10.1038/s41592-024-02362-y (PMID: 39122953)
  10. Chen V, Yang M, Cui W, et al. Applying interpretable machine learning in computational biology—pitfalls, recommendations and opportunities for new developments. Nature Methods. 2024;21(8):1454–1461. https://doi.org/10.1038/s41592-024-02359-7 (PMID: 39122941)
  11. van Smeden M, Moons KGM, de Groot JAH, et al. Sample size for binary logistic prediction models: beyond events per variable criteria. Statistical Methods in Medical Research. 2019;28(8):2455–2474. https://doi.org/10.1177/0962280218784726 (PMID: 29966490)

Let's Talk About Your Science

Tell us:

  • • Your biological question
  • • Data type and size
  • • Timeline constraints

We'll tell you:

  • • What's feasible
  • • How long it will take
  • • Exactly what it will cost
Contact Us

Contact us to start with a free consultation. Need everyday bench calculators? Try our free lab tools.