3. Classification¶
Page 3 of 9. previous: 2. Quality control | index | next: 4. Explainability
Three classifiers were trained on the 10,000 selected CpGs of the 610 training tumours: logistic regression, a random forest, and a small neural network (a multilayer perceptron, MLP, two hidden layers, dropout). Each was trained for two tasks: the four subgroups and the twelve subtypes.
Balanced accuracy is the mean of the per-class recalls. It weights every class equally, so a rare subtype counts as much as a common one. Plain accuracy would be dominated by Group 4. Macro F1 is the mean of the per-class F1 scores. Both are reported.
1. Cross-validation and the held-out test split¶
Five-fold cross-validation on the training split gives an estimate of performance with a spread. The held-out test split, 153 tumours never used for feature selection or training, gives one final number per model.
import json
import pandas as pd
from IPython.display import Image, display
from pathlib import Path
TAB = Path('../results/tables')
FIG = Path('../results/figures')
pd.set_option('display.width', 160)
pd.set_option('display.max_columns', 30)
pd.set_option('display.max_colwidth', 90)
pd.read_csv(TAB / 'cv_summary.csv', header=[0, 1], index_col=[0, 1])
| balanced_accuracy | macro_f1 | accuracy | |||||
|---|---|---|---|---|---|---|---|
| mean | std | mean | std | mean | std | ||
| task | model | ||||||
| group | logreg | 0.978 | 0.020 | 0.981 | 0.017 | 0.982 | 0.017 |
| mlp | 0.977 | 0.018 | 0.980 | 0.019 | 0.982 | 0.015 | |
| rf | 0.978 | 0.015 | 0.984 | 0.011 | 0.985 | 0.011 | |
| subtype | logreg | 0.903 | 0.027 | 0.905 | 0.023 | 0.913 | 0.018 |
| mlp | 0.882 | 0.034 | 0.889 | 0.030 | 0.905 | 0.029 | |
| rf | 0.871 | 0.029 | 0.878 | 0.034 | 0.898 | 0.020 | |
pd.read_csv(TAB / 'test_metrics.csv')
| task | model | balanced_accuracy | macro_f1 | accuracy | |
|---|---|---|---|---|---|
| 0 | group | logreg | 0.970 | 0.968 | 0.974 |
| 1 | group | rf | 0.974 | 0.974 | 0.980 |
| 2 | group | mlp | 0.970 | 0.968 | 0.974 |
| 3 | subtype | logreg | 0.810 | 0.817 | 0.869 |
| 4 | subtype | rf | 0.859 | 0.867 | 0.908 |
| 5 | subtype | mlp | 0.859 | 0.840 | 0.863 |
The four subgroups are close to solved by any model. This matches the literature. The twelve subtypes are harder and the three models are within a few points of each other. The confusion matrices show where the errors sit. Rows are the published label, columns the call.
display(Image(FIG / 'confusion_group_mlp.png'))
display(Image(FIG / 'confusion_subtype_mlp.png'))
rep = pd.read_csv(TAB / 'test_per_class.csv')
rep[rep['model'] == 'mlp'].set_index(['task', 'class'])[['precision', 'recall', 'f1-score', 'support']]
| precision | recall | f1-score | support | ||
|---|---|---|---|---|---|
| task | class | ||||
| group | WNT | 0.933 | 1.000 | 0.966 | 14.0 |
| SHH | 1.000 | 1.000 | 1.000 | 44.0 | |
| Group3 | 0.963 | 0.897 | 0.929 | 29.0 | |
| Group4 | 0.970 | 0.985 | 0.977 | 66.0 | |
| subtype | WNT_alpha | 1.000 | 0.800 | 0.889 | 10.0 |
| WNT_beta | 0.571 | 1.000 | 0.727 | 4.0 | |
| SHH_alpha | 0.714 | 0.769 | 0.741 | 13.0 | |
| SHH_beta | 0.667 | 0.857 | 0.750 | 7.0 | |
| SHH_gamma | 0.800 | 0.444 | 0.571 | 9.0 | |
| SHH_delta | 0.875 | 0.933 | 0.903 | 15.0 | |
| Group3_alpha | 0.929 | 0.929 | 0.929 | 14.0 | |
| Group3_beta | 0.875 | 1.000 | 0.933 | 7.0 | |
| Group3_gamma | 1.000 | 0.875 | 0.933 | 8.0 | |
| Group4_alpha | 0.864 | 0.950 | 0.905 | 20.0 | |
| Group4_beta | 0.913 | 0.955 | 0.933 | 22.0 | |
| Group4_gamma | 0.950 | 0.792 | 0.864 | 24.0 |
Almost all subtype errors are inside SHH and WNT: SHH gamma against SHH alpha and beta, and WNT beta, which has four test tumours. Group 3 and Group 4 subtypes are called well despite the overlap of the two groups in the PCA.
2. Calibration¶
A classifier outputs a probability for each class. It is calibrated when a tumour called with 90 percent confidence is right 90 percent of the time. The expected calibration error (ECE) is the average gap between confidence and observed accuracy across confidence bins. This matters in a clinic: an overconfident model gives no warning on the cases it gets wrong.
display(Image(FIG / 'calibration_mlp.png'))
json.load(open(TAB / 'train_summary.json'))
{'mlp_group_test_ece': 0.023220618955450127,
'mlp_group_test_fraction_confidence_above_0.9': 0.9934640522875817,
'mlp_subtype_test_ece': 0.10308718622899522,
'mlp_subtype_test_fraction_confidence_above_0.9': 0.9019607843137255}
For the four subgroups the MLP is well calibrated: nearly all calls carry a confidence above 0.9 and are nearly always right. For the twelve subtypes most calls still carry a confidence above 0.9 while clearly fewer are right. The subtype model is overconfident. Before anyone trusted its scores it would need temperature scaling or a similar correction.
3. External cohort, Hovestadt 2013¶
The models were trained on Cavalli only and applied to 276 tumours from a different laboratory, with the four-group labels. The cohort is split by material.
pd.read_csv(TAB / 'external_validation.csv')
| model | subset | n | balanced_accuracy | macro_f1 | accuracy | |
|---|---|---|---|---|---|---|
| 0 | logreg | all | 276 | 0.979 | 0.983 | 0.978 |
| 1 | logreg | frozen | 107 | 0.967 | 0.970 | 0.963 |
| 2 | logreg | FFPE | 169 | 0.988 | 0.991 | 0.988 |
| 3 | rf | all | 276 | 0.976 | 0.980 | 0.975 |
| 4 | rf | frozen | 107 | 0.967 | 0.970 | 0.963 |
| 5 | rf | FFPE | 169 | 0.982 | 0.986 | 0.982 |
| 6 | mlp | all | 276 | 0.983 | 0.986 | 0.982 |
| 7 | mlp | frozen | 107 | 0.975 | 0.977 | 0.972 |
| 8 | mlp | FFPE | 169 | 0.988 | 0.991 | 0.988 |
display(Image(FIG / 'confusion_group_mlp_external.png'))
Performance on the external cohort matches the internal test split. FFPE material classifies as well as frozen material. This is the property that makes methylation usable on archival pathology blocks, and it is the reason methylation, not expression, is the clinical reference for brain tumour classification.
4. Sparse coverage¶
Shallow nanopore sequencing is a cheaper and faster way to read methylation, with a result in hours rather than days. It covers a random, sparse subset of CpGs. The simulation removes CpGs at random at test time, 10 repeats per level, and measures four-group balanced accuracy. Missing CpGs are filled with the training mean for the standard models. A third model, the same MLP, was trained with random input masking, so that it learned to decide from whatever subset of CpGs is present.
display(Image(FIG / 'sparse_coverage.png'))
A model trained on complete arrays degrades at half coverage and collapses to chance once fewer than a fifth of the CpGs are seen. The masked-input model keeps high balanced accuracy with a few hundred of the 10,000 CpGs. The idea is simple and the effect is large. Two caveats: real nanopore coverage is not uniform, and nanopore calls come from sequencing reads, not from array beta values.
5. Representation learning¶
A 16-dimensional autoencoder was trained on the training split. It compresses 10,000 beta values into 16 numbers and back. A linear classifier on the 16 numbers is compared with one on the raw CpGs, and the 16 dimensions are drawn with UMAP.
display(Image(FIG / 'umap_latent.png'))
pd.read_csv(TAB / 'latent_vs_raw.csv')
| task | features | cv_balanced_accuracy_mean | cv_balanced_accuracy_std | |
|---|---|---|---|---|
| 0 | group | raw 10,000 CpGs | 0.978 | 0.018 |
| 1 | group | autoencoder latent (16 dims) | 0.975 | 0.022 |
| 2 | subtype | raw 10,000 CpGs | 0.903 | 0.025 |
| 3 | subtype | autoencoder latent (16 dims) | 0.836 | 0.035 |
The latent space separates the four subgroups cleanly and shows the subtypes as sub-clusters. A linear classifier on 16 dimensions comes close to the raw CpGs for the subgroups and loses more for the subtypes. The compression keeps the subgroup structure and drops some of the finer subtype signal.
Page 3 of 9. previous: 2. Quality control | index | next: 4. Explainability