Convex-hull metrics are evaluated across 22 ternary chalcogenide tie-line systems for both full FP relaxation and static FP evaluations on the DFT-relaxed structures. Ground-state agreement measures whether the FP identifies the same lowest-energy phase at each composition as DFT, while within-phase hull-minimum agreement and hull-minimum agreement compare the predicted hull minima. See Error-decomposition metrics for full definitions.
The per-column best value (green) updates when you click a column header. The Version (training) column gives the exact checkpoint evaluated and the dataset it was trained on.
Full FP relaxation
| FP | Average energy error (meV/atom) ↓ |
Ground-state agreement (%) ↑ |
Within-phase hull-minimum agreement (%) ↑ |
Hull-minimum agreement (%) ↑ |
Version (training) |
|---|---|---|---|---|---|
| MACE | 53 | 68.3 | 62.0 | 40.9 | >=v0.3.10 (MACE-MPA-0, medium) (MPTrj + sAlex) |
| CHGNet | 242 | 53.4 | 40.5 | 18.2 | v0.3.0 (MPTrj) |
| M3GNet | 49 | 55.3 | 47.1 | 22.7 | MP-2021.2.8-PES (MP-2021.2.8) |
| UMA | 83 | 62.7 | 60.3 | 36.4 | s-1p1 (OC20+ |
| M3GNet-MatPES | 117 | 50.3 | 55.4 | 22.7 | v2025.1 (MatPES-PBE) |
| TensorNet-MatPES | 89 | 52.8 | 58.7 | 13.6 | v2025.1 (MatPES-PBE) |
| MACE-MatPES | 89 | 49.7 | 59.5 | 18.2 | >=v0.3.10 (MatPES-PBE (fine-tuned)) |
| Orb | 81 | 59.6 | 58.7 | 27.3 | orb-v3-conservative-inf-omat-20250404 (OMat24) |
| SevenNet | 52 | 67.7 | 58.7 | 40.9 | 7net-mf-ompa (modal mpa) (OMat24 + sAlex + MPTrj) |
| MatterSim | 58 | 59.6 | 59.5 | 27.3 | MatterSim-v1.0.0-5M (MatterSim dataset (PBE)) |
| Nequix | 52 | 66.5 | 58.7 | 40.9 | nequix-oam-1 (OMat24 + sAlex + MPTrj) |
| GPTFF | 562 | 41.6 | 34.7 | 13.6 | gptff_v2 (Atomly (PBE)) |
| NEP89 | 162 | 40.4 | 52.1 | 18.2 | nep89_20250409 (mixed QM levels) |
| DPA4 | 84 | 66.5 | 60.3 | 45.5 | DPA4-Plus-OMat24-v20260805 (OMat24) |
| GRACE | 52 | 64.6 | 62.8 | 45.5 | GRACE-3L-OMAT-large-ft-AM (OMat24 → sAlex + MPTrj) |
| eqV2 | 53 | 67.1 | 58.7 | 40.9 | eqV2_31M_omat_mp_salex (OMat24 → MPTrj + sAlex) |
| eSEN | 52 | 68.3 | 58.7 | 40.9 | esen_30m_oam (OMat24 → MPTrj + sAlex) |
Static FP evaluations on the DFT-relaxed structures
| FP | Average energy error (meV/atom) ↓ |
Ground-state agreement (%) ↑ |
Within-phase hull-minimum agreement (%) ↑ |
Hull-minimum agreement (%) ↑ |
|---|---|---|---|---|
| MACE | 15 | 70.2 | 73.6 | 40.9 |
| CHGNet | 246 | 55.9 | 48.8 | 22.7 |
| M3GNet | 33 | 55.9 | 59.5 | 22.7 |
| UMA | 43 | 74.5 | 78.5 | 40.9 |
| M3GNet-MatPES | 109 | 49.1 | 57.0 | 22.7 |
| TensorNet-MatPES | 66 | 49.1 | 63.6 | 13.6 |
| MACE-MatPES | 55 | 53.4 | 71.9 | 18.2 |
| Orb | 46 | 71.4 | 76.0 | 31.8 |
| SevenNet | 11 | 74.5 | 77.7 | 50.0 |
| MatterSim | 22 | 62.1 | 71.1 | 27.3 |
| Nequix | 12 | 69.6 | 76.9 | 45.5 |
| GPTFF | 542 | 39.8 | 42.1 | 31.8 |
| NEP89 | 163 | 44.1 | 55.4 | 22.7 |
| DPA4 | 45 | 77.6 | 78.5 | 54.5 |
| GRACE | 10 | 77.6 | 78.5 | 50.0 |
| eqV2 | 9 | 80.1 | 77.7 | 50.0 |
| eSEN | 10 | 79.5 | 79.3 | 54.5 |
Elemental-ordering metrics are evaluated across 305 ordering groups of 20 candidates each, for both full FP relaxation and static FP evaluations on the DFT-relaxed structures. Top-1 accuracy measures recovery of the DFT-preferred ordering, Recall@k whether it is recovered among the top-ranked candidates, and Spearman ρ overall ranking agreement. See Error-decomposition metrics for full definitions.
The per-column best value (green) updates when you click a column header. The Version (training) column gives the exact checkpoint evaluated and the dataset it was trained on.
Full FP relaxation
| FP | Average energy error (meV/atom) ↓ |
Top-1 accuracy (%) ↑ | Recall@3 (%) ↑ | Recall@10 (%) ↑ | Spearman ρ ↑ | Rate of ranking errors (%) ↓ |
Mean/max ΔEDFT (meV/atom) ↓ |
Version (training) |
|---|---|---|---|---|---|---|---|---|
| MACE | 43 | 23.6 | 32.8 | 60.9 | 0.28 | 38.9 | >=v0.3.10 (MACE-MPA-0, medium) (MPTrj + sAlex) | |
| CHGNet | 272 | 8.2 | 20.1 | 52.8 | 0.09 | 46.8 | v0.3.0 (MPTrj) | |
| M3GNet | 56 | 7.2 | 18.6 | 51.3 | 0.05 | 48.1 | MP-2021.2.8-PES (MP-2021.2.8) | |
| UMA | 65 | 24.3 | 34.6 | 61.9 | 0.31 | 37.6 | s-1p1 (OC20+ | |
| M3GNet-MatPES | 95 | 10.2 | 21.0 | 54.8 | 0.12 | 45.7 | v2025.1 (MatPES-PBE) | |
| TensorNet-MatPES | 69 | 12.8 | 25.4 | 56.1 | 0.17 | 43.6 | v2025.1 (MatPES-PBE) | |
| MACE-MatPES | 69 | 21.3 | 32.7 | 60.5 | 0.28 | 39.1 | >=v0.3.10 (MatPES-PBE (fine-tuned)) | |
| Orb | 65 | 25.2 | 33.7 | 61.3 | 0.29 | 38.5 | orb-v3-conservative-inf-omat-20250404 (OMat24) | |
| SevenNet | 42 | 21.0 | 36.1 | 62.0 | 0.31 | 38.1 | 7net-mf-ompa (modal mpa) (OMat24 + sAlex + MPTrj) | |
| MatterSim | 47 | 18.0 | 31.1 | 58.7 | 0.24 | 41.0 | MatterSim-v1.0.0-5M (MatterSim dataset (PBE)) | |
| Nequix | 41 | 20.7 | 34.0 | 62.2 | 0.31 | 37.6 | nequix-oam-1 (OMat24 + sAlex + MPTrj) | |
| GPTFF | 531 | 7.5 | 17.6 | 50.6 | 0.01 | 49.7 | gptff_v2 (Atomly (PBE)) | |
| NEP89 | 177 | 8.9 | 22.0 | 53.2 | 0.09 | 46.6 | nep89_20250409 (mixed QM levels) | |
| DPA4 | 65 | 22.0 | 34.0 | 61.2 | 0.3 | 38.4 | DPA4-Plus-OMat24-v20260805 (OMat24) | |
| GRACE | 43 | 24.9 | 35.8 | 61.8 | 0.32 | 37.4 | GRACE-3L-OMAT-large-ft-AM (OMat24 → sAlex + MPTrj) | |
| eqV2 | 44 | 21.0 | 35.4 | 61.8 | 0.31 | 38.0 | eqV2_31M_omat_mp_salex (OMat24 → MPTrj + sAlex) | |
| eSEN | 42 | 23.6 | 35.8 | 61.9 | 0.31 | 37.9 | esen_30m_oam (OMat24 → MPTrj + sAlex) |
Static FP evaluations on the DFT-relaxed structures
| FP | Average energy error (meV/atom) ↓ |
Top-1 accuracy (%) ↑ | Recall@3 (%) ↑ | Recall@10 (%) ↑ | Spearman ρ ↑ | Rate of ranking errors (%) ↓ |
Mean/max ΔEDFT (meV/atom) ↓ |
|---|---|---|---|---|---|---|---|
| MACE | 14 | 46.6 | 54.5 | 72.7 | 0.56 | 26.6 | |
| CHGNet | 270 | 20.3 | 29.7 | 59.0 | 0.24 | 40.6 | |
| M3GNet | 41 | 22.3 | 28.7 | 58.7 | 0.23 | 41.0 | |
| UMA | 35 | 55.7 | 65.4 | 79.9 | 0.69 | 19.7 | |
| M3GNet-MatPES | 94 | 19.7 | 29.8 | 60.0 | 0.26 | 40.2 | |
| TensorNet-MatPES | 52 | 27.9 | 41.4 | 65.3 | 0.38 | 34.8 | |
| MACE-MatPES | 43 | 41.0 | 48.0 | 70.0 | 0.50 | 29.6 | |
| Orb | 37 | 54.4 | 60.4 | 77.7 | 0.64 | 22.3 | |
| SevenNet | 10 | 52.1 | 59.6 | 76.9 | 0.64 | 22.8 | |
| MatterSim | 19 | 44.3 | 50.1 | 70.4 | 0.51 | 29.2 | |
| Nequix | 11 | 51.8 | 58.4 | 76.4 | 0.64 | 23.1 | |
| GPTFF | 520 | 13.1 | 27.3 | 57.4 | 0.18 | 43.1 | |
| NEP89 | 171 | 16.4 | 30.9 | 58.8 | 0.26 | 40.2 | |
| DPA4 | 36 | 57.0 | 64.0 | 79.9 | 0.69 | 19.7 | |
| GRACE | 10 | 58.7 | 65.7 | 80.1 | 0.7 | 19.2 | |
| eqV2 | 8 | 52.8 | 64.8 | 79.4 | 0.69 | 20.2 | |
| eSEN | 9 | 59.0 | 66.4 | 80.5 | 0.71 | 18.8 |
RMSD applies only to full FP relaxation --
static evaluation does not produce an FP-relaxed structure to compare against DFT. Computed
via pymatgen StructureMatcher, converted to Å via the geometric-mean cell
volume. See Error-decomposition metrics for full definitions.
The per-column best value (green) updates when you click a column header. The Version (training) column gives the exact checkpoint evaluated and the dataset it was trained on.
| FP | Map success (%) ↑ | Mean/max RMSD (Å) ↓ | RMSD < 0.05 Å (%) ↑ | RMSD < 0.10 Å (%) ↑ | RMSD < 0.20 Å (%) ↑ | Version (training) |
|---|---|---|---|---|---|---|
| MACE | 89.9 | 24.0 | 44.5 | 65.5 | >=v0.3.10 (MACE-MPA-0, medium) (MPTrj + sAlex) | |
| CHGNet | 87.3 | 11.9 | 26.5 | 49.5 | v0.3.0 (MPTrj) | |
| M3GNet | 88.8 | 10.2 | 21.1 | 39.4 | MP-2021.2.8-PES (MP-2021.2.8) | |
| UMA | 89.9 | 24.4 | 45.4 | 66.1 | s-1p1 (OC20+ | |
| M3GNet-MatPES | 89.8 | 12.3 | 26.3 | 51.3 | v2025.1 (MatPES-PBE) | |
| TensorNet-MatPES | 89.9 | 15.6 | 36.7 | 63.1 | v2025.1 (MatPES-PBE) | |
| MACE-MatPES | 89.6 | 23.6 | 44.5 | 64.5 | >=v0.3.10 (MatPES-PBE (fine-tuned)) | |
| Orb | 89.4 | 20.8 | 40.4 | 63.3 | orb-v3-conservative-inf-omat-20250404 (OMat24) | |
| SevenNet | 89.8 | 24.8 | 47.2 | 67.0 | 7net-mf-ompa (modal mpa) (OMat24 + sAlex + MPTrj) | |
| MatterSim | 89.9 | 22.0 | 44.5 | 67.6 | MatterSim-v1.0.0-5M (MatterSim dataset (PBE)) | |
| Nequix | 89.9 | 24.4 | 46.6 | 68.7 | nequix-oam-1 (OMat24 + sAlex + MPTrj) | |
| GPTFF | 89.6 | 12.5 | 24.3 | 46.0 | gptff_v2 (Atomly (PBE)) | |
| NEP89 | 87.3 | 9.0 | 24.0 | 48.2 | nep89_20250409 (mixed QM levels) | |
| DPA4 | 89.6 | 24.7 | 45.8 | 65.6 | DPA4-Plus-OMat24-v20260805 (OMat24) | |
| GRACE | 89.6 | 27.1 | 48.2 | 67.7 | GRACE-3L-OMAT-large-ft-AM (OMat24 → sAlex + MPTrj) | |
| eqV2 | 90.3 | 28.2 | 49.0 | 68.1 | eqV2_31M_omat_mp_salex (OMat24 → MPTrj + sAlex) | |
| eSEN | 89.4 | 25.5 | 45.7 | 65.5 | esen_30m_oam (OMat24 → MPTrj + sAlex) |
Error-decomposition metrics
Evaluated for both full FP relaxation and static FP evaluation on the DFT-relaxed structure.
Convex hull and relative phase stability
| Name | Metrics |
|---|---|
| Average energy error | Energy error relative to DFT, reported as the MAE over all structures in the convex-hull benchmark. |
| Ground-state agreement: lowest-energy phase at each composition. | For each composition, the FP-predicted ground-state phase is compared with DFT. The reported metric is the fraction of compositions, across all tie-line systems, for which the two agree. |
| Within-phase hull-minimum agreement: lowest-energy composition within each phase. | For each phase, the FP-predicted minimum-energy composition is compared with DFT. The reported metric is the fraction of phases, across all tie-line systems, for which the two agree. |
| Hull-minimum agreement: global convex-hull minimum across the tie-line. | For each tie-line system, the FP-predicted convex-hull minimum is compared with DFT. A system with no structure below the tie line has no stable intermediate. The reported metric is the fraction of systems for which the two agree: they identify the same stable intermediate, or both find none and their tie lines are anchored on the same endpoint structures. |
Energy rankings of elemental orderings
| Name | Metrics |
|---|---|
| Average energy error | Energy error relative to DFT, reported as the MAE over all structures in the ordering benchmark. |
| Top-1 accuracy | Fraction of ordering groups for which the FP identifies the same lowest-energy configuration (LEC) as DFT. |
| Recall@k | Fraction of the k lowest-energy DFT orderings retained among the top k FP-ranked orderings within each ordering group. The reported metric is the mean over all ordering groups. |
| Spearman's ρ | Rank correlation between FP and DFT orderings within each ordering group. The reported metric is the mean over all ordering groups. |
| Rate of ranking errors | Fraction of configuration pairs whose FP and DFT rankings disagree, within each ordering group. The reported metric is the mean over all ordering groups. |
| ΔEDFT of misranked pairs | DFT energy difference between configurations that the FP misranks. For each ordering group, the mean and maximum ΔEDFT over its misranked pairs are computed and then averaged across all ordering groups. |
Structural-relaxation effects
Applies to full FP relaxation only (static evaluation does not produce an FP-relaxed structure to compare).
| Name | Metrics |
|---|---|
| Structure relaxation error | For each structure, the RMSD between the FP-relaxed and DFT-relaxed structures is computed. The reported metric is the mean/maximum over all structures in the dataset. |
| Mean/max RMSD | Geometric deviation after alignment (Å). |
| Map success | Fraction of cases where FP and DFT structures remain sufficiently similar for meaningful RMSD evaluation; failures correspond to large structural deviation. |
| RMSD thresholds | Increasingly strict measures of fidelity. |
These metrics are complementary and should be interpreted together rather than combined into a single overall ranking. Click any column heading to sort by that metric.
Benchmark Dataset
Data availability. The FP-DeErr convex-hull and elemental-ordering benchmark subsets are available on Figshare. The underlying ternary chalcogenide DFT data are available from Adams et al. on Dryad.
All DFT energies are raw, uncorrected VASP-PBE energies -- no Materials Project MP2020 compatibility correction is applied anywhere. Model versions and official sources for the 17 FPs evaluated here are documented once on the FP-DeErr home page. See the Phase_stability_ordering README for the full standardized-file schema.
Use FP-DeErr
Provided FP-DeErr reference + new FP calculations
or
User DFT reference + user FP results
↓
standardized reference/results
↓
validation and analysis
↓
hull, ordering, and RMSD tablesEvaluate a new FP on the provided benchmark. Use generation/convexhull_ordering_run_generator.ipynb to generate per-FP jobs and submission scripts against the shipped DFT reference, run them on your cluster, merge the results, then load the merged file directly in the analysis notebook -- your FP appears in every table above.
Apply the analysis functions to another dataset. Call build_phase_stability_ordering_results(...) and the table builders in scripts/convexhull_analysis_utils.py directly with your own DFT reference and FP results.
git clone https://github.com/mogroupumd/FP-DeErr.git
cd FP-DeErr/Phase_stability_ordering
pip install -r requirements.txt
pip install jupyterlab
jupyter lab analysis/convexhull_ordering_analysis_all_models.ipynbSee the Phase_stability_ordering README for the full quick start, the generator and analysis notebooks for complete operational detail, and examples/ for a small runnable slice of real data.
Add a potential
Interested in adding a foundation potential to FP-DeErr? Follow our Adding a Potential guide to evaluate your model, and visit Contribute for example outputs, results formats, and email submission instructions.