FPBench · Ion Migration by NEB

FPBench: Ion Migration by NEB

Migration-barrier and migration-pathway metrics for foundation potentials (FPs), evaluated on 154 real Li- and Na-ion migration pathways using the nudged elastic band (NEB) method.

View Leaderboard Use FPBench Dataset (Figshare) Paper (arXiv) GitHub
▼ lower is better ▲ higher is better green text = best in column cell shading: green = better, gold = middle, red = worse identifiers are never colored

Results are reported for 154 active migration pathways, using the full FP-NEB workflow and static FP evaluations on the finalized DFT-NEB image structures; one excluded pathway is documented in the data README. Barrier error measures migration-barrier accuracy, endpoint energy-difference error and endpoint energy ranking agreement assess the relative endpoint energies, and energy-profile shape agreement measures whether the FP reproduces the DFT migration-profile shape. See Metrics for full definitions.

The per-column best value (green) updates when you click a column header. The Version (training) column gives the exact checkpoint evaluated and the dataset it was trained on.

FP Non-conv. paths
(n/total) ↓		 Barrier error (eV)
full FP-NEB
(MAE / RMSE) ↓		 Barrier error (eV)
static FP on DFT-NEB
(MAE / RMSE) ↓		 Endpoint energy-diff.
error (eV), full
(MAE / RMSE) ↓		 Endpoint energy
rank. agr. (%), full ↑		 Energy-profile
shape agr. (%), full ↑		 Version (training)

Restricted to full FP-NEB converged paths. RMSD is the geometric deviation between the FP-relaxed and DFT-relaxed endpoint structures after alignment. See Metrics for full definitions.

The per-column best value (green) updates when you click a column header. The Version (training) column gives the exact checkpoint evaluated and the dataset it was trained on.

FP Map success (%) ↑ Mean/max RMSD (Å) ↓ RMSD < 0.05 Å (%) ↑ RMSD < 0.10 Å (%) ↑ RMSD < 0.20 Å (%) ↑ Version (training)

Metrics

NameMetrics
Non-converged pathsFraction of FP-NEB calculations that do not satisfy the NEB convergence criterion before reaching the maximum number of optimization steps.
Barrier error, full FP-NEBThe forward and backward barrier errors are computed relative to DFT for each path. The reported MAE/RMSE is the true pooled statistic over the combined forward+backward per-path error array (not each direction's own MAE/RMSE averaged separately), over all converged paths.
Barrier error, static FP evaluations on the DFT-NEB image structuresThe same forward and backward barrier errors, computed from static FP evaluations on the DFT-NEB image structures, pooled the same way, over all paths.
Endpoint energy ranking agreementFraction of converged paths for which FP and DFT identify the same lower-energy endpoint between the two endpoints of the migration path, or both classify the two endpoints as equal in energy.
Endpoint energy-difference errorFor each path, the error in the endpoint energy difference is computed relative to DFT. The reported metric is the MAE/RMSE over all converged paths.
Energy-profile shape agreementFraction of converged paths for which the FP reproduces the Normal-Hill energy profile, as in DFT-NEBs.
Integrated energy-profile differenceFor each path, the integrated absolute energy difference between FP and DFT-NEB energy profiles along the normalized reaction coordinate is computed. The reported metric is the MAE/RMSE over all converged paths.
Endpoint-structure relaxation errorFor each endpoint structure, the RMSD between FP-relaxed and DFT-relaxed endpoint structures is computed. The reported metric is the mean/maximum over all endpoint structures of the converged paths.
Map successFraction of endpoint-structure comparisons, over the converged paths, for which pymatgen's StructureMatcher found a valid structural mapping between the FP-relaxed and DFT-relaxed endpoint structure. A failed mapping is excluded from the mean/maximum RMSD, not counted as RMSD = 0.
Force errors on FP-NEB pathMean force-magnitude error |Δ|F|| and force-angle error Δθ across all atoms for each image structure of the final FP-NEB path.
Force errors on DFT-NEB pathMean force magnitude error |Δ|F|| and force angle error Δθ across all atoms for each image structure of the final DFT-NEB path.

These metrics are complementary and should be interpreted together rather than combined into a single overall ranking. Full FP-NEB and static FP-on-DFT-NEB results are separate protocols; force errors on the FP-NEB path and force errors on the DFT-NEB path are separate diagnostics. Comparing the two protocols' barrier errors (both shown in the table above) separates errors observed from static FP evaluations on the DFT-NEB image structures from additional differences associated with FP endpoint relaxation and FP-NEB pathway optimization. Click any column heading to sort by that metric.

Protocols

WorkflowStructural inputResults branch
Full FP-NEB workflowUnrelaxed source endpoint structuresfull_fp_neb
Static FP evaluations on the DFT-NEB image structuresFinalized DFT-NEB image structuresfp_static_on_dft_neb
DFT static diagnostics on the final FP-NEB image structuresFinal full FP-NEB image structuresdft_static_on_fp_neb

Benchmark Dataset

Ion-migration pathways
Active pathways: 154
DFT-NEB images per pathway: 7
Structurally distinct materials: 106
154 Li- and Na-ion migration pathways spanning 106 structurally distinct materials and a range of chemistries, including oxides (32.1%), sulfides (20.8%), halides, and multi-anion compounds, from the ion-migration dataset of Saravanan et al. DFT-NEB reference calculations were performed for this work.

Data availability. The 154-pathway DFT-NEB reference dataset used in FPBench is available on Figshare. The underlying migration pathways are from the original data release of Saravanan et al.

Endpoint structures were relaxed with a convergence threshold of 0.002 eV/Å and fixed cell shape/volume; the full FP-NEB workflow uses a convergence threshold of 0.05 eV/Å and a maximum of 1000 NEB steps. Model versions and official sources for the seven FPs evaluated here are documented once on the FPBench home page. See the Ion_migration_NEB README for the full standardized-file schema.

Use FPBench

Provided FPBench reference + new FP calculations
                         or
User DFT reference + user FP results
                          ↓
           standardized reference/results
                          ↓
                validation and analysis
                          ↓
     barrier, profile, RMSD, and force-error tables

Evaluate a new FP on the provided benchmark. Use generation/fp_neb_generation_and_run.ipynb to generate full_fp_neb and fp_static_on_dft_neb jobs and submission scripts against the shipped DFT reference, run them on your cluster, merge the results, then load the merged file directly in the analysis notebook. Your FP appears in every table above.

Apply the analysis functions to another dataset. Call build_neb_analysis_results(...) and the table/metric functions in scripts/neb_analysis.py directly with your own DFT reference and FP results.

git clone https://github.com/mogroupumd/FPBench.git
cd FPBench/Ion_migration_NEB
pip install -r requirements.txt
pip install jupyterlab
jupyter lab analysis/neb_analysis.ipynb

See the Ion_migration_NEB README for the full quick start, the generator and analysis notebooks for complete operational detail, and examples/ for a small runnable slice of real data.

Contribute

Interested in evaluating a new foundation potential, or having it considered for inclusion in FPBench? See our Adding a Potential guide to integrate and evaluate a new model with FPBench. For inclusion in the public leaderboard, please contact Prof. Yifei Mo at yfmo@umd.edu with the model name, version/checkpoint, and a link to the official implementation or model weights.