FP-DeErr: Contribute a Foundation Potential

1. Run FP-DeErr

FP-DeErr evaluates foundation potentials through ASE calculators. Adding one means a single entry in the component's POTENTIAL_REGISTRY — no metric or analysis code changes. Full procedure: Adding a Potential.

Register — one entry in the component's POTENTIAL_REGISTRY.
Generate — the generator writes one standalone script per job. Nothing is submitted.
Run — on your own machine or cluster. FP-DeErr generates jobs; it does not execute them.
Merge — validates each job against the shared reference and writes the standardized results file.
Analyze — the analysis notebook reads that file and reproduces the leaderboard tables.

Example: register your potential

A representative entry. my_fp, MyFPCalculator and the paths are illustrative placeholders. Field names differ between components, so copy an existing entry from the generator notebook of the component you are evaluating rather than this one.

"my_fp": {
    "mlip_name":     "my_fp",
    "site_pkgs":     "/path/to/my_fp_env/lib/python3.10/site-packages",
    "venv_activate": "/path/to/my_fp_env/bin/activate",
    "python":        "/path/to/my_fp_env/bin/python",
    "model_path":    "/path/to/checkpoints/my_fp_model.pt",
    "calc_setup":    textwrap.dedent("""
        from my_fp.calculators import MyFPCalculator
        calc = MyFPCalculator(
            model_paths=[MODEL_PATH],
            device="cpu",
            default_dtype="float64",
        )
    """).strip(),
},

Notes on the registry entry

The generator substitutes MODEL_PATH from the entry's model_path. Where a package ships its own weights, set model_path to None. Values still beginning with /path/to/ raise an explicit error rather than producing a job script that fails later. One environment per FP family is recommended, since FP packages pin incompatible PyTorch and ASE versions.

2. Prepare your results

For each component evaluated, submit the final results file produced by the merge step. The tabs below show the expected file format and example analysis code.

File format

One gzipped JSON file per dataset, holding every FP you evaluated on that dataset, with each FP's per-atom error arrays under models. Published files follow <dataset>_force_results_standardized.json, for example matpes_pbe_force_results_standardized.json.

Structure

{
  "schema_version": "1.0",
  "dataset_name": "OMat24-rattled-1000",
  "units": {"force": "eV/Å", "angle": "degree"},
  "generation_metadata": {
    "dataset_label": "OMat24-rattled-1000",
    "n_chunks_expected": 20,
    "models_included": ["My_FP"],
    "raw_structure_count": 117004,
    "raw_atom_count": 1657765,
    "raw_dft_checksum": "7c3e25874ae2cb75...",
    "per_model_summary": {
      "My_FP": {
        "registry_key": "my_fp",
        "n_chunks_found": 20,
        "partial": false,
        "raw_atom_count": 1657765,
        "metric_valid_atom_count": 1657765,
        "excluded_undefined_angle_count": 0,
        "model_path": "/path/to/checkpoints/my_fp_model.pt",
        "structure_id_checksum": "6c24e356fdc27384..."
      }
    }
  },
  "reference_population": {
    "dft_force_magnitude": [0.0739, 0.1021],
    "structure_id":        [29632, 29632],
    "atom_index":          [0, 1]
  },
  "models": {
    "My_FP": {
      "dft_force_magnitude":   [0.0739, 0.1021],
      "fp_force_magnitude":    [0.0720, 0.0994],
      "force_magnitude_error": [-0.0019, -0.0027],
      "force_angle_error":     [1.84, 2.11],
      "force_vector_error":    [0.0031, 0.0035],
      "structure_id":          [29632, 29632],
      "atom_index":            [0, 1]
    }
  }
}

Only two atoms are shown for readability. Submit the full results file.

Fields

Field	Meaning
`models.<key>`	One entry per FP, keyed by the name shown in the leaderboard row.
`dft_force_magnitude` `fp_force_magnitude`	Per-atom DFT and FP force magnitudes.
`force_magnitude_error`	FP minus DFT force magnitude.
`force_angle_error`	Angle between the DFT and FP force vectors. Undefined where either force is numerically zero.
`force_vector_error`	Per-atom force-vector error.
`structure_id`	Identifies the source structure in the original dataset.
`atom_index`	Zero-based position of the atom within its own structure; restarts at 0 for each structure.
Array alignment	Arrays in a model entry are aligned element-wise: index i is the same atom in every array.
`reference_population`	DFT-only population, independent of any FP's filtering.
`excluded_undefined_angle_count`	Atoms the merge step dropped for an undefined angle; `build_force_results` keeps every atom and writes NaN instead.
Units	Forces in eV/Å, angles in degrees.

Cartesian FP forces with structure IDs and matching atom ordering are preferred, as they allow us to calculate force errors directly against the DFT reference.

Example: convert forces to standardized results

Convert Cartesian DFT and FP forces into standardized per-atom force errors.

Run from: Force_error/analysis/
Requires: numpy

import json, sys
sys.path.insert(0, "../scripts")
from force_results import build_force_results

ex = json.load(open("../examples/mace_matpes_cartesian_force_example.json"))

force_results = build_force_results(
    ex["dft_forces"],            # DFT reference forces: one (N_i, 3) array per structure
    {"MACE": ex["fp_forces"]},   # FP forces aligned with the DFT structures and atom order
    ex["structure_ids"],
)
print(len(force_results["MACE"]["dft_force_magnitude"]), "atoms")
# 25 atoms

# For benchmark evaluation, use the official reference data and your model results.
# The resulting object is what the generator serializes as "models" in the standardized file.

Example: calculate force-error metrics

Calculate a metric table using the included example data. To analyze your own results, replace the example-data preparation with the commented file-loading code.

Run from: Force_error/analysis/
Requires: numpy pandas matplotlib

import json, sys
sys.path.insert(0, "../scripts")
from force_results import build_force_results
from force_error_metrics import build_highly_accurate_force_fraction_table

ex = json.load(open("../examples/mace_matpes_cartesian_force_example.json"))
force_results = build_force_results(ex["dft_forces"], {"MACE": ex["fp_forces"]},
                                    ex["structure_ids"])

# To analyze a standardized results file, replace the example-data preparation above with:
# with open("../data/<your-dataset>_force_results_standardized.json") as f:
#     force_results = json.load(f)["models"]

# Map standardized fields to the names expected by the analysis functions.
all_results = {
    model: {"F_dft":      d["dft_force_magnitude"],
            "F_fp":       d["fp_force_magnitude"],
            "deltaF":     d["force_magnitude_error"],
            "deltaTheta": d["force_angle_error"],
            "e_vec":      d["force_vector_error"]}
    for model, d in force_results.items()
}

table = build_highly_accurate_force_fraction_table(all_results, [0.01, 0.05, 0.10])
print(table)
#       0.01  0.05  0.10
# MACE  12.0  56.0  84.0      <- Example output for the 25-atom subset.

table.to_csv("force_highly_accurate_fraction.csv")

Format documentation · Example data · Analysis notebook · Component guide

File format

One gzipped JSON file. Each FP has a hull section for convex-hull candidates and an ordering section for elemental-ordering groups, and each is evaluated under both protocols: full FP relaxation and static FP evaluations on the DFT-relaxed structures.

Structure

{
  "schema_version": "1.0",
  "component": "phase_stability_ordering",
  "units": {"energy": "eV"},
  "reference": {"sha256": "1b6addae634f59ec318d31214e17ecd8..."},
  "models": {
    "my_fp": {
      "metadata": {
        "registry_key": "my_fp",
        "mlip_name": "My_FP",
        "model_path": "/path/to/checkpoints/my_fp_model.pt",
        "reference_sha256": "1b6addae634f59ec318d31214e17ecd8...",
        "protocol": {
          "relax": {
            "label": "Full FP relaxation",
            "optimizer": "ASE FIRE",
            "fmax": 0.01,
            "max_steps": 10000,
            "fixed_cell": true,
            "starting_geometry": "initial_structure"
          },
          "static": {
            "label": "Static FP evaluations on the DFT-relaxed structures",
            "starting_geometry": "relaxed_structure"
          }
        },
        "counts": {
          "hull_unique_candidates_expected": 597,
          "hull_unique_candidates_relax_success": 597,
          "ordering_expected": 6100,
          "ordering_relax_success": 6100
        }
      },
      "hull": {
        "relax": {
          "Bi2Se3_Sb2Se3": {
            "mp-1198150": {
              "status": "success",
              "energy_total": -155.58319976100043,
              "relaxed_structure": {"@module": "pymatgen.core.structure"}
            }
          }
        },
        "static": {
          "Bi2Se3_Sb2Se3": {
            "mp-1198150": {"status": "success", "energy_total": -155.58319976100043}
          }
        }
      },
      "ordering": {
        "relax": {
          "mp-10074_Ge4+1_Sn2+3_Se2-5": {
            "ordered_0": {
              "status": "success",
              "energy_total": -35.87344152891665,
              "relaxed_structure": {"@module": "pymatgen.core.structure"}
            }
          }
        },
        "static": {
          "mp-10074_Ge4+1_Sn2+3_Se2-5": {
            "ordered_0": {"status": "success", "energy_total": -35.87344152891665}
          }
        }
      }
    }
  }
}

One hull candidate and one ordering candidate are shown, with atomic structures shortened for readability. Submit the full results file.

Fields

Field	Meaning
`models.<key>`	One entry per FP, keyed by its stable registry key.
`metadata.protocol`	Settings the numbers were produced with: optimizer, force tolerance, step cap, whether the cell was fixed.
`metadata.counts`	Per-branch record counts, used to assert completeness at merge time.
`reference.sha256`	The exact reference file these results were merged against; re-verified when loaded.
Identifiers	Tie-line, candidate, and ordering-group identifiers must match the reference dataset.
`energy_total`	Total energy of the candidate.
`status`	`success`, `missing`, `failed` or `non_converged`. Only `success` is scored; the rest are reported separately.
`relax` / `static`	Same shape, except `static` has no `relaxed_structure`.
Units	Energies in eV.

Example: validate results and calculate hull metrics

Validate the example results and calculate a hull metric table.

Run from: Phase_stability_ordering/analysis/
Requires: numpy pandas matplotlib pymatgen

import json, sys
sys.path.insert(0, "../scripts")
from convexhull_analysis_utils import (
    build_phase_stability_ordering_results,
    validate_phase_stability_ordering_results,
    build_combined_hull_table,
)

with open("../examples/phase_stability_ordering_demo_data.json") as f:
    demo = json.load(f)

results = build_phase_stability_ordering_results(
    reference_data=demo["reference_data"], fp_results=demo["fp_results"],
)
FPS = list(demo["fp_results"])
MODEL_NAMES = {"mace": "MACE"}

report = validate_phase_stability_ordering_results(
    dft_hull=results["dft_hull"],         fp_hull_by_fp=results["fp_hull"],
    dft_ordering=results["dft_ordering"], fp_ordering_by_fp=results["fp_ordering"],
    fps=FPS,
)
# prints a coverage report: 10/10 hull candidates, 1/1 ordering group,
# with missing and non-success records broken down per FP and protocol

hull_table, hull_details = build_combined_hull_table(
    results["dft_hull"], results["fp_hull"], FPS, MODEL_NAMES,
)
print(hull_table)
# Average energy error 69 meV/atom, ground-state agreement 50.0% (relax)

hull_table.to_csv("phase_hull_table_demo.csv")

# For benchmark evaluation, use the official reference data and your model results.

Format documentation · Example data · Analysis notebook · Component guide

File format

One gzipped JSON file with three protocol branches per FP. The full FP-NEB workflow and static FP evaluations on the DFT-NEB image structures produce every leaderboard metric. The third, DFT static diagnostics on the final FP-NEB image structures, is optional: it requires DFT on your own FP images, and the validator flags its absence without affecting any other analysis.

Structure

{
  "schema_version": "1.0",
  "component": "ion_migration_neb",
  "units": {"energy": "eV", "forces": "eV/angstrom"},
  "models": {
    "My_FP": {
      "full_fp_neb": {
        "pathways": {
          "29966|8": {
            "identifiers": {"icsd_id": "29966", "source_path_id": "8"},
            "neb_status": {
              "neb_converged": true,
              "neb_last_fmax": 0.036414,
              "neb_n_steps": 15,
              "status_source": "convergence_results"
            },
            "endpoint_relaxation": {"initial": {}, "final": {}},
            "convergence": {},
            "steps": {},
            "final_fp_neb_images": {"image00": {}}
          }
        },
        "unsuccessful_pathways": {}
      },
      "fp_static_on_dft_neb": {
        "pathways": {"29966|8": {"identifiers": {}, "images": {}}},
        "unsuccessful_pathways": {}
      },
      "dft_static_on_fp_neb": {
        "pathways": {"29966|8": {"identifiers": {}, "images": {}}},
        "unsuccessful_image_attempts": {}
      }
    }
  }
}

One pathway is shown, with endpoint, convergence and image records omitted for readability. Submit the full results file.

Fields

Field	Meaning
`models.<key>`	One entry per FP, keyed by its name.
Pathway key	`"<icsd_id>\|<source_path_id>"`; must match the pathway's `identifiers`.
`full_fp_neb`	Full FP-NEB workflow: FP-relaxed endpoints and the FP's own climbing-image path.
`fp_static_on_dft_neb`	Static FP evaluations on the DFT-NEB image structures.
`dft_static_on_fp_neb`	DFT static diagnostics on the final FP-NEB image structures; intentionally sparse, never padded. Not required for submission.
`neb_status.neb_converged`	Required on every `full_fp_neb` pathway; never assumed true.
`neb_last_fmax` `neb_n_steps`	Final maximum force and optimizer step count of the NEB run.
`calculation_status`	Whether the job ran, kept separate from whether it converged.
Units	Energies in eV, forces in eV/Å.

Example: validate results and calculate reference barriers

Validate the example results and calculate the DFT reference barriers for one pathway.

Run from: Ion_migration_NEB/analysis/
Requires: numpy pandas matplotlib pymatgen ase scikit-learn

import json, sys
sys.path.insert(0, "../scripts")
from neb_analysis import validate_neb_analysis_inputs, build_neb_analysis_results

with open("../examples/ion_migration_neb_demo_data.json") as f:
    demo = json.load(f)

reference_data = demo["reference_data"]
fp_results     = demo["fp_results"]
fp_order       = list(fp_results["models"].keys())

validation = validate_neb_analysis_inputs(
    reference_data, fp_results, expected_pathways=None, fp_order=fp_order,
)
print("validation ok:", validation.ok)
# validation ok: True

analysis = build_neb_analysis_results(
    reference_data, fp_results, expected_pathways=None,
    fp_order=fp_order, validate=True,
)
print(analysis.dft_path_metrics_df)
#     ICSD  Path  Pathway Topology  energy_forward_barrier  energy_backward_barrier
# 0  29966     8       Normal-Hill                0.165365                 0.139784

analysis.dft_path_metrics_df.to_csv("neb_dft_path_metrics_demo.csv")

# For benchmark evaluation, use the official reference data and your model results.

Format documentation · Example data · Analysis notebook · Component guide

These examples show how to prepare and analyze results. See the analysis notebooks for the complete benchmark analysis.

3. Submit by email

Email Prof. Yifei Mo at yfmo@umd.edu with the information below, together with your complete standardized results files or download links and computed metric tables.

Model name:
Version / checkpoint:
Parameters:
Training data:
License (code):
License (checkpoint):
Implementation:
Checkpoint download:
Paper / DOI:
Environment:             Python version and package versions
Calculator setup:        Code snippet or link
Components evaluated:    Force / Phase stability & ordering / NEB
Datasets evaluated:
Calculation settings:    Configuration file or settings used
Results files:           Download links
Metric tables:           Attached files or links

Mark fields that are unavailable or not applicable accordingly.

Submissions are handled by email. We check submitted files and recompute benchmark metrics before inclusion. The exact checkpoint matters: a later version of the same architecture is a separate entry rather than an update to an existing one.