configuration for 200k or small

[acd1111]

Hi,

Do you have configuration recommendations for training EquiformerV2 on smaller datasets?

My situation:

• Dataset: ~24K training samples (vs 2M in provided config)
• Using: EquiformerV2 N@12_L@6_M@2.yml

Questions:

1. Do you have configs for 20K-200K sample datasets?
2. What parameters should I adjust? Particularly:
   • max_epochs (currently 12 for 2M)
   • Regularization (weight_decay, alpha_drop, etc.)

Any guidance would be appreciated. Thanks!

[yilunliao]

Hi @acd1111

Thanks for your question.

I have some comments below:

1. We can start with smaller degrees (e.g., Lmax = 2 -> 4 -> 6). As datasets become smaller, we might not need higher degrees (i.e., similar accuracy but much slower).
2. For smaller datasets, smaller batch sizes (more randomness) and longer training epochs would be helpful. The number of epochs would simply be an issue of training budget if no overfitting is observed.
3. I think weight decay and dropout should be kept the same as based on my experience, they would not have much performance gain.
4. Another thing you can try is to use denoising. For datasets containing equilibrium structures like QM9 and OC20 IS2RE, the work of Noisy Nodes has shown this can boost performance by a lot. One example of training Equiformer(V1) with Noisy Nodes on OC20 IS2RE can be found here. For other datasets containing non-equilibrium structures (e.g., predicting forces) like OC20 S2EF, the work of DeNS has shown consistent performance gain. The implementation of training EquiformerV2 with DeNS on OC20 S2EF is here.

Let me know if you have further questions!

[acd1111]

Thank you for your response! I now have some additional questions that I’d like to ask. My dataset is a subset of 20M samples, resulting in about 23K training samples, and I’m training EquiformerV2 for the S2EF task. Since I want to train the model on a specific system, such as adsorbates being H, I extracted 23K samples from the 20M dataset. However, due to the large model parameters, the performance is quite poor, and I would greatly appreciate your help.The provided configuration in the project is for a 2M dataset. I would like to know if there are any other specific adjustments I should make for my smaller dataset (23K) besides the parameters you mentioned earlier.Please see the parameters below.

trainer: forces_v2

dataset:

• src: datasets/oc20/s2ef/2M/train/
  normalize_labels: True
  target_mean: -0.7554450631141663
  target_std: 2.887317180633545
  grad_target_mean: 0.0
  grad_target_std: 2.887317180633545
• src: datasets/oc20/s2ef/all/val_id/

logger: wandb

task:
dataset: trajectory_lmdb_v2
description: "Regressing to energies and forces for DFT trajectories from OCP"
type: regression
metric: force_mae
labels:
- potential energy
grad_input: atomic forces
train_on_free_atoms: True
eval_on_free_atoms: True

hide_eval_progressbar: False

model:
name: equiformer_v2

use_pbc: True
regress_forces: True
otf_graph: True
max_neighbors: 20
max_radius: 12.0
max_num_elements: 90

num_layers: 12
sphere_channels: 128
attn_hidden_channels: 64 # [64, 96] This determines the hidden size of message passing. Do not necessarily use 96.
num_heads: 8
attn_alpha_channels: 64 # Not used when use_s2_act_attn is True.
attn_value_channels: 16
ffn_hidden_channels: 128
norm_type: 'layer_norm_sh' # ['rms_norm_sh', 'layer_norm', 'layer_norm_sh']

lmax_list: [2]
mmax_list: [2]
grid_resolution: 18 # [18, 16, 14, None] For None, simply comment this line.

num_sphere_samples: 128

edge_channels: 128
use_atom_edge_embedding: True
share_atom_edge_embedding: False # If True, use_atom_edge_embedding must be True and the atom edge embedding will be shared across all blocks.
distance_function: 'gaussian'
num_distance_basis: 512 # not used

attn_activation: 'silu'
use_s2_act_attn: False # [False, True] Switch between attention after S2 activation or the original EquiformerV1 attention.
use_attn_renorm: True # Attention re-normalization. Used for ablation study.
ffn_activation: 'silu' # ['silu', 'swiglu']
use_gate_act: False # [True, False] Switch between gate activation and S2 activation
use_grid_mlp: True # [False, True] If True, use projecting to grids and performing MLPs for FFNs.
use_sep_s2_act: True # Separable S2 activation. Used for ablation study.

alpha_drop: 0.1 # [0.0, 0.1]
drop_path_rate: 0.05 # [0.0, 0.05]
proj_drop: 0.0

weight_init: 'uniform' # ['uniform', 'normal']

optim:
batch_size: 2 # 6
eval_batch_size: 4 # 6
grad_accumulation_steps: 1 # gradient accumulation: effective batch size = grad_accumulation_steps * batch_size * (num of GPUs)
load_balancing: atoms
num_workers: 8
lr_initial: 0.0002 # [0.0002, 0.0004], eSCN uses 0.0008 for batch size 96

optimizer: AdamW
optimizer_params:
weight_decay: 0.001
scheduler: LambdaLR
scheduler_params:
lambda_type: cosine
warmup_factor: 0.2
warmup_epochs: 0.1
lr_min_factor: 0.01

max_epochs: 20
force_coefficient: 100
energy_coefficient: 2
clip_grad_norm: 100
ema_decay: 0.999
loss_energy: mae
loss_force: l2mae

eval_every: 5000

#slurm:

constraint: "volta32gb"

[yilunliao]

Hi @acd1111

1. Due to the large model parameters, the performance is quite poor

By poor, have you compared with some other methods trained on the same dataset? It might also be because the dataset is too small.

2. Config

Can you also check if the batch size you used is reasonable? If yes, I think you can also check if training errors and validation errors make sense. Specifically, the training error should be quite low given the dataset you used is quite small, and the validation error should be higher than training error.

[acd1111]

Dear Author,

Due to limited GPU resources in our lab, and because my research focuses on a specific adsorbate such as H, I used a small dataset extracted from a 20M dataset for training. The training set contains 23k samples, and the validation set contains 5k samples. Although the results would be much better using the full 20M dataset, in this specific small dataset setting, the modified equiformer_v2 model outperforms other classical models, such as cgcnn, and even exceeds the baseline equiformer_v2 model.

Although the overall performance is not as good as when using the full dataset, the improved model performs better in this specific small dataset environment. Could you please advise whether such experimental results, based on the performance within a specific data context, are sufficient to support publishing a paper?

Is it necessary to include the R² evaluation metric when publishing a paper? I noticed that in your paper and many other similar papers, MAE is used as the evaluation standard, but R² is not. Is it because the complexity and diversity of crystalline materials cause R² to perform poorly?

I look forward to your advice and feedback.

Thank you!

[yilunliao]

Hi @acd1111

1. whether such experimental results, based on the performance within a specific data context, are sufficient to support publishing a paper?

It's hard to tell and depends a lot on where you submit. Generally, I think as long as your proposed method is well-motivated, well-designed and shows non-trivial improvements over baselines, it would be valuable. However, smaller-scale experiments might not show the effectiveness or generality of methods.

2. Is it necessary to include the R² evaluation metric when publishing a paper?

No, I think as long as you report metrics also in previous works, it should be sufficient.

[acd1111]

I hope you're doing well. I have a question regarding the evaluation process for the model, specifically on EvalAI and the validation set.

You mentioned that EvalAI no longer supports testing set evaluation, and that evaluating on the validation set is sufficient. I’m not entirely sure which method would be correct for this:

Should I directly extract the validation metrics from the best checkpoint (best_checkpoint.pt) using a script like:
python -c "import torch; ckpt = torch.load('checkpoints/2025-11-02-20-48-00/best_checkpoint.pt', map_location='cpu'); print('Val Metrics:', ckpt.get('val_metrics', {}))"

Or should I run the validation process again by changing the mode to validate in the .sh script and adding the --checkpoint argument, Does the yml configuration file need to be modified for this approach? like this:
python -u -m torch.distributed.launch --nproc_per_node=4 main_oc20.py \ --distributed \ --num-gpus 4 \ --mode validate \ --config-yml 'oc20/configs/s2ef/2M/equiformer_v2/A100.yml' \ --run-dir 'models/oc20/s2ef/2M/equiformer_v2/N@12_L@2_M@2/bs@32_lr@2e-4_wd@1e-3_epochs@12_warmup-epochs@0.1_g@4' \ --print-every 200 \ --amp \ --checkpoint 'models/oc20/s2ef/2M/equiformer_v2/N@12_L@2_M@2/bs@32_lr@2e-4_wd@1e-3_epochs@12_warmup-epochs@0.1_g@4/checkpoints/2025-11-10-22-21-52/checkpoint.pt'

Additionally, when comparing with other classic models, is the same evaluation procedure used in the fairchem project as well?

I would appreciate any clarification on which method is preferred for evaluating on the validation set.

Thanks a lot for your time!

[yilunliao]

Hi @acd1111

1. Should I directly extract the validation metrics from the best checkpoint (best_checkpoint.pt) using a script like:
   python -c "import torch; ckpt = torch.load('checkpoints/2025-11-02-20-48-00/best_checkpoint.pt', map_location='cpu'); print('Val Metrics:', ckpt.get('val_metrics', {}))"

No, the validation metrics are evaluated on a subset of the whole validation set and are obtained during training. (We usually evaluate on a subset of the validation set during training to keep track of the performance of models.) If you check the validation metrics from the checkpoints, they clearly differ from the numbers reported in papers.

2. Should I run the validation process again by changing the mode to validate in the .sh script and adding the --checkpoint argument, Does the yml configuration file need to be modified for this approach?

Yes, this is correct. The config file should be updated so that you evaluate on all the four sub-splits of the validation set (i.e., ID, OOD-Ads, OOD-Cat and OOD-Both).

3. when comparing with other classic models, is the same evaluation procedure used in the fairchem project as well?

Yes, or something similar is adopted. One way you can double check this is to evaluate GemNet-OC on said ID subsplit and see if the result is the same as what is reported in their paper.

[acd1111]

Hello!

Thank you very much for developing and open-sourcing this project! 🙌

I am currently trying to plot loss curves based on the logs generated during training. However, I noticed that there are multiple loss values recorded under the same epoch (for example, several entries for Epoch 1).

I have two questions and would appreciate your guidance:

Are these loss values recorded per batch, or are they logged at specific intervals during training?
When plotting the loss curve for each epoch, should I average all the loss values within an epoch, or choose a specific value (such as the last one or the minimum) to represent the epoch's loss?
I hope to correctly interpret the logs to avoid misrepresenting the training process.

Thank you very much for your help!

Best regards

[yilunliao]

Hi @acd1111

1. Are these loss values recorded per batch, or are they logged at specific intervals during training?

They should be the average loss computed over 200 training iterations.

2. When plotting the loss curve for each epoch, should I average all the loss values within an epoch, or choose a specific value (such as the last one or the minimum) to represent the epoch's loss?

I think each of them works as long as you plot the loss curves in a consistent manner. I would simply use the loss shown in the log for convenience.

[acd1111]

Hello!

Thank you very much for developing and open-sourcing this project! 🙌

I am working on the S2EF (Structure to Energy and Forces) task and would appreciate some clarification on the following points:

1. Input and Target Definitions:
   • Structure: Does the input structure refer to the atomic coordinates of the combined system (i.e., both the crystal surface and the adsorbate)?
   • Energy: Does the target energy refer to the adsorption energy or the total potential energy of the system?
   • Forces: Do the target forces represent the atomic forces on all individual atoms (), or specifically the interaction forces between the adsorbate and the crystal surface?
2. Performance & Baselines: I conducted an experiment using a small dataset containing only Hydrogen (H) adsorbates, but I observed a very low score.
   • Is this performance expected given the limited data size?
   • Did you calculate/track the metric when training on the larger 2M or 20M datasets? If so, could you share what a typical looks like for those settings? It would be very helpful to have a baseline for comparison.
   Thanks for your help!

[yilunliao]

Hi @acd1111

For your latest questions, the only thing I can answer is that we trained on adsorption energy for OC20 (following previous works) while training on total energy would be better (shown in subsequent works).

You can check the OC20 paper for the rest of your questions.