LA_Matrix_Factorise: cannot factorise, error: 28

[OmPatel-512]

I'm constantly encountering this error:
ompatel@Om:/mnt/e/CASTEP-ML-data-main/Smmol3d/MD/bkup1_train$ gap_fit at_file=train_final.xyz gp_file=GAP.xml energy_parameter_name=energy force_parameter_name=forces do_copy_at_file=F sparse_separate_file=T default_sigma={ 0.005 0.050 0.1 1.0 } e0_method=average gap={ distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 add_species=T delta=1.0 : soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 add_species=True covar
iance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 } sparse_jitter=1.0e-8
libAtoms::Hello World: 2026-03-25 04:26:21
libAtoms::Hello World: git version https://github.com/libAtoms/QUIP.git,v0.10.2-dirty
libAtoms::Hello World: QUIP_ARCH linux_x86_64_gfortran
libAtoms::Hello World: compiled on Mar 20 2026 at 10:00:15
libAtoms::Hello World: Random Seed = 15981528
libAtoms::Hello World: global verbosity = 0

Calls to system_timer will do nothing by default

================================ Input parameters ==============================

config_file =
atoms_filename = //MANDATORY//
at_file = train_final.xyz
gap = " distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 add_species=T delta=1.0 : soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 add_species=True covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5"
e0 = 0.0
local_property0 = 0.0
e0_offset = 0.0
e0_method = average
default_kernel_regularisation = //MANDATORY//
default_sigma = " 0.005 0.050 0.1 1.0"
default_kernel_regularisation_local_property = 0.001
default_local_property_sigma = 0.001
sparse_jitter = 1.0e-8
hessian_displacement = 1.0e-2
hessian_delta = 1.0e-2
baseline_param_filename = quip_params.xml
core_param_file = quip_params.xml
baseline_ip_args =
core_ip_args =
energy_parameter_name = energy
local_property_parameter_name = local_property
force_parameter_name = forces
virial_parameter_name = virial
stress_parameter_name = stress
hessian_parameter_name = hessian
config_type_parameter_name = config_type
kernel_regularisation_parameter_name = sigma
sigma_parameter_name = sigma
force_mask_parameter_name = force_mask
local_property_mask_parameter_name = local_property_mask
parameter_name_prefix =
config_type_kernel_regularisation =
config_type_sigma =
kernel_regularisation_is_per_atom = T
sigma_per_atom = T
do_copy_atoms_file = T
do_copy_at_file = F
sparse_separate_file = T
sparse_use_actual_gpcov = F
gap_file = gap_new.xml
gp_file = GAP.xml
verbosity = NORMAL
rnd_seed = -1
openmp_chunk_size = 0
do_ip_timing = F
template_file = template.xyz
sparsify_only_no_fit = F
dryrun = F
condition_number_norm =
linear_system_dump_file =
mpi_blocksize_rows = 0
mpi_blocksize_cols = 100
mpi_print_all = F
export_covariance = F

======================================== ======================================

============== Gaussian Approximation Potentials - Database fitting ============

Initial parsing of command line arguments finished.
Found 2 GAPs.
Descriptors have been parsed
XYZ file read
Old GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 add_species=T delta=1.0}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={62 62 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 62 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 6 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 8 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 7 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={8 62 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={8 8 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 62 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 6 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 8 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 1 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 7 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 62 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 8 }}
New GAP: { distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 7 }}
Old GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 add_species=True covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5}
New GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=62 species_Z={62 6 8 1 7 }}
New GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=6 species_Z={62 6 8 1 7 }}
New GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=8 species_Z={62 6 8 1 7 }}
New GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=1 species_Z={62 6 8 1 7 }}
New GAP: { soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=7 species_Z={62 6 8 1 7 }}
Multispecies support added where requested

===================== Report on number of descriptors found ====================

---

Descriptor 1: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={62 62 }
Number of descriptors: 100
Number of partial derivatives of descriptors: 600

Descriptor 2: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 62 }
Number of descriptors: 2318
Number of partial derivatives of descriptors: 13908

Descriptor 3: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 6 }
Number of descriptors: 42514
Number of partial derivatives of descriptors: 255084

Descriptor 4: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 8 }
Number of descriptors: 17760
Number of partial derivatives of descriptors: 106560

Descriptor 5: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={6 7 }
Number of descriptors: 6622
Number of partial derivatives of descriptors: 39732

Descriptor 6: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={8 62 }
Number of descriptors: 2020
Number of partial derivatives of descriptors: 12120

Descriptor 7: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={8 8 }
Number of descriptors: 3998
Number of partial derivatives of descriptors: 23988

Descriptor 8: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 62 }
Number of descriptors: 1188
Number of partial derivatives of descriptors: 7128

Descriptor 9: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 6 }
Number of descriptors: 52608
Number of partial derivatives of descriptors: 315648

Descriptor 10: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 8 }
Number of descriptors: 9168
Number of partial derivatives of descriptors: 55008

Descriptor 11: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 1 }
Number of descriptors: 12856
Number of partial derivatives of descriptors: 77136

Descriptor 12: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={1 7 }
Number of descriptors: 3202
Number of partial derivatives of descriptors: 19212

Descriptor 13: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 62 }
Number of descriptors: 400
Number of partial derivatives of descriptors: 2400

Descriptor 14: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 8 }
Number of descriptors: 2088
Number of partial derivatives of descriptors: 12528

Descriptor 15: distance_Nb order=2 n_sparse=20 cutoff=4.5 cutoff_transition_width=1.0 compact_clusters covariance_type=ard_se theta_uniform=1.0 sparse_method=uniform f0=0.0 delta=1.0 Z={7 7 }
Number of descriptors: 200
Number of partial derivatives of descriptors: 1200

Descriptor 16: soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=62 species_Z={62 6 8 1 7 }
Number of descriptors: 100
Number of partial derivatives of descriptors: 5343

Descriptor 17: soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=6 species_Z={62 6 8 1 7 }
Number of descriptors: 3000
Number of partial derivatives of descriptors: 119670

Descriptor 18: soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=8 species_Z={62 6 8 1 7 }
Number of descriptors: 700
Number of partial derivatives of descriptors: 31032

Descriptor 19: soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=1 species_Z={62 6 8 1 7 }
Number of descriptors: 1800
Number of partial derivatives of descriptors: 73146

Descriptor 20: soap n_sparse=100 n_max=8 l_max=4 cutoff=3.5 cutoff_transition_width=0.5 atom_sigma=0.4 covariance_type=dot_product zeta=4 sparse_method=cur_points delta=0.5 n_species=5 Z=7 species_Z={62 6 8 1 7 }
Number of descriptors: 200
Number of partial derivatives of descriptors: 9447

======================================== ======================================

========================= Memory Estimate (per process) ========================

Descriptors
Descriptor 1 :: x 1 100 memory 800 B
Descriptor 1 :: xPrime 1 600 memory 4800 B
Descriptor 2 :: x 1 2318 memory 18 KB
Descriptor 2 :: xPrime 1 13908 memory 111 KB
Descriptor 3 :: x 1 42514 memory 340 KB
Descriptor 3 :: xPrime 1 255084 memory 2040 KB
Descriptor 4 :: x 1 17760 memory 142 KB
Descriptor 4 :: xPrime 1 106560 memory 852 KB
Descriptor 5 :: x 1 6622 memory 52 KB
Descriptor 5 :: xPrime 1 39732 memory 317 KB
Descriptor 6 :: x 1 2020 memory 16 KB
Descriptor 6 :: xPrime 1 12120 memory 96 KB
Descriptor 7 :: x 1 3998 memory 31 KB
Descriptor 7 :: xPrime 1 23988 memory 191 KB
Descriptor 8 :: x 1 1188 memory 9504 B
Descriptor 8 :: xPrime 1 7128 memory 57 KB
Descriptor 9 :: x 1 52608 memory 420 KB
Descriptor 9 :: xPrime 1 315648 memory 2525 KB
Descriptor 10 :: x 1 9168 memory 73 KB
Descriptor 10 :: xPrime 1 55008 memory 440 KB
Descriptor 11 :: x 1 12856 memory 102 KB
Descriptor 11 :: xPrime 1 77136 memory 617 KB
Descriptor 12 :: x 1 3202 memory 25 KB
Descriptor 12 :: xPrime 1 19212 memory 153 KB
Descriptor 13 :: x 1 400 memory 3200 B
Descriptor 13 :: xPrime 1 2400 memory 19 KB
Descriptor 14 :: x 1 2088 memory 16 KB
Descriptor 14 :: xPrime 1 12528 memory 100 KB
Descriptor 15 :: x 1 200 memory 1600 B
Descriptor 15 :: xPrime 1 1200 memory 9600 B
Descriptor 16 :: x 4101 100 memory 3280 KB
Descriptor 16 :: xPrime 4101 5343 memory 175 MB
Descriptor 17 :: x 4101 3000 memory 98 MB
Descriptor 17 :: xPrime 4101 119670 memory 3926 MB
Descriptor 18 :: x 4101 700 memory 22 MB
Descriptor 18 :: xPrime 4101 31032 memory 1018 MB
Descriptor 19 :: x 4101 1800 memory 59 MB
Descriptor 19 :: xPrime 4101 73146 memory 2399 MB
Descriptor 20 :: x 4101 200 memory 6561 KB
Descriptor 20 :: xPrime 4101 9447 memory 309 MB
Subtotal 8028 MB

Covariances
yY 800 17450 memory 111 MB * 2
yy 800 800 memory 5120 KB
A 800 18250 memory 116 MB * 2
Subtotal 462 MB

Peak1 8139 MB
Peak2 462 MB
PEAK 8139 MB

Free system memory 12 GB
Total system memory 14 GB

======================================== ======================================

E0/atom = -0.16395742012327585E+003 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 -0.16395742012327585E+003 -0.16395742012327585E+003 -0.16395742012327585E+003 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 -0.16395742012327585E+003 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000 0.00000000000000000E+000

========== Report on number of target properties found in training XYZ: ========

Number of target energies (property name: energy) found: 50
Number of target local_properties (property name: local_property) found: 0
Number of target forces (property name: forces) found: 17400
Number of target virials (property name: virial) found: 0
Number of target Hessian eigenvalues (property name: hessian) found: 0

================================= End of report ================================

===== Report on per-configuration/per-atom sigma (error parameter) settings ====

Number of per-configuration setting of energy_sigma found: 0
Number of per-configuration setting of force_sigma found: 0
Number of per-configuration setting of virial_sigma found: 0
Number of per-configuration setting of hessian_sigma found: 0
Number of per-configuration setting of local_propery_sigma found:0
Number of per-atom setting of force_atom_sigma found: 0
Number of per-component setting of force_component_sigma found: 0
Number of per-component setting of virial_component_sigma found: 0

================================= End of report ================================

Started CUR decomposition
cur_decomposition: iteration: 1, error: .11667918879304737E-020
Finished CUR decomposition
Started CUR decomposition
cur_decomposition: iteration: 1, error: 1.0472707694160066
Finished CUR decomposition
Started CUR decomposition
cur_decomposition: iteration: 1, error: .50348265414339399
Finished CUR decomposition
Started CUR decomposition
cur_decomposition: iteration: 1, error: 1.6016962290107417
Finished CUR decomposition
Started CUR decomposition
cur_decomposition: iteration: 1, error: .71861738419818808E-003
Finished CUR decomposition
Cartesian coordinates transformed to descriptors
Started sparse covariance matrix calculation of coordinate 1

Finished sparse covariance matrix calculation of coordinate 1
TIMER: gpFull_covarianceMatrix_sparse_Coordinate1_sparse done in .70969999999874744E-002 cpu secs, .71043339848984033E-002 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate1 done in .72260000000028413E-002 cpu secs, .72357309982180595E-002 wall clock secs.
Started sparse covariance matrix calculation of coordinate 2

Finished sparse covariance matrix calculation of coordinate 2
TIMER: gpFull_covarianceMatrix_sparse_Coordinate2_sparse done in .43451000000004569E-001 cpu secs, .43452134006656706E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate2 done in .43518000000005941E-001 cpu secs, .43517663987586275E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 3

Finished sparse covariance matrix calculation of coordinate 3
TIMER: gpFull_covarianceMatrix_sparse_Coordinate3_sparse done in .78493600000001607 cpu secs, .78495232400018722 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate3 done in .78498999999999342 cpu secs, .78500646399334073 wall clock secs.
Started sparse covariance matrix calculation of coordinate 4

Finished sparse covariance matrix calculation of coordinate 4
TIMER: gpFull_covarianceMatrix_sparse_Coordinate4_sparse done in .26605200000000195 cpu secs, .26605215598829091 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate4 done in .26612299999999323 cpu secs, .26612245000433177 wall clock secs.
Started sparse covariance matrix calculation of coordinate 5

Finished sparse covariance matrix calculation of coordinate 5
TIMER: gpFull_covarianceMatrix_sparse_Coordinate5_sparse done in .11059199999999691 cpu secs, .11059286998352036 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate5 done in .11066499999998314 cpu secs, .11066448802012019 wall clock secs.
Started sparse covariance matrix calculation of coordinate 6

Finished sparse covariance matrix calculation of coordinate 6
TIMER: gpFull_covarianceMatrix_sparse_Coordinate6_sparse done in .29222000000004300E-001 cpu secs, .29222235025372356E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate6 done in .29271999999991749E-001 cpu secs, .29271692998008803E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 7

Finished sparse covariance matrix calculation of coordinate 7
TIMER: gpFull_covarianceMatrix_sparse_Coordinate7_sparse done in .82250999999985197E-001 cpu secs, .82251229992834851E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate7 done in .82300999999972646E-001 cpu secs, .82300653011770919E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 8

Finished sparse covariance matrix calculation of coordinate 8
TIMER: gpFull_covarianceMatrix_sparse_Coordinate8_sparse done in .21955999999988762E-001 cpu secs, .21971695998217911E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate8 done in .22015000000010332E-001 cpu secs, .22030479012755677E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 9

Finished sparse covariance matrix calculation of coordinate 9
TIMER: gpFull_covarianceMatrix_sparse_Coordinate9_sparse done in .59922599999998738 cpu secs, .59922679097508080 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate9 done in .59927200000004177 cpu secs, .59927212001639418 wall clock secs.
Started sparse covariance matrix calculation of coordinate 10

Finished sparse covariance matrix calculation of coordinate 10
TIMER: gpFull_covarianceMatrix_sparse_Coordinate10_sparse done in .10838000000001102 cpu secs, .10838152200449258 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate10 done in .10845299999999725 cpu secs, .10845384999993257 wall clock secs.
Started sparse covariance matrix calculation of coordinate 11

Finished sparse covariance matrix calculation of coordinate 11
TIMER: gpFull_covarianceMatrix_sparse_Coordinate11_sparse done in .25515000000001464 cpu secs, .25515077001182362 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate11 done in .25519299999996292 cpu secs, .25519335098215379 wall clock secs.
Started sparse covariance matrix calculation of coordinate 12

Finished sparse covariance matrix calculation of coordinate 12
TIMER: gpFull_covarianceMatrix_sparse_Coordinate12_sparse done in .37932000000012067E-001 cpu secs, .37931995990220457E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate12 done in .37971999999967920E-001 cpu secs, .37972691992763430E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 13

Finished sparse covariance matrix calculation of coordinate 13
TIMER: gpFull_covarianceMatrix_sparse_Coordinate13_sparse done in .97979999999893153E-002 cpu secs, .97979830170515925E-002 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate13 done in .98590000000058353E-002 cpu secs, .98582089995034039E-002 wall clock secs.
Started sparse covariance matrix calculation of coordinate 14

Finished sparse covariance matrix calculation of coordinate 14
TIMER: gpFull_covarianceMatrix_sparse_Coordinate14_sparse done in .30221999999980653E-001 cpu secs, .30222918023355305E-001 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate14 done in .30273999999963053E-001 cpu secs, .30273698997916654E-001 wall clock secs.
Started sparse covariance matrix calculation of coordinate 15

Finished sparse covariance matrix calculation of coordinate 15
TIMER: gpFull_covarianceMatrix_sparse_Coordinate15_sparse done in .87580000000002656E-002 cpu secs, .87584950088057667E-002 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate15 done in .88049999999952888E-002 cpu secs, .88047680037561804E-002 wall clock secs.
Started sparse covariance matrix calculation of coordinate 16
Covariance matrix 100% || 4.0 / 4.0 s
Finished sparse covariance matrix calculation of coordinate 16
TIMER: gpFull_covarianceMatrix_sparse_Coordinate16_sparse done in 4.0340349999999603 cpu secs, 4.0340813499933574 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate16 done in 4.2949220000000423 cpu secs, 4.2949975550000090 wall clock secs.
Started sparse covariance matrix calculation of coordinate 17
Covariance matrix 100% || 89.5 / 89.5 s
Finished sparse covariance matrix calculation of coordinate 17
TIMER: gpFull_covarianceMatrix_sparse_Coordinate17_sparse done in 89.548076999999978 cpu secs, 89.550126427988289 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate17 done in 90.125247999999999 cpu secs, 90.127439133008011 wall clock secs.
Started sparse covariance matrix calculation of coordinate 18
Covariance matrix 100% || 22.2 / 22.2 s
Finished sparse covariance matrix calculation of coordinate 18
TIMER: gpFull_covarianceMatrix_sparse_Coordinate18_sparse done in 22.222975999999960 cpu secs, 22.223544915003004 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate18 done in 22.287811999999974 cpu secs, 22.288385518011637 wall clock secs.
Started sparse covariance matrix calculation of coordinate 19
Covariance matrix 100% || 53.7 / 53.7 s
Finished sparse covariance matrix calculation of coordinate 19
TIMER: gpFull_covarianceMatrix_sparse_Coordinate19_sparse done in 53.689488999999980 cpu secs, 53.690745783998864 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate19 done in 53.819550000000049 cpu secs, 53.820811473007780 wall clock secs.
Started sparse covariance matrix calculation of coordinate 20
Covariance matrix 100% |********************| 6.9 / 6.9 s
Finished sparse covariance matrix calculation of coordinate 20
TIMER: gpFull_covarianceMatrix_sparse_Coordinate20_sparse done in 6.9120389999999361 cpu secs, 6.9340069669997320 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_Coordinate20 done in 6.9282469999999421 cpu secs, 6.9502176049863920 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse_FunctionValues done in .12100000003556488E-003 cpu secs, .12096899445168674E-003 wall clock secs.
TIMER: gpFull_covarianceMatrix_sparse done in 179.91783399999997 cpu secs, 179.92684430300142 wall clock secs.
SYSTEM ABORT: Traceback (most recent call last)
File "../src/libAtoms/linearalgebra.F90", line 2348 kind unspecified
LA_Matrix_Factorise: cannot factorise, error: 28
STOP 1

What could possibly be the reason for it?

Here is the first frame of the system :

"
116
Lattice="12.110000 0.000000 0.000000 4.604120 11.298959 0.000000 4.297149 0.328956 12.412076" Properties=species:S:1:pos:R:3:forces:R:3 energy=-19018.287747 pbc="T T T"
Sm 10.0095754464 5.5101105805 8.0959549602 0.402217 0.449283 0.391315
Sm 11.1674278735 6.0151215706 4.3412311820 -0.396490 0.409651 -0.453053
C 12.3777789948 3.6217097314 6.4869510573 0.596237 -0.887282 1.407474
C 8.8058479255 7.8609267392 5.9139412566 -0.942015 1.087461 0.489077
C 13.0671238174 2.2764988867 6.6047801402 -0.217359 -0.340800 1.144678
C 8.0459323151 9.2039877786 5.8215842044 0.900016 -1.669504 0.471308
C 2.1821380679 0.3907176901 5.6096458999 0.147583 -0.232492 -0.399393
C 19.0766350045 11.0508220339 6.8879191154 -0.436376 -0.194161 0.643004
C 13.5655764892 1.6080569528 5.5125190858 0.606711 -0.452970 0.508255
C 7.6301121208 9.7475317008 7.0126298348 -1.661312 1.971064 0.018264
C 9.0422554160 2.1084671443 10.9770529221 0.668959 -0.742950 0.180773
C 11.8461601935 9.9977698310 1.7128904739 -1.448867 0.800732 -0.022630
C 8.6611217597 3.0473780764 9.9019294650 0.602205 -0.399505 0.026296
C 12.2044685118 8.8810603637 2.6251979465 -0.107912 -0.664822 0.331137
C 7.9194017133 9.8242981556 4.5590802572 0.242189 -0.286973 0.245673
C 13.2394335555 1.6786188036 7.8956724266 1.081053 -0.396336 -1.538748
C 7.3245144103 11.0608706810 4.4392379369 0.452723 -0.289046 -0.014112
C 13.9139250565 0.4652446655 8.0006254166 -0.305300 -0.218942 0.111332
C 10.3741614090 1.7366288130 11.0918303771 -0.906975 -0.150614 0.568543
C 10.4775617986 10.3223662081 1.5781533468 1.117182 0.534216 -0.959247
C 10.7384542154 0.8074284634 12.1488043189 0.355888 2.162447 -1.790099
C 10.0473166637 11.2495384517 0.5541137564 -0.382707 -1.573138 1.528449
C 16.7048710088 11.9115406970 12.4066723862 0.036766 -0.933071 0.292812
C 4.0760039383 0.2755790153 0.3194461019 0.617880 -0.872094 0.631805
C 12.8260784775 10.6673198666 0.9547556433 -1.551352 1.093288 -1.106421
C 8.0602223726 1.5032851024 11.7530491869 -0.659294 -0.589039 0.969217
C 9.0139191545 3.9627829422 4.9497028683 0.142271 -0.788414 1.960330
C 11.8926215920 7.8414765892 7.4561091670 -0.625965 -0.939385 -0.290384
C 8.3218859405 2.7064379274 5.3942276416 0.075766 -0.820899 -0.959281
C 12.3621050486 9.2418029361 7.1784196390 -0.166642 -0.534005 -0.267523
C 8.9382187038 1.9587359664 6.4201647812 0.632553 1.347959 0.486664
C 11.7934481507 9.9613809198 6.1249385750 0.121366 -0.713817 0.783995
C 8.5642385255 0.6427144048 6.7453013165 -0.907008 -0.203944 -0.866480
C 12.1167900506 11.3097997450 5.9806063154 0.312251 1.233034 -0.070979
C 13.7416380913 11.1403691005 7.7254487886 2.100435 0.683610 1.798234
C 6.8321356434 0.8032583000 5.0569200132 0.424571 2.298415 -1.509981
C 13.3296732144 9.8625690116 8.0113726209 -0.356112 0.413543 -1.193769
C 7.2602130412 2.1281106459 4.6710471480 -0.750512 -2.538747 1.208836
C 7.8603791751 7.2416797847 10.1331542545 -0.818420 -1.095773 2.759337
C 13.0000947709 4.0018569819 2.1109860330 -0.712859 -0.622682 1.674084
C 7.0330073045 7.5508677251 11.2809090585 0.566892 0.844880 -2.370625
C 13.6311941310 3.4402318949 0.9874326377 -1.189431 1.500734 0.838930
C 3.0314291557 6.8215271824 0.1330922240 -0.865507 0.372389 0.108840
C 17.4619910304 4.2349939361 12.1936849741 0.896601 -0.268317 -0.789957
C 4.1722222107 6.0654085510 0.2593796441 2.471516 -1.566768 0.190110
C 16.4579352821 5.2301299444 12.0902006182 0.504424 -0.650695 0.382407
C 4.6527926564 5.5650213851 1.5123887001 0.082738 1.342851 1.430255
C 16.0712205642 5.7756638772 10.8349537731 -0.110299 0.497499 0.432083
C 5.8640786458 4.9507037464 1.7112248223 -1.542024 -0.213128 -0.724029
C 15.0814882504 6.7088102211 10.8399610355 0.486695 -0.136003 -2.587024
C 6.5907516952 4.6156155837 0.5604567605 2.816185 -0.516284 -0.148283
C 14.3631401942 7.1020699117 11.9501369273 -1.057996 1.176434 0.684552
C 7.8800364382 3.9686636755 0.6949310392 0.551530 0.383510 -0.124796
C 13.2352159485 7.9786469665 11.8219476901 0.216446 -0.280245 1.953536
C 12.9526427578 3.9877630292 11.9749227806 -2.008784 1.139745 0.233853
C 8.3068616340 7.9492475641 0.6149043694 -1.270885 0.374058 -0.084060
C 12.4772328453 4.4695429137 10.7147134202 0.125662 -0.312017 1.050299
C 8.7402623835 7.4105052444 1.8473922616 -0.262212 -0.486384 -0.441898
C 10.5356783842 5.3967426345 11.6937581225 0.041502 -0.829740 -1.437554
C 10.4613946289 6.3104491403 0.8193352369 0.204077 -0.690192 0.576729
C 9.2783509045 6.0947030022 11.5244098774 0.184449 -0.127060 -0.805380
C 11.5588000676 5.3540378297 0.8891328269 -0.257053 -0.336422 0.295104
O 7.7992352337 4.8958209102 7.9551866449 -0.353243 0.341128 0.447234
O 13.3347215610 6.4386428532 3.8903973813 -0.028829 -0.525385 -0.727932
O 11.7177243870 4.0262237841 7.5247063227 0.657464 -0.131676 -1.028518
O 9.2284351638 7.3624206906 4.8201798594 -0.100245 -0.298726 0.817083
O 9.6676617514 3.5833489617 9.3663074692 0.144747 0.303575 -0.772951
O 11.3907325175 8.5499510453 3.5443933106 0.189992 -0.424016 -0.106034
O 12.6226364215 4.2227824894 5.4177315779 -1.037372 0.964136 -0.935055
O 8.9188729410 7.4545885227 7.1278825583 0.381789 -0.673585 -0.967108
O 9.2946220197 4.2609350147 3.7767850487 0.013404 0.001064 -0.621571
O 11.5955333219 7.5229629767 8.6527417978 0.318252 0.175509 -0.911938
O 9.4653072020 4.6133544771 5.9716008293 0.710354 0.872813 -0.563949
O 11.6944704449 6.9964639287 6.4546833909 0.213496 0.899596 0.916886
O 7.4539929025 3.1247983431 9.6130861849 -0.925217 0.496301 -0.398289
O 13.2707607403 8.2867692989 2.2761149247 0.092341 -0.253578 0.206714
H 7.6871425545 4.1053100367 7.4057373904 -0.161800 -0.269150 -0.143117
H 13.5870407136 7.0044489266 4.6211873251 0.546118 0.748118 0.690310
H 13.4786173561 2.0521320481 4.5555986931 -0.188147 0.442494 -1.064176
H 7.6418923456 9.2092355534 7.9971127918 0.317748 0.642103 -1.072046
H 8.3690351824 9.1996662232 3.7691073575 -0.255909 0.790939 -0.016377
H 13.0263860940 2.3162271971 8.7213262709 -0.606433 0.009755 0.937534
H 2.7792570740 0.2617232636 3.4662080499 -0.482182 0.117831 0.161579
H 18.5480944229 11.2686799783 8.9666206995 0.291473 0.042908 0.336527
H 10.9806724300 2.2730795335 10.3438239394 0.535827 -0.647082 0.652508
H 9.8007762647 9.8154040949 2.2427158183 -0.416899 -0.036981 0.143029
H 12.6967487077 1.2019384627 11.7315926855 1.016081 -0.075641 0.082029
H 7.9633315048 10.9864598778 0.9147274106 -0.137804 0.582752 -0.266880
H 13.8390312666 10.4819705072 1.1112040297 1.766669 -0.410221 0.333099
H 7.0341242045 1.7420033127 11.6028402951 -0.834459 0.251575 -0.356107
H 9.7354659911 2.5064875689 7.0001783527 -0.667998 -0.914426 -0.490957
H 11.1342211633 9.5137484014 5.3791758784 -0.049463 -0.084402 0.351705
H 10.0051752679 0.2558059646 8.3269396399 -0.943151 -0.354010 -0.641254
H 10.5958381246 11.6752907000 4.5359313729 -0.359317 -0.213472 -0.287228
H 13.8942257640 9.3406610269 8.7417330506 0.156243 -0.771183 0.844236
H 6.7074769895 2.5875653207 3.8599276428 0.148391 0.378209 -0.070259
H 7.5285808546 7.4773501765 9.1638614877 -0.328432 0.468603 -1.337761
H 13.2534619348 3.6793871389 3.1446737798 0.209595 0.181269 -0.637682
H 18.3300089520 8.1898798797 11.0038437390 -0.836715 0.378086 0.365017
H 2.3521063783 2.8020638418 1.1974517319 0.509528 -0.591092 -0.457102
H 14.5249416100 7.0699630103 1.0030368397 0.018227 -0.001247 -0.158754
H 5.8624285673 3.8899862561 11.2930480043 -0.195124 0.032008 -0.285123
H 4.1566709474 5.9647079338 2.4036479997 -0.426774 -0.407292 -0.040219
H 16.5690599162 5.5673303526 9.8928081717 0.243741 -0.423902 0.075108
H 6.1731857602 4.5758491541 2.6771392996 0.155405 0.200692 0.304747
H 14.8163563296 7.1434371187 9.8648955859 -0.120600 -0.023859 0.253987
H 8.2990155069 3.9629586766 1.7300111699 -0.255194 -0.741107 -0.662987
H 12.8468132050 8.3705639049 10.8892441938 0.143606 -0.145343 -0.335107
H 13.8723761199 3.4518115181 11.9711532899 0.873429 -0.251411 0.429370
H 7.3532483087 8.5460313252 0.6690611779 0.953712 -0.434585 -0.310652
H 13.1389122648 4.3561118000 9.8510952963 -0.377095 -0.021089 -0.205118
H 8.1319836501 7.5303601479 2.7398278020 0.107315 0.219347 0.135495
N 8.9123120952 6.4453012473 10.2387524489 0.544508 0.364772 0.724633
N 11.9493376168 4.8317465471 2.1120763496 0.160163 0.686285 -1.158136
N 11.3332474819 5.1137545534 10.5697091542 -0.838460 1.114784 0.452429
N 9.7913940054 6.6062697087 1.9751316904 0.601106 0.142760 -0.360783
"

[albapa]

It's hard to tell. Can you share all 50 frames you used? Probably unrelated but your delta=1 values for the 2-body terms are too large: these will most likely dominate over the soap terms.

[OmPatel-512]

train_sub.xyz.txt
I tried with lesser delta value, it's resulting in same error.
Here it is. Just a side note but I do feel it's lapack library issue and I'm not able to catch that. Any idea on how to resolve that one?

[albapa]

This runs fine for me making debugging somewhat difficult.

I tried with lesser delta value, it's resulting in same error.

Like I said, this is likely not related, but would most likely result in a suboptimal model.

I do feel it's lapack library issue and I'm not able to catch that. Any idea on how to resolve that one?

Did you compile it locally? Can you use a different LAPACK/BLAS?

Are you running this on small machine e.g. a laptop or similar? While I note the predicted memory usage (8GB) is smaller than what's nominally available (12 GB) this doesn't include what LAPACK would use in which case you may be right and it is a LAPACK issue but ideally you would do this on a server.

[OmPatel-512]

I ran in both my laptop and HPC. Their also it's resulting in same error. I posted the snippets from my laptop though. How to resolve this lapack issue and which lapack will be more suitable? openblas or scalapack?

[albapa]

You didn't say if you compiled it locally or e.g. used the executable from the pip wheel.

I used openblas (which includes LAPACK). Scalapack is a different thing (the distributed extension of LAPACK) and is only needed if you are using the MPI-parallel gap_fit.

[OmPatel-512]

I compiled it locally.
Ok, I'll try with openblas now one more time. Does it work well with MKL? because installing openblas will interfere with other programs

[albapa]

It should work with MKL (it's worked in the past).

[lormio]

I am encountering the same (or at least similar) issue with the gap_fit executable compiled with meson, with MPI turned on:

SYSTEM ABORT: proc=0 Traceback (most recent call last)
File "../src/libAtoms/linearalgebra.F90", line 2348 kind unspecified
LA_Matrix_Factorise: cannot factorise, error: 8

However, when building gap_fit using QUIP_ARCH=linux_x86_64_gfortran_openmpi and make config (using -lopenblas -lscalapack and no extra link options), the produced gap_fit does not have any problem running the fit to the end, using the same training file and config_file.

I am compiling quip inside a conda environment with openblas, scalapack and compilers downloaded from conda itself.

[albapa]

I will investigate what happens if I build with meson.

[OmPatel-512]

I am encountering the same (or at least similar) issue with the gap_fit executable compiled with meson, with MPI turned on:

SYSTEM ABORT: proc=0 Traceback (most recent call last) File "../src/libAtoms/linearalgebra.F90", line 2348 kind unspecified LA_Matrix_Factorise: cannot factorise, error: 8

However, when building gap_fit using QUIP_ARCH=linux_x86_64_gfortran_openmpi and make config (using -lopenblas -lscalapack and no extra link options), the produced gap_fit does not have any problem running the fit to the end, using the same training file and config_file.

I am compiling quip inside a conda environment with openblas, scalapack and compilers downloaded from conda itself.

Thanks, let me do the same, I compiled it with meson as well as ifort but it's returning same error, i'll try it once with makefile

I compiled it with makefile andnow it's working for me as well

[OmPatel-512]

testrun.zip

But now I'm encountering here for the test run

[albapa]

The reason for the original failure could be that the program was linked dynamically and some mismatch may have occurred at runtime. I will try to reproduce that in the future and fix it but without having access to the same environment this is not trivial.

For your second attempt, you don't say what I should be looking at. I assume you meant test_si/testrun.o17069. There I can see the program crashes when trying to do OpenMP parallelisation. I suggest trying to compile QUIP/GAP without any parallelisation first. I also note that you're using 128 OpenMP threads which is certainly too much and you're way past any speedup. Our OpenMP parallelisation works over atoms, and you have 16 atoms only!

Please note that I will be away for a fortnight and won't be able to respond.

[OmPatel-512]

testrun.zip

hybrid-md, quip, gap_fit is working fine individually
I'm not able to catch this error

This is coming since very long time