How can directly download the entire expression matrix (4.0 million single cells × 30,000 genes)

[Counts-Xin]

Great resource! I was wondering how I can directly download the entire expression matrix (4.0 million single cells × 30,000 genes).

[danielsf]

Because the data is so massive, we split the data into several .h5ad files, roughly divided based on the anatomical region from which the samples were taken. Furthermore, data sampled from 10Xv3, 10Xv2 and 10XMulti rigs are stored separately. To download all of these files, you want to do something like this.

import pathlib
from abc_atlas_access.abc_atlas_cache.abc_project_cache import AbcProjectCache


def main():

    # specify the directory to which to download the data
    cache_dir = pathlib.Path('ABC_atlas_data')
    abc_cache = AbcProjectCache.from_cache_dir(cache_dir)

    # load a specific version of the data release
    abc_cache.load_manifest('releases/20250131/manifest.json')

    # download all. of the relevant metadata CSV files
    for dir_name in ('WMB-taxonomy', 'WMB-10X', 'WMB-neighborhoods'):
        abc_cache.get_directory_metadata(dir_name)

    # download all of the .h5ad files
    for dir_name in ('WMB-10Xv2', 'WMB-10Xv3', 'WMB-10XMulti'):
        abc_cache.get_directory_data(dir_name)


if __name__ == "__main__":
    main()

This will be on the order of 400 GB (data is stored both as raw counts and as log2(CPM+1) normalized counts).

The files you download will be separated into different directories by rig version (WMB-10Xv2, WMB-10Xv3, WMB-10XMulti). To get the path to a file in a given directory you would do something like

>>> abc_cache.get_data_path(directory='WMB-10Xv2', file_name='WMB-10Xv2-OLF/log2')

To see all of the valid file_names for a directory, do

>>> cache.list_data_files(directory='WMB-10Xv2')

Note: list_data_files will work before you download the data, so you can use this to figure out which files to download if you don't want to download all 400 GB of data. (If a file has not already been downloaded, the call to get_data_path downloads the file, so you could conceivably assemble a list of (directory, file_name) pairs you want to download and loop over that list, calling get_data_path for each to download only the data in which you are interested).

Let me know if anything does not work/is unclear.