|
1 | 1 | --- |
2 | 2 | name: tooluniverse-data-wrangling |
3 | | -description: Universal data access reference for scientific research. Teaches how to download bulk data, parse domain-specific formats, paginate APIs, and handle authentication for ANY data source. Covers 10 API pattern categories for all ToolUniverse data sources plus major uncovered sources. Use when ToolUniverse tools return metadata but you need raw/bulk data, when data is in a format tools don't parse, when multi-step API workflows are needed, or when the source has no ToolUniverse tool. |
| 3 | +description: Universal data access reference for scientific research. Teaches how to download bulk data, parse any scientific file format (VCF, h5ad, mzML, PDB, FASTA, XPT, NIfTI, and 30+ more), paginate REST APIs, and handle authentication. Covers 24 domain API patterns across ALL life science data sources — genomics, proteomics, clinical, imaging, ecology, and more. Use this skill whenever you need to download raw data, parse a file format, access a bulk API, write a multi-step data retrieval workflow, or when a ToolUniverse tool returns metadata but you need the actual data. Also use when the data source has no ToolUniverse tool at all. Even if the user doesn't say "data wrangling" — if their task requires getting data from a scientific database or parsing a scientific file format, this is the skill to use. |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | # Data Wrangling: Universal Access Patterns |
@@ -275,227 +275,26 @@ for f in record["files"]: |
275 | 275 | content = requests.get(f["links"]["self"]).content # download each file |
276 | 276 | ``` |
277 | 277 |
|
278 | | -### 11. Proteomics (PRIDE, MassIVE, PeptideAtlas, ProteomeXchange) |
279 | | -Tools: `PRIDE_*`, `MassIVE_*`, `PeptideAtlas_*` |
280 | | -```python |
281 | | -# PRIDE project files — search then download raw/processed data |
282 | | -project = requests.get("https://www.ebi.ac.uk/pride/ws/archive/v2/projects/PXD012345").json() |
283 | | -files = requests.get(f"https://www.ebi.ac.uk/pride/ws/archive/v2/projects/PXD012345/files").json() |
284 | | -for f in files: |
285 | | - if f["fileName"].endswith(".mzML"): # mass spec data |
286 | | - download_url = f["publicFileLocations"][0]["value"] |
287 | | - |
288 | | -# ProteomeXchange: search across PRIDE + MassIVE + jPOST |
289 | | -px = requests.get("https://proteomecentral.proteomexchange.org/cgi/GetDataset?ID=PXD012345&outputMode=JSON").json() |
290 | | -``` |
291 | | - |
292 | | -### 12. Metabolomics (MetaboLights, Metabolomics Workbench, HMDB) |
293 | | -Tools: `MetaboLights_*`, `MetabolomicsWorkbench_*`, `HMDB_*` |
294 | | -```python |
295 | | -# MetaboLights study download |
296 | | -study_id = "MTBLS1234" |
297 | | -study = requests.get(f"https://www.ebi.ac.uk/metabolights/ws/studies/{study_id}").json() |
298 | | -# Download metabolite assignment file |
299 | | -files = requests.get(f"https://www.ebi.ac.uk/metabolights/ws/studies/{study_id}/files").json() |
300 | | - |
301 | | -# Metabolomics Workbench REST |
302 | | -mw = requests.get("https://www.metabolomicsworkbench.org/rest/study/study_id/ST001234/analysis").json() |
303 | | - |
304 | | -# HMDB metabolite data |
305 | | -hmdb = requests.get("https://hmdb.ca/metabolites/HMDB0000001.xml").text # XML format |
306 | | -``` |
307 | | - |
308 | | -### 13. Microbiome & Metagenomics (MGnify, GMREPO) |
309 | | -Tools: `MGnify_*`, `GMREPO_*` |
310 | | -```python |
311 | | -# MGnify: search analyses, download taxonomy/function profiles |
312 | | -analyses = requests.get("https://www.ebi.ac.uk/metagenomics/api/v1/analyses", |
313 | | - params={"study_accession": "MGYS00001234", "page_size": 100}).json() |
314 | | -# Download OTU/taxonomy TSV |
315 | | -for a in analyses["data"]: |
316 | | - tax_url = f"https://www.ebi.ac.uk/metagenomics/api/v1/analyses/{a['id']}/downloads" |
317 | | - downloads = requests.get(tax_url).json() |
318 | | - |
319 | | -# GMREPO: gut microbiome phenotype associations |
320 | | -gmrepo = requests.get("https://gmrepo.humangut.info/api/getAssociatedSpeciesByMeshID", |
321 | | - params={"meshID": "D003920"}).json() # diabetes |
322 | | -``` |
323 | | - |
324 | | -### 14. Ecology & Biodiversity (GBIF, iNaturalist, OBIS) |
325 | | -Tools: `GBIF_*`, `iNaturalist_*`, `OBIS_*` |
326 | | -```python |
327 | | -# GBIF occurrence data (millions of records, paginated) |
328 | | -all_records = [] |
329 | | -offset = 0 |
330 | | -while True: |
331 | | - resp = requests.get("https://api.gbif.org/v1/occurrence/search", |
332 | | - params={"scientificName": "Panthera tigris", "limit": 300, "offset": offset}).json() |
333 | | - all_records.extend(resp["results"]) |
334 | | - if resp["endOfRecords"]: break |
335 | | - offset += 300 |
336 | | - |
337 | | -# iNaturalist observations |
338 | | -obs = requests.get("https://api.inaturalist.org/v1/observations", |
339 | | - params={"taxon_name": "Danaus plexippus", "per_page": 200, "geo": "true"}).json() |
340 | | -``` |
341 | | - |
342 | | -### 15. Model Organisms (FlyBase, WormBase, ZFIN, RGD, MGI) |
343 | | -Tools: `FlyBase_*`, `WormBase_*`, `ZFIN_*`, `RGD_*` |
344 | | -```python |
345 | | -# FlyBase gene data |
346 | | -fb = requests.get("https://api.flybase.org/api/v1.0/gene/FBgn0000490").json() # dpp gene |
347 | | - |
348 | | -# WormBase gene info |
349 | | -wb = requests.get("https://wormbase.org/rest/widget/gene/WBGene00006763/overview", |
350 | | - headers={"Accept": "application/json"}).json() |
351 | | - |
352 | | -# ZFIN (zebrafish) gene expression |
353 | | -zfin = requests.get("https://zfin.org/action/api/marker/ZDB-GENE-980526-166/expression").json() |
354 | | - |
355 | | -# RGD (rat) — disease annotations for a gene |
356 | | -rgd = requests.get("https://rest.rgd.mcw.edu/rgdws/genes/Tp53/9606").json() # human TP53 |
357 | | -``` |
358 | | - |
359 | | -### 16. Pathways & Networks (Reactome, STRING, BioGRID, WikiPathways) |
360 | | -Tools: `Reactome_*`, `STRING_*`, `BioGRID_*`, `WikiPathways_*` |
361 | | -```python |
362 | | -# Reactome pathway participants |
363 | | -pathway_id = "R-HSA-109582" # Hemostasis |
364 | | -participants = requests.get(f"https://reactome.org/ContentService/data/participants/{pathway_id}").json() |
365 | | - |
366 | | -# STRING protein-protein interactions (bulk) |
367 | | -proteins = "9606.ENSP00000269305%0d9606.ENSP00000344818" # TP53, MDM2 |
368 | | -network = requests.get(f"https://string-db.org/api/json/network?identifiers={proteins}&species=9606").json() |
369 | | - |
370 | | -# BioGRID interactions for a gene (tab-delimited bulk) |
371 | | -url = "https://webservice.thebiogrid.org/interactions/?searchNames=true&geneList=BRCA1&taxId=9606&format=json&accesskey=YOUR_KEY" |
372 | | - |
373 | | -# WikiPathways: download pathway as GPML or GMT |
374 | | -gpml = requests.get("https://www.wikipathways.org/wikipathways/wpi/wpi.php?action=downloadFile&type=gpml&pwTitle=Pathway:WP254").text |
375 | | -``` |
376 | | - |
377 | | -### 17. Ontologies (OLS, Gene Ontology, HPO, Disease Ontology) |
378 | | -Tools: `ols_search_terms`, `ols_get_term_info`, `Gene_Ontology_*` |
379 | | -```python |
380 | | -# OLS term search + hierarchy traversal |
381 | | -terms = requests.get("https://www.ebi.ac.uk/ols4/api/search", |
382 | | - params={"q": "apoptosis", "ontology": "go", "rows": 20}).json() |
383 | | - |
384 | | -# Get children of a GO term |
385 | | -children = requests.get("https://www.ebi.ac.uk/ols4/api/ontologies/go/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FGO_0006915/children").json() |
386 | | - |
387 | | -# HPO: phenotype-to-disease annotations |
388 | | -hpo = requests.get("https://hpo.jax.org/api/hpo/term/HP:0001250/diseases").json() |
389 | | - |
390 | | -# Gene Ontology annotations bulk (GAF format) |
391 | | -# Download from: http://current.geneontology.org/annotations/goa_human.gaf.gz |
392 | | -``` |
393 | | - |
394 | | -### 18. Immunology (IEDB, VDJdb, ImmPort) |
395 | | -Tools: `IEDB_*`, `VDJdb_*` |
396 | | -```python |
397 | | -# IEDB epitope search |
398 | | -epitopes = requests.get("https://query-api.iedb.org/epitope_search", |
399 | | - params={"linear_sequence": "SIINFEKL", "limit": 50}).json() |
400 | | - |
401 | | -# VDJdb: T-cell receptor specificity database |
402 | | -vdjdb = pd.read_csv("https://raw.githubusercontent.com/antigenomics/vdjdb-db/master/latest-version.zip", |
403 | | - compression="zip", sep="\t") |
404 | | - |
405 | | -# ImmPort shared data (requires registration) |
406 | | -# Search at: https://www.immport.org/shared/search |
407 | | -``` |
408 | | - |
409 | | -### 19. Drug & Pharmacology (DrugBank, PharmGKB, SIDER, DGIdb) |
410 | | -Tools: `PharmGKB_*`, `DGIdb_*`, `SIDER_*`, `DrugCentral_*` |
411 | | -```python |
412 | | -# DGIdb drug-gene interactions |
413 | | -dgi = requests.get("https://dgidb.org/api/v2/interactions.json", |
414 | | - params={"genes": "EGFR", "interaction_sources": "DrugBank,ChEMBL"}).json() |
415 | | - |
416 | | -# PharmGKB clinical annotations for a gene |
417 | | -pgkb = requests.get("https://api.pharmgkb.org/v1/data/clinicalAnnotation", |
418 | | - params={"view": "base", "location.genes.symbol": "CYP2D6"}).json() |
419 | | - |
420 | | -# SIDER side effect frequencies (bulk download) |
421 | | -# Download from: http://sideeffects.embl.de/media/download/meddra_freq.tsv.gz |
422 | | -``` |
423 | | - |
424 | | -### 20. Imaging & Atlases (TCIA, HPA, Allen Brain Atlas, BioImage Archive) |
425 | | -Tools: `TCIA_*`, `HPA_*`, `AllenBrainAtlas_*` |
426 | | -```python |
427 | | -# TCIA: cancer imaging collections |
428 | | -collections = requests.get("https://services.cancerimagingarchive.net/nbia-api/services/v1/getCollectionValues").json() |
429 | | -# Get series for a patient |
430 | | -series = requests.get("https://services.cancerimagingarchive.net/nbia-api/services/v1/getSeries", |
431 | | - params={"Collection": "TCGA-BRCA", "PatientID": "TCGA-A1-A0SB"}).json() |
432 | | - |
433 | | -# Human Protein Atlas: tissue expression |
434 | | -hpa = requests.get("https://www.proteinatlas.org/api/search_download.php?search=TP53&format=json&columns=g,t,up").json() |
435 | | - |
436 | | -# Allen Brain Atlas: gene expression in brain regions |
437 | | -aba = requests.get("https://api.brain-map.org/api/v2/data/query.json?criteria=model::Gene,rma::criteria,[acronym$eq'BDNF']").json() |
438 | | -``` |
439 | | - |
440 | | -### 21. Protein Structure Download (RCSB PDB, AlphaFold, CATH) |
441 | | -Tools: `RCSB_*`, `AlphaFold_*`, `PDBe_*`, `CATH_*` |
442 | | -```python |
443 | | -# Download PDB/CIF structure files directly |
444 | | -pdb_id = "1A2B" |
445 | | -pdb_content = requests.get(f"https://files.rcsb.org/download/{pdb_id}.pdb").text |
446 | | -cif_content = requests.get(f"https://files.rcsb.org/download/{pdb_id}.cif").text |
447 | | - |
448 | | -# AlphaFold predicted structure by UniProt ID |
449 | | -uniprot_id = "P04637" # TP53 |
450 | | -af_pdb = requests.get(f"https://alphafold.ebi.ac.uk/files/AF-{uniprot_id}-F1-model_v4.pdb").text |
451 | | -af_cif = requests.get(f"https://alphafold.ebi.ac.uk/files/AF-{uniprot_id}-F1-model_v4.cif").text |
452 | | - |
453 | | -# RCSB advanced search (GraphQL) for bulk queries |
454 | | -query = {"query": {"type": "terminal", "service": "text", "parameters": {"attribute": "rcsb_entity_source_organism.taxonomy_lineage.name", "operator": "exact_match", "value": "Homo sapiens"}}, "return_type": "entry", "request_options": {"paginate": {"start": 0, "rows": 100}}} |
455 | | -results = requests.post("https://search.rcsb.org/rcsbsearch/v2/query", json=query).json() |
456 | | -``` |
457 | | - |
458 | | -### 22. Clinical Genomics & Variant Databases (ClinVar, ClinGen, CIViC, OncoKB) |
459 | | -Tools: `ClinVar_*`, `ClinGen_*`, `CIViC_*`, `OncoKB_*` |
460 | | -```python |
461 | | -# ClinVar bulk download (variant_summary, ~200MB) |
462 | | -# df = pd.read_csv("https://ftp.ncbi.nlm.nih.gov/pub/clinvar/tab_delimited/variant_summary.txt.gz", sep="\t") |
463 | | - |
464 | | -# CIViC GraphQL API — all evidence items for a gene |
465 | | -query = '{"query": "{ gene(entrezId: 673) { name variants { nodes { name evidenceItems { nodes { description evidenceLevel } } } } } }"}' |
466 | | -civic = requests.post("https://civicdb.org/api/graphql", json={"query": query}).json() |
467 | | - |
468 | | -# ClinGen allele registry |
469 | | -allele = requests.get("https://reg.clinicalgenome.org/allele?hgvs=NM_000059.4:c.68_69del").json() |
470 | | -``` |
471 | | - |
472 | | -### 23. Single-Cell Portals (cellxgene, ARCHS4, Cell Marker) |
473 | | -Tools: `cellxgene_*`, `ARCHS4_*` |
474 | | -```python |
475 | | -# cellxgene Census — query human single-cell data at scale |
476 | | -import cellxgene_census # requires cellxgene-census |
477 | | -census = cellxgene_census.open_soma() |
478 | | -adata = cellxgene_census.get_anndata(census, organism="Homo sapiens", |
479 | | - obs_value_filter="tissue_general == 'lung' and disease == 'normal'") |
480 | | - |
481 | | -# ARCHS4 gene expression (pre-computed from SRA) |
482 | | -archs4 = requests.get("https://maayanlab.cloud/archs4/search/loadExpressionTSV.php", |
483 | | - params={"search": "BRCA1", "species": "human"}).text |
484 | | - |
485 | | -# Without cellxgene_census: download h5ad directly from cellxgene data portal |
486 | | -# Browse collections at https://cellxgene.cziscience.com/collections |
487 | | -``` |
488 | | - |
489 | | -### 24. Toxicology & Environmental (CTD, EPA, Tox21) |
490 | | -Tools: `CTD_*`, `EPA_*` |
491 | | -```python |
492 | | -# CTD: chemical-gene-disease interactions |
493 | | -ctd = requests.get("https://ctdbase.org/tools/batchQuery.go", |
494 | | - params={"inputType": "chem", "inputTerms": "Bisphenol A", "report": "genes_curated", "format": "json"}).json() |
495 | | - |
496 | | -# EPA CompTox Dashboard |
497 | | -comptox = requests.get("https://comptox.epa.gov/dashboard/api/search/chemical/equal/Bisphenol%20A").json() |
498 | | -``` |
| 278 | +### 11-24. Specialized Domains |
| 279 | + |
| 280 | +For these 14 additional domains, read [references/specialized-domains.md](references/specialized-domains.md) when you need the specific API pattern: |
| 281 | + |
| 282 | +| # | Domain | Key APIs/Tools | When to Read | |
| 283 | +|---|--------|---------------|--------------| |
| 284 | +| 11 | Proteomics | PRIDE, MassIVE, ProteomeXchange | Mass spec data download | |
| 285 | +| 12 | Metabolomics | MetaboLights, Metabolomics Workbench, HMDB | Metabolite/spectra data | |
| 286 | +| 13 | Microbiome | MGnify, GMREPO | Metagenome profiles | |
| 287 | +| 14 | Ecology | GBIF, iNaturalist, OBIS | Species occurrence data | |
| 288 | +| 15 | Model Organisms | FlyBase, WormBase, ZFIN, RGD | Gene data for non-human species | |
| 289 | +| 16 | Pathways & Networks | Reactome, STRING, BioGRID | Network/pathway export | |
| 290 | +| 17 | Ontologies | OLS, GO, HPO | Term hierarchy traversal | |
| 291 | +| 18 | Immunology | IEDB, VDJdb, ImmPort | Epitope/receptor data | |
| 292 | +| 19 | Drug & Pharma | PharmGKB, DGIdb, SIDER | Drug-gene interactions | |
| 293 | +| 20 | Imaging & Atlases | TCIA, HPA, Allen Brain Atlas | Imaging collections | |
| 294 | +| 21 | Protein Structure | RCSB PDB, AlphaFold | PDB/CIF file download | |
| 295 | +| 22 | Clinical Genomics | ClinVar, ClinGen, CIViC | Variant interpretation bulk | |
| 296 | +| 23 | Single-Cell | cellxgene, ARCHS4 | scRNA-seq data portals | |
| 297 | +| 24 | Toxicology | CTD, EPA CompTox | Chemical-gene-disease | |
499 | 298 |
|
500 | 299 | --- |
501 | 300 |
|
|
0 commit comments