Skip to content

Commit 7e744e5

Browse files
committed
refactor: restructure data-wrangling per skill-creator standards
- Move categories 11-24 to references/specialized-domains.md (progressive disclosure — loaded only when agent needs that specific domain) - SKILL.md now 397 lines (under 500-line recommendation), with compact index table pointing to references for specialized domains - Description rewritten to be "pushy" for better triggering — explicitly lists contexts like "parse a file format", "download raw data", etc. - Added evals/evals.json with 3 realistic test prompts - Also improved descriptions for dataset-discovery and epidemiological- analysis skills to be more trigger-friendly
1 parent fdb41d4 commit 7e744e5

5 files changed

Lines changed: 289 additions & 224 deletions

File tree

skills/tooluniverse-data-wrangling/SKILL.md

Lines changed: 21 additions & 222 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
name: tooluniverse-data-wrangling
3-
description: Universal data access reference for scientific research. Teaches how to download bulk data, parse domain-specific formats, paginate APIs, and handle authentication for ANY data source. Covers 10 API pattern categories for all ToolUniverse data sources plus major uncovered sources. Use when ToolUniverse tools return metadata but you need raw/bulk data, when data is in a format tools don't parse, when multi-step API workflows are needed, or when the source has no ToolUniverse tool.
3+
description: Universal data access reference for scientific research. Teaches how to download bulk data, parse any scientific file format (VCF, h5ad, mzML, PDB, FASTA, XPT, NIfTI, and 30+ more), paginate REST APIs, and handle authentication. Covers 24 domain API patterns across ALL life science data sources — genomics, proteomics, clinical, imaging, ecology, and more. Use this skill whenever you need to download raw data, parse a file format, access a bulk API, write a multi-step data retrieval workflow, or when a ToolUniverse tool returns metadata but you need the actual data. Also use when the data source has no ToolUniverse tool at all. Even if the user doesn't say "data wrangling" — if their task requires getting data from a scientific database or parsing a scientific file format, this is the skill to use.
44
---
55

66
# Data Wrangling: Universal Access Patterns
@@ -275,227 +275,26 @@ for f in record["files"]:
275275
content = requests.get(f["links"]["self"]).content # download each file
276276
```
277277

278-
### 11. Proteomics (PRIDE, MassIVE, PeptideAtlas, ProteomeXchange)
279-
Tools: `PRIDE_*`, `MassIVE_*`, `PeptideAtlas_*`
280-
```python
281-
# PRIDE project files — search then download raw/processed data
282-
project = requests.get("https://www.ebi.ac.uk/pride/ws/archive/v2/projects/PXD012345").json()
283-
files = requests.get(f"https://www.ebi.ac.uk/pride/ws/archive/v2/projects/PXD012345/files").json()
284-
for f in files:
285-
if f["fileName"].endswith(".mzML"): # mass spec data
286-
download_url = f["publicFileLocations"][0]["value"]
287-
288-
# ProteomeXchange: search across PRIDE + MassIVE + jPOST
289-
px = requests.get("https://proteomecentral.proteomexchange.org/cgi/GetDataset?ID=PXD012345&outputMode=JSON").json()
290-
```
291-
292-
### 12. Metabolomics (MetaboLights, Metabolomics Workbench, HMDB)
293-
Tools: `MetaboLights_*`, `MetabolomicsWorkbench_*`, `HMDB_*`
294-
```python
295-
# MetaboLights study download
296-
study_id = "MTBLS1234"
297-
study = requests.get(f"https://www.ebi.ac.uk/metabolights/ws/studies/{study_id}").json()
298-
# Download metabolite assignment file
299-
files = requests.get(f"https://www.ebi.ac.uk/metabolights/ws/studies/{study_id}/files").json()
300-
301-
# Metabolomics Workbench REST
302-
mw = requests.get("https://www.metabolomicsworkbench.org/rest/study/study_id/ST001234/analysis").json()
303-
304-
# HMDB metabolite data
305-
hmdb = requests.get("https://hmdb.ca/metabolites/HMDB0000001.xml").text # XML format
306-
```
307-
308-
### 13. Microbiome & Metagenomics (MGnify, GMREPO)
309-
Tools: `MGnify_*`, `GMREPO_*`
310-
```python
311-
# MGnify: search analyses, download taxonomy/function profiles
312-
analyses = requests.get("https://www.ebi.ac.uk/metagenomics/api/v1/analyses",
313-
params={"study_accession": "MGYS00001234", "page_size": 100}).json()
314-
# Download OTU/taxonomy TSV
315-
for a in analyses["data"]:
316-
tax_url = f"https://www.ebi.ac.uk/metagenomics/api/v1/analyses/{a['id']}/downloads"
317-
downloads = requests.get(tax_url).json()
318-
319-
# GMREPO: gut microbiome phenotype associations
320-
gmrepo = requests.get("https://gmrepo.humangut.info/api/getAssociatedSpeciesByMeshID",
321-
params={"meshID": "D003920"}).json() # diabetes
322-
```
323-
324-
### 14. Ecology & Biodiversity (GBIF, iNaturalist, OBIS)
325-
Tools: `GBIF_*`, `iNaturalist_*`, `OBIS_*`
326-
```python
327-
# GBIF occurrence data (millions of records, paginated)
328-
all_records = []
329-
offset = 0
330-
while True:
331-
resp = requests.get("https://api.gbif.org/v1/occurrence/search",
332-
params={"scientificName": "Panthera tigris", "limit": 300, "offset": offset}).json()
333-
all_records.extend(resp["results"])
334-
if resp["endOfRecords"]: break
335-
offset += 300
336-
337-
# iNaturalist observations
338-
obs = requests.get("https://api.inaturalist.org/v1/observations",
339-
params={"taxon_name": "Danaus plexippus", "per_page": 200, "geo": "true"}).json()
340-
```
341-
342-
### 15. Model Organisms (FlyBase, WormBase, ZFIN, RGD, MGI)
343-
Tools: `FlyBase_*`, `WormBase_*`, `ZFIN_*`, `RGD_*`
344-
```python
345-
# FlyBase gene data
346-
fb = requests.get("https://api.flybase.org/api/v1.0/gene/FBgn0000490").json() # dpp gene
347-
348-
# WormBase gene info
349-
wb = requests.get("https://wormbase.org/rest/widget/gene/WBGene00006763/overview",
350-
headers={"Accept": "application/json"}).json()
351-
352-
# ZFIN (zebrafish) gene expression
353-
zfin = requests.get("https://zfin.org/action/api/marker/ZDB-GENE-980526-166/expression").json()
354-
355-
# RGD (rat) — disease annotations for a gene
356-
rgd = requests.get("https://rest.rgd.mcw.edu/rgdws/genes/Tp53/9606").json() # human TP53
357-
```
358-
359-
### 16. Pathways & Networks (Reactome, STRING, BioGRID, WikiPathways)
360-
Tools: `Reactome_*`, `STRING_*`, `BioGRID_*`, `WikiPathways_*`
361-
```python
362-
# Reactome pathway participants
363-
pathway_id = "R-HSA-109582" # Hemostasis
364-
participants = requests.get(f"https://reactome.org/ContentService/data/participants/{pathway_id}").json()
365-
366-
# STRING protein-protein interactions (bulk)
367-
proteins = "9606.ENSP00000269305%0d9606.ENSP00000344818" # TP53, MDM2
368-
network = requests.get(f"https://string-db.org/api/json/network?identifiers={proteins}&species=9606").json()
369-
370-
# BioGRID interactions for a gene (tab-delimited bulk)
371-
url = "https://webservice.thebiogrid.org/interactions/?searchNames=true&geneList=BRCA1&taxId=9606&format=json&accesskey=YOUR_KEY"
372-
373-
# WikiPathways: download pathway as GPML or GMT
374-
gpml = requests.get("https://www.wikipathways.org/wikipathways/wpi/wpi.php?action=downloadFile&type=gpml&pwTitle=Pathway:WP254").text
375-
```
376-
377-
### 17. Ontologies (OLS, Gene Ontology, HPO, Disease Ontology)
378-
Tools: `ols_search_terms`, `ols_get_term_info`, `Gene_Ontology_*`
379-
```python
380-
# OLS term search + hierarchy traversal
381-
terms = requests.get("https://www.ebi.ac.uk/ols4/api/search",
382-
params={"q": "apoptosis", "ontology": "go", "rows": 20}).json()
383-
384-
# Get children of a GO term
385-
children = requests.get("https://www.ebi.ac.uk/ols4/api/ontologies/go/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FGO_0006915/children").json()
386-
387-
# HPO: phenotype-to-disease annotations
388-
hpo = requests.get("https://hpo.jax.org/api/hpo/term/HP:0001250/diseases").json()
389-
390-
# Gene Ontology annotations bulk (GAF format)
391-
# Download from: http://current.geneontology.org/annotations/goa_human.gaf.gz
392-
```
393-
394-
### 18. Immunology (IEDB, VDJdb, ImmPort)
395-
Tools: `IEDB_*`, `VDJdb_*`
396-
```python
397-
# IEDB epitope search
398-
epitopes = requests.get("https://query-api.iedb.org/epitope_search",
399-
params={"linear_sequence": "SIINFEKL", "limit": 50}).json()
400-
401-
# VDJdb: T-cell receptor specificity database
402-
vdjdb = pd.read_csv("https://raw.githubusercontent.com/antigenomics/vdjdb-db/master/latest-version.zip",
403-
compression="zip", sep="\t")
404-
405-
# ImmPort shared data (requires registration)
406-
# Search at: https://www.immport.org/shared/search
407-
```
408-
409-
### 19. Drug & Pharmacology (DrugBank, PharmGKB, SIDER, DGIdb)
410-
Tools: `PharmGKB_*`, `DGIdb_*`, `SIDER_*`, `DrugCentral_*`
411-
```python
412-
# DGIdb drug-gene interactions
413-
dgi = requests.get("https://dgidb.org/api/v2/interactions.json",
414-
params={"genes": "EGFR", "interaction_sources": "DrugBank,ChEMBL"}).json()
415-
416-
# PharmGKB clinical annotations for a gene
417-
pgkb = requests.get("https://api.pharmgkb.org/v1/data/clinicalAnnotation",
418-
params={"view": "base", "location.genes.symbol": "CYP2D6"}).json()
419-
420-
# SIDER side effect frequencies (bulk download)
421-
# Download from: http://sideeffects.embl.de/media/download/meddra_freq.tsv.gz
422-
```
423-
424-
### 20. Imaging & Atlases (TCIA, HPA, Allen Brain Atlas, BioImage Archive)
425-
Tools: `TCIA_*`, `HPA_*`, `AllenBrainAtlas_*`
426-
```python
427-
# TCIA: cancer imaging collections
428-
collections = requests.get("https://services.cancerimagingarchive.net/nbia-api/services/v1/getCollectionValues").json()
429-
# Get series for a patient
430-
series = requests.get("https://services.cancerimagingarchive.net/nbia-api/services/v1/getSeries",
431-
params={"Collection": "TCGA-BRCA", "PatientID": "TCGA-A1-A0SB"}).json()
432-
433-
# Human Protein Atlas: tissue expression
434-
hpa = requests.get("https://www.proteinatlas.org/api/search_download.php?search=TP53&format=json&columns=g,t,up").json()
435-
436-
# Allen Brain Atlas: gene expression in brain regions
437-
aba = requests.get("https://api.brain-map.org/api/v2/data/query.json?criteria=model::Gene,rma::criteria,[acronym$eq'BDNF']").json()
438-
```
439-
440-
### 21. Protein Structure Download (RCSB PDB, AlphaFold, CATH)
441-
Tools: `RCSB_*`, `AlphaFold_*`, `PDBe_*`, `CATH_*`
442-
```python
443-
# Download PDB/CIF structure files directly
444-
pdb_id = "1A2B"
445-
pdb_content = requests.get(f"https://files.rcsb.org/download/{pdb_id}.pdb").text
446-
cif_content = requests.get(f"https://files.rcsb.org/download/{pdb_id}.cif").text
447-
448-
# AlphaFold predicted structure by UniProt ID
449-
uniprot_id = "P04637" # TP53
450-
af_pdb = requests.get(f"https://alphafold.ebi.ac.uk/files/AF-{uniprot_id}-F1-model_v4.pdb").text
451-
af_cif = requests.get(f"https://alphafold.ebi.ac.uk/files/AF-{uniprot_id}-F1-model_v4.cif").text
452-
453-
# RCSB advanced search (GraphQL) for bulk queries
454-
query = {"query": {"type": "terminal", "service": "text", "parameters": {"attribute": "rcsb_entity_source_organism.taxonomy_lineage.name", "operator": "exact_match", "value": "Homo sapiens"}}, "return_type": "entry", "request_options": {"paginate": {"start": 0, "rows": 100}}}
455-
results = requests.post("https://search.rcsb.org/rcsbsearch/v2/query", json=query).json()
456-
```
457-
458-
### 22. Clinical Genomics & Variant Databases (ClinVar, ClinGen, CIViC, OncoKB)
459-
Tools: `ClinVar_*`, `ClinGen_*`, `CIViC_*`, `OncoKB_*`
460-
```python
461-
# ClinVar bulk download (variant_summary, ~200MB)
462-
# df = pd.read_csv("https://ftp.ncbi.nlm.nih.gov/pub/clinvar/tab_delimited/variant_summary.txt.gz", sep="\t")
463-
464-
# CIViC GraphQL API — all evidence items for a gene
465-
query = '{"query": "{ gene(entrezId: 673) { name variants { nodes { name evidenceItems { nodes { description evidenceLevel } } } } } }"}'
466-
civic = requests.post("https://civicdb.org/api/graphql", json={"query": query}).json()
467-
468-
# ClinGen allele registry
469-
allele = requests.get("https://reg.clinicalgenome.org/allele?hgvs=NM_000059.4:c.68_69del").json()
470-
```
471-
472-
### 23. Single-Cell Portals (cellxgene, ARCHS4, Cell Marker)
473-
Tools: `cellxgene_*`, `ARCHS4_*`
474-
```python
475-
# cellxgene Census — query human single-cell data at scale
476-
import cellxgene_census # requires cellxgene-census
477-
census = cellxgene_census.open_soma()
478-
adata = cellxgene_census.get_anndata(census, organism="Homo sapiens",
479-
obs_value_filter="tissue_general == 'lung' and disease == 'normal'")
480-
481-
# ARCHS4 gene expression (pre-computed from SRA)
482-
archs4 = requests.get("https://maayanlab.cloud/archs4/search/loadExpressionTSV.php",
483-
params={"search": "BRCA1", "species": "human"}).text
484-
485-
# Without cellxgene_census: download h5ad directly from cellxgene data portal
486-
# Browse collections at https://cellxgene.cziscience.com/collections
487-
```
488-
489-
### 24. Toxicology & Environmental (CTD, EPA, Tox21)
490-
Tools: `CTD_*`, `EPA_*`
491-
```python
492-
# CTD: chemical-gene-disease interactions
493-
ctd = requests.get("https://ctdbase.org/tools/batchQuery.go",
494-
params={"inputType": "chem", "inputTerms": "Bisphenol A", "report": "genes_curated", "format": "json"}).json()
495-
496-
# EPA CompTox Dashboard
497-
comptox = requests.get("https://comptox.epa.gov/dashboard/api/search/chemical/equal/Bisphenol%20A").json()
498-
```
278+
### 11-24. Specialized Domains
279+
280+
For these 14 additional domains, read [references/specialized-domains.md](references/specialized-domains.md) when you need the specific API pattern:
281+
282+
| # | Domain | Key APIs/Tools | When to Read |
283+
|---|--------|---------------|--------------|
284+
| 11 | Proteomics | PRIDE, MassIVE, ProteomeXchange | Mass spec data download |
285+
| 12 | Metabolomics | MetaboLights, Metabolomics Workbench, HMDB | Metabolite/spectra data |
286+
| 13 | Microbiome | MGnify, GMREPO | Metagenome profiles |
287+
| 14 | Ecology | GBIF, iNaturalist, OBIS | Species occurrence data |
288+
| 15 | Model Organisms | FlyBase, WormBase, ZFIN, RGD | Gene data for non-human species |
289+
| 16 | Pathways & Networks | Reactome, STRING, BioGRID | Network/pathway export |
290+
| 17 | Ontologies | OLS, GO, HPO | Term hierarchy traversal |
291+
| 18 | Immunology | IEDB, VDJdb, ImmPort | Epitope/receptor data |
292+
| 19 | Drug & Pharma | PharmGKB, DGIdb, SIDER | Drug-gene interactions |
293+
| 20 | Imaging & Atlases | TCIA, HPA, Allen Brain Atlas | Imaging collections |
294+
| 21 | Protein Structure | RCSB PDB, AlphaFold | PDB/CIF file download |
295+
| 22 | Clinical Genomics | ClinVar, ClinGen, CIViC | Variant interpretation bulk |
296+
| 23 | Single-Cell | cellxgene, ARCHS4 | scRNA-seq data portals |
297+
| 24 | Toxicology | CTD, EPA CompTox | Chemical-gene-disease |
499298

500299
---
501300

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
{
2+
"skill_name": "tooluniverse-data-wrangling",
3+
"evals": [
4+
{
5+
"id": 1,
6+
"prompt": "I need to download all TCGA-BRCA clinical data and run a Cox proportional hazards model on overall survival stratified by TP53 mutation status. Can you get the data and run the analysis?",
7+
"expected_output": "Agent should: (1) use GDC API to download clinical data with pagination, (2) parse JSON into DataFrame, (3) get TP53 mutation data, (4) merge and run Cox PH regression, (5) report hazard ratio with CI and p-value",
8+
"files": []
9+
},
10+
{
11+
"id": 2,
12+
"prompt": "Download the GWAS Catalog associations for type 2 diabetes, filter to genome-wide significant hits, and tell me which genes have the most associations.",
13+
"expected_output": "Agent should: (1) download full GWAS Catalog TSV, (2) filter by DISEASE/TRAIT containing diabetes AND p-value < 5e-8, (3) group by MAPPED_GENE, (4) report top genes with counts",
14+
"files": []
15+
},
16+
{
17+
"id": 3,
18+
"prompt": "I have a VCF file with variants from whole exome sequencing. Can you parse it and annotate each variant with ClinVar significance and predicted functional impact from VEP?",
19+
"expected_output": "Agent should: (1) parse VCF using the pandas pattern from Format Cookbook, (2) for each variant, query Ensembl VEP REST API, (3) optionally check ClinVar, (4) produce annotated DataFrame",
20+
"files": []
21+
}
22+
]
23+
}

0 commit comments

Comments
 (0)