Skip to content

Commit fad6901

Browse files
tien-tongtien-tong
committed
change get_data
1 parent 23cb8d0 commit fad6901

10 files changed

Lines changed: 120 additions & 44 deletions

File tree

‎_site/docs/datasets/HBN/index.html‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -352,7 +352,7 @@ <h3 id="about-the-dataset">About the dataset</h3>
352352
v25-0-0
353353

354354
<br />
355-
<span style="font-weight:normal;">(3736/3887 complete)</span>
355+
<span style="font-weight:normal;">(3736/3813 complete)</span>
356356

357357
</th>
358358
<th style="font-weight:normal">
@@ -393,7 +393,7 @@ <h3 id="about-the-dataset">About the dataset</h3>
393393
v0-1-2
394394

395395
<br />
396-
<span style="font-weight:normal;">(3736/3887 complete)</span>
396+
<span style="font-weight:normal;">(3736/3813 complete)</span>
397397

398398
</th>
399399
<th style="font-weight:normal">

‎_site/docs/datasets/NKI/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,7 @@ <h3 id="about-the-dataset">About the dataset</h3>
337337
<a href="https://github.com/ReproBrainChart/NKI_BIDS/blob/main/study-NKI_desc-T1_qc.json">, View T1w QC Variables</a>
338338

339339

340-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
340+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
341341

342342
</th>
343343
<th style="font-weight:normal"></th>

‎_site/docs/datasets/PNC/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -338,7 +338,7 @@ <h3 id="about-the-dataset">About the dataset</h3>
338338
<a href="https://github.com/ReproBrainChart/PNC_BIDS/blob/main/study-PNC_desc-T1_qc.json">, View T1w QC Variables</a>
339339

340340

341-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
341+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
342342

343343
</th>
344344
<th style="font-weight:normal"></th>

‎_site/docs/datasets/dCCNP/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -337,7 +337,7 @@ <h3 id="about-the-dataset">About the dataset</h3>
337337
<a href="https://github.com/ReproBrainChart/CCNP_BIDS/blob/main/study-CCNP_desc-T1_qc.json">, View T1w QC Variables</a>
338338

339339

340-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
340+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
341341

342342
</th>
343343
<th style="font-weight:normal"></th>

‎_site/docs/get_data.html‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -235,17 +235,20 @@ <h2 id="1-request-access-to-a-cubic-project">1. Request access to a CUBIC projec
235235

236236
<p><span style="color: #7556b7;">🌟 <strong>Check the <code class="language-plaintext highlighter-rouge">DUA</code> section on the dataset-specific page and obtain the appropriate DUA</strong> (Dataset-specific links listed <a href="/AI2D/docs/datasets">here</a>).</span></p>
237237

238-
<p>Then, follow these steps:</p>
238+
<p><strong>If <code class="language-plaintext highlighter-rouge">DUA</code> is <code class="language-plaintext highlighter-rouge">None</code></strong>, all CUBIC users have read-access to that <code class="language-plaintext highlighter-rouge">CUBIC Project</code>.</p>
239+
240+
<p><strong>If <code class="language-plaintext highlighter-rouge">DUA</code> is NOT <code class="language-plaintext highlighter-rouge">None</code></strong>, follow these steps:</p>
239241

240242
<ol>
241-
<li>Send Dr. Satterthwaite a request including the following:
243+
<li>Send Tien Tong (<span style="color: blue;"><strong>Tien.Tong@PennMedicine.upenn.edu</strong></span>) a request including the following:
242244
<ul>
243245
<li>The level of access, in this case, you will need <strong>read-only access</strong></li>
244246
<li>The name of the user[s] to be given access (full name or login name). If you plan to store and analyze data in a CUBIC project, you should use <strong>your project username</strong>, not personal CUBIC username.</li>
245247
<li>The full path to the project you need access to, <code class="language-plaintext highlighter-rouge">/cbica/projects/&lt;project_name&gt;</code>.</li>
248+
<li>Any DUA requirements.</li>
246249
</ul>
247250
</li>
248-
<li>Submit the DUA (if required) and Dr. Satterthwaite’s approval (PDF) — specifying access level, requesting users/project users, and the full project path — to <span style="color: blue;"><strong>help@cbica.upenn.edu</strong></span>.</li>
251+
<li>Tien will forward your request to the CUBIC admins and notify you when access is granted.</li>
249252
</ol>
250253

251254
<p><span style="color: #7556b7;">🌟 <strong>Once you have access to the project, check the <code class="language-plaintext highlighter-rouge">BABS</code> section on the dataset-specific page to determine whether the dataset was processed with <a href="/AI2D/docs/imaging/image_babs">BABS</a></strong> (Dataset-specific links listed <a href="/AI2D/docs/datasets">here</a>).</span></p>

‎_site/docs/imaging/image_xcpd/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -429,7 +429,7 @@ <h2 id="quality-control">Quality Control</h2>
429429
</details>
430430

431431
<p><br />
432-
We recommend using a <strong>median Framewise Displacement (FD) threshold of <code class="language-plaintext highlighter-rouge">&gt; 0.2</code> for run-level exclusion</strong>, as implemented in <a href="https://doi.org/10.1101/2025.02.24.639850">the Reproducible Brain Chart study</a>. Check the <code class="language-plaintext highlighter-rouge">XCP-D</code> section on the dataset-specific page (dataset-specific links listed <a href="/AI2D/docs/datasets">here</a>) for the path to our recommended <strong>XCP-D fMRI QC file</strong>.</p>
432+
We recommend using a <strong>median Framewise Displacement (FD) threshold of <code class="language-plaintext highlighter-rouge">&gt; 0.2</code> for run-level exclusion</strong>, as implemented in <a href="https://doi.org/10.1016/j.neuron.2025.08.026">the Reproducible Brain Chart study</a>. Check the <code class="language-plaintext highlighter-rouge">XCP-D</code> section on the dataset-specific page (dataset-specific links listed <a href="/AI2D/docs/datasets">here</a>) for the path to our recommended <strong>XCP-D fMRI QC file</strong>.</p>
433433

434434
<div class="alert alert-warning" role="alert">
435435
<strong>⚠️ WARNING:</strong> Please note that our general recommendation is to use a run-level threshold of <strong>median FD &gt; 0.2</strong>. However, depending on the characteristics of a given dataset, higher or lower thresholds may be more appropriate. To determine the most suitable threshold for a specific dataset (and assess the percentage of retained data), we encourage you to review the <strong>XCP-D fMRI QC file</strong> listed in the <code>XCP-D</code> section of the dataset-specific page. Dataset-specific links are listed <a href="/AI2D/docs/datasets">here</a>.

‎_site/docs/team/index.html‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,7 +210,7 @@ <h2 style="text-align: center;">AI2D Data Infrastructure Leadership</h2>
210210
</a>
211211
<div class="media-body" style="word-wrap: break-word; overflow-wrap: break-word;">
212212
<p style="margin: 0px; font-size: 20px; text-align: left; font-weight: bold;"><a href="https://pennlinc.io" class="off">Ted Satterthwaite</a></p>
213-
<p style="margin: 0px; font-size: 16px; text-align: left;">McLure II Professor of Psychiatry & Behavioral Research</p>
213+
<p style="margin: 0px; font-size: 16px; text-align: left;">PennLINC Director<br>McLure II Professor of Psychiatry & Behavioral Research</p>
214214
<p style="margin: 0px; font-size: 16px; text-align: left;">University of Pennsylvania</p>
215215
</div>
216216
</div>

‎_site/feed.xml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.3">Jekyll</generator><link href="http://localhost:4000/AI2D/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/AI2D/" rel="alternate" type="text/html" /><updated>2025-09-22T12:08:55-04:00</updated><id>http://localhost:4000/AI2D/feed.xml</id><title type="html">AI2D | PennLINC | Shared Data Resources</title><subtitle>AI2D Shared Data Resources</subtitle><author><name>{&quot;links&quot;=&gt;[{&quot;label&quot;=&gt;&quot;Email&quot;, &quot;icon&quot;=&gt;&quot;fas fa-fw fa-envelope-square&quot;}, {&quot;label&quot;=&gt;&quot;Website&quot;, &quot;icon&quot;=&gt;&quot;fas fa-fw fa-link&quot;}, {&quot;label&quot;=&gt;&quot;Twitter&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-twitter-square&quot;}, {&quot;label&quot;=&gt;&quot;Facebook&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-facebook-square&quot;}, {&quot;label&quot;=&gt;&quot;GitHub&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-github&quot;}, {&quot;label&quot;=&gt;&quot;Instagram&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-instagram&quot;}]}</name></author><entry><title type="html">Midnight Scan Club</title><link href="http://localhost:4000/AI2D/docs/datasets/midnight_scan_club/" rel="alternate" type="text/html" title="Midnight Scan Club" /><published>2025-08-16T00:00:00-04:00</published><updated>2025-08-16T00:00:00-04:00</updated><id>http://localhost:4000/AI2D/docs/datasets/midnight_scan_club</id><content type="html" xml:base="http://localhost:4000/AI2D/docs/datasets/midnight_scan_club/"><![CDATA[<div style="text-align: left;">
1+
<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.9.3">Jekyll</generator><link href="http://localhost:4000/AI2D/feed.xml" rel="self" type="application/atom+xml" /><link href="http://localhost:4000/AI2D/" rel="alternate" type="text/html" /><updated>2025-09-29T09:10:35-04:00</updated><id>http://localhost:4000/AI2D/feed.xml</id><title type="html">AI2D | PennLINC | Shared Data Resources</title><subtitle>AI2D Shared Data Resources</subtitle><author><name>{&quot;links&quot;=&gt;[{&quot;label&quot;=&gt;&quot;Email&quot;, &quot;icon&quot;=&gt;&quot;fas fa-fw fa-envelope-square&quot;}, {&quot;label&quot;=&gt;&quot;Website&quot;, &quot;icon&quot;=&gt;&quot;fas fa-fw fa-link&quot;}, {&quot;label&quot;=&gt;&quot;Twitter&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-twitter-square&quot;}, {&quot;label&quot;=&gt;&quot;Facebook&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-facebook-square&quot;}, {&quot;label&quot;=&gt;&quot;GitHub&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-github&quot;}, {&quot;label&quot;=&gt;&quot;Instagram&quot;, &quot;icon&quot;=&gt;&quot;fab fa-fw fa-instagram&quot;}]}</name></author><entry><title type="html">Midnight Scan Club</title><link href="http://localhost:4000/AI2D/docs/datasets/midnight_scan_club/" rel="alternate" type="text/html" title="Midnight Scan Club" /><published>2025-08-16T00:00:00-04:00</published><updated>2025-08-16T00:00:00-04:00</updated><id>http://localhost:4000/AI2D/docs/datasets/midnight_scan_club</id><content type="html" xml:base="http://localhost:4000/AI2D/docs/datasets/midnight_scan_club/"><![CDATA[<div style="text-align: left;">
22
<img src="/AI2D/assets/images/logos/WashU.png" style="width: auto; height: 10vw;" />
33
</div>
44

@@ -1617,7 +1617,7 @@ Taylor Salo and M. Dylan Tisdall and Lia Brodrick and Adam Czernuszenko and Davi
16171617
v25-0-0
16181618

16191619
<br />
1620-
<span style="font-weight:normal;">(3736/3887 complete)</span>
1620+
<span style="font-weight:normal;">(3736/3813 complete)</span>
16211621

16221622
</th>
16231623
<th style="font-weight:normal">
@@ -1658,7 +1658,7 @@ Taylor Salo and M. Dylan Tisdall and Lia Brodrick and Adam Czernuszenko and Davi
16581658
v0-1-2
16591659

16601660
<br />
1661-
<span style="font-weight:normal;">(3736/3887 complete)</span>
1661+
<span style="font-weight:normal;">(3736/3813 complete)</span>
16621662

16631663
</th>
16641664
<th style="font-weight:normal">
@@ -2100,7 +2100,7 @@ Alexander LM, Escalera J, Ai L, Andreotti C, Febre K, Mangone A, Vega-Potler N,
21002100
<a href="https://github.com/ReproBrainChart/NKI_BIDS/blob/main/study-NKI_desc-T1_qc.json">, View T1w QC Variables</a>
21012101

21022102

2103-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
2103+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
21042104

21052105
</th>
21062106
<th style="font-weight:normal"></th>
@@ -2615,7 +2615,7 @@ Tobe RH, MacKay-Brandt A, Lim R, Kramer M, Breland MM, Tu L, Tian Y, Trautman KD
26152615
<a href="https://github.com/ReproBrainChart/PNC_BIDS/blob/main/study-PNC_desc-T1_qc.json">, View T1w QC Variables</a>
26162616

26172617

2618-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
2618+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
26192619

26202620
</th>
26212621
<th style="font-weight:normal"></th>
@@ -3136,7 +3136,7 @@ RC2MH089983, RC2MH089924, <a href="https://reporter.nih.gov/search/e9_-eAc68EOR6
31363136
<a href="https://github.com/ReproBrainChart/CCNP_BIDS/blob/main/study-CCNP_desc-T1_qc.json">, View T1w QC Variables</a>
31373137

31383138

3139-
<a href="https://doi.org/10.1101/2025.02.24.639850">, View RBC Paper for T1w QC Method</a>
3139+
<a href="https://doi.org/10.1016/j.neuron.2025.08.026">, View RBC Paper for T1w QC Method</a>
31403140

31413141
</th>
31423142
<th style="font-weight:normal"></th>

‎_site/scripts/unzip_files.sh‎

Lines changed: 95 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -2,34 +2,53 @@
22

33
# Function to display help message
44
show_help() {
5-
echo "Usage: $0 <input_dir> <output_dir> [sub_list|file_pattern] [file_pattern]"
6-
echo ""
7-
echo "Extract all .zip files from the specified input directory to the output directory using 7z."
8-
echo "Options:"
9-
echo " sub_list Path to a text file with subject IDs (one per line)."
10-
echo " file_pattern A pattern to extract specific files from each archive (using 7z's -r flag)."
11-
echo ""
12-
echo "Examples:"
13-
echo " # Extract all zip files and overwrite existing files:"
14-
echo " $0 /path/to/input /path/to/output"
15-
echo ""
16-
echo " # Extract only files matching subjects in the list and overwrite existing files:"
17-
echo " $0 /path/to/input /path/to/output subject_list.txt"
18-
echo ""
19-
echo " # Extract only files matching the given pattern and overwrite existing files:"
20-
echo " $0 /path/to/input /path/to/output \"*/sub-*/anat/*_space-MNI152NLin6Asym_res-2_desc-preproc_T1w.nii.gz\""
21-
echo ""
22-
echo " # Use both subject list and file pattern (overwrite existing files):"
23-
echo " $0 /path/to/input /path/to/output subject_list.txt \"*/sub-*/anat/*_space-MNI152NLin6Asym_res-2_desc-preproc_T1w.nii.gz\""
24-
exit 0
5+
cat <<'EOF'
6+
Usage: unzip_files.sh [--add_ses_freesurfer] <input_dir> <output_dir> [sub_list|file_pattern] [file_pattern]
7+
8+
Extract all .zip files from the specified input directory to the output directory using 7z.
9+
Options:
10+
--add_ses_freesurfer After extracting, rename sourcedata/freesurfer/sub-<id> to
11+
sourcedata/freesurfer/sub-<id>_ses-<ses> using the ses from the zip filename.
12+
sub_list Path to a text file with subject IDs (one per line).
13+
file_pattern A pattern to extract specific files from each archive (using 7z's -r flag).
14+
15+
Important notes:
16+
* When --add_ses_freesurfer is enabled, your subject_list must include both subject and session IDs.
17+
For example:
18+
sub-1_ses-1
19+
sub-1_ses-2
20+
sub-1_ses-3
21+
22+
Examples:
23+
unzip_files.sh /path/to/input /path/to/output
24+
unzip_files.sh --add_ses_freesurfer /path/to/input /path/to/output
25+
unzip_files.sh /path/to/input /path/to/output subject_list.txt
26+
unzip_files.sh /path/to/input /path/to/output "*/sub-*/anat/*_space-MNI152NLin6Asym_res-2_desc-preproc_T1w.nii.gz"
27+
unzip_files.sh /path/to/input /path/to/output subject_list.txt "*/sub-*/anat/*_space-MNI152NLin6Asym_res-2_desc-preproc_T1w.nii.gz"
28+
EOF
29+
exit 0
2530
}
2631

2732
# Check if help is requested
2833
if [ "$1" == "-h" ] || [ "$1" == "--help" ]; then
2934
show_help
3035
fi
3136

32-
# Check if the correct number of arguments is provided (minimum 2, maximum 4)
37+
# Optional flag parsing
38+
add_ses_freesurfer=false
39+
while true; do
40+
case "$1" in
41+
--add_ses_freesurfer)
42+
add_ses_freesurfer=true
43+
shift
44+
;;
45+
*)
46+
break
47+
;;
48+
esac
49+
done
50+
51+
# Check if the correct number of arguments is provided (minimum 2, maximum 4 after flags)
3352
if [ "$#" -lt 2 ] || [ "$#" -gt 4 ]; then
3453
echo "Error: Invalid number of arguments."
3554
show_help
@@ -65,18 +84,70 @@ fi
6584
# Ensure output directory exists
6685
mkdir -p "$output_dir"
6786

87+
# Post-extraction fix for freesurfer directory naming
88+
post_extract_fix() {
89+
local zip_file="$1"
90+
local base
91+
base="$(basename "$zip_file")"
92+
93+
# Extract sub-<id> and ses-<id> from the zip filename
94+
local subj
95+
local ses
96+
subj="$(echo "$base" | sed -n 's/.*\(sub-[0-9A-Za-z]\+\).*/\1/p')"
97+
ses="$(echo "$base" | sed -n 's/.*\(ses-[0-9A-Za-z]\+\).*/\1/p')"
98+
99+
if [ -z "$subj" ] || [ -z "$ses" ]; then
100+
echo "Warning: Could not parse subject or session from $base. Skipping freesurfer rename."
101+
return
102+
fi
103+
104+
# Locate freesurfer subject dir anywhere under output_dir (handles extra top-level folder)
105+
local src_dir
106+
src_dir="$(find "$output_dir" -type d -path "*/sourcedata/freesurfer/$subj" -print -quit)"
107+
108+
if [ -z "$src_dir" ]; then
109+
echo "Note: No freesurfer dir found for $subj under $output_dir"
110+
return
111+
fi
112+
113+
# Build destination alongside the found source dir
114+
local dst_dir="${src_dir%/$subj}/${subj}_${ses}"
115+
116+
if [ -d "$dst_dir" ]; then
117+
echo "Note: Destination $dst_dir already exists. Merging contents from $src_dir."
118+
(
119+
shopt -s dotglob
120+
mv "$src_dir"/* "$dst_dir"/ 2>/dev/null || true
121+
)
122+
rmdir "$src_dir" 2>/dev/null || true
123+
else
124+
mv "$src_dir" "$dst_dir"
125+
fi
126+
echo "Renamed freesurfer directory to ${subj}_${ses}."
127+
}
128+
68129
# Define extraction command based on whether a file pattern is provided.
69130
if [ -n "$file_pattern" ]; then
70131
extract_cmd() {
71132
local zip_file="$1"
72133
echo "Extracting (with file pattern): $zip_file"
73-
7z e "$zip_file" -aoa -o"$output_dir" -r "$file_pattern"
134+
if [ "$add_ses_freesurfer" = true ]; then
135+
# Preserve paths so freesurfer tree exists for rename
136+
7z x "$zip_file" -aoa -o"$output_dir" -r "$file_pattern"
137+
post_extract_fix "$zip_file"
138+
else
139+
# Default: flatten when not renaming freesurfer
140+
7z e "$zip_file" -aoa -o"$output_dir" -r "$file_pattern"
141+
fi
74142
}
75143
else
76144
extract_cmd() {
77145
local zip_file="$1"
78146
echo "Extracting: $zip_file"
79147
7z x "$zip_file" -aoa -o"$output_dir"
148+
if [ "$add_ses_freesurfer" = true ]; then
149+
post_extract_fix "$zip_file"
150+
fi
80151
}
81152
fi
82153

@@ -86,9 +157,9 @@ if [ -n "$sub_list" ]; then
86157
echo "Error: Subject list file '$sub_list' not found."
87158
exit 1
88159
fi
89-
while read -r subid; do
90-
# Skip empty lines
160+
while IFS= read -r subid || [ -n "$subid" ]; do
91161
[ -z "$subid" ] && continue
162+
subid=${subid%$'\r'} # strip carriage return if present
92163
found=false
93164
for zip_file in "$input_dir"/*${subid}*.zip; do
94165
if [ -f "$zip_file" ]; then

0 commit comments

Comments
 (0)