Skip to content

fix: forward delete_old in Yahoo data updates - #2335

Open
Rishabh Raj (aerraj) wants to merge 1 commit into
microsoft:mainfrom
aerraj:fix/yahoo-update-delete-old
Open

fix: forward delete_old in Yahoo data updates#2335
Rishabh Raj (aerraj) wants to merge 1 commit into
microsoft:mainfrom
aerraj:fix/yahoo-update-delete-old

Conversation

@aerraj

Copy link
Copy Markdown

Description

Add delete_old: bool = True to the end of Run.update_data_to_bin and forward it to GetData.qlib_data when the base dataset needs to be downloaded. Document the option in the method and Yahoo collector README.

The default remains unchanged. --delete_old False skips deletion of existing data directories before archive extraction; it does not prevent archive entries from overwriting matching files. Complete local datasets still skip the base download.

Motivation and Context

Fixes #1121.

Current main already forwards exists_skip, but delete_old is still missing from the update method's signature and download call. This addresses the remaining part of the report without changing the existing download condition or update pipeline.

How Has This Been Tested?

  • Added 12 offline cases in tests/test_yahoo_collector.py: default/explicit deletion settings, both exists_skip values, CLI boolean parsing through Fire, and skipping the base download when data already exists. Downloads, normalization, binary writes, and index collection are mocked; calendar input uses a temporary directory.
  • Before the fix, the initial 10 regression cases failed on upstream code (missing argument forwarding or an unexpected delete_old keyword).
  • python -m pytest tests/test_yahoo_collector.py tests/misc/test_index_data.py tests/misc/test_sepdf.py -q: 20 passed, with one existing pandas SettingWithCopyWarning in test_sepdf.py.
  • cd tests && python -m pytest test_yahoo_collector.py -q: 12 passed, matching the CI working directory.
  • python -m black --check --diff -l 120 scripts/data_collector/yahoo/collector.py tests/test_yahoo_collector.py: passed.
  • Flake8 on the new test file and git diff --check: passed. The collector has four existing F841 warnings on lines 116, 146, 175, and 185; compared against upstream and confirmed unchanged.
  • Checked that python scripts/data_collector/yahoo/collector.py update_data_to_bin --help lists --delete_old with default True.
  • Full pipeline tests were not run; they require external market datasets and model training. No live Yahoo download was performed.

Environment: macOS arm64, Python 3.12.14, NumPy 1.26.4, pandas 2.3.3, editable source install.

Types of changes

  • Fix bugs
  • Add new feature
  • Update documentation

@aerraj

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

exists_skip and delete_old ignored

1 participant