feat: download cache for assets and asset eodag:download_link creation#1929
Draft
anesson-cs wants to merge 12 commits intodevelopfrom
Draft
feat: download cache for assets and asset eodag:download_link creation#1929anesson-cs wants to merge 12 commits intodevelopfrom
anesson-cs wants to merge 12 commits intodevelopfrom
Conversation
anesson-cs
force-pushed
the
downloadlink-property-to-asset
branch
from
0e1d1db to
9bb0439
Compare
Contributor
Test Results 2 files - 2 2 suites - 2 49s ⏱️ - 2m 26s For more details on these errors, see this check. Results for commit ae07435. ± Comparison against base commit da1039b. This pull request removes 622 and adds 49 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
Collaborator
|
Code Coverage (Ubuntu)DetailsDiff against developResults for commit: ae07435 Minimum allowed coverage is ♻️ This comment has been updated with latest results |
it works in library mode for 'get' download method with 'HTTPDownload' plugin with provider that can contains single assets and the 'downloadLink' asset at the same time remove 'ignore_assets' from 'HTTPDownload' plugin configuration since it is naturally handled now
… matching property
…in HTTP download plugin
fix type hints
anesson-cs
force-pushed
the
downloadlink-property-to-asset
branch
from
7792189 to
ae07435
Compare
anesson-cs
changed the title
anesson-cs
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1619
The purpose of this PR is to handle individual asset downloads. This leads to these sub-goals:
add cache to assets downloads in their own record filename
Asset.locationandAsset.remote_locationattributesEOProduct.locationandEOProduct.remote_locationattributes_prepare_download(), useassetinstead ofproductin parametersonly use assets to download a product
download_linkfor providers which do not have any asset or contain a "true" download link. Use search plugin attributeassets_mappingto do it.archiveto assetdownload_link, and that will be a way to know if an asset is a full producteodag:download_linkAsset.download()is called inEOProduct.download()instead of the opposite. Then, in methodEOProduct.download():assetis set, download matching asset(s) or only assetdownload_linkif exists among these matching assetsassetis not set, download assetdownload_linkif exists or all other assetsassetinstead ofproductin parameters of methods_prepare_download()and_finalize()download_linkare kept, with pluginAwsDownload, setbuild_safetoTrue_finalize(asset, flatten_top_dirs=False)first, then use a new method_finalize_product(product)which handleflatten_top_dirsandbuild_safe. Otherwise, use only_finalize(asset)Asset.downloaderandAsset.downloader_authand methodAsset.register_downloader(), use the ones of the product to which the asset belongs toHTTPDownload._download_assets()andHTTPDownload.download()intoHTTPDownload.download()which implies:HTTPDownload._stream_download_assets()andHTTPDownload._stream_download()intoHTTPDownload._stream_download()outside of these goals:
EOProduct, update attributeassetsofEOProductwithget_assets_from_mapping()and remove the ones which have the value ofhrefset toNOT_AVAILABLEdescriptionto assetdownload_linkdownload_linkmay be set directly in the code if they are common to all of providers to reduceassets_mappingsize inproviders.yml(e.g.role=["data", "archive"])What must be corrected:
_prepare_download(), do not return a dictionary of record filenames with assets name as keysassetsfromEOProductproperties to its attributes at the initialization of anEOProductdownload_linkhas been removed if it existed after the search query, undo it as it will not exist anymoreWhat is missing?
Asset.locationandAsset.remote_locationhave been added to the class, but they are not applied in the codeIssues to create:
overviewand for thumbnails calledthumbnailAsset.downloaderandAsset.downloader_authand methodAsset.register_downloader()when there will be different authentication and download plugins between assets of a same productusgs) with a different name thandownload_linkto not create confusion. With rolearchive, it should be possible.EOProduct.assets