You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The basic idea here is to have a provider-level setting (off by default) that directs apko_config to prefetch the APKs to warm the go-apk cache. Something like:
provider"apko" {
# Have apko_config prefetch the APKs for the relevant architectures.prefetch_apks=true
}
This would shift some of the latency from apko_build to apko_config, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:
Today the apko_config rules in the above diagram will be very quick, and the apko_build rules will be scheduled in parallel. Both of these builds overlap in the packages for the latest variant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.
However, if we shift to prefetching the APKs during the config step, since latest-dev depends on latest, we will see latest prefetch all of the shared packages, and only then latest-dev will fetch the things that it additionally needs (likely all hits from other parallel dev builds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.
The basic idea here is to have a provider-level setting (off by default) that directs
apko_configto prefetch the APKs to warm thego-apkcache. Something like:This would shift some of the latency from
apko_buildtoapko_config, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:Today the
apko_configrules in the above diagram will be very quick, and theapko_buildrules will be scheduled in parallel. Both of these builds overlap in the packages for thelatestvariant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.However, if we shift to prefetching the APKs during the config step, since
latest-devdepends onlatest, we will seelatestprefetch all of the shared packages, and only thenlatest-devwill fetch the things that it additionally needs (likely all hits from other paralleldevbuilds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.cc @jonjohnsonjr