Skip to content

Enable APK prefetching in apko_config #134

Description

@mattmoor

The basic idea here is to have a provider-level setting (off by default) that directs apko_config to prefetch the APKs to warm the go-apk cache. Something like:

provider "apko" {
  # Have apko_config prefetch the APKs for the relevant architectures.
  prefetch_apks = true
}

This would shift some of the latency from apko_build to apko_config, but we think that practically this will improve throughput because the dependency graph for a common image roughly looks like:

flowchart TB

LC(apko_config.latest) --> LB[apko_build.latest]
LCD(apko_config.latest-dev) --> LBD[apko_build.latest-dev]
LC --> LCD
Loading

Today the apko_config rules in the above diagram will be very quick, and the apko_build rules will be scheduled in parallel. Both of these builds overlap in the packages for the latest variant, but if the cache is empty they will likely both try to fill it at the same time and have a very low cache hit rate.

However, if we shift to prefetching the APKs during the config step, since latest-dev depends on latest, we will see latest prefetch all of the shared packages, and only then latest-dev will fetch the things that it additionally needs (likely all hits from other parallel dev builds in a mega-module execution). Then once each builds packages have been prefetched then their builds can be scheduled in parallel without one artificially waiting on the other.

cc @jonjohnsonjr

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions