add flag to make build reproducible (in particular deterministic, non-hardcoded picking of UUIDs)

[wucke13]

I'd like to generate images which are reproducible (running the build twice generates the same image, down to the bit). I guess that it is utopic to make this work for all image formats and all filesystems right away. But, for some of image formats (such as fat32) this can be done. The common sources of in-determinism that I'm aware of are:

• Dates/Timestamps
  • Create/modify/access times of files
  • Creation time of filesystem itself
  • Creation time of partition tables? (Does GPT store it somewhere?)
• IDs/UUIDs
  • Most filesystems have a ID/UUID
  • In GPT for example, each partition has UUID as well

For my limited use-cases (mostly fat partitions on GPT disks), there does not seem to be an issue with the dates & timestamps. The UUIDs can be hardcoded. However, for the UUIDs it would be nice to have a collision free, non-hardcoded mechanism. I believe that UUIDv5 (which deterministically computes a UUID from a namespace + a name) would be a good fit here. A naive approach to gather namespace and name could be:

• namespace = sha256 of the entire genimage input config file
• name = partition start address + sha256 of the partition's image

This choice would ensure:

• Two differing genimage configs are guaranteed collision free
• Two identical genimage configs are guaranteed to be reproducible
• Two identical images in the same config get different UUIDs

Depending on the filesystem, the FS itself will also carry a UUID. This again must be generated from a configuration of the genimage config and the filesystem contents.

[nrclark]

@michaelolbrich I'm thinking of implementing a reduced version of this:

• Swap out stdlib's random() for an LFSR random number generator.
• Add an optional config option called randomseed that initalizes the RNG to a known state.
• Still uses a timestamp-based seed if randomseed is empty or unspecified.

This design doesn't address anything filesystem related, but it does provide reproducible generated UUIDs for users that want it.

Would you be open to a PR like that?

[nrclark]

Opened this PR: #314

[michaelolbrich]

Having a global flag make sense.

A lot of this depends on the various mkfs tools. I think some of them already use SOURCE_DATE_EPOCH if set but I'm not sure. Either way, that's always a good standard to use here.
@nrclark I think we should use it to seed the RNG if it is set.

For filesystem contents, I think a separate option makes sense. In my experience, if you want full reproducibility, then that includes the content of the files as well, and if your build chain before calling genimage is already reproducible, then it's likely that the files timestamps are reproducible as well.

For UUIDs: Maybe we can hash SOURCE_DATE_EPOCH with a predictable unique seed for the usage. e.g. include image name, partition name, etc.

[nrclark]

@wucke13, support for the randomseed option just got merged (thanks @michaelolbrich!). It doesn't address filesystem creation, but it does give you the ability to get reproducible UUIDs in your GPT table. To use it, pass whatever string/int you want to the randomseed option.