Consider including protoc source in release artifacts

[mnm364]

Since #1876 (released in 1.33.0), building swift-protobuf also builds protoc. In development, git submodules are used to get the protoc source; however, the release artifacts (at https://github.com/apple/swift-protobuf/releases) do not include this protoc source. This means that swift-protobuf cannot actually be built from the release artifacts. For example, the following fails:

curl -L https://github.com/apple/swift-protobuf/archive/refs/tags/1.33.3.tar.gz | tar -xz
cd swift-protobuf-1.33.3
swift build -c release

The alternative is to use git-tooling, such as

git clone https://github.com/apple/swift-protobuf.git
git checkout tags/[tag_name]
git submodule init
git submodule update
swift build -c release

This makes it difficult to utilize the source releases for development tooling or CI at all as the binary cannot be built from them! Alternatively or in addition, releasing binary artifacts as in #1197 would also help alleviate this pain point.

Could we consider including the protoc source in the release artifact?

Thanks so much for your time!

cc @FranzBusch

[thomasvl]

I think you are talking about the archives that github makes for ever tag and projects have no control over. I believe this has been talked about in a bunch of places (like github discussion), but I don't think repo owners actually have any controls over how these are made.

[mnm364]

Mmm I was not aware of the underlying process; that makes sense but is quite frustrating!

In that discussion there does seem to be a workaround - albeit manual - proposed:

"manually create a complete .zip yourself on your own computer, then 'Edit' your release, and upload your complete .zip back into the release."

I still think that could be worth consideration if that works.

Without doing so, I would actually say that there is risk that older versions may become broken over time!

Also, as the time goes more and more of github releases in the current form will become broken. It's naive to assume that the referenced dependencies somehow will always be around.

And, not to be repetitive, but tangentially, releasing binary artifacts may also benefit these workflows: #1197.

Thanks for your time and quick response!

[thomasvl]

[...]
Without doing so, I would actually say that there is risk that older versions may become broken over time!

How are you thinking they would become broken?

They could also break if/when the future version of Swift don't support the 5/6 language versions. We already have old releases that likely don't compile with Swift 6. (just like any old release of any project might not work with the newer compilers/toolchains)

Also, as the time goes more and more of github releases in the current form will become broken. It's naive to assume that the referenced dependencies somehow will always be around.

I assume you are referring to the comment in the referenced thread? I'm guessing the mean the git submodules? I guess that's possible, but then again any project (or its posted releases) could also go away, no? In the case of this project, if upstream protobuf goes away, I'm not so sure what the use of this project would be. 😄

[thomasvl]

Just to clarify, I'm not against this, just was curious to understand the concerns vs. other things.

Anyway, like #1197, it's up folks on the Apple side since it probably means workflow/etc. setup to help auto manager this when releases are made.

[Lukasa]

I think it would be reasonable to consider an enhancement in the form of a github action that can generate a complete tarball for any tag and attach it to the release. I suspect this can be done entirely automatically. @rnro does that sound right to you?

[FranzBusch]

I wasn't aware that those auto-generated releases don't contain the contents of submodules. That is quite disappointing. It does sound to me like a Swift package registry could help us here though as an alternative to pull down a copy of the sources.

[rnro]

@Lukasa I haven't used those release asset APIs personally but a quick bit of research suggests it should be doable with the right perms.

[mnm364]

Hey all, just wondering if this ever got done or was planned for future releases? Thanks so much!

[mnm364]

I did some more digging and don't think there is a reasonable way forward here for my request. It looks like the Github source-code in a release cannot be overridden. I presume Github is using git archive internally, which does not support including submodule source in the archive. Furthermore the size of the source with the submodule is 14MB versus 2.3MB without, which is non-trivial.

For now, I have forked this repo and made my own release with the source code with the submodule code: https://github.com/WonderInventions/swift-protobuf/releases/tag/1.36.1. This works for my needs for now.

However, I still do believe that releasing binary targets would help swift-protobuf more seamlessly used with developer and CI/CD tooling. #1197

Appreciate all the discussion already.