Improve the cloning progress reporting

[vmarkovtsev]

Currently, git.CloneOptions.Progress only reports the server's messages, as clearly stated in the documentation. However, server-side usually takes 1% of the whole cloning time, and the rest is receiving the objects and decompressing/resolving deltas. This renders Progress nearly useless: 99% of the time it shows Compressing objects: 100% (3000/3000), done. or similar.

I propose to add the local messages to Progress - at least the ratio of fetched objects and anything else which is relevant.

[mcuadros]

The Progress is a server side communication system, the information written there is reported as an implementation detail of it.

This feature IMHO felts outside of the scope of a library. This can be done wrapping object storer. (In order to retrieve the count)

[orirawlings]

I think this is a interesting observation and something that I noticed as well while looking into creating a patch for #528.

Progress is strictly used for data received by the client from the server via the sideband channel (of the pack protocol). The type of the Progress fields in the various option structs does suggest this (i.e. sideband.Progress). In some sense, it might be an equally confusing API if non-sideband data was written to a sideband.Progress.

In other situations the sideband will carry additional data. For example, it might include the output of custom git hooks which are running on the server side during push or fetch.

That being said, I can see where having some other progress reporting of client side processing is useful.

[vmarkovtsev]

@mcuadros I will be grateful if you show me how to wrap the Storer object. Feel free to close this afterwards.

[strib]

I'm interested in more go-git (non-sideband) status information as well, not just for clone/fetch but also for push. I've written up a summary of what I think is useful here: #570 (comment).

A cross-post of the stages I think are useful for clone/fetch (from the perspective of someone using the "file" protocol to fetch from a local git repo):

• Fetch: decoding objects being transferred from the remote, and writing them to the Storer:
  • Number of objects decoded so far / total number of objects expected.
  • The numerator could maybe be figured out from a Storer implementation, through the SetEncodedObject method, though it would need to be aware about what stage it was in.
• Index: building the index has three phases, which each object being touched in each phase. So for purposes of tracking progress, this needs to be broken down into three separate stages:
  • IndexHash: Writing the hash of each object to the packfile.
    • Number of hashes written so far / total number of hashes expected.
  • IndexCRC: Writing the CRC of each object to the packfile.
    • Number of CRCs written so far / total number of CRCs expected.
  • IndexOffset: Writing the offset of each object to the packfile.
    • Number of offsets written so far / total number of offsets expected.

[tyru]

I wrote a quick example.
It only supports http, https yet.
https://gist.github.com/tyru/82cae8bad2b116f442d08eeef456e23e