Maintenance suggestion

[soywod]

As discussed by email with @jonhoo, some new IMAP libraries are really promising and may replace imap over time:

• imap-types and imap-codec: they can be considered a direct replacement of imap-proto. They provide solid types and codecs for dealing with IMAP requests and responses.
• imap-next: a thin sans I/O abstraction over IMAP's distinct protocol flows, based on imap-{types,codec}.
• imap-client: a direct replacement of imap. While it is already used in production by Himalaya CLI, it is not advised to use it yet as the API is not well defined nor documented. It may evolve towards a sans I/O structure. In this case, imap would become a first standard (blocking) I/O connector of imap-client. It would also open doors to async IMAP (tokio, async-std, smol etc).

I just initiate the thread now so we can discuss more about it when the final imap-client API design will start (within few months).

[jonhoo]

Thanks for opening this!

On imap-types + imap-codec, I'm not fundamentally opposed to adopting those over imap-proto, though I think we should carefully weigh the trade-offs, especially since this crate doesn't need any of the server-side bits that appear to be one of the main differentiators of those crates. The stronger typing is nice though, and I'd indeed be very interested if we could build an ergonomic client on top of it. One thing to note though: if the intent is to expose types from imap-types in the public API of this crate, not that that will tightly couple the semver of this crate with that of imap-types. If a new major version of imap-types is released, that would also require a major version bump of imap. Avoiding that tight coupling is exactly why imap (or rather, the 3.0 alpha) has a types submodule that defines all the types. No types from imap-proto are exposed directly. You will almost certainly want to do the same if adopting imap-types unless you specifically believe that that crate will have near-zero breaking changes. Which in turn depends a lot on maintainer effort in that direction.

On imap-next/imap-client, I'm very skeptical of the sans I/O approach. I think it often struggles a lot with support both synchronous and asynchronous code at the same time, especially in a very strongly typed language like Rust. It also tends to get quite tricky once you want support for things like streaming compute, background tasks like heartbeats, etc. Not to mention it tends to make the public API needlessly complicated. In general, I think generalizeability over I/O should be a non-goal if you're optimizing for ergonomics, which is probably right to do for a crate like imap. I think the path forward should almost certainly be to move to async only. Being compatible across executors, especially for wasm and embedded, is interesting, but I suspect likely not something to focus on for a first release since the ecosystem writ-large hasn't figured out how that should work yet.

Ultimately, I don't think imap should become "the blocking connector for imap-client". If anything I'd expect it to be the other way around: that imap becomes an ergonomic wrapper around a lower-level-and-less-ergonomic SansIoImap + AsyncIoConnector or something. In a way, the path I'm most excited about, at least on a theoretical level, is to end up with something like a hyper/reqwest split where one crate is "the IMAP protocol at a low level" (probably in both directions, but not sans-I/O), and one wraps the other to provide higher-level abstractions and more ergonomic interfaces.

That all said, I'd also be quite sad if we ended up with multiple Rust IMAP clients. The IMAP protocol, like the HTTP protocol, is filled with enough awkward corners that splitting efforts and knowledge is likely to be quite detrimental. If the choice ends up being between the path I've outlined above and "a single IMAP client crate", I would much rather have the latter :)

[jonhoo]

Wanted to capture some thoughts from an email thread from a while back as well:

Second, on the particular path you're proposing: I'm not opposed to replacing the entire implementation of imap with a from-scratch async re-write (which it sounds like you've done) for the next major version. I do, however, have some thoughts about what should and shouldn't be done in such a re-write in line with some design principles for the original imap crate:

1. The crate should not do connection pooling internally. One client should hold one connection, and methods should take &mut self so that they can directly access the underlying connection without having to copy arguments and read bytes. This is very commonly violated in async libraries, and tends to lead to poor performance over time.

2. Extensions should be implemented, but should be optional. It should be possible to use the crate to connect to servers that implement none of the extensions (even common ones). Ideally, it should be possible to compile the crate without most of the extensions to speed up compile-time (i.e., with features).

3. The interface and documentation should, as far as is possible, mirror what is in the original spec. The intent is for the imap crate to be a lower-level crate that specifically allows working with the IMAP protocol in a maximally flexible way. It is better to expose a slightly awkward-to-use interface than one that's easier-to-use but won't let you express what the protocol allows you to.

Aside from these, I also generally prefer strong typing where possible for things like distinguishing SEQIDs from UIDs so that callers can't accidentally mix them up.

On 2 (because I got questions about it):

Ah, so, I'm not proposing that we provide fallbacks for any of these commands. Instead, I'm proposing the following two "rules":

1.) the imap crate can be used to interact with servers that implement no extensions.

2.) if you are not using a particular extension in a program, you should be able to avoid building the code for that extension when building the imap crate.

1. is what I mean by extensions being optional.
2. is where I'd like them to be feature-gated. It doesn't matter so much for any one extension, but if I'm writing some code on top of imap that uses only, say, a single extension, my compile time can be meaningfully improved if I could not compile the N other extensions that the crate implements (and that I don't need).

I don't think we should provide fallbacks for any of the extensions if they are missing. That feels like a whole pandora's box. Though maybe we could provide a separate crate that is like imap-polyfills which implements equivalent functionality to various imap extensions by using other commands?

And on 3:

I'm not proposing we give just a bytes-in-bytes-in kind of API. I'm a big fan of using types to make the API harder to misuse (such as strongly typing the response to FETCH or builder for constructing clients/requests). When it comes to polyfilling functionality though, like injecting extra commands to resolve SEQs, or providing higher-level builders for sequences of operations, I'd prefer those to live "on top of" imap. Ideally in a separate crate (though can be in the same repository). That way we make it clear what is spec and what is our implementation, which tends to aid in testing, ability to use the crate for unforeseen things, and operating against weird servers.

[jonhoo]

As pointed out in #311 (comment), I feel like I no longer have the capacity to be a good maintainer for this crate, especially since I don't use it regularly myself any more. @artob has also offered to help out in that thread, so I've added both @soywod and @artob as maintainers of the crate to forge a path ahead. I'll hold back on crates.io publishing privileges for now, but once you feel like we've gotten to that point, please ping me!

I am of course happy to help chart a path forward, but as I also pointed out in an earlier comment, don't think of me as a blocker here. I would much rather the imap crate be brought back to life in a direction that I may disagree with than see it wither because my opinions keep other maintainers away :)

[soywod]

Thank you for the invitation. I was quite busy, hence the delay. I need to refactor my IMAP prototype (I hope in the nearest weeks), it will be a perfect occasion to discuss with you @artob if you are interested in. I may open a branch on that repo in order to attempt a naive merge of this crate and my prototype.

[duesee]

Hello 👋🏻, maintainer of imap-{types,codec,next,proxy,sec} here.

First of all... thank you, @jonhoo! I'm a long time fan of your open-source work and the "Crust of Rust" series. You really helped me to become a better programmer. So... it's a bit exciting to imagine that "your crate" could (!) potentially use something I created :-) Crazy...

Anyway ...

... especially since this crate doesn't need any of the server-side bits that appear to be one of the main differentiators of those crates.

I think about it this way: type-wise, you probably always want the client- and server-side. As a client to create commands and to interpret responses. As a server to interpret commands and to create responses. There is quite some overlap, too, e.g., both sides need Tags, Mailboxs, Flags, etc.

Still, a client serializes commands, and parses responses only, and a server parses commands, and serializes responses only. So, the parsing and serialization routines could be feature-gated in imap-codec to avoid some compilation.

But: Having both sides has another huge advantage: We can do great fuzz-testing ...

let bytes = arbitrary();
assert_eq!(bytes, serialize(parse(bytes)));

let object = arbitrary();
assert_eq!(object, parse(serialize(object)));

ROS looked at this recently, and their pentest report shows that this kind of fuzz-testing is rather effective.

I guess I should update the FAQ about the differences and slowly take a bit more proud in "testing" :-)

One thing to note though: if the intent is to expose types from imap-types in the public API of this crate, not that that will tightly couple the semver of this crate with that of imap-types. If a new major version of imap-types is released, that would also require a major version bump of imap. Avoiding that tight coupling is exactly why imap (or rather, the 3.0 alpha) has a types submodule that defines all the types. No types from imap-proto are exposed directly. You will almost certainly want to do the same if adopting imap-types unless you specifically believe that that crate will have near-zero breaking changes. Which in turn depends a lot on maintainer effort in that direction

We have automated SemVer checks in place (via cargo-semver-checks) and should not break SemVer accidentally. Regarding required breaking changes: I hope for imap-types to become the http of the Rust IMAP ecosystem.

Not sure how well I did but imap-types' types should be minimal. The crate doesn't provide any parsing or serialization logic but barely the code required to implement a codec (such as imap-codec). The types allow to construct every valid message (but not more). So, ignoring naming, and isomorphisms, I hope that most types can't be improved by a lot.

Having said that... I would love to go with someone smarter than me through imap-types and see where we can improve. I remember that http uses some clever tricks to remain open for additions w/o breaking changes, and maybe we can use those tricks here and there, too.

I still agree that depending on how "high-level" rust-imap should be, it may not always be needed to re-export a type -- literals come to mind. Still... some types are so simple, that rust-imap could arguably just re-export them. Other types are so complex, that... well... it may not make sense to completely "redefine" them in rust-imap, either. I'm really a bit torn here...

I definitely have to take a look at 3.0.0!

[jonhoo]

@duesee Thank you for the added context, that's very helpful! The challenge in terms of semver is that if any type from imap-types is exposed in the imap public API, then any major version bump in imap-types requires a breaking change in imap. Even if the breaking change in imap-types isn't to a type that's exposed in imap, that remains true. This is why I completely hid imap-proto from imap's public API 😅

[duesee]

Yeah... I understand the issue. It's just that (so far) I don't know if there is a better solution. axum, for example, re-exposes http. So... my thinking was: if we can make imap-types as good as http (and basically never (scary) have to do breaking changes) maybe it could become the standard library for IMAP that is "safe" to re-export.

Edit: Ah, got it. I said "not always be needed ... to re-export". I didn't mean we should expose only a few types from imap-types -- bad wording :-) I was just thinking about situations where a "redefinition" of a type of imap-types in imap would make sense. And I feel most of these redefinitions would be almost copy&paste as many of the types are just large "dumb" enums that don't really allow to abstract from anymore.

Maybe some clever wrapping could do the trick.