limited request creation via headers method

[Cosm-gedenke]

Is your feature request related to a problem? Please describe.
I'm trying to utilize proxelar proxy's for a project of mine, so far it has been incredible however,
when trying to build a request into a fully entirely editable request, I need to grab every value of the headerMap

/// Returns the request headers.
    pub const fn headers(&self) -> &HeaderMap {
        &self.headers
    }

parameter, however all the options I have are:

pub fn get<K>(&self, key: K) -> Option<&T>
    where
        K: AsHeaderName,
    {
        self.get2(&key)
    }

which doesn't worked due to the arbitrary nature of possible fields values, and simply holding all of the possible headers
in the HTTP RFC is bound to create issues with new parameters, making it very unmantainable in my scenario
and there is also this

pub fn iter(&self) -> Iter<'_, T> {
        Iter {
            map: self,
            entry: 0,
            cursor: self.entries.first().map(|_| Cursor::Head),
        }
    }

which has an arbitrary order, which is also not good for representing and handling a request entirely

Describe the solution you'd like
I wish to preserve the order of the header in the requests

somethign like this on burpsuite, as an example.

Describe alternatives you've considered
an alternative representation of a proxied request so it is easier to handle

[Cosm-gedenke]

as a note, I'd totally love to contribute to this too lf possible, so if a solution is not possible, i'd totally love to know where I can tackle this issue myself :D

[emanuele-em]

@Cosm-gedenke feel free to send a PR!

[Cosm-gedenke]

@emanuele-em in such case, do you have any suggestion regarding this feature? adding a new attribute to the ProxiedRequest struct might fix it in the simplest manner, however it would also take far more memory to hold one instance when it is not necessarily being held or used at all. So I'm thinking of maybe adding a boolean parameter to the start function of the proxelar proxy that would give you acess to the entireRequest attribute or something with that name haha, which would be something of type Option. This approach would keep it's modularity for the other uses.

I'm only asking to not make a PR outright that would change something you wouldn't agree with as I really respect the work done here :)

[emanuele-em]

@Cosm-gedenke
We can take inspiration from mitmproxy with something like this:

#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct HeaderField {
    name: Bytes,
    value: Bytes,
}

#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)]
pub struct HeaderBlock {
    fields: Vec<HeaderField>,
}

And changing the proxiedRequest accordingly:

pub struct ProxiedRequest {
    method: Method,
    uri: Uri,
    version: Version,
    headers: HeaderBlock, //instead of HeaderMap
    body: Bytes,
    body_metadata: BodyMetadata,
    time: i64,
}

So you can also implement methods for headerBlock like "fields", "get(name)", "grt_all()", "append(name, value)", "insert(index, name, value)", "remove_all(name)", set_all(name, values)"

If we choose this approach, everything should be adapted to the new headerBlock structure: TUI parser, web API, intercept protocol, session serialisation, replay logic etc.

One limitation is that Proxelar currently receives headers from Hyper after they have already been parsed into a HeaderMap. Fully preserving the original incoming HTTP/1 order would therefore also require Hyper to expose the original ordered header fields, for example through a request extension.

What do you think about this?

[Cosm-gedenke]

seems good!
I'll go ahead and start implementing it as I have time.
Thank you for the suggestion and hopefully a neat and tidy PR is coming haha

[Cosm-gedenke]

@emanuele-em
one thing I do not understand however is how Hyper's extensions properly work, I understand I need to add an option to directly expose the header bytes, or maybe even the entire request, however this would require proxelar to use a private fork of Hyper, are there alternatives here? I don't really understand how to approach this specifically without adding this layer, beyond that I can't really make up a solution here.

[emanuele-em]

@emanuele-em one thing I do not understand however is how Hyper's extensions properly work, I understand I need to add an option to directly expose the header bytes, or maybe even the entire request, however this would require proxelar to use a private fork of Hyper, are there alternatives here? I don't really understand how to approach this specifically without adding this layer, beyond that I can't really make up a solution here.

@Cosm-gedenke You're absolutely right. The relevant methods are private in Hyper 😢. I think the most viable long-term solution is to implement a small, purpose-built HTTP/1 transport for the proxy path, preserving header order and casing without forking Hyper. Hyper can remain for HTTP/2 and components that do not require wire-level fidelity. This is broadly similar to mitmproxy's approach, which owns its HTTP/1 handling while relying on specialized libraries for other protocols.

If you have a better and simpler idea, I’d be happy to hear it.

[Cosm-gedenke]

@emanuele-em
the only other option that makes sense to me is to fork Hyper, however mitmproxy does use it's own custom HTTP/1 stack, this however would require a high amount of effort and also afterwards maintenance from the proxelar maintainers, due to how handling persistent connections, chunked encoding, trailers, upgrades, etc.
all of this might be a bit of an overreach for a simple issue and would entail a lot of work onto the future of the tool.
not to mention it will take me far more time to get something half working hahahaha 😅

[Cosm-gedenke]

however it's something that would be a LOT of fun!

[emanuele-em]

@Cosm-gedenke I'm cooking something to replace Hyper and create adapters to handle headers autonomously (h1, h2 and maybe also h3), of course this is not only for this small feature, but to have more control (and customization) in the future.

Thanks for your advice. I had also thought about a fork of Hyper, but maybe it's better to “simplify” things rather than maintins a version of Hyper.

btw stay tuned, I'll need your help soon 😉 (I'll request a review if it's ok for you)

[Cosm-gedenke]

@emanuele-em in such case adapting that to the headerBlock would also fix this issue! in that case maybe i'll wait for your dish( ahahah get it cuz you're cooking 😄 ).

feel free to request any reviews on this project, I love to contribute to low level projects :D

[emanuele-em]

@Cosm-gedenke

[Cosm-gedenke]

@emanuele-em Beautiful 🥹