HTTPS proxy support

[olivier-thatch]

Hello,

The docs at https://honeyryderchuck.gitlab.io/httpx/wiki/Proxy say that HTTPX does support HTTPS proxies:

# Use an HTTPS `CONNECT` proxy to tunnel secure HTTP(S) requests.
HTTPX.plugin(:proxy).with_proxy(uri: "https://10.10.0.1:51432").get("https://google.pt")

However, running the above code produces the following error:

irb(main):001> require "httpx"
=> true

irb(main):002> HTTPX.plugin(:proxy).with_proxy(uri: "https://10.10.0.1:51432").get("https://google.pt")
/Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/plugins/proxy.rb:128:in `find_connection': https: unsupported proxy protocol (HTTPX::HTTPProxyError)
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:131:in `block in send_request'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:130:in `catch'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:130:in `send_request'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:242:in `block in _send_requests'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:241:in `each'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:241:in `_send_requests'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:233:in `send_requests'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/session.rb:72:in `request'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/httpx-1.2.3/lib/httpx/chainable.rb:10:in `get'
	from (irb):2:in `<main>'
	from <internal:kernel>:187:in `loop'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/irb-1.11.2/exe/irb:9:in `<top (required)>'
	from /Users/olivier/.rbenv/versions/3.3.0/bin/irb:25:in `load'
	from /Users/olivier/.rbenv/versions/3.3.0/bin/irb:25:in `<top (required)>'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `load'
	from /Users/olivier/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/bundler-2.5.3/lib/bundler/cli/exec.rb:58:in `kernel_load'
	... 13 levels...

Taking a quick look at the source, it does look like HTTPS is indeed unsupported and the docs are incorrect.

Can you confirm whether HTTPX supports HTTPS proxies or not, and if not, whether this feature is planned for a future release?

[HoneyryderChuck]

Hi, thx for the report.

Can you define the proxy as "http://10...." and try again? Also, use HTTPX_DEBUG=2 env var and copy the output here, if the error persists.

[olivier-thatch]

It works with http://, but I'm specifically asking about using an HTTPS proxy (not using an HTTP proxy to connect to an HTTPS host).

It's not supported by Ruby's net/http (cf. https://bugs.ruby-lang.org/issues/16482), and I was hoping that HTTPX would be able to support that, but it doesn't look like that's currently the case.

It's really hard to google for this issue, but my understanding is that support for HTTPS proxies is still very uncommon. curl supports it (curl -x https://my.proxy https://google.com works), and it was added to the .NET runtime last year (dotnet/runtime#87638).

[HoneyryderChuck]

I see, that's clear now. Indeed, it does not yet support it. I tried once in the past, but didn't manage to add a test in CI, and ultimately thouhjt the community wouldnt care much.

I'm open to adding it. Do you think you can take a stab at it? If not, is there a publicly available https proxy.i could test things with?

[olivier-thatch]

Another HTTPS proxy implementation, this one in JS/TypeScript: https://github.com/TooTallNate/proxy-agents/blob/main/packages/https-proxy-agent/src/index.ts

I did take a brief stab at it yesterday, but didn't get very far as I'm completely unfamiliar with HTTPX's internals. I'm under a bit of a time constraint so I'm just going to switch to a regular HTTP proxy to unblock the project I'm working on right now.

Unfortunately the HTTPS proxy I was using to test is a private one so I can't share it. If you want to work on this feature, I think your best bet would be to set up a local HTTPS proxy on your own machine... though then you might also have to deal with self-signed certificates, unless you can use Let's Encrypt or some other approach to avoid having to disable SSL verification or mess with the CA store.

I do think this feature would be nice to have, and would fill a gap in the Ruby ecosystem, but I agree that most people don't care for this as HTTPS proxies are probably very uncommon.

[HoneyryderChuck]

I'm using squid and 3proxy in CI, which is already using self-signed certificates for https tests. I don't think that's a problem really, I just need one httpsproxy image using any OS available https-enabled proxy server I can use to reproduce the missing feature, that'd ramp up development greatly.

[Aubermean]

Hi. You can do this using gost:

gost -L=:8080

Also with docker docker run --rm ginuerzh/gost -L=:8080. This runs it in 'auto' mode that supports various protocols, more info in the docs. It's also configurable by yaml or json.

https://github.com/ginuerzh/gost

3proxy sucks and it is ancient, the chinese are several steps ahead :)

[HoneyryderChuck]

Thx for the suggestion. I've managed to set that up (see here), but ultimately can't move on until openssl supports non-fd based sessions. Feel free to nudge there if this interests you.