Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Custom curl user agent when HOMEBREW_CURLRC is set #19353

Open
1 task done
eptr opened this issue Feb 22, 2025 · 1 comment
Open
1 task done

Custom curl user agent when HOMEBREW_CURLRC is set #19353

eptr opened this issue Feb 22, 2025 · 1 comment
Labels
features New features help wanted We want help addressing this

Comments

@eptr
Copy link

eptr commented Feb 22, 2025

Verification

Provide a detailed description of the proposed feature

When ~/.curlrc is enabled by setting HOMEBREW_CURLRC=1, Brew still uses the HOMEBREW_USE_AGENT_CURL value as curl's user agent. This may cause the download of certain packages to fail.

Proposed Feature:
When HOMEBREW_CURLRC=1 is set, Brew should update curl's arguments with those from ~/.curlrc after processing its internal argument parser logic.

What is the motivation for the feature?

When Brew attempts to download certain packages (e.g., mailmaster), the server returns a 403 Forbidden error.
This issue is caused by the server's user-agent detection, which blocks access when the user-agent string contains the term curl.
This can be demonstrated by the following test.

$ curl -v https://res.126.net/dl/client/macmail/dashi/mail5.dmg
* Host res.126.net:443 was resolved.
* IPv6: (none)
* IPv4: 198.18.0.61
*   Trying 198.18.0.61:443...
* Connected to res.126.net (198.18.0.61) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=CN; ST=Zhejiang; L=Hangzhou; O=NetEase (Hangzhou) Network Co., Ltd; CN=mimg.127.net
*  start date: Aug 20 00:00:00 2024 GMT
*  expire date: Sep 14 23:59:59 2025 GMT
*  subjectAltName: host "res.126.net" matched cert's "res.126.net"
*  issuer: C=US; O=DigiCert, Inc.; CN=GeoTrust G2 TLS CN RSA4096 SHA256 2022 CA1
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://res.126.net/dl/client/macmail/dashi/mail5.dmg
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: res.126.net]
* [HTTP/2] [1] [:path: /dl/client/macmail/dashi/mail5.dmg]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET /dl/client/macmail/dashi/mail5.dmg HTTP/2
> Host: res.126.net
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 403 
< server: openresty
< date: Sat, 22 Feb 2025 02:31:35 GMT
< content-type: text/html
< content-length: 150
< request-id: a33c67b937077827444cc93abcbbcd4f
< 
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>openresty</center>
</body>
</html>
* Connection #0 to host res.126.net left intact

$ curl -v -A 'firefox' https://res.126.net/dl/client/macmail/dashi/mail5.dmg
* Host res.126.net:443 was resolved.
* IPv6: (none)
* IPv4: 198.18.0.61
*   Trying 198.18.0.61:443...
* Connected to res.126.net (198.18.0.61) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=CN; ST=Zhejiang; L=Hangzhou; O=NetEase (Hangzhou) Network Co., Ltd; CN=mimg.127.net
*  start date: Aug 20 00:00:00 2024 GMT
*  expire date: Sep 14 23:59:59 2025 GMT
*  subjectAltName: host "res.126.net" matched cert's "res.126.net"
*  issuer: C=US; O=DigiCert, Inc.; CN=GeoTrust G2 TLS CN RSA4096 SHA256 2022 CA1
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://res.126.net/dl/client/macmail/dashi/mail5.dmg
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: res.126.net]
* [HTTP/2] [1] [:path: /dl/client/macmail/dashi/mail5.dmg]
* [HTTP/2] [1] [user-agent: firefox]
* [HTTP/2] [1] [accept: */*]
> GET /dl/client/macmail/dashi/mail5.dmg HTTP/2
> Host: res.126.net
> User-Agent: firefox
> Accept: */*
> 
* Request completely sent off
< HTTP/2 200 
< server: openresty
< date: Sat, 22 Feb 2025 02:33:14 GMT
< content-type: application/octet-stream
< content-length: 103853070
< last-modified: Mon, 13 Jan 2025 01:42:38 GMT
< cache-control: max-age=3600
< ctl-cache-status: HIT from sh-putuo1-ca28, HIT from ha-kaifeng2-ca14, HIT from fj-xiamen10-ca49
< request-id: a33c67b9376a78276a10d3937e79cb2a
< etag: "67846f8e-630ac0e"
< expires: Sat, 22 Feb 2025 03:05:59 GMT
< age: 0
< 
Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.
* Failure writing output to destination, passed 3565 returned 4294967295
* process_pending_input: nghttp2_session_mem_recv() returned -902:The user callback function failed
* Connection #0 to host res.126.net left intact

How will the feature be relevant to at least 90% of Homebrew users?

Any package provider may detect the user agent used by Brew and block user access. This fix mitigates that risk.

What alternatives to the feature have been considered?

Directly load HOMEBREW_USER_AGENT_CURL from environment variables when it exists, rather than generating it before every request.
@eptr eptr added the features New features label Feb 22, 2025
@eptr eptr changed the title Custom curl user agent not working when HOMEBREW_CURLRC is set Custom curl user agent when HOMEBREW_CURLRC is set Feb 22, 2025
@MikeMcQuaid MikeMcQuaid added the help wanted We want help addressing this label Feb 24, 2025
@Bo98
Copy link
Member

Bo98 commented Feb 24, 2025

If you're having user-agent issues with particular casks, really it should be fixed in the cask itself rather than requiring every user to configure their own user-agent: https://docs.brew.sh/Cask-Cookbook#additional-url-parameters. Casks can be configured to mimic a browser user-agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
features New features help wanted We want help addressing this
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MikeMcQuaid @Bo98 @eptr