Using cURL with a Proxy Server

If you are located inside a network that uses a proxy and authentication for making HTTP requests to external servers, you need to pass proxy parameters and authentication credentials when using cURL with a proxy server. I used cURL on macOS Sierra for the examples here. An alternative to using cURL’s proxy options is to use environment variables. You can read about that approach using the link below.

In the following examples, I am using the parameters shown in table 1. The Yahoo Finance API URL returns stock data in JSON format.

Hostname
proxy.myorg.com
Port
8080
Username
dave
Active Directory Domain
ABC
URL
https://query1.finance.yahoo.com/v8/finance/chart/AAPL
Table 1: Values used in this post’s examples

Example 1: HTTP GET Request via Proxy without Authentication

The simplest command, using cURl’s -x option to specify a proxy server that does not require authentication is:

curl -x proxy.myorg.com https://query1.finance.yahoo.com/v8/finance/chart/AAPL

If the proxy server uses a port other than the standard HTTP port, you must append it to the hostname with a colon:

curl -x proxy.myorg.com:8080 https://query1.finance.yahoo.com/v8/finance/chart/AAPL

Example 2: HTTP GET Request via Proxy with Authentication

If the proxy requires authentication with a username and password, use the -U option and separate the values with a colon. Note that special characters such as the @ character need to be escaped with a backslash \ character.

curl -U dave:P\@ssW0rd -x proxy.myorg.com:8080 https://query1.finance.yahoo.com/v8/finance/chart/AAPL

The backslash character needs to be escaped as well. So, for passing a network domain name in the -U option, use \\.

curl -U ABC\\dave:P\@ssW0rd -x proxy.myorg.com:8080 https://query1.finance.yahoo.com/v8/...

Additional special characters that need to be escaped with a backslash are listed in the following table.

&
Ampersand
*
Asterisk
'
Apostrophe
@
At Symbol
\
Backslash
:
Colon
!
Exclamation Mark
`
Grave Accent
{
Left Brace
[
Left Bracket
(
Left Parenthesis
#
Number Sign
?
Question Mark
"
Quotation Mark
}
Right Brace
]
Right Bracket
)
Right Parenthesis
;
Semicolon
/
Slash
~
Tilde
<
Less-Than Sign
>
Greater-than Sign
$
Dollar Sign
|
Vertical Bar
Table 2: Terminal special characters. Source scriptingosx.com.

Example 3: Setting Proxy Authentication Scheme

In the previous example, the Basic authentication scheme was used by default. If your proxy requires NTLM authentication, it can be specified using the --proxy-ntlm option. You can also use the --proxy-anyauth to let cURL determine the proxy authentication scheme by evaluating the 407 response of the proxy.

curl -U ABC\\dave:P\@ssW0rd -x proxy.myorg.com:8080 --proxy-ntlm https://query1.finance.yahoo.com/v8/...

Debugging cURL HTTP Requests with Verbose Option

Use the -v option so that cURL prints out verbose information while running. This can be used for evaluating the HTTP request and response exchanges between cURL and the proxy.

bash-3.2$ curl -U dave:P\@ssW0rd -v -x proxy.myorg.com:8080 https://query1.finance.yahoo.com/v8/finance/chart/AAPL * Trying 10.89.4.75... * TCP_NODELAY set * Connected to proxy.aexp.com (10.89.4.75) port 8080 (#0) * Establish HTTP proxy tunnel to query1.finance.yahoo.com:443 * Proxy auth using Basic with user 'dave' > CONNECT query1.finance.yahoo.com:443 HTTP/1.1 > Host: query1.finance.yahoo.com:443 > Proxy-Authorization: Basic QURTXHNicnVobnM6V2ludGVyQDIwMTk= > User-Agent: curl/7.54.0 > Proxy-Connection: Keep-Alive > < HTTP/1.0 200 Connection established < * Proxy replied OK to CONNECT request * TLS 1.2 connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 * Server certificate: *.yahoo.com * Server certificate: DigiCert SHA2 High Assurance Server CA * Server certificate: DigiCert High Assurance EV Root CA > GET /v8/finance/chart/AAPL HTTP/1.1 > Host: query1.finance.yahoo.com > User-Agent: curl/7.54.0 > Accept: */* > < HTTP/1.1 200 OK < X-Yahoo-Request-Id: a4cu6apedh7m3 < Cache-Control: max-age=10, stale-while-revalidate=15 < Content-Type: application/json;charset=utf-8 < Vary: Origin < Date: Sun, 12 May 2019 22:31:31 GMT < Age: 0 < Server: ATS < X-Frame-Options: SAMEORIGIN < Referrer-Policy: no-referrer-when-downgrade < Strict-Transport-Security: max-age=15552000 < Transfer-Encoding: chunked < Connection: keep-alive < Expect-CT: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only" < X-XSS-Protection: 1; mode=block < X-Content-Type-Options: nosniff < Public-Key-Pins-Report-Only: max-age=2592000; pin-sha256="2fRAUXyxl4A1/XHrKNBmc8bTkzA7y4FB/GLJuNAzCqY="; pin-sha256="I/Lt/z7ekCWanjD0Cvj5EqXls2lOaThEA0H2Bg4BT/o="; pin-sha256="Wd8xe/qfTwq3ylFNd3IpaqLHZbh2ZNCLluVzmeNkcpw="; pin-sha256="WoiWRyIOVNa9ihaBciRSC7XHjliYS9VwUGOIud4PB18="; pin-sha256="i7WTqTvh0OioIruIfFR4kMPnBqrS2rdiVPl/s2uC/CY="; pin-sha256="r/mIkG3eEpVdm+u/ko/cwxzOMo1bk4TyHIlByibiA5E="; pin-sha256="uUwZgwDOxcBXrQcntwu+kYFpkiVkOaezL0WYEZ3anJc="; pin-sha256="dolnbtzEBnELx/9lOEQ22e6OZO/QNb6VSSX2XHA3E7A="; includeSubdomains; report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-hpkp-report-only" <{   "chart":     {       "result":       [         {           "meta":           {             "currency":"USD",             "symbol":"AAPL",             "exchangeName":"NMS",             "instrumentType":"EQUITY",             "firstTradeDate":345459600,             "gmtoffset":-14400             ....

For more information on using cURL with a proxy server, refer to the cURL documentation at https://ec.haxx.se/usingcurl-proxies.html.s cURL supports many protocols. To see how cURL can be used to transfer data via FTP, follow the link:

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.