HTTP GET and POST Requests in Java with NTLM Proxy

Java provides the java.net package for implementing networking applications and for making HTTP GET and POST requests in Java to a web server. This post shows how to use some of these classes for making HTTP GET and POST requests without using a proxy. Therefore, the approach used here does not rely on third party libraries such as the Apache HTTP Components. If you are able to leverage such libraries, I strongly encourage you to do so. For the examples shown here, I used Java 10 (10.0.2). Running the examples with older Java versions may result in errors when making requests through a proxy that requires NTLM authentication (…see exception at the bottom of this post).

Java Class for Handling HTTP Connections

The below HttpConnector class provides a basic implementation for making HTTP GET and POST Requests in Java, using the java.net package, specifically the java.net.HttpURLConnection and java.net.URL classes. It refers to a few other classes that are defined in the following sections and provide support for HTTP request / response handling and proxy authentication.

package tech.stefanbruhns.java.http.connect;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.net.HttpURLConnection;
import java.net.Proxy;
import java.net.URL;
import java.util.Scanner;

import tech.stefanbruhns.java.http.request.HttpGETRequest;
import tech.stefanbruhns.java.http.request.HttpPOSTRequest;
import tech.stefanbruhns.java.http.request.HttpRequest;
import tech.stefanbruhns.java.http.response.HttpResponse;

public class HttpConnector {
    protected Proxy proxy;
    protected HttpURLConnection connection; 

    public HttpConnector(final Proxy proxy) {
        this.proxy = proxy;
    }

    public HttpResponse makeRequest(final HttpRequest request) throws Exception {
        connection = getURLConnection(request);
        HttpResponse response = new HttpResponse();
        StringBuffer sb = new StringBuffer();
        Scanner input = null;

        // Set HTTP method
        connection.setRequestMethod(request.getMethod());

        // Set headers
        for (String key : request.headers.keySet())
            connection.addRequestProperty(key, request.headers.get(key));

        connection.setUseCaches(false);

        // Only call connect() here for GET
        if (request instanceof HttpGETRequest) 
            connection.connect();

        // If POST, write the request data
        if (request instanceof HttpPOSTRequest)
            writePostRequest((HttpPOSTRequest)request);

        response.contentType = connection.getContentType();
        response.code = connection.getResponseCode();
        response.message = connection.getResponseMessage();

        // Read the response
           try {
            input = new Scanner(connection.getInputStream());

            while (input.hasNextLine())
                sb.append(input.nextLine());
            
            response.content = sb.toString().getBytes();

        } catch (IOException e) {
            throw new Exception(e.getMessage(), e);

        } finally {
            // clean up resources
            if (input != null) input.close();
            if (connection != null) connection.disconnect();
        }
           
        return response;
    }

    protected HttpURLConnection getURLConnection(final HttpRequest request) throws Exception {
        String protocol = request.isHttps ? "https" : "http";
        String resource = request.getResource();
        URL url = new URL(protocol, request.domain, request.port, resource);
        return (HttpURLConnection)url.openConnection(this.proxy);
    }
    
    protected void writePostRequest(HttpPOSTRequest request) throws Exception {
        PrintWriter writer = null;

        try {
            connection.setDoInput(true);
            connection.setDoOutput(true);
            writer = new PrintWriter(new OutputStreamWriter(connection.getOutputStream(), "UTF-8"));
            writer.print(request.getData());

        } catch (Exception e) {
            throw new Exception("Error while writing POST request: " + e.getMessage(), e);

        } finally {
            // clean up resources
            if (writer != null) {
                writer.flush();
                writer.close();
            }
        }
    }
}

Abstract Java Class for Encapsulating Common HTTP Request Properties

In the code below, the abstract HttpRequest class is used to hold basic HTTP request parameters, such as the protocol, domain, port and resource.

package tech.stefanbruhns.java.http.request;

import java.util.HashMap;
import java.util.Map;

public abstract class HttpRequest {
    public boolean isHttps = false;       // Use http or https protocol
    public String domain;                 // Domain part of the URL, e.g. www.google.com
    public int port = 80;                 // http port (e.g.: 80 = http or 443 = https)
    protected String resource;            // Resource part of the URL (e.g.: /quote)
    public Map<String, String> params;    // Map for holding request parameters
    public Map<String, String> headers;   // Map for holding request headers
    
    public abstract String getMethod();   // Return HTTP method, such as GET or POST
    public abstract String getResource(); // Return resource, incl. query String for GET
    
    public HttpRequest() {
        params = new HashMap<String, String>();
        headers = new HashMap<String, String>();
    }
    
    public void setResource(String resource) {
    	this.resource = resource;
    }
}

Concrete Java Classes for Handling HTTP GET and POST Requests

The class HttpGETRequest provides a method named getQueryString() for generating a URL query string based on the list of query parameters that is defined in the abstract class HttpRequest.

package tech.stefanbruhns.java.http.request;

public class HttpGETRequest extends HttpRequest {
    
    @Override
    public String getMethod() {
        return "GET";
    }

    @Override
    public String getResource() {
        return super.resource += getQueryString();
    }
    
    public String getQueryString() {
        String urlParams = "";
        boolean isFirstParam = true;

        for (final String key : super.params.keySet()) {
            
            if (isFirstParam) {
                urlParams += "?" + key + "=" + super.params.get(key);
                isFirstParam = false;
            } else {
                urlParams += "&" + key + "=" + super.params.get(key);
            }
        }
        
        return urlParams;
    }
}

The following code snippet shows the HttpPOSTRequest class, which provides two methods to set and get POST request payload data.

package tech.stefanbruhns.java.http.request;

public class HttpPOSTRequest extends HttpRequest {
    
    private String data;
    
    @Override
    public String getMethod() {
        return "POST";
    }
    
    @Override
    public String getResource() {
        return super.resource;
    }
    
    public String getData() {
        return data;
    }
 
    public void setData(String data) {
        this.data = data;
    }
}

In addition, the simple Java class named HttpResponse provides variables for holding HTTP response parameter values.

package tech.stefanbruhns.java.http.response;

public class HttpResponse {
    public int code;
    public String message;
    public String contentType;
    public byte[] content;
}

Java Classes for Handling NTLM Proxy Authentication

When trying to make HTTP GET and POST Requests in Java from behind a proxy, you can use the below ProxyFactory class. It allows calling classes to obtain configured java.net.Proxy instances. That is, this class returns a configured java.net.Proxy object. Most importantly, note that setting a CookieHandler as shown in line 28 is needed to support the NTLM proxy authentication process.

package tech.stefanbruhns.java.http.auth;

import java.net.Authenticator;
import java.net.CookieHandler;
import java.net.CookieManager;
import java.net.CookiePolicy;
import java.net.InetSocketAddress;
import java.net.Proxy;

public class ProxyFactory {

    private Authenticator authenticator;
    private String hostname;
    private int port;
    
    public ProxyFactory(String hostname, int port) {
        this.hostname = hostname;
        this.port = port;
    }

    public ProxyFactory(String hostname, int port, Authenticator auth) {
        this.hostname = hostname;
        this.port = port;
        this.authenticator = auth;
        
        // Set system-wide cookie handler (default in-memory here)
        if (CookieHandler.getDefault() == null) {
            CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL));
        }
    }

    public Proxy getProxy() {
        Proxy proxy = Proxy.NO_PROXY;

        // Use a proxy if parameters are set
        if (hostname != null && port > 0) {
            InetSocketAddress inetAddr = new InetSocketAddress(hostname, port);
            proxy = new Proxy(Proxy.Type.HTTP, inetAddr);
            
            // Use authentication
            if (authenticator != null) {
                Authenticator.setDefault(authenticator);
            }
        }

        return proxy;
    }
}

For handling proxy authentication, another class is used. It is named ProxyAuthenticator and extends java.net.Authenticator for this purpose. This class is used to pass authentication credentials, namely username and password, to proxies that require them.

package tech.stefanbruhns.java.http.auth;

import java.net.Authenticator;
import java.net.PasswordAuthentication;

public class ProxyAuthenticator  extends Authenticator {
    
    private final String username;
    private final char[] password;

    public ProxyAuthenticator(final String username, final String password) {
        super();
        this.username = username;
        this.password = password.toCharArray(); 
    }
    
    @Override
    public PasswordAuthentication getPasswordAuthentication() {
        System.out.println("ProxyAuthenticator::Providing authentication credentials");
        return new PasswordAuthentication(username, password);
    }
}

Example 1 – HTTP GET Request without Proxy

In the below example, a direct HTTPS call is made to the Yahoo Finance stock quote API to obtain the data for Tesla’s stock. The code below make a HTTPS GET request to the following URL:

package tech.stefanbruhns.java.http.test;

import java.net.Proxy;

import tech.stefanbruhns.java.http.connect.HttpConnector;
import tech.stefanbruhns.java.http.request.HttpGETRequest;
import tech.stefanbruhns.java.http.response.HttpResponse;

public class TestLocalGETRequest {
    
    public static void main(final String... args) throws Exception {
        HttpGETRequest request = new HttpGETRequest();
        request.domain = "query1.finance.yahoo.com";
        request.port = 443;
        request.isHttps = true;
        request.setResource("/v8/finance/chart/TSLA");
        
        HttpConnector connector = new HttpConnector(Proxy.NO_PROXY);
        HttpResponse response = connector.makeRequest(request);
        System.out.println(new String(response.content));
    }
}

Run this main class, e.g. from within Eclipse. I used Eclipse 2018-12 (4.10.0). The response, in this case JSON text, will be printed to the console as shown below.

Eclipse 2018-09 (4.10.0) - Make HTTPS GET request with no proxy
Eclipse 2018-09 (4.10.0) – Make HTTPS GET request with no proxy

Example 2 – HTTP GET Request with Proxy and Authentication (NTLM)

If you need to run the same request through a proxy that also requires authentication, you can change the existing example to use the ProxyAuthenticator and ProxyFactory classes. Provide the domain name, user name and password as shown in line 20, if required. If your proxy does not require authentication, use the other constructor in the ProxyFactory class that only has 2 arguments (see line 16).

package tech.stefanbruhns.java.http.test;

import tech.stefanbruhns.java.http.auth.ProxyFactory;

import tech.stefanbruhns.java.http.auth.ProxyAuthenticator;
import tech.stefanbruhns.java.http.connect.HttpConnector;
import tech.stefanbruhns.java.http.request.HttpGETRequest;
import tech.stefanbruhns.java.http.response.HttpResponse;

public class TestProxyGETRequest {

    public static void main(final String... args) throws Exception {
        HttpGETRequest request = new HttpGETRequest();
        request.domain = "query1.finance.yahoo.com";
        request.port = 443;
        request.isHttps = true;
        request.setResource("/v8/finance/chart/MSFT");

        ProxyAuthenticator authenticator = new ProxyAuthenticator("domain\\stefan", "Password123");
        ProxyFactory proxyFactory = new ProxyFactory("proxy.example.com", 8080, authenticator);

        HttpConnector connector = new HttpConnector(proxyFactory.getProxy());
        HttpResponse response = connector.makeRequest(request);
        System.out.println(new String(response.content));
    }
}

In my case, I am using a NTLM proxy that requires authentication. For details on the NTLM authentication process, refer to this Microsoft NTLM article. The console output shows that the getPasswordAuthentication of ProxyAuthenticator is invoked during the NTLM authentication process.

Eclipse 2018-09 (4.10.0) - Make HTTPS GET request with NTLM proxy
Eclipse 2018-09 (4.10.0) – Make HTTPS GET request with NTLM proxy

Older Java versions such as Java 8 may result in the below exception that is caused by a NTLM authentication failure. For more details on this issue have a look at the given links.

Exception in thread "main" java.lang.NullPointerException
at com.sun.security.ntlm.Client.type3(Client.java:161)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.buildType3Msg(NTLMAuthentication.java:250)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.setHeaders(NTLMAuthentication.java:225)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2114)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at tech.stefanbruhns.java.http.connect.HttpConnector.makeRequest(HttpConnector.java:39)
at tech.stefanbruhns.java.http.test.TestProxyGETRequest.main(TestProxyGETRequest.java:25)

Example 3 – HTTP POST Request without Proxy

To test HTTP Post requests without a proxy, I am hosting the below Servlet on a local Tomcat web server. It simply handles POST requests and prints the payload data to the console. Using a proxy works the same way as shown earlier for HTTP GET requests.

package tech.stefanbruhns.java.web;

import java.io.BufferedReader;
import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

@SuppressWarnings("serial")
@WebServlet("/PostTestServlet")
public class PostTestServlet extends HttpServlet {

    protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        StringBuffer sb = new StringBuffer();
        BufferedReader br = request.getReader();

        while (br.ready()) {
            sb.append(br.readLine());
        }

        System.out.println(sb.toString());
    }
}

This code shows how to make the HTTP POST request to that Servlet. The payload here is a plaint text, saying Hello from the Client!.

package tech.stefanbruhns.java.http.test;

import java.net.Proxy;

import tech.stefanbruhns.java.http.connect.HttpConnector;
import tech.stefanbruhns.java.http.request.HttpPOSTRequest;
import tech.stefanbruhns.java.http.response.HttpResponse;

public class TestMakePOSTRequest {
    
    public static void main(final String... args) throws Exception {
        HttpPOSTRequest request = new HttpPOSTRequest();
        request.domain = "localhost";
        request.port = 8080;
        request.isHttps = false;
        request.setResource("/stb/PostTestServlet");

        request.headers.put("Content-Type", "text/plain");
        request.setData("Hello from the client!");

        HttpConnector connector = new HttpConnector(Proxy.NO_PROXY);
        HttpResponse response = connector.makeRequest(request);
        System.out.println("HTTP Response Code: " + response.code);
    }
}

Running this code produces one line of console output, showing the HTTP response code, here HTTP Response Code: 200.

Eclipse 2018-09 (4.10.0) - Make HTTP POST request without proxy
Eclipse 2018-09 (4.10.0) – Make HTTP POST request without proxy

The Servlet simply prints the data that it receives from the client.

Eclipse 2018-09 (4.10.0) - Run HTTP POST Servlet with Tomcat Web Server
Eclipse 2018-09 (4.10.0) – Run HTTP POST Servlet with Tomcat Web Server

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.