-
public final class HttpUrlA uniform resource locator (URL) with a scheme of either
{@code http}or{@code https}. Use this class to compose and decompose Internet addresses. For example, this code will compose and print a URL for Google search:
which prints:{@code * * HttpUrl url = new HttpUrl.Builder() * .scheme("https") * .host("www.google.com") * .addPathSegment("search") * .addQueryParameter("q", "polar bears") * .build(); * System.out.println(url); * }
As another example, this code prints the human-readable query parameters of a Twitter search:{@code * * https://www.google.com/search?q=polar%20bears * }
which prints:{@code * * HttpUrl url = HttpUrl.parse("https://twitter.com/search?q=cute%20%23puppies&f=images"); * for (int i = 0, size = url.querySize(); i < size; i++) { * System.out.println(url.queryParameterName(i) + ": " + url.queryParameterValue(i)); * } * }
In addition to composing URLs from their component parts and decomposing URLs into their component parts, this class implements relative URL resolution: what address you'd reach by clicking a relative link on a specified page. For example:{@code * * q: cute #puppies * f: images * }
which prints:{@code * * HttpUrl base = HttpUrl.parse("https://www.youtube.com/user/WatchTheDaily/videos"); * HttpUrl link = base.resolve("../../watch?v=cbP2N1BQdYc"); * System.out.println(link); * }
What's in a URL?A URL has several components. Scheme{@code * * https://www.youtube.com/watch?v=cbP2N1BQdYc * }Sometimes referred to as protocol, A URL's scheme describes what mechanism should be used to retrieve the resource. Although URLs have many schemes (
Username and Password{@code mailto},{@code file},{@code ftp}), this class only supports{@code http}and{@code https}. Use java.net.URI for URLs with arbitrary schemes.Username and password are either present, or the empty string
Host{@code ""}if absent. This class offers no mechanism to differentiate empty from absent. Neither of these components are popular in practice. Typically HTTP applications use other mechanisms for user identification and authentication.The host identifies the webserver that serves the URL's resource. It is either a hostname like
{@code square.com}or{@code localhost}, an IPv4 address like{@code 192.168.0.1}, or an IPv6 address like{@code ::1}.Usually a webserver is reachable with multiple identifiers: its IP addresses, registered domain names, and even
Port{@code localhost}when connecting from the server itself. Each of a webserver's names is a distinct URL and they are not interchangeable. For example, even if{@code * http://square.github.io/dagger}and{@code http://google.github.io/dagger}are served by the same IP address, the two URLs identify different resources.The port used to connect to the webserver. By default this is 80 for HTTP and 443 for HTTPS. This class never returns -1 for the port: if no port is explicitly specified in the URL then the scheme's default is used.
PathThe path identifies a specific resource on the host. Paths have a hierarchical structure like "/square/okhttp/issues/1486" and decompose into a list of segments like ["square", "okhttp", "issues", "1486"].
This class offers methods to compose and decompose paths by segment. It composes each path from a list of segments by alternating between "/" and the encoded segment. For example the segments ["a", "b"] build "/a/b" and the segments ["a", "b", ""] build "/a/b/".
If a path's last segment is the empty string then the path ends with "/". This class always builds non-empty paths: if the path is omitted it defaults to "/". The default path's segment list is a single empty string: [""].
QueryThe query is optional: it can be null, empty, or non-empty. For many HTTP URLs the query string is subdivided into a collection of name-value parameters. This class offers methods to set the query as the single string, or as individual name-value parameters. With name-value parameters the values are optional and names may be repeated.
FragmentThe fragment is optional: it can be null, empty, or non-empty. Unlike host, port, path, and query the fragment is not sent to the webserver: it's private to the client.
EncodingEach component must be encoded before it is embedded in the complete URL. As we saw above, the string
Percent encoding{@code cute #puppies}is encoded as{@code cute%20%23puppies}when used as a query parameter value.Percent encoding replaces a character (like
{@code \ud83c\udf69}) with its UTF-8 hex bytes (like{@code %F0%9F%8D%A9}). This approach works for whitespace characters, control characters, non-ASCII characters, and characters that already have another meaning in a particular context.Percent encoding is used in every URL component except for the hostname. But the set of characters that need to be encoded is different for each component. For example, the path component must escape all of its
{@code ?}characters, otherwise it could be interpreted as the start of the URL's query. But within the query and fragment components, the{@code ?}character doesn't delimit anything and doesn't need to be escaped.
This prints:{@code * * HttpUrl url = HttpUrl.parse("http://who-let-the-dogs.out").newBuilder() * .addPathSegment("_Who?_") * .query("_Who?_") * .fragment("_Who?_") * .build(); * System.out.println(url); * }
When parsing URLs that lack percent encoding where it is required, this class will percent encode the offending characters. IDNA Mapping and Punycode encoding{@code * * http://who-let-the-dogs.out/_Who%3F_?_Who?_#_Who?_ * }Hostnames have different requirements and use a different encoding scheme. It consists of IDNA mapping and Punycode encoding.
In order to avoid confusion and discourage phishing attacks, IDNA Mapping transforms names to avoid confusing characters. This includes basic case folding: transforming shouting
{@code SQUARE.COM}into cool and casual{@code square.com}. It also handles more exotic characters. For example, the Unicode trademark sign (™) could be confused for the letters "TM" in{@code http://ho™mail.com}. To mitigate this, the single character (™) maps to the string (tm). There is similar policy for all of the 1.1 million Unicode code points. Note that some code points such as "\ud83c\udf69" are not mapped and cannot be used in a hostname.Punycode converts a Unicode string to an ASCII string to make international domain names work everywhere. For example, "σ" encodes as "xn--4xa". The encoded string is not human readable, but can be used with classes like InetAddress to establish connections.
Why another URL model?Java includes both java.net.URL and java.net.URI. We offer a new URL model to address problems that the others don't.
Different URLs should be differentAlthough they have different content,
{@code java.net.URL}considers the following two URLs equal, and the equals() method between them returns true:- http://square.github.io/
- http://google.github.io/
{@code java.net.URL}unusable for many things. It shouldn't be used as a Map key or in a Set. Doing so is both inefficient because equality may require a DNS lookup, and incorrect because unequal URLs may be equal because of how they are hosted. Equal URLs should be equalThese two URLs are semantically identical, but
{@code java.net.URI}disagrees:- http://host:80/
- http://host
{@code :80}) and the absent trailing slash ({@code /}) cause URI to bucket the two URLs separately. This harms URI's usefulness in collections. Any application that stores information-per-URL will need to either canonicalize manually, or suffer unnecessary redundancy for such URLs.Because they don't attempt canonical form, these classes are surprisingly difficult to use securely. Suppose you're building a webservice that checks that incoming paths are prefixed "/static/images/" before serving the corresponding assets from the filesystem.
By canonicalizing the input paths, they are complicit in directory traversal attacks. Code that checks only the path prefix may suffer!{@code * * String attack = "http://example.com/static/images/../../../../../etc/passwd"; * System.out.println(new URL(attack).getPath()); * System.out.println(new URI(attack).getPath()); * System.out.println(HttpUrl.parse(attack).encodedPath()); * }
If it works on the web, it should work in your application{@code * * /static/images/../../../../../etc/passwd * /static/images/../../../../../etc/passwd * /etc/passwd * }The
Paths and Queries should decompose{@code java.net.URI}class is strict around what URLs it accepts. It rejects URLs like "http://example.com/abc|def" because the '|' character is unsupported. This class is more forgiving: it will automatically percent-encode the '|', yielding "http://example.com/abc%7Cdef". This kind behavior is consistent with web browsers.{@code HttpUrl}prefers consistency with major web browsers over consistency with obsolete specifications.Neither of the built-in URL models offer direct access to path segments or query parameters. Manually using
Plus a modern API{@code StringBuilder}to assemble these components is cumbersome: do '+' characters get silently replaced with spaces? If a query parameter contains a '&', does that get escaped? By offering methods to read and write individual query parameters directly, application developers are saved from the hassles of encoding and decoding.The URL (JDK1.0) and URI (Java 1.4) classes predate builders and instead use telescoping constructors. For example, there's no API to compose a URI with a custom port without also providing a query and fragment.
Instances of HttpUrl are well-formed and always have a scheme, host, and path. With
{@code java.net.URL}it's possible to create an awkward URL like{@code http:/}with scheme and path but no hostname. Building APIs that consume such malformed values is difficult!This class has a modern API. It avoids punitive checked exceptions: parse() returns null if the input is an invalid URL. You can even be explicit about whether each component has been encoded already.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description public final classHttpUrl.Builder
-
Method Summary
Modifier and Type Method Description URLurl()Returns this URL as a java.net.URL. URIuri()Returns this URL as a java.net.URI. Stringscheme()Returns either "http" or "https". booleanisHttps()StringencodedUsername()Returns the username, or an empty string if none is set. Stringusername()Returns the decoded username, or an empty string if none is present. StringencodedPassword()Returns the password, or an empty string if none is set. Stringpassword()Returns the decoded password, or an empty string if none is present. Stringhost()Returns the host address suitable for use with getAllByName. intport()Returns the explicitly-specified port if one was provided, or the default port for this URL'sscheme. static intdefaultPort(String scheme)Returns 80 if {@code scheme.equals("http")}, 443 if{@code scheme.equals("https")}and -1otherwise.intpathSize()Returns the number of segments in this URL's path. StringencodedPath()Returns the entire path of this URL encoded for use in HTTP resource resolution. List<String>encodedPathSegments()Returns a list of encoded path segments like {@code ["a", "b", "c"]}for the URL{@code * http://host/a/b/c}.List<String>pathSegments()Returns a list of path segments like {@code ["a", "b", "c"]}for the URL{@code * http://host/a/b/c}.StringencodedQuery()Returns the query of this URL, encoded for use in HTTP resource resolution. Stringquery()Returns this URL's query, like {@code "abc"}for{@code http://host/?abc}.intquerySize()Returns the number of query parameters in this URL, like 2 for {@code * http://host/?a=apple&b=banana}.StringqueryParameter(String name)Returns the first query parameter named {@code name}decoded using UTF-8, or null if there isno such query parameter.Set<String>queryParameterNames()Returns the distinct query parameter names in this URL, like {@code ["a", "b"]}for{@code * http://host/?a=apple&b=banana}.List<String>queryParameterValues(String name)Returns all values for the query parameter {@code name}ordered by their appearance in thisURL.StringqueryParameterName(int index)Returns the name of the query parameter at {@code index}.StringqueryParameterValue(int index)Returns the value of the query parameter at {@code index}.StringencodedFragment()Returns this URL's encoded fragment, like {@code "abc"}for{@code http://host/#abc}.Stringfragment()Returns this URL's fragment, like {@code "abc"}for{@code http://host/#abc}.Stringredact()Returns a string with containing this URL with its username, password, query, and fragmentstripped, and its path replaced with {@code /...}.HttpUrlresolve(String link)Returns the URL that would be retrieved by following {@code link}from this URL, or null ifthe resulting URL is not well-formed.HttpUrl.BuildernewBuilder()HttpUrl.BuildernewBuilder(String link)Returns a builder for the URL that would be retrieved by following {@code link}from this URL,or null if the resulting URL is not well-formed.static HttpUrlparse(String url)Returns a new {@code HttpUrl}representing{@code url}if it is a well-formed HTTP or HTTPSURL, or null if it isn't.static HttpUrlget(URL url)Returns an HttpUrl for {@code url}if its protocol is{@code http}or{@code https}, ornull if it has any other protocol.static HttpUrlget(URI uri)booleanequals(@Nullable() Object other)inthashCode()StringtoString()StringtopPrivateDomain()Returns the domain name of this URL's host that is one level beneath the publicsuffix by consulting the public suffix list. -
-
Method Detail
-
url
URL url()
Returns this URL as a java.net.URL.
-
uri
URI uri()
Returns this URL as a java.net.URI. Because
{@code URI}is more strict than thisclass, the returned URI may be semantically different from this URL:- Characters forbidden by URI like
{@code [}and{@code |}will be escaped. - Invalid percent-encoded sequences like
{@code %xx}will be encoded like{@code %25xx}. - Whitespace and control characters in the fragment will be stripped.
These differences may have a significant consequence when the URI is interpreted by awebserver. For this reason the URI class and this method should be avoided.
- Characters forbidden by URI like
-
isHttps
boolean isHttps()
-
encodedUsername
String encodedUsername()
Returns the username, or an empty string if none is set.
-
encodedPassword
String encodedPassword()
Returns the password, or an empty string if none is set.
-
host
String host()
Returns the host address suitable for use with getAllByName. Maybe:
- A regular host name, like
{@code android.com}. - An IPv4 address, like
{@code 127.0.0.1}. - An IPv6 address, like
{@code ::1}. Note that there are no square braces. - An encoded IDN, like
{@code xn--n3h.net}.
- A regular host name, like
-
port
int port()
Returns the explicitly-specified port if one was provided, or the default port for this URL'sscheme. For example, this returns 8443 for
{@code https://square.com:8443/}and 443 for{@code * https://square.com/}. The result is in{@code [1..65535]}.
-
defaultPort
static int defaultPort(String scheme)
Returns 80 if
{@code scheme.equals("http")}, 443 if{@code scheme.equals("https")}and -1otherwise.
-
pathSize
int pathSize()
Returns the number of segments in this URL's path. This is also the number of slashes in theURL's path, like 3 in
{@code http://host/a/b/c}. This is always at least 1.
-
encodedPath
String encodedPath()
Returns the entire path of this URL encoded for use in HTTP resource resolution. The returnedpath will start with
{@code "/"}.
-
encodedPathSegments
List<String> encodedPathSegments()
Returns a list of encoded path segments like
{@code ["a", "b", "c"]}for the URL{@code * http://host/a/b/c}. This list is never empty though it may contain a single empty string.
-
pathSegments
List<String> pathSegments()
Returns a list of path segments like
{@code ["a", "b", "c"]}for the URL{@code * http://host/a/b/c}. This list is never empty though it may contain a single empty string.
-
encodedQuery
@Nullable() String encodedQuery()
Returns the query of this URL, encoded for use in HTTP resource resolution. The returned stringmay be null (for URLs with no query), empty (for URLs with an empty query) or non-empty (allother URLs).
-
query
@Nullable() String query()
Returns this URL's query, like
{@code "abc"}for{@code http://host/?abc}. Most callers shouldprefer queryParameterName and queryParameterValue because these methods offerdirect access to individual query parameters.
-
querySize
int querySize()
Returns the number of query parameters in this URL, like 2 for
{@code * http://host/?a=apple&b=banana}. If this URL has no query this returns 0. Otherwise it returnsone more than the number of{@code "&"}separators in the query.
-
queryParameter
@Nullable() String queryParameter(String name)
Returns the first query parameter named
{@code name}decoded using UTF-8, or null if there isno such query parameter.
-
queryParameterNames
Set<String> queryParameterNames()
Returns the distinct query parameter names in this URL, like
{@code ["a", "b"]}for{@code * http://host/?a=apple&b=banana}. If this URL has no query this returns the empty set.
-
queryParameterValues
List<String> queryParameterValues(String name)
Returns all values for the query parameter
{@code name}ordered by their appearance in thisURL. For example this returns{@code ["banana"]}for{@code queryParameterValue("b")}on{@code * http://host/?a=apple&b=banana}.
-
queryParameterName
String queryParameterName(int index)
Returns the name of the query parameter at
{@code index}. For example this returns{@code "a"}for{@code queryParameterName(0)}on{@code http://host/?a=apple&b=banana}. This throws if{@code index}is not less than the query size.
-
queryParameterValue
String queryParameterValue(int index)
Returns the value of the query parameter at
{@code index}. For example this returns{@code * "apple"}for{@code queryParameterName(0)}on{@code http://host/?a=apple&b=banana}. Thisthrows if{@code index}is not less than the query size.
-
encodedFragment
@Nullable() String encodedFragment()
Returns this URL's encoded fragment, like
{@code "abc"}for{@code http://host/#abc}. Thisreturns null if the URL has no fragment.
-
fragment
@Nullable() String fragment()
Returns this URL's fragment, like
{@code "abc"}for{@code http://host/#abc}. This returns nullif the URL has no fragment.
-
redact
String redact()
Returns a string with containing this URL with its username, password, query, and fragmentstripped, and its path replaced with
{@code /...}. For example, redacting{@code * http://username:password@example.com/path}returns{@code http://example.com/...}.
-
resolve
@Nullable() HttpUrl resolve(String link)
Returns the URL that would be retrieved by following
{@code link}from this URL, or null ifthe resulting URL is not well-formed.
-
newBuilder
HttpUrl.Builder newBuilder()
-
newBuilder
@Nullable() HttpUrl.Builder newBuilder(String link)
Returns a builder for the URL that would be retrieved by following
{@code link}from this URL,or null if the resulting URL is not well-formed.
-
parse
@Nullable() static HttpUrl parse(String url)
Returns a new
{@code HttpUrl}representing{@code url}if it is a well-formed HTTP or HTTPSURL, or null if it isn't.
-
get
@Nullable() static HttpUrl get(URL url)
Returns an HttpUrl for
{@code url}if its protocol is{@code http}or{@code https}, ornull if it has any other protocol.
-
hashCode
int hashCode()
-
topPrivateDomain
@Nullable() String topPrivateDomain()
Returns the domain name of this URL's host that is one level beneath the publicsuffix by consulting the public suffix list. Returnsnull if this URL's host is an IP address or is considered a public suffix by thepublic suffix list.
In general this method should not be used to test whether a domain is validor routable. Instead, DNS is the recommended source for that information.
-
-
-
-