Understanding Integrated Web Filtering

With integrated Web filtering, the firewall intercepts every HTTP request in a TCP connection and extracts the URL from the HTTP request. The decision making is done on the device after it identifies a category for a URL either from user-defined categories or from the SurfControl category server. Each individual HTTP request is blocked or permitted based on URL filtering profiles defined by you.

You can permit or block access to a requested site by binding a Web filtering profile to a firewall policy. A Web filtering profile specifies URL categories and the action the device takes (permit or block) when it receives a request to access a URL in each category. A URL category is a list of URLs grouped by content. URL categories are predefined and maintained by SurfControl or are defined by you.

• SurfControl URL Categories

  SurfControl maintains about 40 predefined categories. To view SurfControl global categories, type the following command:

  user@host# show groups junos-defaults

  You can also look for custom-url-category

• User-Defined URL Categories

  When defining your own URL categories, you can group URLs and create categories specific to your needs. Each category can have a maximum of 20 URLs. When you create a category, you can add either the URL or the IP address of a site. When you add a URL to a user-defined category, the device performs DNS lookup, resolves the host name into IP addresses, and caches this information. When a user tries to access a site with the IP address of the site, the device checks the cached list of IP addresses and tries to resolve the hostname. Many sites have dynamic IP addresses, meaning that their IP addresses change periodically. A user attempting to access a site can type an IP address that is not in the cached list on the device. Therefore, if you know the IP addresses of sites you are adding to a category, enter both the URL and the IP address(es) of the site.

  You define your own categories using URL pattern list and custom URL category list custom objects. Once defined, you can select your categories when you configure your Web filtering profile as follows:

  user@host# set security utm feature-profile Web-filtering surf-control-integrated profile <profile_name> category

Note: If a URL appears in both a user-defined category and a predefined category, the device matches the URL to the user-defined category.

Note: Web filtering is performed on all the methods defined in HTTP 1.0 and HTTP 1.1.

Integrated Web Filtering: Process Overview

This is a general description of how Web traffic is intercepted and acted upon by the Web filtering module.

1. The device intercepts a TCP connection.
2. The device intercepts each HTTP request in the TCP connection.
3. The device extracts each URL in the HTTP request and checks its URL filter cache.
4. Global Web filtering white and black lists are checked first for block or permit.
5. If the HTTP request URL is allowed based on cached parameters, it is forwarded to the Web server. If there is no cache match, a request for categorization is sent to the SurfControl server. (If the HTTP request URL is blocked, the request is not forwarded and a notification message is logged.)
6. In the allowed case, the SurfControl server responds with the corresponding category.
7. Based on the identified category, if the URL is permitted, the device forwards the HTTP request to the Web server. If the URL is not permitted, then a deny page is sent to the HTTP client.

Integrated Web Filtering Cache

By default, the device retrieves and caches the URL categories from the SurfControl CPA server. This process reduces the overhead of accessing the SurfControl CPA server each time the device receives a new request for previously requested URLs. You can configure the size and duration of the cache, according to the performance and memory requirements of your networking environment. The lifetime of cached items is configurable between 1 and 1800 seconds with a default value of 300 seconds.

Note: Caches are not preserved across device reboots or power losses.

Web Filtering Profiles

You configure Web filtering profiles that permit or block URLs according to defined categories. A Web filtering profile consists of a group of URL categories assigned one of the following actions:

• Permit — The device always allows access to the Web sites in this category.
• Block — The device blocks access to the Web sites in this category. When the device blocks access to this category of Web sites, it displays a message in your browser indicating the URL category.
• Black List — The device always blocks access to the Web sites in this list. You can create a user-defined category or use a predefined category.
• White List — The device always allows access to the Web sites in this list. You can create a user-defined category or use a predefined category.

Note: A predefined profile is provided and can be used if you choose not to define your own profile.

A Web filtering profile may contain one black list or one white list, multiple user-defined and/or predefined categories each with a permit or block action, and an Other category with a permit or block action. You can define an action for all Other categories in a profile to specify what to do when the incoming URL does not belong to any of the categories defined in the profile. If the action for the Other category is block, the incoming URL is blocked if it does not match any of the categories explicitly defined in the profile. If an action for the Other category is not specified, the default action of permit is applied to the incoming URL not matching any category.

Profile Matching Precedence

When a profile employs several categories for URL matching, those categories are checked for matches in the following order:

1. If present, the global black list is checked first. If a match is made, the URL is blocked. If no match is found...
2. The global white list is checked next. If a match is made, the URL is permitted. If no match is found...
3. User-defined categories are checked next. If a match is made, the URL is blocked or permitted as specified. If no match is found...
4. Predefined categories are checked next. If a match is made, the URL is blocked or permitted as specified. If no match is found...
5. The Other category is checked next. If a match is made, the URL is blocked or permitted as specified.