Local Web Filtering

 

The Web filtering lets you to manage Internet usage by preventing access to inappropriate Web content. There are three types of Web filtering solutions:

  • Redirect Web filtering

  • Local Web filtering

  • Enhanced Web filtering

For more information, see the following topics:

Understanding Local Web Filtering

Local web filtering allows you to define custom URL categories, which can be included in blacklists and whitelists that are evaluated on the SRX Series device. All URLs for each category in a blacklist are denied, while all URLs for each category in a whitelist are permitted.

With local Web filtering, a firewall intercepts every HTTP request in a TCP connection and extracts the URL. A decision is made by the device after it looks up a URL to determine whether it is in the whitelist or blacklist based on its user-defined category. A URL is first compared to the blacklist URLs. If a match is found, the request is blocked. If no match is found, the URL is compared to the whitelist. If a match is found, the request is permitted. If the URL is not in either list, the custom category is taken (block, log-and-permit, or permit). If the URL is not in custom category, the defined default action is taken (block, log-and-permit, or permit). You can permit or block access to a requested site by binding a Web filtering profile to a firewall policy. Local Web filtering provides basic Web filtering without requiring an additional license or external category server.

This topic contains the following sections:

Local Web Filtering Process

The following section describes on how Web traffic is intercepted and acted upon by the Web filtering module.

  1. The device intercepts a TCP connection.
  2. The device intercepts each HTTP request in the TCP connection.
  3. The device extracts each URL in the HTTP request and checks its URL against the user-defined whitelist and blacklist.
  4. If the URL is found in the blacklist, the request is not permitted and a deny page is sent to the http client. If the URL is found in the whitelist, the request is permitted.
  5. If the URL is not found in the whitelist or blacklist, the configured default fallback action is applied. If no fallback action is defined, then the request is permitted.

User-Defined Custom URL Categories

To perform local Web filtering, you must define a blacklist and whitelist content that can be applied to the profile.

When defining your own URL categories, you can group URLs and create categories specific to your needs. Each category can have a maximum of 20 URLs. When you create a category, you can add either the URL or the IP address of a site. When you add a URL to a user-defined category, the device performs DNS lookup, resolves the hostname into IP addresses, and caches this information. When a user tries to access a site with the IP address of the site, the device checks the cached list of IP addresses and tries to resolve the hostname. Many sites have dynamic IP addresses, meaning that their IP addresses change periodically. A user attempting to access a site can type an IP address that is not in the cached list on the device. Therefore, if you know the IP addresses of sites you are adding to a category, enter both the URL and the IP address(es) of the site.

You define your own categories using URL pattern list and custom URL category list custom objects. Once defined, you assign your categories to the global user-defined url-blacklist (block) or url-whitelist (permit) categories.

Note

Web filtering is performed on all the methods defined in HTTP 1.0 and HTTP 1.1.

Local Web Filtering Profiles

You configure Web filtering profiles that permit or block URLs according to defined custom categories. A Web filtering profile consists of a group of URL categories assigned one of the following actions:

  • Blacklist — The device always blocks access to the websites in this list. Only user-defined categories are used with local Web filtering.

  • Whitelist — The device always allows access to the websites in this list. Only user-defined categories are used with local Web filtering.

A Web filtering profile can contain one blacklist or one whitelist with multiple user-defined categories each with a permit or block action. You can define a default fallback action when the incoming URL does not belong to any of the categories defined in the profile. If the action for the default category is block, the incoming URL is blocked if it does not match any of the categories explicitly defined in the profile. If an action for the default action is not specified, the default action of permit is applied to the incoming URL not matching any category.

Note

Starting with Junos OS Release 17.4R1, custom category configuration is supported for local Web filtering. The custom-message option is also supported in a category for local Web filtering and Websense redirect profiles. Users can create multiple URL lists (custom categories) and apply them to a UTM Web filtering profile with actions such as permit, permit and log, block, and quarantine. To create a global whitelist or blacklist, apply a local Web filtering profile to a UTM policy and attach it to a global rule.

User Messages and Redirect URLs for Web Filtering on SRX Series devices

Starting with Junos OS Release 17.4R1, a new option, custom-message, is added for the custom-objects statement that enables you to configure user messages and redirect URLs to notify users when a URL is blocked or quarantined for each EWF category. The custom-message option has the following mandatory attributes:

  • Name: Name of the custom message; maximum length is 59 ASCII characters.

  • Type: Type of custom message: user-message or redirect-url.

  • Content: Content of the custom message; maximum length is 1024 ASCII characters.

You configure a user message or redirect URL as a custom object and assign the custom object to an EWF category.

  • User messages indicate that website access has been blocked by an organization's access policy. To configure a user message, include the type user-message content message-text statement at the [edit security utm custom-objects custom-message message] hierarchy level.

  • Redirect URLs redirect a blocked or quarantined URL to a user-defined URL. To configure a redirect URL, include the type redirect-url content redirect-url statement at the [edit security utm custom-objects custom-message message] hierarchy level.

The custom-message option provides the following benefits:

  • You can configure a separate custom message or redirect URL for each EWF category.

  • The custom-message option enables you to fine-tune messages to support your polices to know which URL is blocked or quarantined.

Profile Matching Precedence

When a profile employs several categories for URL matching, those categories are checked for matches in the following order:

  1. If present, the global blacklist is checked first. If a match is made, the URL is blocked. If no match is found...
  2. The global whitelist is checked next. If a match is made, the URL is permitted. If no match is found...
  3. User-defined categories are checked next. If a match is made, the URL is blocked or permitted as specified.

Example: Configuring Local Web Filtering

This example shows how to configure local Web filtering for managing website access.

Requirements

This example uses the following hardware and software components:

  • SRX1500 device

  • Junos OS Release 12.1X46-D10 or later

Before you begin, learn more about Web filtering. See Web Filtering Overview.

Overview

In this example you configure local Web filtering custom objects, local Web filtering feature profiles, and local Web filtering UTM policies. You also attach local Web filtering UTM policies to security policies. Table 1 shows information about local Web filtering configuration type, steps, and parameters used in this example.

Table 1: Local Web filtering Configuration Type, Steps, and Parameters

Configuration Type

Configuration Steps

Configuration Parameters

URL pattern and custom objects

Configure a URL pattern list of URLs or addresses that you want to bypass.

Create a custom object called urllis1 that contains the pattern [http://www.example1.net 192.0.2.0]

Create a custom object called urllist2 that contains the pattern [http://www.example2.net 192.0.2.3]

Create a custom object called urllist3 that contains the pattern [http://www.example3.net 192.0.2.9]

Create a custom object called urllist4 that contains the pattern [http://www.example4.net 192.0.2.8]

  • [http://www.example1.net 192.0.2.0]

  • [http://www.example2.net 192.0.2.3]

  • [http://www.example3.net 192.0.2.9]

  • [http://www.example4.net 192.0.2.8]

  • value urllist3

  • value urllist4

The urllist1 and urllist2 custom objects are then added to the custom URL categories cust-black-list, and cust-permit-list respectively.

  • value urllist1

  • value urllist2

Feature profiles

Configure the Web filtering feature profile:

 
  • Set the URL blacklist filtering category to custurl4 and the URL whitelist filtering category to custurl3. Set the type of Web filtering engine to juniper-local.

  • custurl3

  • custurl4

  • type juniper-local

  • Create a juniper-local profile name called localprofile1. Select a default action (permit, log-and-permit, block) for this profile for requests that experience errors. This example sets the default action to permit. Add category cust-permit-list with log-and-permit action and cus-black-list with block action.

  • localprofile1

  • Action: block

  • Action: log-and-permit

  • cust-black-list

  • cust-permit-list

  • Define redirect url. Enter a custom message to be sent when HTTP requests are blocked.

  • Select fallback settings (block or log-and-permit) for this profile, in case errors occur in each configured category. This example sets fallback settings to block.

  • block-message type custom-redirect-url

  • block-message url 192.0.2.10

  • custom-block-message “**Access to this site is not permitted**”.

  • fallback-settings:

    • block

    • log-and-permit

UTM policies

Create the UTM policy utmp5 and attach it to the profile localprofile1. In the final configuration example, attach the UTM policy utmp5 to the security policy p5.

  • utm policy utmp5

  • policy p5

Configuration

Configuring Local Web Filtering Custom Objects and URL Patterns

CLI Quick Configuration

To quickly configure this section of the example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Starting in Junos OS Release 15.1X49-D110, the “* “ in a wildcard syntax, used for URL pattern Web filtering profile, matches all subdomains. For example, *.example.net matches:

  • http://a.example.net

  • http://example.net

  • aaa.example.net

Step-by-Step Procedure

To configure local Web filtering using the CLI:

  1. Configure a URL pattern list custom object by creating the list name and adding values to it as follows:Note

    Because you use URL pattern lists to create custom URL category lists, you must configure URL pattern list custom objects before you configure custom URL category lists.

    Note
    • The guideline to use a URL pattern wildcard is as follows: Use \*\.[]\?* and precede all wildcard URLs with http://. You can use “*” only if it is at the beginning of the URL and is followed by “.”. You can use “?” only at the end of the URL.

    • The following wildcard syntaxes are supported: http://*.example.net, http://www.example.ne?, http://www.example.n??. The following wildcard syntaxes are not supported: *.example.???, http://*example.net, http://?.

  2. Applying the URL pattern to a custom URL category.

Results

From configuration mode, confirm your configuration by entering the show security utm custom-objects command. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

If you are done configuring the device, enter commit from configuration mode.

Apply Custom Objects to the Feature Profiles

CLI Quick Configuration

To quickly configure this section of the example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Step-by-Step Procedure

The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.

To configure local Web filtering feature profiles:

  1. Configure the Web filtering URL blacklist, URL whitelist, and the Web filtering engine.
  2. Create a profile name, and select a category from the included permit and blacklist categories. The custom category action could be block, permit, log-and-permit, and quarantine.
  3. Define a redirect URL server so that instead of the device sending a block page with plain text HTML, the device send an HTTP 302 redirect to this redirect server with special variables embedded in the HTTP redirect location field. These special variables are parsed by the redirect server and serve as a special block page to the client with images and a clear text format.
  4. Enter a custom message to be sent when HTTP requests are blocked.
  5. Specify a default action (permit, log and permit, block, or quarantine) for the profile, when no other explicitly configured action (blacklist, whitelist, custom category, predefined category actions, or site reputation actions) is matched .
  6. Configure fallback settings (block or log and permit) for this profile.

Results

From configuration mode, confirm your configuration by entering the show security utm feature-profile command. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

If you are done configuring the device, enter commit from configuration mode.

Attaching Web Filtering UTM Policies to Security Policies

CLI Quick Configuration

To quickly configure this section of the example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Step-by-Step Procedure

To configure a UTM policy:

  1. Create the UTM policy referencing a profile. Apply the Web filtering profile to the UTM policy.

Results

From configuration mode, confirm your configuration by entering the show security utm command. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

For brevity, this show command output includes only the configuration that is relevant to this example. Any other configuration on the system has been replaced with ellipses (...).

If you are done configuring the device, enter commit from configuration mode.

Attaching Local Web Filtering UTM Policies to Security Policies

CLI Quick Configuration

To quickly configure this section of the example, copy the following commands, paste them into a text file, remove any line breaks, change any details necessary to match your network configuration, copy and paste the commands into the CLI at the [edit] hierarchy level, and then enter commit from configuration mode.

Step-by-Step Procedure

To attach a UTM policy to a security policy:

  1. Create and configure the security policy.
  2. Apply the UTM policy to the security policy.

Results

From configuration mode, confirm your configuration by entering the show security policies command. If the output does not display the intended configuration, repeat the configuration instructions in this example to correct it.

If you are done configuring the device, enter commit from configuration mode.

Verification

To confirm that the configuration is working properly, perform the following task:

Verifying the Statistics of UTM Web Filtering

Purpose

Verify the Web filtering statistics for connections including whitelist and blacklist hits and custom category hits.

Action

From operational mode, enter the show security utm web-filtering statistics command.

Sample Output

user@host>show security utm web-filtering statistics
Release History Table
Release
Description
Starting with Junos OS Release 17.4R1, custom category configuration is supported for local Web filtering. The custom-message option is also supported in a category for local Web filtering and Websense redirect profiles. Users can create multiple URL lists (custom categories) and apply them to a UTM Web filtering profile with actions such as permit, permit and log, block, and quarantine. To create a global whitelist or blacklist, apply a local Web filtering profile to a UTM policy and attach it to a global rule.
Starting with Junos OS Release 17.4R1, a new option, custom-message, is added for the custom-objects statement that enables you to configure user messages and redirect URLs to notify users when a URL is blocked or quarantined for each EWF category.
Starting in Junos OS Release 15.1X49-D110, the “* “ in a wildcard syntax, used for URL pattern Web filtering profile, matches all subdomains.