Integrated Web Filtering
Enhanced Web Filtering (EWF) with Websense is an integrated URL filtering solution. When you enable the solution on the device, the firewall intercepts the HTTP and the HTTPS requests and sends the HTTP URL or the HTTPS source IP to the Websense ThreatSeeker Cloud (TSC). For more information, see the following topics:
Understanding Integrated Web Filtering
The Integrated Web Filtering is not supported from Junos OS Release 15.1X49-D10 and Junos OS Release 17.3R1 onwards. For previous releases, with integrated Web filtering, the firewall intercepts every HTTP request in a TCP connection and extracts the URL from the HTTP request. Each individual HTTP request is blocked or permitted based on URL filtering profiles defined by you. The decision making is done on the device after it identifies a category for a URL.
The Surf-Control feature is not supported from Junos OS Release 15.1X49-D10 and Junos OS Release 17.3R1 onwards.
A URL category is a list of URLs grouped by content. URL categories are predefined and maintained by Surf-Control or are defined by you. Surf-Control maintains about 40 predefined categories. When defining your own URL categories, you can group URLs and create categories specific to your needs.
You define your own categories using URL pattern list and custom URL category list custom objects. Once defined, you can select your categories when you configure your Web filtering profile. Each category can have a maximum of 20 URLs. When you create a category, you can add either the URL or the IP address of a site. When you add a URL to a user-defined category, the device performs DNS lookup, resolves the host name into IP addresses, and caches this information. When a user tries to access a site with the IP address of the site, the device checks the cached list of IP addresses and tries to resolve the hostname. Many sites have dynamic IP addresses, meaning that their IP addresses change periodically. A user attempting to access a site can type an IP address that is not in the cached list on the device. Therefore, if you know the IP addresses of sites you are adding to a category, enter both the URL and the IP address(es) of the site.
If a URL appears in both a user-defined category and a predefined category, the device matches the URL to the user-defined category.
Web filtering is performed on all the methods defined in HTTP 1.0 and HTTP 1.1.
The integrated Web filtering solution intercepts every HTTP request in a TCP connection. In this case, the decision making is done on the device after it identifies the category for a URL either from user-defined categories or from a category server (SurfControl Content Portal Authority provided by Websense). The Integrated Web filtering is not supported from Junos OS Release 15.1X49-D10 onwards.
The integrated Web filtering feature is a separately licensed subscription service. When the license key for Web filtering has expired, no URLs are sent to the category server for checking, only local user-defined categories are checked.
Integrated Web filtering solution is supported only on SRX210, SRX220, SRX240, SRX550, and SRX650 devices.
This topic contains the following sections:
- Integrated Web Filtering Process
- Integrated Web Filtering Cache
- Integrated Web Filtering Profiles
- Profile Matching Precedence
Integrated Web Filtering Process
This is a general description of how Web traffic is intercepted and acted upon by the Web filtering module.
The device intercepts a TCP connection.
The device intercepts each HTTP request in the TCP connection.
The device extracts each URL in the HTTP request and checks its URL filter cache.
Global Web filtering allowlists and blocklists are checked first for block or permit.
If the HTTP request URL is allowed based on cached parameters, it is forwarded to the webserver. If there is no cache match, a request for categorization is sent to the SurfControl server. (If the HTTP request URL is blocked, the request is not forwarded and a notification message is logged.)
In the allowed case, the SurfControl server responds with the corresponding category.
Based on the identified category, if the URL is permitted, the device forwards the HTTP request to the webserver. If the URL is not permitted, then a deny page is sent to the HTTP client.
Integrated Web Filtering Cache
By default, the device retrieves and caches the URL categories from the SurfControl CPA server. This process reduces the overhead of accessing the SurfControl CPA server each time the device receives a new request for previously requested URLs. You can configure the size and duration of the cache, according to the performance and memory requirements of your networking environment. The lifetime of cached items is configurable between 1 and 1800 seconds with a default value of 300 seconds.
Caches are not preserved across device reboots or power losses.
Integrated Web Filtering Profiles
You configure Web filtering profiles that permit or block URLs according to defined categories. A Web filtering profile consists of a group of URL categories assigned one of the following actions:
Permit — The device always allows access to the websites in this category.
Block — The device blocks access to the websites in this category. When the device blocks access to this category of websites, it displays a message in your browser indicating the URL category.
Blocklist — The device always blocks access to the websites in this list. You can create a user-defined category.
Allowlist — The device always allows access to the websites in this list. You can create a user-defined category.
A predefined profile is provided and can be used if you choose not to define your own profile.
A Web filtering profile may contain one blocklist or one allowlist, multiple user-defined and/or predefined categories each with a permit or block action, and an Other category with a permit or block action. You can define an action for all Other categories in a profile to specify what to do when the incoming URL does not belong to any of the categories defined in the profile. If the action for the Other category is block, the incoming URL is blocked if it does not match any of the categories explicitly defined in the profile. If an action for the Other category is not specified, the default action of permit is applied to the incoming URL not matching any category.
Profile Matching Precedence
When a profile employs several categories for URL matching, those categories are checked for matches in the following order:
If present, the global blocklist is checked first. If a match is made, the URL is blocked. If no match is found...
The global allowlist is checked next. If a match is made, the URL is permitted. If no match is found...
User-defined categories are checked next. If a match is made, the URL is blocked or permitted as specified. If no match is found...
Predefined categories are checked next. If a match is made, the URL is blocked or permitted as specified. If no match is found...
The Other category is checked next. If a match is made, the URL is blocked or permitted as specified.
See Also
Example: Configuring Integrated Web Filtering
The Integrated Web Filtering is not supported from Junos OS Release 15.1X49-D10 and Junos OS Release 17.3R1 onwards. For previous releases, this example shows how to configure integrated Web filtering.
Requirements
Before you begin, learn more about Web filtering. See Web Filtering Overview.
Overview
In this example you configure integrated Web filtering custom objects, integrated Web filtering feature profiles, and integrated Web filtering UTM policies. You also attach integrated Web filtering UTM policies to security policies.
In the first example configuration you create a custom object called urllist3 that contains the pattern http://www.example.net 1.2.3.4. The urllist3 custom object is then added to the custom URL category custurl3.
In the second example configuration, you configure the Web filtering
feature profile. You set the URL blocklist filtering category to custblacklist, set the allowlist filtering category to custwhitelist and the type of Web filtering engine to surf-control-integrated.
Then you set the cache size parameters for Web filtering to 500 KB,
which is the default, and the cache timeout parameters to 1800.
You name the Surf Control server as surfcontrolserver and enter 8080 as the port number for communicating with it. (Default ports are 80, 8080, and 8081.) Then you create a surf-control-integrated profile name called surfprofile1.
Next you select a category from the included allowlist and blocklist categories or select a custom URL category list you created for filtering against. Then you enter an action (permit, log and permit, block) to go with the filter. You do this as many times as necessary to compile your allowlists and blocklists and their accompanying actions. This example blocks URLs in the custurl3 category.
Then you enter a custom message to be sent when HTTP requests are blocked. This example configures the device to send an ***access denied*** message. You select a default action (permit, log and permit, block) for this profile for requests that experience errors. This example sets the default action to block. You select fallback settings (block or log and permit) for this profile, in case errors occur in each configured category. This example sets fallback settings to block.
Finally, you enter a timeout value in seconds. Once this limit is reached, fail mode settings are applied. The default is 10 seconds, and you can enter a value from 10 to 240 seconds. This example sets the timeout value to 10.
In the third example configuration, you create UTM policy utmp5 and attach it to profile surfprofile1.
In the final example configuration, you attach the UTM policy utmp5 to the security policy p5.
Configuration
- Configuring Integrated Web Filtering Custom Objects
- Configuring the Integrated Web Filtering Feature Profiles
- Configuring Integrated Web Filtering UTM Policies
- Attaching Integrated Web Filtering UTM Policies to Security Policies
Configuring Integrated Web Filtering Custom Objects
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm custom-objects url-pattern urllist3 value http://www.example.net set security utm custom-objects url-pattern urllist3 value 1.2.3.4 set security utm custom-objects url-pattern urllistblack value http://www.untrusted.com set security utm custom-objects url-pattern urllistblack value 13.13.13.13 set security utm custom-objects url-pattern urllistwhite value http://www.trusted.com set security utm custom-objects url-pattern urllistwhite value 7.7.7.7 set security utm custom-objects custom-url-category custurl3 value urllist3 set security utm custom-objects custom-url-category custblacklist value urllistblack set security utm custom-objects custom-url-category custwhiltelist value urllistwhite
Custom category does not take precedence over predefined categories when it has the same name as one of the predefined categories. We do not recommend having a custom category name be the same as the predefined category name.
Step-by-Step Procedure
To configure integrated Web filtering:
Create custom objects and create the URL pattern list.
[edit security utm] user@host# set custom-objects url-pattern urllist3 value [http://www.example.net 1.2.3.4]
Configure the custom URL category list custom object using the URL pattern list.
[edit security utm] user@host# set custom-objects custom-url-category custurl3 value urllist3
Create a list of untrusted sites.
[edit security utm] user@host# set custom-objects url-pattern urllistblack value [http://www.untrusted.com 13.13.13.13]
Configure the custom URL category list custom object using the URL pattern list of untrusted sites.
[edit security utm] user@host# set custom-objects custom-url-category custblacklist value urllistblack
Create a list of trusted sites.
[edit security utm] user@host# set custom-objects url-pattern urllistwhite value [http://www.trusted.com 7.7.7.7]
Configure the custom URL category list custom object using the URL pattern list of trusted sites.
[edit security utm] user@host# set custom-objects custom-url-category custwhitelist value urllistwhite
Results
From configuration mode, confirm your configuration
by entering the show security utm custom-objects command.
If the output does not display the intended configuration, repeat
the configuration instructions in this example to correct it.
For brevity, this show command output includes only the configuration that is relevant to this example. Any other configuration on the system has been replaced with ellipses (...).
[edit]
userhost#show security utm custom-objects
url-pattern {
urllist3 {
value [ http://www.example.net ];
}
urllistblack {
value [ http://www.untrusted.com 13.13.13.13 ];
}
urllistwhite {
value [ http://www.trusted.com 7.7.7.7 ];
}
}
custom-url-category {
custurl3 {
value urllist3;
}
custblacklist {
value urllistblack;
}
custwhiltelist {
value urllistwhite;
}
}
If you are done configuring the device, enter commit from configuration mode.
Configuring the Integrated Web Filtering Feature Profiles
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm feature-profile web-filtering url-whitelist custwhitelist set security utm feature-profile web-filtering url-blacklist custblacklist set security utm feature-profile web-filtering surf-control-integrated cache timeout 1800 set security utm feature-profile web-filtering surf-control-integrated cache size 500 set security utm feature-profile web-filtering surf-control-integrated server host surfcontrolserver set security utm feature-profile web-filtering surf-control-integrated server port 8080 set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 category custurl3 action block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 default block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 custom-block-message "***access denied ***" set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 fallback-settings default block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 fallback-settings server-connectivity block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 fallback-settings timeout block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 fallback-settings too-many-requests block set security utm feature-profile web-filtering surf-control-integrated profile surfprofile1 timeout 10 set security utm feature-profile content-filtering profile contentfilter1
Step-by-Step Procedure
The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.
To configure integrated Web filtering feature profiles:
Configure the Web filtering URL Blocklist.
[edit security utm feature-profile web-filtering] user@host# set url-blacklist custblacklist
Configure the Web filtering URL Allowlist.
[edit security utm feature-profile web-filtering] user@host# set url-whitelist custwhitelist
Specify the surf-control-integrated Web filtering engine and set the cache size parameters.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated cache size 500
Set the cache timeout parameters.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated cache timeout 1800
Set the server name or IP address.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated server host surfcontrolserver
Enter the port number for communicating with the server.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated server port 8080
Create a profile name and select a category from the included allowlist and blocklist categories.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated profile surfprofile1 category custurl3 action block
Enter a custom message to be sent when HTTP requests are blocked.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated profile surfprofile1 custom-block-message “***access denied***”
Select a default action (permit, log and permit, block) for this profile for requests that experience errors.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated profile surfprofile1 default block
Select fallback settings (block or log and permit) for this profile.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated profile surfprofile1 fallback-settings default block
user@host# set surf-control-integrated profile surfprofile1 fallback-settings server-connectivity block user@host# set surf-control-integrated profile surfprofile1 fallback-settings timeout block user@host# set surf-control-integrated profile surfprofile1 fallback-settings too-many-requests block
Enter a timeout value, in seconds.
[edit security utm feature-profile web-filtering] user@host# set surf-control-integrated profile surfprofile1 timeout 10
Results
From configuration mode, confirm your configuration
by entering the show security utm feature-profile command.
If the output does not display the intended configuration, repeat
the configuration instructions in this example to correct it.
[edit]
userhost#show security utm feature-profile
web-filtering {
url-whitelist custwhitelist;
url-blacklist custblacklist;
type juniper-local;
surf-control-integrated {
cache {
timeout 1800;
size 500;
}
server {
host surfcontrolserver;
port 8080;
}
profile surfprofile1 {
category {
custurl3 {
action block;
}
}
default block;
custom-block-message "***access denied ***";
fallback-settings {
default block;
server-connectivity block;
timeout block;
too-many-requests block;
}
timeout 10;
}
}
content-filtering {
profile contentfilter1;
}
If you are done configuring the device, enter commit from configuration mode.
Configuring Integrated Web Filtering UTM Policies
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm utm-policy utmp5 web-filtering http-profile surfprofile1
Step-by-Step Procedure
To configure a UTM policy:
Create the UTM policy referencing a profile.
[edit] user@host# set security utm utm-policy utmp5 web-filtering http-profile surfprofile1
Results
From configuration mode, confirm your configuration
by entering the show security utm utm-policy command.
If the output does not display the intended configuration, repeat
the configuration instructions in this example to correct it.
[edit]
userhost#show security utm utm-policy
...
utm-policy utmp5 {
content-filtering {
http-profile contentfilter1;
}
web-filtering {
http-profile surfprofile1;
}
}
If you are done configuring the device, enter commit from configuration mode.
Attaching Integrated Web Filtering UTM Policies to Security Policies
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security policies from-zone trust to-zone untrust policy p5 match source-address any set security policies from-zone trust to-zone untrust policy p5 match destination-address any set security policies from-zone trust to-zone untrust policy p5 match application junos-http set security policies from-zone trust to-zone untrust policy p5 then permit application-services utm-policy utmp5
Step-by-Step Procedure
To attach a UTM policy to a security policy:
Create and configure the security policy.
[edit security policies from-zone trust to-zone untrust policy p5] user@host# set match source-address any user@host# set match destination-address any user@host# set match application junos-http
Attach the UTM policy to the security policy.
[edit security policies from-zone trust to-zone untrust policy p5] user@host# set then permit application-services utm-policy utmp5
Results
From configuration mode, confirm your configuration
by entering the show security policies command. If the
output does not display the intended configuration, repeat the configuration
instructions in this example to correct it.
[edit]
userhost#show security policies
from-zone trust to-zone untrust {
policy p5 {
match {
source-address any;
destination-address any;
application junos-http;
}
then {
permit {
application-services {
utm-policy utmp5;
}
}
}
}
}
If you are done configuring the device, enter commit from configuration mode.
Verification
To confirm that the configuration is working properly, perform these tasks:
- Verifying the Configuration of Integrated Web Filtering Custom Objects
- Verifying the Configuration of Integrated Web Filtering Feature Profiles
- Verifying the Configuration of Integrated Web Filtering UTM Policies
- Verifying the Attachment of Integrated Web Filtering UTM Policies to Security Policies
Verifying the Configuration of Integrated Web Filtering Custom Objects
Purpose
Verify the configuration of integrated Web filtering custom objects.
Action
From the top of the configuration in configuration mode,
enter the show security utm custom-objects command.
Verifying the Configuration of Integrated Web Filtering Feature Profiles
Purpose
Verify the configuration of integrated Web filtering feature profiles.
Action
From the top of the configuration in configuration mode,
enter the show security utm feature-profile command.
Verifying the Configuration of Integrated Web Filtering UTM Policies
Purpose
Verify the configuration of integrated Web filtering UTM policies.
Action
From the top of the configuration in configuration mode,
enter the show security utm command.
Displaying Global SurfControl URL Categories
Purpose
The Surf-Control feature is not supported from Junos OS Release 15.1X49-D10 and Junos OS Release 17.3R1 onwards. For previous releases, view global URL categories defined and maintained by SurfControl.
Action
Enter the user@host# show groups junos-defaults CLI command. You can also look for custom-url-category.