Local Web Filtering
The Web filtering lets you to manage Internet usage by preventing access to inappropriate Web content. There are four types of Web filtering solutions. For more information, see the following topics:
Understanding Local Web Filtering
Local web filtering allows you to define custom URL categories, which can be included in blocklists and allowlists that are evaluated on the SRX Series Firewall. All URLs for each category in a blocklist are denied, while all URLs for each category in a allowlist are permitted.
With local Web filtering, a firewall intercepts every HTTP and HTTPS request in a TCP connection and extracts the URL. A decision is made by the device after it looks up a URL to determine whether it is in the allowlist or blocklist based on its user-defined category. A URL is first compared to the blocklist URLs. If a match is found, the request is blocked. If no match is found, the URL is compared to the allowlist. If a match is found, the request is permitted. If the URL is not in either list, the custom category is taken (block, log-and-permit, or permit). If the URL is not in custom category, the defined default action is taken (block, log-and-permit, or permit). You can permit or block access to a requested site by binding a Web filtering profile to a firewall policy. Local Web filtering provides basic Web filtering without requiring an additional license or external category server.
This topic contains the following sections:
- Local Web Filtering Process
- User-Defined Custom URL Categories
- Local Web Filtering Profiles
- User Messages and Redirect URLs for Web Filtering
- Profile Matching Precedence
Local Web Filtering Process
The following section describes on how Web traffic is intercepted and acted upon by the Web filtering module.
The device intercepts a TCP connection.
-
The device intercepts each HTTP and HTTPS request in the TCP connection.
-
The device extracts each URL in the HTTP and HTTPS request and checks its URL against the user-defined allowlist and blocklist.
-
If the URL is found in the blocklist, the request is not permitted and a deny page is sent to the http or https client. If the URL is found in the allowlist, the request is permitted.
If the URL is not found in the allowlist or blocklist, the configured default fallback action is applied. If no fallback action is defined, then the request is permitted.
User-Defined Custom URL Categories
To perform local Web filtering, you must define a blocklist and allowlist content that can be applied to the profile.
When defining your own URL categories, you can group URLs and create categories specific to your needs. Each category can have a maximum of 20 URLs. When you create a category, you can add either the URL or the IP address of a site. When you add a URL to a user-defined category, the device performs DNS lookup, resolves the hostname into IP addresses, and caches this information. When a user tries to access a site with the IP address of the site, the device checks the cached list of IP addresses and tries to resolve the hostname. Many sites have dynamic IP addresses, meaning that their IP addresses change periodically. A user attempting to access a site can type an IP address that is not in the cached list on the device. Therefore, if you know the IP addresses of sites you are adding to a category, enter both the URL and the IP address(es) of the site.
You define your own categories using URL pattern list and custom URL category list custom objects. Once defined, you assign your categories to the global user-defined url-blocklist (block) or url-allowlist (permit) categories.
Web filtering is performed on all the methods defined in HTTP 1.0 and HTTP 1.1.
Local Web Filtering Profiles
You configure Web filtering profiles that permit or block URLs according to defined custom categories. A Web filtering profile consists of a group of URL categories assigned one of the following actions:
Blocklist — The device always blocks access to the websites in this list. Only user-defined categories are used with local Web filtering.
Allowlist — The device always allows access to the websites in this list. Only user-defined categories are used with local Web filtering.
A Web filtering profile can contain one blocklist or one allowlist with multiple user-defined categories each with a permit or block action. You can define a default fallback action when the incoming URL does not belong to any of the categories defined in the profile. If the action for the default category is block, the incoming URL is blocked if it does not match any of the categories explicitly defined in the profile. If an action for the default action is not specified, the default action of permit is applied to the incoming URL not matching any category.
Starting with Junos OS Release 17.4R1, custom category configuration is supported for local Web
filtering. The custom-message option is also supported in a category for
local Web filtering and Websense redirect profiles. Users can create multiple URL lists
(custom categories) and apply them to a Content Security Web filtering profile with actions
such as permit, permit and log, block, and quarantine. To create a global allowlist or
blocklist, apply a local Web filtering profile to a Content Security policy and attach it to
a global rule.
User Messages and Redirect URLs for Web Filtering
Starting with Junos
OS Release 17.4R1, a new option, custom-message, is added
for the custom-objects statement that enables you to configure
user messages and redirect URLs to notify users when a URL is blocked
or quarantined for each EWF category. The custom-message option has the following mandatory attributes:
Name: Name of the custom message; maximum length is 59 ASCII characters.
Type: Type of custom message:
user-messageorredirect-url.Content: Content of the custom message; maximum length is 1024 ASCII characters.
You configure a user message or redirect URL as a custom object and assign the custom object to an EWF category.
User messages indicate that website access has been blocked by an organization's access policy. To configure a user message, include the
type user-message content message-textstatement at the[edit security utm custom-objects custom-message message]hierarchy level.Redirect URLs redirect a blocked or quarantined URL to a user-defined URL. To configure a redirect URL, include the
type redirect-url content redirect-urlstatement at the[edit security utm custom-objects custom-message message]hierarchy level.
The custom-message option provides the following
benefits:
You can configure a separate custom message or redirect URL for each EWF category.
The
custom-messageoption enables you to fine-tune messages to support your polices to know which URL is blocked or quarantined.
Profile Matching Precedence
When a profile employs several categories for URL matching, those categories are checked for matches in the following order:
If present, the global blocklist is checked first. If a match is made, the URL is blocked. If no match is found...
The global allowlist is checked next. If a match is made, the URL is permitted. If no match is found...
User-defined categories are checked next. If a match is made, the URL is blocked or permitted as specified.
See Also
Example: Configuring Local Web Filtering
This example shows how to configure local Web filtering for managing website access.
Requirements
This example uses the following hardware and software components:
SRX1500 device
Junos OS Release 12.1X46-D10 or later
Before you begin, learn more about Web filtering. See Web Filtering Overview.
Overview
In this example you configure local Web filtering custom objects, local Web filtering feature profiles, and local Web filtering Content Security policies. You also attach local Web filtering Content Security policies to security policies. Table 1 shows information about local Web filtering configuration type, steps, and parameters used in this example.
Configuration Type |
Configuration Steps |
Configuration Parameters |
|---|---|---|
URL pattern and custom objects |
Configure a URL pattern list of URLs or addresses that you want to bypass. Create a custom object called urllis1 that contains the pattern [http://www.example1.net 192.0.2.0] Create a custom object called urllist2 that contains the pattern [http://www.example2.net 192.0.2.3] Create a custom object called urllist3 that contains the pattern [http://www.example3.net 192.0.2.9] Create a custom object called urllist4 that contains the pattern [http://www.example4.net 192.0.2.8] |
|
The urllist1 and urllist2 custom objects are then added to the custom URL categories cust-blocklist, and cust-permit-list respectively. |
|
|
Feature profiles |
Configure the Web filtering feature profile: |
|
|
|
|
|
|
|
|
|
|
Content Security policies |
Create the Content Security policy |
|
Configuration
- Configuring Local Web Filtering Custom Objects and URL Patterns
- Apply Custom Objects to the Feature Profiles
- Attaching Web Filtering Content Security Policies to Security Policies
- Attaching Local Web Filtering Content Security Policies to Security Policies
Configuring Local Web Filtering Custom Objects and URL Patterns
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm custom-objects url-pattern urllist1 value http://www.example1.net set security utm custom-objects url-pattern urllist1 value 192.0.2.0 set security utm custom-objects url-pattern urllist2 value http://www.example2.net set security utm custom-objects url-pattern urllist2 value 192.0.2.3 set security utm custom-objects url-pattern urllist3 value http://www.example3.net set security utm custom-objects url-pattern urllist3 value 192.0.2.9 set security utm custom-objects url-pattern urllist4 value http://www.example4.net set security utm custom-objects url-pattern urllist4 value 192.0.2.8 set security utm custom-objects custom-url-category cust-black-list value urllist1 set security utm custom-objects custom-url-category cust-permit-list value urllist2 set security utm custom-objects custom-url-category custurl3 value urllist3 set security utm custom-objects custom-url-category custurl4 value urllist4
Starting in Junos OS Release 15.1X49-D110, the “* “ in a wildcard syntax, used for URL pattern Web filtering profile, matches all subdomains. For example, *.example.net matches:
http://a.example.net
http://example.net
aaa.example.net
Step-by-Step Procedure
To configure local Web filtering using the CLI:
Configure a URL pattern list custom object by creating the list name and adding values to it as follows:
Note:Because you use URL pattern lists to create custom URL category lists, you must configure URL pattern list custom objects before you configure custom URL category lists.
[edit] user@host# set security utm custom-objects url-pattern urllist1 value [http://www.example1.net 192.0.2.0] user@host# set security utm custom-objects url-pattern urllist2 value [http://www.example2.net 192.0.2.3] user@host# set security utm custom-objects url-pattern urllist3 value [http://www.example3.net 192.0.2.9] user@host# set security utm custom-objects url-pattern urllist4 value [http://www.example4.net 192.0.2.8]
Note:The guideline to use a URL pattern wildcard is as follows: Use \*\.[]\?* and precede all wildcard URLs with http://. You can use “*” only if it is at the beginning of the URL and is followed by “.”. You can use “?” only at the end of the URL.
The following wildcard syntaxes are supported: http://*.example.net, http://www.example.ne?, http://www.example.n??. The following wildcard syntaxes are not supported: *.example.???, http://*example.net, http://?.
Applying the URL pattern to a custom URL category.
[edit] user@host# set security utm custom-objects custom-url-category cust-black-list value urllist1 user@host# set security utm custom-objects custom-url-category cust-permit-list value urllist2 user@host# set security utm custom-objects custom-url-category custurl3 value urllist3 user@host# set security utm custom-objects custom-url-category custurl4 value urllist4
Results
From configuration mode, confirm your configuration
by entering the show security utm custom-objects command.
If the output does not display the intended configuration, repeat
the configuration instructions in this example to correct it.
[edit]
userhost#show security utm custom-objects
url-pattern {
urllist1 {
value [ http://www.example1.net 192.0.2.0 ];
}
urllist2 {
value [ http://www.example2.net 192.0.2.3 ];
}
urllist3 {
value [ http://www.example3.net 192.0.2.9 ];
}
urllist4 {
value [ http://www.example4.net 192.0.2.8 ];
}
}
custom-url-category {
cust-black-list {
value urllist1;
}
cust-permit-list {
value urllist2;
}
custurl3 {
value urllist3;
}
custurl4 {
value urllist4;
}
}
If you are done configuring the device, enter commit from configuration mode.
Apply Custom Objects to the Feature Profiles
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm feature-profile web-filtering juniper-local profile localprofile1 category cust-black-list action block set security utm feature-profile web-filtering juniper-local profile localprofile1 category cust-permit-list action log-and-permit set security utm feature-profile web-filtering juniper-local profile localprofile1 block-message type custom-redirect-url set security utm feature-profile web-filtering juniper-local profile localprofile1 block-message url http://192.0.2.10 set security utm feature-profile web-filtering juniper-local profile localprofile1 custom-block-message "Access to this site is not permitted." set security utm feature-profile web-filtering juniper-local profile localprofile1 default log-and-permit set security utm feature-profile web-filtering juniper-local profile localprofile1 fallback-settings default block set security utm feature-profile web-filtering juniper-local profile localprofile1 fallback-settings too-many-requests block
Step-by-Step Procedure
The following example requires you to navigate various levels in the configuration hierarchy. For instructions on how to do that, see Using the CLI Editor in Configuration Mode in the CLI User Guide.
To configure local Web filtering feature profiles:
Create a profile name, and select a category from the included permit and blocklist categories. The custom category action could be block, permit, log-and-permit, and quarantine.
[edit security utm feature-profile web-filtering] user@host# set juniper-local profile localprofile1 category cust-black-list action block user@host# set juniper-local profile localprofile1 category cust-permit-list action log-and-permit
Define a redirect URL server so that instead of the device sending a block page with plain text HTML, the device send an HTTP 302 redirect to this redirect server with special variables embedded in the HTTP redirect location field. These special variables are parsed by the redirect server and serve as a special block page to the client with images and a clear text format.
[edit security utm feature-profile web-filtering] user@host# set security utm feature-profile web-filtering juniper-local profile localprofile1 block-message type custom-redirect-url user@host# set security utm feature-profile web-filtering juniper-local profile localprofile1 block-message url http://192.0.2.10
-
Enter a custom message to be sent when HTTP or HTTPS requests are blocked.
[edit security utm feature-profile web-filtering] user@host# set juniper-local profile localprofile1 custom-block-message “Access to this site is not permitted”
Specify a default action (permit, log and permit, block, or quarantine) for the profile, when no other explicitly configured action (blocklist, allowlist, custom category, predefined category actions, or site reputation actions) is matched .
[edit security utm feature-profile web-filtering] user@host# set juniper-local profile localprofile1 default log-and-permit
Configure fallback settings (block or log and permit) for this profile.
[edit security utm feature-profile web-filtering] user@host# set juniper–local profile localprofile1 fallback-settings default block user@host# set juniper–local profile localprofile1 fallback-settings too-many-requests block
Results
From configuration mode, confirm your configuration
by entering the show security utm feature-profile command.
If the output does not display the intended configuration, repeat
the configuration instructions in this example to correct it.
[edit]
userhost#show security utm feature-profile
web-filtering {
juniper-local {
profile localprofile1 {
default log-and-permit;
category {
cust-black-list {
action block;
}
cust-permit-list {
action log-and-permit;
}
}
custom-block-message "Access to this site is not permitted.";
block-message {
type custom-redirect-url;
url http://192.0.2.10;
}
fallback-settings {
default block;
too-many-requests block;
}
}
}
}
}
}
If you are done configuring the device, enter commit from configuration mode.
Attaching Web Filtering Content Security Policies to Security Policies
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security utm utm-policy utmp5 web-filtering http-profile localprofile1
Step-by-Step Procedure
To configure a Content Security policy:
-
Create the Content Security policy referencing a profile. Apply the Web filtering profile to the Content Security policy.
[edit] user@host# set security utm utm-policy utmp5 web-filtering http-profile localprofile1
Results
From configuration mode, confirm your configuration
by entering the show security utm command. If the output
does not display the intended configuration, repeat the configuration
instructions in this example to correct it.
For brevity, this show command output includes only the configuration that is relevant to this example. Any other configuration on the system has been replaced with ellipses (...).
[edit]
userhost# show security utm
utm-policy utmp5 {
web-filtering {
http-profile localprofile1;
}
}
If you are done configuring the device, enter commit from configuration mode.
Attaching Local Web Filtering Content Security Policies to Security Policies
CLI Quick Configuration
To quickly configure this section of the example,
copy the following commands, paste them into a text file, remove any
line breaks, change any details necessary to match your network configuration,
copy and paste the commands into the CLI at the [edit] hierarchy
level, and then enter commit from configuration mode.
set security policies from-zone trust to-zone untrust policy p5 match source-address any set security policies from-zone trust to-zone untrust policy p5 match destination-address any set security policies from-zone trust to-zone untrust policy p5 match application junos-http set security policies from-zone trust to-zone untrust policy p5 then permit application-services utm-policy utmp5
Step-by-Step Procedure
To attach a Content Security policy to a security policy:
Create and configure the security policy.
[edit security policies from-zone trust to-zone untrust policy p5] user@host# set match source-address any user@host# set match destination-address any user@host# set match application junos-http
-
Apply the Content Security policy to the security policy.
[edit security policies from-zone trust to-zone untrust policy p5] user@host# set then permit application-services utm-policy utmp5
Results
From configuration mode, confirm your configuration
by entering the show security policies command. If the
output does not display the intended configuration, repeat the configuration
instructions in this example to correct it.
[edit]
userhost# show security policies
from-zone trust to-zone untrust {
policy p5 {
match {
source-address any;
destination-address any;
application junos-http;
}
then {
permit {
application-services {
utm-policy utmp5;
}
}
}
}
}
If you are done configuring the device, enter commit from configuration mode.
Verification
To confirm that the configuration is working properly, perform the following task:
Verifying the Statistics of Content Security Web Filtering
Purpose
Verify the Web filtering statistics for connections including allowlist and blocklist hits and custom category hits.
Action
From operational mode, enter the show security
utm web-filtering statistics command.
Sample Output
command-name
user@host>show security utm web-filtering statistics
UTM web-filtering statistics:
Total requests: 0
white list hit: 0
Black list hit: 0
Custom category permit: 0
Custom category block: 0
Custom category quarantine: 0
Custom category qurantine block: 0
Custom category quarantine permit: 0
Web-filtering sessions in total: 0
Web-filtering sessions in use: 0
Fallback: log-and-permit block
Default 0 0
Timeout 0 0
Connectivity 0 0
Too-many-requests 0 0custom-message option is also supported in a
category for local Web filtering and Websense redirect profiles. Users can create multiple
URL lists (custom categories) and apply them to a Content Security Web filtering profile
with actions such as permit, permit and log, block, and quarantine. To create a global
allowlist or blocklist, apply a local Web filtering profile to a Content Security policy
and attach it to a global rule.custom-message, is added
for the custom-objects statement that enables you to configure
user messages and redirect URLs to notify users when a URL is blocked
or quarantined for each EWF category.