Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

URL Filtering

 

URL Filtering Overview

You can use URL filtering to determine which Web content is not accessible to users.

Components of this feature include the following:

  • URL filter database file

  • Configuration of one or more templates (up to eight per profile)

  • URL Filter Plug-in (jservices-urlf)

  • URL filtering daemon (url-filterd)

The URL filter database file is stored on the Routing Engine and contains all the disallowed URLs. Configured templates define which traffic to monitor, what criteria to match, and which actions to take. You configure the templates and the location of the URL filter database file in a profile.

Starting in Junos OS Release 17.2R2 and 17.4R1, for Adaptive Services, you can disable the filtering of HTTP traffic that contains an embedded IP address (for example, http:/10.1.1.1) belonging to a disallowed domain name in the URL filter database.Starting in Junos OS Release 19.3R2, this same functionaly is supported for Next Gen Services on MX240, MX480, and MX960.

To enable the URL filtering feature, you must configure jservices-urlf as the package-name at the [edit chassis fpc slot-number pic pic-number adaptive-services service-package extension-provider] hierarchy level. Once enabled, jservices-urlf maintains the URL filtering profile and receives all traffic to be filtered, the filtering criteria, and the action to be taken on the filtered traffic.

The URL filtering daemon (url-filterd), which also resides on the Routing Engine, resolves the domain name of each URL in the URL filter database to a list of IPv4 and IPv6 addresses. It then downloads the list of IP addresses to the service PIC, which runs jservices-urlf. Then url-filterd interacts with the Dynamic Firewall process (dfwd) to install filters on the Packet Forwarding Engine to punt the selected traffic from the Packet Forwarding Engine to the service PIC.

As new HTTP and HTTPS traffic reaches the router, a decision is made based on the information in the URL filter database file. The filtering rules are checked and either the router accepts the traffic and passes it on or blocks the traffic. If the traffic is blocked, one of the following configured actions is taken:

  • An HTTP redirect is sent to the user.

  • A custom page is sent to the user.

  • An HTTP status code is sent to the user.

  • A TCP reset is sent.

Accept is also an option. In this case, the traffic is not blocked.

For more details on the URL filtering feature, see the following sections:

URL Filter Database File

The URL filter database file contains entries of URLs and IP addresses. Create the URL filter database file in the format indicated in Table 1 and locate it on the Routing Engine in the /var/db/url-filterd directory.

Table 1: URL Filter Database File Format

Entry

Description

Example

FQDN

Fully qualified domain name.

www.badword.com/jjj/bad.jpg

URL

Full string URL without the Layer 7 protocol.

www.yahoo.com/*badword*/

IPv4 address

HTTP request on a specific IPv4 address.

10.1.1.199

IPv6 address

HTTP request on a specific IPv6 address.

1::1

You must specify a custom URL filter database in the profile. If needed, you can also assign a custom URL filter database file with any template, and that database takes precedence over the database configured at the profile level.

If you change the contents of the URL filter database file, use the request services (url-filter | web-filter) update command. Other commands to help maintain the URL filter database file include the following:

  • request services (url-filter | web-filter) delete

  • request services (url-filter | web-filter) force

  • request services (url-filter | web-filter) validate

URL Filter Profile Caveats

The URL filter profile consists of from one to eight templates. Each template consists of a set of configured logical interfaces where traffic is monitored for URL filtering and one or more terms.

A term is a set of match criteria with actions to be taken if the match criteria is met. You must configure at least one term to configure URL filtering. Each term consists of a from statement and a then statement, where the from statement defines the source IP prefixes and destination ports that are monitored. The then statement specifies the action to be taken. If you omit the from statement, any source IP prefix and any destination port are considered to match. But you can omit only one from statement per template or per profile.

Example configuration of multiple terms without from statements

If you omit more than one from statement per template, you will get the following error message on commit:

Configuring URL Filtering

URL filtering is configured on a service PIC. The interfaces you are dealing with are services interfaces (which use the ms prefix) or aggregated multiservices (AMS) interfaces (which use the ams prefix). For more information on AMS interfaces, see the Adaptive Services Interfaces User Guide for Routing Devices starting with Understanding Aggregated Multiservices Interfaces.

To configure the URL filtering feature, you must first configure jservices-urlf as the package-name at the [edit chassis fpc slot-number pic pic-number adaptive-services service-package extension-provider] hierarchy level. For more information on configuring the extension-provider package package-name configuration statement, see the package (Loading on PIC) statement.

A URL filtering profile is a collection of templates. Each template consists of a set of criteria that defines which URLs are disallowed and how the recipient is notified.

To configure the URL profile:

  1. Assign a name to the URL profile.

    Starting in Junos OS Release 18.3R1, for Adaptive Services. configure the profile at the [edit services web-filter] hierarchy level. Before Junos OS Release 18.3R1, configure the profile at the [edit services url-filter] hierarchy level.Starting in Junos OS Release 19.3R2, this same functionality is available for Next Gen Serices on MX240, MX480, and MX960.

  2. Specify the name of the URL filter database to use.
  3. Configure one or more templates for the profile.

    To configure each template:

    1. Name the template.
      Note

      Starting in Junos OS Release 18.3R1, configure the template with the url-filter-template statement. Before Junos OS Release 18.3R1, configure the template with the template statement.

    2. Go to that new template hierarchy level.
    3. Specify the name of the URL filter database to use.
    4. Specify the loopback interface for which the source IP address is picked for sending DNS queries.
    5. Disable the filtering of HTTP traffic that contains an embedded IP address (for example, http:/10.1.1.1) belonging to a disallowed domain name in the URL filter database.
    6. Configure the DNS resolution time interval in minutes.
    7. Configure the number of retries for a DNS query in case the query fails or times out.
    8. Specify the IP addresses (IPv4 or IPv6) of DNS servers to which the DNS queries are sent.
    9. Specify the client-facing logical interfaces on which the URL filtering is configured.
    10. Specify the server-facing logical interfaces on which the URL filtering is configured.
    11. Specify the routing instance on which the URL filtering is configured.
    12. Specify the routing instance on which the DNS server is reachable.
  4. Configure the term information.

    Terms are used in filters to segment the policy or filter into small match and action pairs.

    1. Name the term.
    2. Go to the new term hierarchy level.
    3. Specify the source IP address prefixes for traffic you want to filter.
    4. Specify the destination ports for traffic you want to filter.
    5. Configure an action to take.

      The action can be one of the following:

      custom-page custom-pageSend a custom page string to the user.
      http-status-code http-status-codeSend an HTTP status code to the user.
      redirect-url redirect-urlSend an HTTP redirect to the user.
      tcp-resetSend a TCP reset to the user.
  5. Associate the URL profile with a next-hop service set.Note

    For URL filtering, you must configure the service set as a next-hop service set.

    Note

    The service interface can also be of the ams prefix. If you are using ams interfaces at the [edit services service-set service-set-name] hierarchy level for the URL filter, you must also configure the load-balancing-options hash-keys statement at the [edit interfaces ams-interface-name unit number] hierarchy level. .

    Note

    Starting in Junos OS Release 18.3R1, configure the service set with the web-filter-profile statement. Before Junos OS Release 18.3R1, configure the service set with the url-filter-profile statement.

DNS Request Filtering for Disallowed Website Domains

Overview of DNS Request Filtering

Starting in Junos OS Release 18.3R1, you can configure DNS filtering to identify DNS requests for disallowed website domains. Starting in Junos OS Release 19.3R2, you can configure DNS filtering if you are running Next Gen Services with the MX-SPC3 services card. Next Gen Services are supported on MX240, MX480 and MX960 routers. For DNS request types A, AAAA, MX, CNAME, TXT, SRV, and ANY, you configure the action to take for a DNS request for a disallowed domain. You can either:

  • Block access to the website by sending a DNS response that contains the IP address or fully qualified domain name (FQDN) of a DNS sinkhole server. This ensures that when the client attempts to send traffic to the disallowed domain, the traffic instead goes to the sinkhole server (see Figure 1).

  • Log the request and allow access.

For other DNS request types for a disallowed domain, the request is logged and access is allowed.

The actions that the sinkhole server takes are not controlled by the DNS request filtering feature; you are responsible for configuring the sinkhole server actions. For example, the sinkhole server could send a message to the requestor that the domain is not reachable and prevent access to the disallowed domain.

Figure 1: DNS Request for Disallowed Domain
 DNS Request for Disallowed
Domain

Benefits

DNS filtering redirects DNS requests for disallowed website domains to sinkhole servers, while preventing anyone operating the system from seeing the list of disallowed domains. This is because the disallowed domain names are in an encrypted format.

Disallowed Domain Filter Database File

DNS request filtering requires a disallowed domain filter database .txt file, which identifies each disallowed domain name, the action to take on a DNS request for the disallowed domain, and the IP address or fully qualified domain name (FQDN) of a DNS sinkhole server.

DNS Filter Profile

You configure a DNS filter profile to specify which disallowed domain filter database file to use. You can also specify the interfaces on which DNS request filtering is performed, limit the filtering to requests for specific DNS servers, and limit the filtering to requests from specific source IP address prefixes.

How to Configure DNS Request Filtering

To filter DNS requests for disallowed website domains, perform the following:

How to Configure a Domain Filter Database

Create one or more domain filter database files that include an entry for each disallowed domain. Each entry specifies what to do with a DNS request for a disallowed website domain.

To configure a domain filter database file:

  1. Create the name for the file. The database file name can have a maximum length of 64 characters and must have a .txt extension.
  2. Add a file header with a format such as 20170314_01:domain,sinkhole_ip,v6_sinkhole,sinkhole_fqdn,id,action.
  3. Add an entry in the file for each disallowed domain. You can include a maximum of 10,000 domain entries. Each entry in the database file has the following items:

    hashed-domain-name,IPv4 sinkhole address, IPv6 sinkhole address, sinkhole FQDN, ID, action

    where:

    • hashed-domain-name is a hashed value of the disallowed domain name (64 hexadecimal characters). The hash method and hash key that you use to produce the hashed domain value are needed when you configure DNS filtering with the Junos OS CLI.

    • IPv4 sinkhole address is the address of the DNS sinkhole server for IPv4 DNS requests.

    • IPv6 sinkhole address is the address of the DNS sinkhole server for IPv6 DNS requests.

    • sinkhole FQDN is the fully qualified domain name of the DNS sinkhole server.

    • ID is a 32-bit number that uniquely associates the entry with the hashed domain name.

    • action is the action to apply to a DNS request that matches the disallowed domain name. If you enter replace, the MX Series router sends the client a DNS response with the IP address or FQDN of the DNS sinkhole server. If you enter report, the DNS request is logged and then sent to the DNS server.

  4. In the last line of the file, include the file hash, which you calculate by using the same key and hash method that you used to produce the hashed domain names.
  5. Save the database files on the Routing Engine in the /var/db/url-filterd directory.
  6. Validate the domain filter database file.
  7. If you make any changes to the database file, apply the changes.

How to Configure a DNS Filter Profile

A DNS filter profile includes general settings for filtering DNS requests for disallowed website domains, and includes up to 32 templates. The template settings apply to DNS requests on specific uplink and downlink logical interfaces or routing instances, or to DNS requests from specific source IP address prefixes, and override the corresponding settings at the DNS profile level. You can configure up to eight DNS filter profiles.

To configure a DNS filter profile:

  1. Configure the name for a DNS filter profile:

    The maximum number of profiles is 8.

  2. Configure the interval for logging per-client statistics for DNS filtering. The range is 0 through 60 minutes and the default is 5 minutes.
  3. Configure general DNS filtering settings for the profile. These values are used if a DNS request does not match a specific template.
    1. Specify the name of the domain filter database to use when filtering DNS requests.
    2. (Optional) To limit DNS filtering to DNS requests that are destined for specific DNS servers, specify up to three IP addresses (IPv4 or IPv6).
    3. Specify the format for the hash key.
    4. Specify the hash key that you used to create the hashed domain name in the domain filter database file.
    5. Specify the hash method that was used to create the hashed domain name in the domain filter database file.

      The only supported hash method is hmac-sha2-256.

    6. Configure the interval for logging statistics for DNS requests and for sinkhole actions performed for each customer IP address. The range is 1 through 60 minutes and the default is 5 minutes.
    7. Configure the time to live while sending the DNS response after taking the DNS sinkhole action. The range is 0 through 86,400 seconds and the default is 1800.
    8. Configure the level of subdomains that are searched for a match. The range is 0 through 10. A value of 0 indicates that subdomains are not searched.

      For example, if you set the wildcarding-level to 4 and the database file includes an entry for example.com, the following comparisons are made for a DNS request that arrives with the domain 198.51.100.0.example.com:

      • 198.51.100.0.example.com: no match

      • 51.100.0.example.com: no match for one level down

      • 100.0.example.com: no match for two levels down

      • 0.example.com: no match for three levels down

      • example.com: match for four levels down

  4. Configure a template. You can configure a maximum of 8 templates in a profile. Each template identifies filter settings for DNS requests on specific uplink and downlink logical interfaces or routing instances, or for DNS requests from specific source IP address prefixes.
    1. Configure the name for the template.
    2. (Optional) Specify the client-facing logical interfaces (uplink) to which the DNS filtering is applied.
    3. (Optional) Specify the server-facing logical interfaces (downlink) to which the DNS filtering is applied.
    4. (Optional) Specify the routing instance for the client-facing logical interface to which the DNS filtering is applied.
    5. (Optional) Specify the routing instance for the server-facing logical interface to which DNS filtering is applied.
      Note

      If you configure the client and server interfaces or the client and server routing instances, implicit filters are installed on the interfaces or routing instances to direct DNS traffic to the services PIC for DNS filtering. If you configure neither the client and server interfaces nor the routing instances, you must provide a way to direct DNS traffic to the services PIC (for example, via routes).

    6. Specify the name of the domain filter database to use when filtering DNS requests.
    7. (Optional) To limit DNS filtering to DNS requests that are destined for specific DNS servers, specify up to three IP addresses (IPv4 or IPv6).
    8. Specify the hash method that was used to create the hashed domain name in the domain filter database file.

      The only supported hash method is hmac-sha2-256.

    9. Specify the hash key that was used to create the hashed domain name in the domain filter database file.
    10. Configure the interval for logging statistics for DNS requests and for sinkhole actions performed for each customer IP address. The range is 1 through 60 minutes and the default is 5 minutes.
    11. Configure the time to live while sending the DNS response after taking the DNS sinkhole action. The range is 0 through 86,400 seconds and the default is 1800.
    12. Configure the level of subdomains that are searched for a match. The range is 0 through 10. A value of 0 indicates that subdomains are not searched.

      For example, if you set the wildcarding-level to 4 and the database file includes an entry for example.com, the following comparisons are made for a DNS request that arrives with the domain 198.51.100.0.example.com:

      • 198.51.100.0.example.com: no match

      • 51.100.0.example.com: no match for one level down

      • 100.0.example.com: no match for two levels down

      • 0.example.com: no match for three levels down

      • example.com: match for four levels down

    13. (Optional) Specify the response error code for SRV and TXT query types.
    14. Configure a term for the template. You can configure a maximum of 64 terms in a template.
    15. (Optional) Specify the source IP address prefixes of DNS requests you want to filter. You can configure a maximum of 64 prefixes in a term.
    16. Specify that the sinkhole action identified in the domain filter database is performed on disallowed DNS requests.

How to Configure a Service Set for DNS Filtering

  • Associate the DNS filter profile with a next-hop service set and enable logging for DNS filtering. The service interface can be an ms- or vms- interface Next Gen Services with MX-SPC3 services card), or it can be an aggregated multiservices (AMS) interface.

Integration of Juniper ATP Cloud and Web filtering on MX Routers

Overview

Juniper Sky™ Advanced Threat Prevention (Juniper ATP Cloud) is integrated with MX series routers to protect all hosts in your network against evolving security threats by employing cloud-based threat detection software with a next-generation firewall system.

This topic provides an overview of Juniper ATP Cloud, Policy Enforcer, Security Intelligence, Web filtering, and their benefits when integrated on MX Series routers (MX240, MX480 and MX960).

Benefits

  • Simplifies deployment and enhances the anti-threat capabilities when integrated with the MX routers.

  • Delivers protection against “zero-day” threats using a combination of tools to provide robust coverage against sophisticated, evasive threats.

  • Checks inbound and outbound traffic with policy enhancements that allow users to stop malware, quarantine infected systems, prevent data exfiltration, and disrupt lateral movement.

  • Supports High Availability to provide uninterrupted service.

  • Provides scalability to handle increasing loads that require more computing resources, increased network bandwidth to receive more customer submissions, and a large storage for malware.

  • Provides deep inspection, actionable reporting, and inline malware blocking.

Understanding Policy Enforcer and Juniper ATP Cloud

Juniper Networks Security Director comprises a feature called the Policy Enforcer (PE) that enables it to learn from threat conditions, automate the policy creation, and to dynamically deploy enforcement to Juniper devices in the network.

Figure 2 illustrates the traffic flow between the PE, the Juniper ATP Cloud, and the MX router which functions as a firewall.

  • Policy Enforcer (PE) learns from threat conditions, automates the policy creation, and deploys enforcement to Juniper devices in the network.

  • Juniper Sky™ Advanced Threat Prevention (Juniper ATP Cloud) protects all hosts in your network by employing cloud-based threat detection software with a next-generation firewall system.

  • MX router fetches the threat intelligence feeds from Policy Enforcer (PE) and implements those policies to quarantine compromised hosts. It comprises of the following important components:

    • Security Intelligence process

    • Web Filtering process

    • Firewall process

Figure 2: System Architecture
System Architecture
System Architecture

To understand the functionality of the system architecture consider the following example—if a user downloads a file from the Internet and that file passes through an MX firewall, the file can be sent to the Juniper ATP Cloud cloud for malware inspection (depending on your configuration settings.) If the file is determined to be malware, PE identifies the IP address and MAC address of the host that downloaded the file. Based on a user-defined policy, that host can be put into a quarantine VLAN or blocked from accessing the Internet.

MX Series routers (MX240, MX480, and MX960) can be integrated with the Juniper ATP Cloud to prevent compromised hosts (botnets) from communicating with command and control servers:

  • Starting in Junos OS Release 18.4R1 with the Adaptive Services as an Inline security capability

  • Starting in Junos OS Release 19.3R2 with the Next Gen Services as an Inline security capability

Security Intelligence (SecIntel) - Overview

The Security Intelligence process (IPFD), is responsible for downloading the security intelligence feeds and parsing from the feed connector or ATP Cloud cloud feed server. The IPFD process on the MX platforms fetches the command and control IPv4/IPv6 feeds from Policy Enforcer. C&C feeds are essentially a list of servers that are known command and control servers for botnets. The list also includes servers that are known sources for malware downloads. The information thus fetched is saved in a file (urlf_si_cc_db.txt) created under the /var/db/url-filterd directory.

The file format of the disallowed IPs sent by IPFD to the web filtering process is as follows:

IPv4 address | IPv6 address, threat-level.

The threat-level is an integer ranging from 1 to 10 to indicate the threat level of files scanned for malware and for infected hosts. Here, 1 represents the lowest threat level and 10 represents the highest threat level.

For example: 178.10.19.20, 4

Here, 178.10.19.20 indicates the disallowed IP and 4 indicates the threat-level.

The C&C feed database is synced onto the backup Routing Engine. IPFD then shares the information to the web filtering process (url-filterd). The web filtering process reads the file contents and configures the filters accordingly.

Configuring Security Intelligence to Download the CC Feed from Policy Enforcer

To download the command and control IPv4/IPv6 feeds from Juniper ATP Cloud/Policy Enforcer, include the security-intelligence statement at the [edit services] hierarchy as shown in the following example:

Web Filtering (URL-Filterd) - Overview

The web filtering process reads the file contents fetched from the IPFD and configures the filters on the Packet Forwarding Engine accordingly. The web filtering process enforces the command and control feeds by programming the filters in the Packet Forwarding Engine to block the packets destined to the blocked IP addresses and to generate logs for reporting the incident.

Figure 3 illustrates the way C&C feed is fetched by the IPFD and then processed by the web filtering process.

Figure 3: Web Filtering
Web Filtering
Web Filtering

The web filter profile can have more than one templates. Each template consists of a set of configured logical interfaces for Web filtering and one or more terms. A term is a set of match criteria with actions to be taken if the match criteria is met. To configure the web filter profile to use dynamically fetched C&C feed, you can configure the security-intelligence-policy command under the [edit services web-filter profile profile-name hierarchy level. You need not configure a term for a security-intelligence-policy based web filter profiles.

You can configure the following threat level actions for the web filter profile at the edit web-filter profile profile-name security-intelligence-policy threat-level threat-level threat-action hierarchy level:

  • drop

  • drop-and-log

  • log

You can configure only one threat-action for each threat level. If the threat-action is not configured for a particular threat level, the default threat-action is accept.

Configuring the Web Filter Profile for Sampling

Starting in Junos OS Release 19.3R1, web filtering process (url-filterd) supports inline sampling of packets as a threat level action. The packets are dropped, logged, and sampled based on the threat-action you configure. For scaled scenarios, sampling of packets is preferred over the logging option. Along with the existing threat level actions, you can configure the following threat level actions on the web filter profile at the edit web-filter profile profile-name security-intelligence-policy threat-level threat-level threat-action hierarchy level:

  • drop-and-sample

  • drop-log-and-sample

  • log-and-sample

  • sample

The inline flow monitoring samples the packets and sends the flow records in IPFIX format to a flow collector. You can derive the threat level for the sampled packets received at the external collector by matching the received IP from the sampled packets with the corresponding IP entry in /var/db/url-filterd/urlf_si_cc_db.txt. You can configure sampling using any of the following methods:

  • Associate a sampling instance with the FPC on which the media interface is present at the [edit chassis] hierarchy level. If you are configuring sampling of IPv4 flows, IPv6 flows, or VPLS flows, you can configure the flow hash table size for each family.

  • Configure the template properties for inline flow monitoring at the [edit services flow-monitoring hierarchy level.

  • Configure a sampling instance and associate the flow-server IP address, port number, flow export rate, and specify the collectors at the [edit forwarding-options hierarchy level.

Associate a Sampling Instance with the FPC

To associate the defined instance with a particular FPC, MPC, or DPC, you include the sampling-instance statement at the [edit chassis fpc number] hierarchy level, as shown in the following example:

Configure a Sampling Instance and Associate the Template With the Sampling Instance.

To configure the template properties for inline flow monitoring, include the following statements at the edit services flow-monitoring hierarchy level as shown in the following example:

Configure the sample instance and associate the flow-server IP address and other parameters.

To configure a sampling instance and associate the flow-server IP address and other parameters. include the following statements at the [edit forwarding-options] hierarchy, as shown in the following example:

Example: Configuring Web-filter Profile to Define Different Threat-Levels

Release History Table
Release
Description
Starting in Junos OS Release 19.3R2, this same functionaly is supported for Next Gen Services on MX240, MX480, and MX960.
Starting in Junos OS Release 19.3R2, this same functionality is available for Next Gen Serices on MX240, MX480, and MX960.
Starting in Junos OS Release 19.3R2, you can configure DNS filtering if you are running Next Gen Services with the MX-SPC3 services card. Next Gen Services are supported on MX240, MX480 and MX960 routers.
Starting in Junos OS Release 19.3R2 with the Next Gen Services as an Inline security capability
Starting in Junos OS Release 19.3R1, web filtering process (url-filterd) supports inline sampling of packets as a threat level action
Starting in Junos OS Release 18.4R1 with the Adaptive Services as an Inline security capability
Starting in Junos OS Release 18.3R1, for Adaptive Services. configure the profile at the [edit services web-filter] hierarchy level. Before Junos OS Release 18.3R1, configure the profile at the [edit services url-filter] hierarchy level.
Starting in Junos OS Release 17.2R2 and 17.4R1, for Adaptive Services, you can disable the filtering of HTTP traffic that contains an embedded IP address (for example, http:/10.1.1.1) belonging to a disallowed domain name in the URL filter database.