URL Filtering

You can implement URL filtering using access control rules.

URL Filtering Overview

Use the URL filtering feature to control the websites that users on your network can access:

  • Category and reputation-based URL filtering—With a URL Filtering license, you can control access to websites based on the URL’s general classification (category) and risk level (reputation). This is the recommended option.

  • Manual URL filtering—With any license, you can manually specify individual URLs, groups of URLs, and URL lists and feeds to achieve granular, custom control over web traffic. For more information, see Manual URL Filtering.

See also Security Intelligence, a similar but different feature for blocking malicious URLs, domains, and IP addresses.

About URL Filtering with Category and Reputation

With a URL Filtering license, you can control access to websites based on the category and reputation of requested URLs:

  • Category—A general classification for the URL. For example, ebay.com belongs to the Auctions category, and monster.com belongs to the Job Search category.

    A URL can belong to more than one category.

  • Reputation—How likely the URL is to be used for purposes that might be against your organization’s security policy. Reputations range from Unknown risk (level 0) or Untrusted (level 1) to Trusted (level 5).

Benefits of Category and Reputation-Based URL Filtering

URL categories and reputations help you quickly configure URL filtering. For example, you can use access control to block untrusted URLs in the Hacking category. Or, you can use QoS to rate limit traffic from sites in the Streaming Video category. There are also categories for types of threats, such as a Spyware and Adware category.

Using category and reputation data simplifies policy creation and administration. It grants you assurance that the system controls web traffic as expected. Because Cisco continually updates its threat intelligence with new URLs, as well as new categories and risks for existing URLs, the system uses up-to-date information to filter requested URLs. Sites that (for example) represent security threats, or that serve undesirable content, may appear and disappear faster than you can update and deploy new policies.

Some examples of how the system can adapt include:

  • If an access control rule blocks all gaming sites, as new domains get registered and classified as Games, the system can block those sites automatically. Similarly, if a QoS rule rate limits all streaming video sites, the system can automatically limit traffic to new Streaming Video sites.

  • If an access control rule blocks all malware sites and a shopping page gets infected with malware, the system can recategorize the URL from Shopping to Malware Sites and block that site.

  • If an access control rule blocks untrusted social networking sites and somebody posts a link on their profile page that contains links to malicious payloads, the system can change the reputation of that page from Favorable to Untrusted and block it.

Limitations of category-based filtering in decryption policy Do Not Decrypt rules

You can optionally choose to include categories in your decryption policies. These categories, also referred to as URL filtering, are updated by the Cisco Talos intelligence group. Updates are based on machine learning and human analysis according to content that is retrievable from the website destination and sometimes from its hosting and registration information. Categorization is not based on the declared company vertical, intent, or security.


Note

Don't confuse URL filtering with application detection, which relies on reading some of the packet from a website to determine more specifically what it is (for example, Facebook Message or Salesforce). For more information, see Best Practices for Configuring Application Control.

For more information, see Use Categories in URL Filtering.

URL Filtering Data from the Cisco Cloud

Adding a URL Filtering license automatically enables the URL filtering feature. This allows traffic handling based on a website’s general classification, or category, and risk level, or reputation.

By default, when users browse to an URL whose category and reputation is not in a local cache of previously accessed websites, the system submits it to the cloud for threat intelligence evaluation and adds the result to the cache.

Optionally, you can use a local URL dataset of categories and reputations, which can make web browsing faster. When you enable (or re-enable) URL filtering, the management center automatically queries Cisco for URL data and pushes the dataset to managed devices. Then, when users browse to an URL, the system checks the local dataset and the cache for category and reputation information before submitting it to the cloud for threat intelligence evaluation. To see your options for using the local dataset, including how to disable individual cloud lookups altogether, see URL Filtering Options.

Automatic updates of URL data is enabled by default; we strongly recommend you do not disable these updates.

The set of URL categories may change periodically. When you receive a change notification, review your URL filtering configurations to make sure traffic is handled as expected. For more information, see If the URL Category Set Changes, Take Action.

Best Practices for URL Filtering

Keep in mind the following guidelines and limitations for URL filtering:

Filter by Category and Reputation

Follow the instructions in How to Configure URL Filtering with Category and Reputation.

Configure Your Policy to Inspect Packets That Must Pass Before a URL Can Be Identified

The system cannot filter URLs before:

  • A monitored connection is established between a client and server.

  • The system identifies the DNS, HTTP or HTTPS application in the session.

  • The system identifies the requested domain or URL (for encrypted sessions, from a non-encrypted domain name, the ClientHello message or the server certificate).

This identification should occur within 3 to 5 packets, or after the server certificate exchange in the TLS/SSL handshake if the traffic is encrypted.

Important! To ensure that your system examines these initial packets that would otherwise pass, see Inspection of Packets That Pass Before Traffic Is Identified and subtopics.

If early traffic matches all other rule conditions but identification is incomplete, the system allows the packet to pass and the connection to be established (or the TLS/SSL handshake to complete). After the system completes its identification, the system applies the appropriate rule action to the remaining session traffic.

Block Threat Categories

Be sure that your policies specifically address Threat categories, which identify known malicious sites. Do this in addition to blocking sites with poor reputations.

For example, to protect your network from malicious sites, you must block all Threat categories. Additionally, Talos recommends that you block only sites with Poor category. You can block questionable reputations if you have an aggressive security posture, but this may result in a higher amount of false positives.

For specifics, see Threat Categories at the URL in URL Category and Reputation Descriptions.

URL Conditions and Rule Order

  • Position URL rules after all other rules that must be hit.

  • URLs can belong to more than one category. It is possible to want to allow one category of websites and block another—whether explicitly or by relying on the default action. In this case, make sure you create and order URL rules so you get the desired effect, depending on whether the allow or the block should take precedence.

For additional guidelines for rules, see the following topics: Best Practices for Access Control Rules.

Uncategorized or Reputationless URLs

When you build a URL rule, you first choose the category you want to match. If you explicitly choose Uncategorized URLs, you cannot further constrain by reputation.

Uncategorized URLs with Untrusted reputation are handled by the Malicious Sites category. If you want to block uncategorized sites with any other reputation level (such as Questionable), you must block all uncategorized sites.

After selecting a category and a reputation level, you can optionally select Apply to unknown reputation. For example, you can create a rule that applies to sites with Untrusted, Questionable, and unknown reputations.

You cannot manually assign categories and reputations to URLs, but in access control and QoS policies, you can manually block specific URLs. See Manual URL Filtering. See also Dispute URL Category and Reputation.

URL Filtering for Encrypted Web Traffic

When performing URL filtering on encrypted web traffic, the system:

  • (If DNS filtering is enabled) Checks to see if the system has previously seen the originating domain or the domain is in the local reputation database, and if so, takes action based on the reputation and category of the domain. Otherwise, the system processes the traffic based on your configurations for encrypted traffic, even if Retry URL cache miss lookup is enabled in the access control policy's advanced settings.

  • Disregards the encryption protocol; a rule matches both HTTPS and HTTP traffic if the rule has a URL condition but not an application condition that specifies the protocol.

  • Does not use URL lists. You must use URL objects and groups instead.

  • Matches HTTPS traffic based on the subject common name in the public key certificate used to encrypt the traffic, and also evaluates the reputation of any other URLs presented at any time during the transaction, including the post-decryption HTTP URL.

  • Disregards subdomains within the subject common name.

  • Does not display an HTTP response page for encrypted connections blocked by access control rules (or any other configuration); see Limitations to HTTP Response Pages.

URL Filtering and TLS Server Identity Discovery

The latest version of the Transport Layer Security (TLS) protocol 1.3, defined by RFC 8446, is the preferred protocol for many web servers to provide secure communications. Because the TLS 1.3 protocol encrypts the server's certificate for additional security, and the certificate is needed to match application and URL filtering criteria in access control rules, the Firepower System provides a way to extract the server certificate without decrypting the entire packet.

Access control policy advanced settings offer an Early application detection and URL categorization option for TLS Server Identity Discovery.

We strongly recommend enabling it for any traffic you want to match on application or URL criteria, especially if you want to perform deep inspection of that traffic. A decryption policy is not required because traffic is not decrypted in the process of extracting the server certificate.


Note

  • Because the certificate is decrypted, TLS server identity discovery can reduce performance depending on the hardware platform.

  • TLS server identity discovery is not supported in inline tap mode or passive mode deployments.

  • Enabling TLS server identity discovery is not supported on any Secure Firewall Threat Defense Virtual deployed to AWS. If you have any such managed devices managed by the Secure Firewall Management Center, the connection event PROBE_FLOW_DROP_BYPASS_PROXY increments every time the device attempts to extract the server certificate.

  • TLS Server Identity Discovery also operates on TLS 1.2 sessions.

For more information, see Access Control Policy Advanced Settings.

HTTP/2

The system can extract HTTP/2 URLs from TLS certificates, but not from a payload.

Manual URL Filtering

  • Specify URLs using a custom Security Intelligence list or feed object. Do not use a URL object or directly enter a URL into the rule. For details, see Manual URL Filtering Options.

  • If you manually filter specific URLs using URL objects or by entering URLs directly into the rule, carefully consider other traffic that might be affected. To determine whether network traffic matches a URL condition, the system performs a simple substring match. If the requested URL matches any part of the string, the URLs are considered to match.

  • If you use manual URL filtering to create exceptions to other rules, position the specific rule with the exceptions above the general rule that would otherwise apply.

Search Query Parameters in URLs

The system does not use search query parameters in the URL to match URL conditions. For example, consider a scenario where you block all shopping traffic. In that case, using a web search to search for amazon.com is not blocked, but browsing to amazon.com is.

URL Filtering in High Availability Deployments

For guidelines for URL filtering with Firepower Management Centers in high availability, see URL Filtering and Security Intelligence in the Cisco Secure Firewall Management Center Administration Guide.

Memory Limitations for Selected Device Models

  • Device models with less memory store less URL data locally, and the system may therefore check the cloud more frequently to determine category and reputation for sites that are not in the local database.

    Lower-memory devices include:

    • Firepower 1010

    • Threat Defense Virtual with 8 GB of RAM

URL Matching for TLS session Resumption on Threat Defense

Use URL matching with Snort 2 under the following conditions:

  • If there is no TLS session resumption and SSL policy is enabled or the Client Hello message contains Server Name Indication (SNI) extension.

  • If there is TLS session resumption and SSL policy is not enabled or the Client Hello message does not contain SNI extension.

Filtering HTTPS Traffic

To filter encrypted traffic, the system determines the requested URL based on information passed during the TLS/SSL handshake: the subject common name in the public key certificate used to encrypt the traffic.

HTTPS filtering, unlike HTTP filtering, disregards subdomains within the subject common name. Do not include subdomain information when manually filtering HTTPS URLs in access control or QoS policies. For example, use example.com rather than www.example.com.


Tip

In an decryption policies, you can handle and decrypt traffic to specific URLs by defining a distinguished name decryption policy rule condition. The common name attribute in a certificate’s subject distinguished name contains the site’s URL. Decrypting HTTPS traffic allows access control rules to evaluate the decrypted session, which improves URL filtering.

Controlling Traffic by Encryption Protocol

The system disregards the encryption protocol (HTTP vs HTTPS) when performing URL filtering in access control or QoS policies. This occurs for both manual and reputation-based URL conditions. In other words, URL filtering treats traffic to the following websites identically:

  • http://example.com/

  • https://example.com/

To configure a rule that matches only HTTP or HTTPS traffic, add an application condition to the rule. For example, you could allow HTTPS access to a site while disallowing HTTP access by constructing two access control rules, each with an application and URL condition.

The first rule allows HTTPS traffic to the website:

  • Action: Allow
  • Application: HTTPS
  • URL: example.com

The second rule blocks HTTP access to the same website:

  • Action: Block
  • Application: HTTP
  • URL: example.com

Use Categories in URL Filtering

Limitations of categories in Do Not Decrypt rules

You can optionally choose to include categories in your decryption policies. These categories, also referred to as URL filtering, are updated by the Cisco Talos intelligence group. Updates are based on machine learning and human analysis according to content that is retrievable from the website destination and sometimes from its hosting and registration information. Categorization is not based on the declared company vertical, intent, or security. While we strive to continuously update and improve URL filtering categories, it is not an exact science. Some websites are not categorized at all and it's possible some websites might be improperly categorized.

Avoid overusing categories in do not decrypt rules to avoid decrypting traffic without a reason; for example, the Health and Medicine category includes the WebMD website, which does not threaten patient privacy.

Following is a sample decryption policy that can prevent decryption for websites in the Health and Medicine categories but allows decryption for WebMD and everything else. General information about decryption rules can be found in Guidelines for Using TLS/SSL Decryption.

Sample decryption policy that exempts websites in the Health and Medicine categories

Note

Don't confuse URL filtering with application detection, which relies on reading some of the packet from a website to determine more specifically what it is (for example, Facebook Message or Salesforce). For more information, see Best Practices for Configuring Application Control.

License Requirements for URL Filtering

Threat Defense License

  • Category and reputation filtering—URL Filtering

  • Manual filtering—No additional license.

Classic License

  • Category and reputation filtering—URL Filtering

  • Manual filtering—No additional license.

URL Filtering Licenses for Threat Defense Devices

See URL Licenses in the Licenses chapter in the Cisco Secure Firewall Management Center Administration Guide.

Requirements and Prerequisites for URL Filtering

Model Support

Any

Supported Domains

Any

User Roles

  • Admin

  • Access Admin

  • Network Admin

How to Configure URL Filtering with Category and Reputation

Do This

More Information

Step 1

Ensure that you have the correct licenses.

Assign the URL Filtering license to each managed device that will filter URLs.

Step 2

Ensure that your management center can communicate with the cloud to obtain URL filtering data.

Internet Access Requirements and Communication Port Requirements in the Cisco Secure Firewall Management Center Administration Guide.

Step 3

Understand limitations and guidelines and take any necessary actions.

Best Practices for URL Filtering

Step 4

Enable the URL Filtering feature.

Enable URL Filtering Using Category and Reputation

Step 5

Configure rules to filter URLs by category and reputation.

Configuring URL Conditions

For the best protection against malicious sites, you must block sites by reputation AND block URLs in all Threat categories.

(Optional) Supplement or Selectively Override Category and Reputation-Based URL Filtering

Step 6

(Optional) Allow users to bypass a website block by clicking through a warning page.

Configure HTTP Response Pages

Step 7

Order your rules so that traffic hits key rules first.

URL Rule Order

Step 8

(Optional) Modify advanced options related to URL filtering.

Generally, use the defaults unless you have a specific reason to change them.

For information about advanced options, including the following, see Access Control Policy Advanced Settings.

  • Maximum URL characters to store in connection events

  • Allow an Interactive Block to bypass blocking for (seconds)

  • Retry URL cache miss lookup

  • Enable reputation enforcement on DNS traffic

Step 9

Deploy your changes.

Deploy Configuration Changes

Step 10

Ensure that your system receives future URL data updates as expected

Configure URL Filtering Health Monitors

Step 11

Be sure you have enabled other features that protect your network from malicious sites

See Security Intelligence.

Enable URL Filtering Using Category and Reputation

You must be an Admin user to perform this task.

Before you begin

Complete prerequisites described in How to Configure URL Filtering with Category and Reputation.

Procedure

Step 1

Choose Integration > Other Integrations.

Step 2

Click Cloud Services.

Step 3

Configure URL Filtering Options.

Step 4

Click Save.

URL Filtering Options

Adding a URL Filtering license automatically enables the URL filtering feature. This allows traffic handling based on a website’s general classification, or category, and risk level, or reputation.

Although by default the system is configured to submit all URLs to the cloud for threat intelligence evaluation, using a local dataset of category and reputation data can make web browsing faster. When you enable (or re-enable) URL filtering, the management center automatically queries Cisco for URL data and pushes the dataset to managed devices. This process may take some time.

If you use SSL rules to handle encrypted traffic, also see Decryption Rule Guidelines and Limitations.

Enable Automatic Updates

If you Enable Automatic Updates (the default), the management center checks the cloud every 30 minutes for updates. If you need strict control over when the system contacts external resources, disable automatic updates and instead create a recurring task using the scheduler. See Automated URL Filtering Updates Using a Scheduled Task in the Cisco Secure Firewall Management Center Administration Guide .

Update Now

Click Update Now to perform a one-time, on-demand URL data update. You cannot start an on-demand update if an update is already in progress. Although daily updates tend to be small, if it has been more than five days since your last update, new URL data may take up to 20 minutes to download, depending on your bandwidth. Then, it may take up to 30 minutes to perform the update itself.

URL Query Source

You can choose how the system assigns a category and reputation to the URLs that your users browse to. You can choose:

  • Local Database Only: Uses the local dataset only. Use this option if you do not want to submit your uncategorized URLs (category and reputation not in the local dataset) to Cisco, for example, for privacy reasons. However, note that connections to uncategorized URLs do not match rules with category or reputation-based URL conditions. You cannot assign categories or reputations to URLs manually.

  • Local Database and Cisco Cloud: Uses the local dataset when possible, which can make web browsing faster. When users browse to an URL whose category and reputation is not in the local dataset or a cache of previously accessed websites, the system submits it to the cloud for threat intelligence evaluation and adds the result to the cache.

  • Cisco Cloud Only (default): Does not use the local dataset. When users browse to an URL whose category and reputation is not in a local cache of previously accessed websites, the system submits it to the cloud for threat intelligence evaluation and adds the result to the cache. This option guarantees the most up-to-date category and reputation information.

    This option requires threat defense Version 7.3. If you enable this option, devices running earlier versions will use the Local Database and Cisco Cloud option.

Cached URLs Expire

Caching category and reputation data makes web browsing faster. By default, cached data for URLs never expires, for fastest performance.

To minimize instances of URLs matching on stale data, you can set URLs in the cache to expire. For greater accuracy and currency of threat data, choose a shorter expiration time. A cached URL refreshes after the first time a user on the network accesses it after the specified time has passed. The first user does not see the refreshed result, but the next user who visits this URL does see the refreshed result.

Configuring URL Conditions

Protect your network by controlling access to sites based on URL category and reputation.

Before you begin


Attention

As a prerequisite, ensure that you create at least a Monitor rule at the top of your access control policy, containing Category or Reputation parameters. This is essential to see ANY category or reputation data for ANY URLs that hit the particular access control policy.

If there is no rule in the access control policy with the category or reputation parameters configured, the Connection Events page in the management center shows no data for Category or Reputation for any URL traffic that hits the access control policy.

Procedure

Step 1

In the rule editor, click the following for URL conditions:

  • Access control or QoS—Click URLs.
  • SSL—Click Category.

Step 2

Find and choose the URL categories that you want to control:

In an access control or QoS rule, click Category.

For effective protection from malicious sites, you must block URLs in all Threat categories. Additionally, Talos recommends that you block only sites with Poor category. You can block questionable reputations if you have an aggressive security posture, but this may result in a higher amount of false positives. For a list of Threat categories, see URL Category and Reputation Descriptions.

Be sure to click the arrows at the bottom of the list to see all available categories.

Step 3

(Optional) Constrain URL categories by choosing a Reputation.

Note that if you explicitly match Uncategorized URLs, you cannot further constrain by reputation. Choosing a reputation level also includes other reputations either more or less severe than the level you choose, depending on the rule action:

  • Includes less severe reputations—If the rule allows or trusts web traffic. For example, if you configure an access control rule to allow Favorable (level 4), it also automatically allows Trusted (level 5) sites.

  • Includes more severe reputations—If the rule rate limits, decrypts, blocks, or monitors web traffic. For example, if you configure an access control rule to block Questionable sites (level 2), it also blocks Untrusted (level 1) sites.

If you change the rule action, the system automatically changes the reputation levels in URL conditions.

Optionally, select Apply to unknown reputation.

Step 4

Click Add URL or Add to Rule, or drag and drop.

Step 5

(Optional) To choose predefined URL objects, or URL lists and feeds in an access control or QoS rule, click URL, select the objects, and add them to the destination.

These objects implement manual URL filtering rather than category-based filtering.

Step 6

Save or continue editing the rule.

Example: URL Condition in an Access Control Rule

The following graphic shows the URL condition for an access control rule that blocks all malware sites, all untrusted sites, and all social networking sites with a reputation level of Neutral or worse.


Sample URL condition.

The following table summarizes how you build the condition.

Blocked URL

Category

Reputation

Malware sites, regardless of reputation

Malware Sites

Any

Any untrusted URL (level 1)

Any

1 - Untrusted

Social networking sites with a reputation level of Neutral or worse (levels 1 through 3)

Social Network

3 - Neutral

Rules with URL Conditions

The following table lists rules that support URL conditions, and the types of filtering that each rule type supports.

Rule Type

Supports Category and Reputation Filtering?

Supports Manual Filtering?

Access control

Yes

Yes

Decryption policy

Yes

No; use distinguished name conditions instead

QoS

Yes

Yes

To use URL filtering in a decryption policy that has Do Not Decrypt rule conditions, see Use Categories in URL Filtering.

URL Rule Order

For the most effective URL matching, place rules that include URL conditions before other rules, particularly if the URL rules are block rules and the other rules meet both of the following criteria:

  • They include application conditions.

  • The traffic to be inspected is encrypted.

If you configure exceptions to a rule, put the exception above the other rule.

DNS Filtering: Identify URL Reputation and Category During DNS Lookup

The Enable reputation enforcement on DNS traffic option is enabled by default on the Advanced tab of each new access control policy. This option slightly modifies URL filtering behavior and is applicable only when URL filtering is enabled and configured.

When this option is enabled:

  • The system evaluates domain category and reputation early in URL transactions, when the browser looks up the domain name to get the IP address

  • Category and reputation of encrypted traffic can often be determined without decryption

    If DNS filtering cannot determine the URL of encrypted traffic, that traffic is processed using your configurations for encrypted traffic.

Enable DNS Filtering to Identify URLs During Domain Lookup

DNS filtering is enabled by default in new access control policies. However, additional configurations may be required in order for this setting to take effect.

Before you begin

  • URL filtering using category and reputation must be licensed, enabled, and configured.

    (DNS filtering does not use the following settings in the URLs tab: URL groups, URL objects, URL lists and feeds, and URLs entered into the "Enter URL" text box.)

  • See limitations at DNS Filtering Limitations.

Procedure

Step 1

In your access control policy's advanced settings, select Enable reputation enforcement on DNS traffic.

Step 2

In the same policy, for each access control rule that has URL category and reputation blocking configured:

  • Application conditions—If the application condition is anything other than any (or empty), add DNS to that list. Other DNS-related options are not relevant for this purpose.

  • Port condition—If the port/protocol condition is anything other than any (or empty), add DNS_over_TCP and DNS_over_UDP.

Step 3

Save your changes.
What to do next

If you are done making changes: Deploy Configuration Changes.

DNS Filtering Limitations

Traffic that matches rules having action Block with reset, Interactive Block, or Interactive Block with reset will be treated as if the rule action were Block.

End users trying to access a blocked URL will experience this as an unexplained inability to connect to their page; the connection will spin and then time out.

DNS Filtering and Events

Connection events generated by DNS filtering are logged using the following fields: DNS Query, URL Category, URL Reputation, and Destination Port. The DNS Query field holds the domain name; the URL field will be blank for DNS filtering matches. The Destination Port will be 53.

Also:

  • When the access control rule action is Allow or Trust, two connection events will be generated for the same traffic, one for DNS filtering (with the DNS Query field populated) and one for URL filtering (with the URL field populated).

  • The first time the system encounters a particular URL, you will see two events for that single session: One event showing uncategorized/reputationless for the DNS Query, and one event showing the actual category and reputation for the URL, which were retrieved during the DNS Query and applied to the session while processing using standard URL filtering.

Manual URL Filtering

In access control and QoS rules, you can supplement or selectively override category and reputation-based URL filtering by manually filtering individual URLs, groups of URLs, or URL lists and feeds.

For example, you might use access control to block a category of websites that are not appropriate for your organization. However, if the category contains a website that is appropriate, and to which you want to provide access, you can create a manual Allow rule for that site and place it before the Block rule for the category.

You can perform this type of URL filtering without a special license.

Manual URL filtering is not supported in SSL rules; instead, use distinguished name conditions.


Caution

Depending on how you implement manual URL filtering, URL matching may not be what you intend. See Manual URL Filtering Options.

Manual URL Filtering Options

There are several ways to specify URLs for manual URL filtering:

Option

Description

(Best practice)

Use custom Security Intelligence URL list or feed objects.

This is the recommended method for manual URL filtering.

You can create a new list or feed, or choose an existing one in an access control or QoS rule.

For more information, see Custom Security Intelligence Lists and Feeds and subtopics.

Use URL objects, individually or as groups. URL objects are described at URL.

Or

Enter URLs directly into the access control rule. (The Enter URL option on the rule page in the web interface.)

If you do not include a path (that is, there are no / characters in the URL), the match is based on the server’s hostname only. If you include one or more / character, the entire URL string is used for a substring match. Then, a URL is considered a match if any of the following are true:

  • The string is at the beginning of the URL.

  • The string follows a dot.

  • The string contains a dot in the beginning.

  • The string follows the :// characters.

For example, ign.com matches ign.com or www.ign.com, but not versign.com.

Note

 

We recommend that you do not use manual URL filtering to block or allow individual web pages or parts of sites (that is, URL strings with / characters), as servers can be reorganized and pages moved to new paths.

The Enter URL option does not support wildcards.

Supplement or Selectively Override Category and Reputation-Based URL Filtering

In access control or QoS rules, you can use Security Intelligence URL lists and feeds to supplement, or to specify exceptions to, your category and reputation-based URL filtering rules.

Important! If the list or feed you are configuring in this procedure contains exceptions to category- or reputation-based rules, put this rule above those rules in the rule order.

In SSL rules, use distinguished name conditions to configure parallel behavior.

Before you begin

Procedure

Step 1

Navigate to the access control or QoS policy in which you will define the rule.

Step 2

Create or edit the rule in which you will add the new condition:

  • If you are supplementing a category- or reputation-based URL filtering rule, edit the existing rule.
  • If you are overriding or creating exceptions to a category- or reputation-based URL filtering rule, create a new rule.

Step 3

Select the list or feed you created as the destination URL criteria.

Step 4

Save the rule.

Configure HTTP Response Pages

As part of access control, you can configure an HTTP response page to display when the system blocks web requests, using either access control rules or the access control policy default action.

The response page displayed depends on how you block the session:

  • Block Response Page: Overrides the default browser or server page that explains that the connection was denied.

  • Interactive Block Response Page: Warns users, but also allows them to click a button (or refresh the page) to load the originally requested site. Users may have to refresh after bypassing the response page to load page elements that did not load.

If you do not choose a response page, the system blocks sessions without interaction or explanation.

Limitations to HTTP Response Pages

  • The system displays a response page only for unencrypted or decrypted HTTP/HTTPS connections blocked (or interactively blocked) either by access control rules or by the access control policy default action. The system does not display a response page for connections that are blocked by any other policy or mechanism.

  • The system cannot display a response page if the connection is reset (RST packet sent). If you enable response pages, the system prioritizes that configuration. Even if you choose Block with reset or Interactive Block with reset as the rule action, the system displays the response page and does not reset matching web connections. To ensure that blocked web connections are reset, you must disable response pages.

    Note that all non-web traffic that matches the rule is blocked with reset.

  • The system does not display a response page for encrypted connections that are blocked by access control rules (or any other configuration). Access control rules evaluate encrypted connections if you did not configure an SSL policy, or your SSL policy passes encrypted traffic.

    For example, the system cannot decrypt HTTP/2 or SPDY sessions. If web traffic encrypted using one of these protocols reaches access control rule evaluation, the system does not display a response page if the session is blocked.

    However, the system does display a response page for connections that are decrypted by the SSL policy, then blocked (or interactively blocked) either by access control rules or by the access control policy default action. In these cases, the system encrypts the response page and sends it at the end of the reencrypted SSL stream.

  • The system does not display a response page when web traffic is blocked because of a promoted access control rule (an early-placed blocking rule with only simple network conditions).

  • If a URL is entered without specifying "http" or "https", and the browser initiates the connection on port 80, and the user clicks through a response page, and the connection is subsequently redirected to port 443, the user will not see a second interactive response page because the response to this URL is already cached.

  • The system does not display a response page when web traffic is blocked before the system identifies the requested URL; see Best Practices for URL Filtering.

  • The system does not display a response page if the block URL access control rule is configured after the allow application rule.

Requirements and Prerequisites for HTTP Response Pages

Model Support

Any

Supported Domains

Any

User Roles

  • Admin

  • Access Admin

  • Network Admin

Choosing HTTP Response Pages

Reliable display of HTTP response pages depends on your network configuration, traffic loads, and size of the page. Smaller pages are more likely to display successfully.

Procedure

Step 1

In the access control policy editor, select HTTP Responses from the More drop-down arrow at the end of the packet flow line.

If the controls are dimmed, settings are inherited from an ancestor policy, or you do not have permission to modify the configuration. If the configuration is unlocked, uncheck Inherit from base policy to enable editing.

Step 2

Choose the Block Response Page and Interactive Block Response Page:

  • System-provided—Displays a generic response. Click View (View button) to view the code for this page.
  • Custom—Create a custom response page. A pop-up window appears, prepopulated with system-provided code that you can replace or modify by clicking Edit (edit icon). A counter shows how many characters you have used.
  • None—Disables the response page and blocks sessions without interaction or explanation. To quickly disable interactive blocking for the whole access control policy, choose this option.

Step 3

Click Save to save the policy.

What to do next

  • Deploy configuration changes.

Configure Interactive Blocking with HTTP Response Pages

When you configure interactive blocking, users can load an originally requested site after reading a warning. Users may have to refresh after bypassing the response page to load page elements that did not load.


Tip

To quickly disable interactive blocking for the whole access control policy, display neither the system-provided page nor a custom page. The system then blocks all connections without interaction.

If a user does not bypass an interactive block, matching traffic is denied without further inspection. If a user bypasses an interactive block, the access control rule allows the traffic, although the traffic may still be subject to deep inspection and blocking.

By default, a user bypass is in effect for 10 minutes (600 seconds) without displaying the warning page on subsequent visits. You can set the duration to as long as a year, or you can force the user to bypass the block every time. This limit applies to every Interactive Block rule in the policy. You cannot set the limit per rule.

Logging options for interactively blocked traffic are identical to those in allowed traffic, but if a user does not bypass the interactive block, the system can log only beginning-of-connection events. When the system initially warns the user, it marks any logged beginning-of-connection event with the Interactive Block or Interactive Block with reset action. If the user bypasses the block, additional connection events logged for the session have an action of Allow.

Configuring Interactive Blocking

The following procedure explains how to allow users to bypass URL filtering rules.

Procedure

Step 1

As part of access control, configure an access control rule that matches web traffic; see Create and Edit Access Control Rules:

Step 2

(Optional) In the access control policy HTTP Responses, choose a custom interactive-block HTTP response page; see Choosing HTTP Response Pages.

Step 3

(Optional) In access control policy Advanced settings, change the user bypass timeout; see Setting the User Bypass Timeout for a Blocked Website.

After a user bypasses a block, the system allows the user to browse to that page without warning until the timeout period elapses.

Step 4

Save the access control policy.

Step 5

Deploy configuration changes.

Setting the User Bypass Timeout for a Blocked Website

The following procedure explains how to set the time allowed for browsing after the user bypasses a URL filtering block. After the timeout expires, the user must bypass the block again.

Procedure

Step 1

Click Policies > Access Control and edit the policy.

Step 2

Select Advanced Settings from the More drop-down arrow at the end of the packet flow line.

Step 3

Click Edit (edit icon) next to General Settings.

If View (View button) appears instead, settings are inherited from an ancestor policy, or you do not have permission to modify the settings. If the configuration is unlocked, uncheck Inherit from base policy to enable editing.

Step 4

In the Allow an Interactive Block to bypass blocking for (seconds) field, enter the number of seconds that must elapse before the user bypass expires.

Setting this value to 0 means the interactive block response is displayed once and the user bypass never expires.

Step 5

Click OK.

Step 6

Click Save to save the policy.

What to do next

  • Deploy configuration changes.

Configure URL Filtering Health Monitors

The following health policies alert if the system has problems obtaining or updating URL category and reputation data.

  • URL Filtering Monitor

  • Threat Data Updates on Device

To ensure that these are configured the way you want them, see Health Modules and Configuring Health Monitoring in the Cisco Secure Firewall Management Center Administration Guide.

Dispute URL Category and Reputation

If you disagree with a category or reputation assigned by Talos, you can submit a request for re-evaluation.

Before you begin

You will need your Cisco account credentials.

Procedure

Step 1

In the management center web interface, do one of the following:

Location of Dispute Option

Path to Dispute Option

Cloud Services configuration page

a. Navigate to the Integration > Other Integration > Cloud Services page.

b. Select Dispute URL categories and reputations.

Manual URL Lookup page

a. Navigate to the manual URL Lookup page: Analysis > Advanced > URL.

b. Look up the URL in question.

c. To see Dispute at the end of the table row, hover over the relevant entry in the list of results, then click dispute.

URL Connection Event

a. Navigate to any page under the Analysis > Connections menu that has a table that includes URLs.

b. Right-click an item in the URL Category or URL Reputation column (show hidden columns if needed) and select an option.

The Talos web site opens in a separate browser window.

Step 2

Sign in to the Talos site with your Cisco credentials.

Step 3

Review the information and follow the instructions on the Talos page.

Step 4

Look for information on the Talos site about how submitted disputes are handled and what response to expect, if any.

The dispute process is independent of Firepower products.

If the URL Category Set Changes, Take Action

The set of URL Filtering categories may occasionally change, in order to accommodate new web trends and evolving usage patterns.

These changes affect both policies and events.

Shortly before URL category changes are scheduled to occur, and after they occur, you will see alerts in the list of rules in any access control, SSL, and QoS policy that is affected by the changes, and on URL or Category in rules that you edit.

You should take action when you see these alerts.


Note

Updates to the URL category set as described in this topic are distinct from the changes that simply add new URLs to existing categories or re-classify misclassified URLs. This topic does not apply to category changes for individual URLs.

Procedure

Step 1

If you see an alert beside a rule in an access control policy, hover over the alert to see details.

Step 2

If the alert mentions changes to URL categories, edit the rule to see further details.

Step 3

Hover over the URL or Category in the rule dialog to see general information about the type of changes.

Step 4

If you see an alert beside a category, click the alert to view details.

Step 5

If you see a "More information" link in the description of a change, click it to view information about the category on the Talos web site.

Alternately, see a list and descriptions all categories at the link in URL Category and Reputation Descriptions.

Step 6

Depending on the type of change, take appropriate action:

Type of Category Change

What The System Will Do

What You Should Do

Existing category will soon be deprecated

Nothing yet. You have a few weeks to change affected rules.

If you do not take action in that time, the system eventually will not be able to redeploy the policy.

 Remove this category from all rules that include it. If there is a similar new category, consider using that category instead.

New category is added

By default, the system does not use newly added categories.

Consider creating new rules for the new category.

Existing category is deleted

The category will appear in the rule in strikethrough text (that is, with a line through the category name.)

You must delete the obsolete category from the rule before you can deploy the policy.

Step 7

Check your SSL rules (Category) for these changes and take action as needed.

Step 8

Check your QoS rules (URL) for these changes and take action as needed.

What to do next

Deploy configuration changes.

URL Category and Reputation Changes: Effect on Events

  • When URL categories change, events that the system processed before the category change will be associated with their original category names and will be labeled with Legacy. Events that the system processed after the category change will be associated with the new categories.

    Older, legacy events will age out of the system over time.

  • If a URL does not have a reputation at the time it was processed, the URL Reputation column in the event viewer will be empty.

Troubleshoot URL Filtering

Expected URL Category is Missing from the Categories List

The URL filtering feature uses a different set of categories than the Security Intelligence feature; the category that you expect to see may be a Security Intelligence category. To see those categories, look at the URLs tab on the Security Intelligence tab in an access control policy.

Initial Packets Are Passing Uninspected

See Inspection of Packets That Pass Before Traffic Is Identified and subtopics.

See also DNS Filtering: Identify URL Reputation and Category During DNS Lookup.

Health alert: "URL Filtering registration failure"

Verify that your management center and any proxies can connect to the Cisco cloud. You may need information about URL Filtering and URL categories in the following topics: Internet Access Requirements and Communication Port Requirements in the Cisco Secure Firewall Management Center Administration Guide.

How can I find the category and reputation of a particular URL?

Do a manual lookup. See Finding URL Category and Reputation in the Cisco Secure Firewall Management Center Administration Guide.

Error when attempting a manual lookup: "Cloud Lookup Failure for <URL>"

Make sure the feature is properly enabled. See the prerequisites in Finding URL Category and Reputation in the Cisco Secure Firewall Management Center Administration Guide.

URL appears to be incorrectly handled based on its URL category and reputation

Problem: The system does not handle the URL correctly based on its URL category and reputation.

Solutions:

If you want to immediately update the local URL category and reputation database, go to Integration > Other Integrations, click Cloud Services, then click Update Now. For more information, see URL Filtering Options.

A URL category or reputation is not correct

For access control or QoS rules: Use manual filtering, paying careful attention to rule order. See Manual URL Filtering and Configuring URL Conditions.

For SSL rules: Manual filtering is not supported. Instead, use distinguished name conditions.

See also Dispute URL Category and Reputation.

Web pages are slow to load

There is a tradeoff between security and performance. Some options:

Events Do Not Include URL Category and Reputation

  • Make sure you have included applicable URL rules in an access control policy, the rules are active, and the policies have been deployed to the relevant devices.

  • URL category and reputation do not appear in an event if the connection is processed before it matches a URL rule.

  • The rule that handles the connection must be configured for URL category and reputation.

  • Even if you have configured URL categories in the Categories tab in an SSL rule, you must also configure the URLs tab in a rule in your access control policy.

DNS Filtering is not working

Make sure you have completed all prerequisites and steps in Enable DNS Filtering to Identify URLs During Domain Lookup.

An End User Tries to Access a Blocked URL and the Page Just Spins and Times Out

When DNS Filtering is enabled and end users access a URL that is blocked, the page will spin but not load. End users are not notified that the page is blocked. This is currently a limitation when DNS filtering is enabled.

See DNS Filtering Limitations.

Events Include URL Category and Reputation but URL Field is Blank

If the DNS Query field is populated and the URL field is empty, this is expected when the DNS filtering feature is enabled.

See DNS Filtering and Events.

Multiple Events are Generated for a Single Transaction

A single web transaction sometimes generates two connection events, one for DNS filtering and one for URL filtering. This is expected when DNS filtering is enabled and:

  • the access control rule action for the traffic is Allow or Trust.

  • the system encounters a URL for the first time.

See DNS Filtering and Events.