[Dr. WoW Season 2] [No 8] Classification of Massive URLs---URL Filtering Configuration

Latest reply: Jun 1, 2018 06:29:11 799 1 1 0

Configuring URL filtering is not as simple as it seems. The business characteristics of enterprise users and Internet access control policies determine URL filtering requirements. However, the URL filtering process determines how to configure URL filtering.

Let's brush up on the URL filtering process. The gray-shaded parts shown in the flowchart require special attention during URL filtering configuration. You can mix and match the configuration options for different URL filtering scenarios.

20180601102625477001.png

 

URLs can be complex. Some sub-websites are defined as second-level domain names, as shown in example 1; some sub-websites are defined as paths, as shown in example 2; and some URLs include other website addresses in their parameter, as shown in example 3.

20180601102626910002.png

 

In addition, some web pages may contain links to other websites. For example, some pages of a shopping website may have links to product images hosted on another website. Blocking these links will prevent the images from being displayed on the shopping website.

When configuring URL filtering, you need to analyze URLs to understand the website architecture and the information in web pages. All these factors need to be considered to ensure satisfactory URL filtering results.

URL Filtering Dependencies

URL filtering involves several functional modules working together, as shown in the following figure:

20180601102626988003.png

The major parts of URL filtering are URL filtering profile and security policy. A URL filtering profile defines the URL blacklist and whitelist and the actions for URL categories. A security policy references a URL filtering profile and defines the matching conditions (used to identify traffic subject to URL filtering) and an action, which must be set to allow. Other modules are described as follows:

20180601102627503004.pngNGFWs are shipped with a predefined URL category file, which is automatically loaded during NGFW startup. This file can also be manually loaded.

20180601102628033005.pngThe parameters of the URL category database must be configured for the NGFW to remotely query Huawei Security Service Centers for URL categories. These parameters include the country/region where the NGFW resides and the action to be taken on URL query timeout. Note that remote query requires a license.

20180601102628143006.pngUser-defined URLs and categories can be created to supplement predefined ones. You can create categories on the NGFW and add URLs to them, or add URLs to predefined categories.

20180601102629247007.pngWeb pages are pushed to users if the requested URLs are blocked by the NGFW. The NGFW pushes web pages for different actions, such as block by the blacklist, block for matching a predefined URL or category, block for matching a user-defined URL or category, and block by the default action. You can customize the pushed web pages or use the default ones provided by the NGFW.

20180601102630032008.pngURL filtering logs indicate whether the URL filtering results meet your expectations. You can then modify the URL filtering configuration accordingly.

Description of the Web Configuration Page

The following figure is a snapshot of the web UI for configuring URL filtering profiles (the USG6000 V100R001C30SPC100 is used as an example). Each configuration item on the UI is subsequently described.

20180601102631379009.jpg

 

The URL filtering profile configuration page is divided into three areas. Area 1 is the basic configuration area. In this area, you can set the profile name, a description, action mode, and default action. Area 2 is the URL blacklist and whitelist configuration area. In this area, you can configure URL blacklist and whitelist, which apply only in the profile in which they are created. Area 3 is the URL filtering level configuration area. In this area, you can configure the actions for predefined and user-defined categories.

For predefined categories, the NGFW provides three default URL filtering levels. These levels define the actions for the categories and are set by administrators.

l  High: All social networking and video sharing websites, as well as the websites blocked at Middle and Low levels, are blocked.

l  Middle: Gambling, weapons-related, ***ographic, and illegal websites, as well as the websites blocked at Low level, are blocked.

l  Low: ***ographic and drug- and violence-related websites are blocked.

Administrators can manually change the actions for predefined URL categories if necessary. Note that the action for a sub-category prevails over the action for the category. Administrators can apply the action for a category to all the sub-categories under the category or specify the actions for the sub-categories individually, as shown in the following figure:

20180601102631872010.png

The default action for user-defined URL categories is allow. You can change this if required.

URL Filtering Configuration Examples

Now let's see some real-world examples. Before configuring URL filtering, ensure that the predefined URL category file has been loaded (and the remote query function has been deployed if possible).

As shown in the following figure, the NGFW is deployed between the enterprise user and web server. URL filtering is configured on the NGFW to control the web resources accessible to enterprise users. HTTP is used in this example. If HTTPS is used, an SSL decryption policy must be configured on the NGFW.

20180601102632938011.png

Example 1: Use predefined URL categories to allow/block user access requests

An enterprise needs to allow access to search engines and portal websites but block access to job websites. To meet these needs, the administrator can use predefined URL categories.

The administrator must configure a URL filtering profile, set the action for the Job Search category to Block, and set the action for the Search Engines/Portals category to Allow, as shown in the following figure.

 

20180601102633547012.jpg

 

A URL filtering profile takes effect only after it is committed.. Committing a profile takes time, so it is recommended to configure all URL filtering profiles before you click Commit..

The URL filtering profile is then referenced in a security policy, and the web page information to be pushed for blocked URL access requests is configured.

After the preceding configurations are complete, the employees of the enterprise can access search engines, such as www.google.com. However, access to job websites, such as www.usajobs2go.com, is blocked by the NGFW. When such a website is blocked, the NGFW pushes a web page to inform the employees.

If the URL filtering does not work as expected and some websites are incorrectly allowed or blocked, incorrect URL categorization is the likely cause. In this case, we need to check URL filtering logs and determine the predefined category of the incorrectly allowed or blocked URL.

To check the existing category of a URL, enter the URL in the search box of the URL category web interface.

20180601102633501013.jpg

The following are two examples:

20180601102634870014.jpg

 

20180601102635272015.jpg

 

If the category of a URL is incorrect, we can change the action for the category. Alternatively, we can create a user-defined category and add the URL to the user-defined category. User-defined categories have a higher priority than predefined categories. Therefore, the actions defined for user-defined categories will prevail.

If no category is displayed for a URL, the URL is not included in the predefined URL category file that is loaded by default. If remote query is unavailable, the default action will be taken for the URL. To correctly allow or block the URL, the remote query service needs to be purchased or a user-defined category needs to be created.

The following configuration script illustrates how to use predefined URL categories to allow/block user access:

#

profile type url-filter name profile_url_filter_example1

 category pre-defined subcategory-id 125 action block

 category pre-defined subcategory-id 126 action allow

 category pre-defined subcategory-id 190 action allow

#

security-policy

rule name policy_url_filter_example1

  source-zone trust

  destination-zone untrust

  source-address 192.168.0.0 mask 255.255.255.0

  profile url-filter profile_url_filter_example1

  action permit

#

Example 2: Use user-defined URL categories to allow/block user access requests

An enterprise needs to block access to www.example1.com and www.example2.com. However, these URLs are not included in the predefined URL category file that is loaded by default, and remote query is unavailable.

Therefore, the administrator must create a user-defined category and add the URLs to it. Note that the NGFW is case-insensitive in URL filtering and so the URLs are typed in lowercase.

20180601102635729016.jpg

 

The administrator must then configure a URL filtering profile and set the action to Block for the user-defined category. After the configurations are complete, the administrator must click Commit and reference the profile in the security policy.

20180601102636730017.jpg

 

If the employees of the enterprise attempt to access www.example1.com and www.example2.com, the NGFW blocks the attempt and pushes a web page to inform the employees.

The following configuration script illustrates how to use user-defined URL categories to allow/block user access:

#

url-filter category user-defined name user_define_url

 add url www.example1.com

 add url www.example2.com

#

profile type url-filter name profile_url_filter_example2

 category user-defined name user_define_url action block

#

security-policy

rule name policy_url_filter_example2

  source-zone trust

  destination-zone untrust

  source-address 192.168.0.0 mask 255.255.255.0

  profile url-filter profile_url_filter_example2

  action permit

#

Example 3: Allow access to a specific website and block access to all other websites

An e-commerce enterprise needs to allow access to www.ebay.com (an e-commerce website) and block access to all other websites. To meet this requirement, the administrator needs to whitelist the website, set the action to Block for all categories, and set the default action to Block.

The administrator must create a URL filtering profile, add www.ebay.com to the whitelist, set the action to Block for all categories, and then set the default action to Block. If remote query is used, the administrator must set the action on remote query timeout to Block. After the configurations are complete, the administrator must click Commit and reference the profile in the security policy.

20180601102637213018.jpg

 

The employees of the enterprise can then access www.ebay.com. However, access to all other websites is blocked by the NGFW, which pushes a web page to inform the employees if they attempt to access such websites.

If www.ebay.com contains domain names like abc.ebay.com, such domain names do not match the URL whitelist and will be blocked. In this case, we can use the host whitelist, as shown in the following figure:

20180601102638983019.jpg

 

The host whitelist ensures that web pages like xxx.ebay.com will match the whitelist and can be accessed by the employees.

If we create a URL whitelist entry like *.ebay.com, the URL whitelist does not work in this situation because only the part after the wildcard (*) will be used in whitelist matching. Therefore, URLs like www.ebay.com/1.asp will not match the URL whitelist entry.

There is another special situation. Only the top-level domain of the website will match *.ebay.com, but the pages on the website may have image and video resources linked from other websites. Such websites will be blocked and, as a result, some image and video resources may be unavailable. Therefore, such websites also need to be added to the whitelist.

The following configuration script illustrates how to allow access to a specific website and block access to all other websites:

#

profile type url-filter name profile_url_filter_example3

 add whitelist host *.ebay.com

 category pre-defined subcategory-id 101 action block

 ……    //Code omitted for brevity.

 default action block

#

security-policy

rule name policy_url_filter_example3

  source-zone trust

  destination-zone untrust

  source-address 192.168.0.0 mask 255.255.255.0

  profile url-filter profile_url_filter_example3

  action permit

#

Example 4: Allow access to all websites except specified websites

To prevent employees from accessing www.ebay.com (an e-commerce and shopping website) and www.facebook.com (social media website) and consequent productivity loss, an enterprise decides to block these two websites and allow access to all other websites. To meet this requirement, the administrator needs to blacklist the two websites, set the action to Allow for all categories, and set the default action to Allow.

The administrator must configure a URL filtering profile, and add the two URLs to the host blacklist so that websites like xxx.ebay.com and xxx.facebook.com match the blacklist entries.

20180601102638091020.jpg

 

The administrator must then set the action to Allow for all categories and set the default action to Allow. If remote query is used, the administrator must set the action on remote query timeout to Block. After the configurations are complete, the administrator must click Commit and reference the profile in the security policy.

These configurations meet the requirements of the enterprise as they allow the employees to access all websites except those that match the *.ebay.com and *.facebook.com blacklist entries.

The following configuration script illustrates how to allow access to all websites except specified websites:

#

profile type url-filter name profile_url_filter_example4

 add blacklist host *.ebay.com

 add blacklist host *.facebook.com

#

security-policy

rule name policy_url_filter_example4

  source-zone trust

  destination-zone untrust

  source-address 192.168.0.0 mask 255.255.255.0

  profile url-filter profile_url_filter_example4

  action permit

#

Example 5: Allow access to specified sub-websites of a website and block access to all other sub-websites

An enterprise needs to allow access to maps.google.com and news.google.com and block access to all other sub-websites, such as Google Books, Google Developers, and Google Store. The administrator needs to use blacklist and whitelist to meet these requirements.

The administrator must configure a URL filtering profile, add the two sub-websites to the whitelist, and add the website to the blacklist. After the configurations are complete, the administrator must click Commit and reference the profile in the security policy.

20180601102639763021.jpg

 

The two sub-websites can be added to URL whitelist or host whitelist, but the website can be added only to the host blacklist. This is because of the same reason explained in example 3.

These configurations meet the requirements of the enterprise and allow the employees to access maps.google.com and news.google.com but prevent them from accessing any other sub-websites of Google.

The following configuration script illustrates how to allow access to all websites except specified websites:

#

profile type url-filter name profile_url_filter_example5

 add blacklist host *.google.com

 add whitelist host maps.google.com

 add whitelist host news.google.com

#

security-policy

rule name policy_url_filter_example5

  source-zone trust

  destination-zone untrust

  source-address 192.168.0.0 mask 255.255.255.0

  profile url-filter profile_url_filter_example5

  action permit

#

 

 

To view the list of all Dr. WoW technical posts, click here.
This post was last edited by dr.wow at 2018-06-01 02:26.
  • x
  • convention:

WheatGrass
Created Jun 1, 2018 06:29:11 Helpful(0) Helpful(0)

Thanks for sharing [Dr. WoW Season 2] [No 8] Classification of Massive URLs---URL Filtering Configuration-2676533-1
  • x
  • convention:

Reply

Reply
You need to log in to reply to the post Login | Register

Notice Notice: To protect the legitimate rights and interests of you, the community, and third parties, do not release content that may bring legal risks to all parties, including but are not limited to the following:
  • Politically sensitive content
  • Content concerning pornography, gambling, and drug abuse
  • Content that may disclose or infringe upon others ' commercial secrets, intellectual properties, including trade marks, copyrights, and patents, and personal privacy
Do not share your account and password with others. All operations performed using your account will be regarded as your own actions and all consequences arising therefrom will be borne by you. For details, see " Privacy."
If the attachment button is not available, update the Adobe Flash Player to the latest version!
Login and enjoy all the member benefits

Login and enjoy all the member benefits

Login