Content discovery

This function can be used to discover content and functionality which is not linked from visible content that you can browse to or spider.

To access this function, select an HTTP request anywhere within Burp, or any part of the Target site map, and choose "Discover content" within "Engagement tools" in the context menu.

Burp uses various techniques to discover content, including name guessing, web crawling, and extrapolation from naming conventions observed in use within the application. Discovered content is displayed within a special site map that is specific to the discovery session, and can also optionally be added to the main suite site map.

Control

This tab shows you the current status of the discovery session.

The toggle button indicates whether the session is running, and lets you pause and restart the session.

The following information is displayed about the progress of the discovery session:

• Number of requests made
• Number of bytes transferred in server responses
• Number of network errors
• Number of discovery tasks queued
• Number of spider requests queued
• Number of responses queued for analysis

The individual discovery tasks that are queued are shown in a table. The discovery engine works recursively, and when a new directory or file is discovered, further tasks are derived from this, depending on the configuration. For example, when a new directory is discovered, Burp might add tasks to look for sub-directories and files within that directory; or, when a new file is discovered, Burp might add a task to check for the same base filename with different file extensions. Newly added tasks are prioritized according to their likelihood of quickly discovering new content.

Target

These options let you define the start directory for the content discovery session, and whether files or directories should be targeted. The following options are available:

• Start directory - This is the location where Burp will start looking for content. Only items within this path and its subdirectories will be requested during the session.
• Discover - This option determines whether the session will look for files or directories or both. If you are checking for directories, you can choose whether and how deep to recurse into discovered subdirectories.

Filenames

These options let you configure the sources that Burp should use for generating filenames to test. The following options are available

• Built-in short file list
• Built-in short directory list
• Built-in long file list
• Built-in long directory list
• Custom file list
• Custom directory list
• Names discovered in use on the target site. If this option is selected, Burp will maintain a list of all directories and filename stems that have been discovered on the target site, and will also check for these in each new directory that is tested.
• Derivations based on discovered items. If this option is selected, Burp will attempt to guess item names based on those that have already been discovered. For example, if the directory AnnualReport2018 is discovered, Burp will also check for AnnualReport2019, AnnualReport2020, etc.

File extensions

These settings control how the discovery session adds file extensions to file stems that are being tested. The file stems themselves are derived according to the filenames options. When each file stem is tested, Burp check for various different extensions, according to these settings. The following options are available:

• Test these extensions - This option lets you configure a list of extensions that Burp will always check for. You can fine-tune the default list based on the technologies known to be in use on the target application.
• Test all extensions observed on target site - If this option is selected, then Burp will automatically check for file extensions that have been observed in use on the target site. This option is useful when you don't know exactly what extensions or technologies are in use. You can also configure a list of extensions that you don't want to check for even if found to be in use (such as image files).
• Test these variant extensions on discovered files - This option lets you configure a list of extensions that Burp will additionally check for using the stems of discovered filenames. This option is useful to check for backup copies of existing files.
• Test file stems with no extension - If this option is selected, Burp will check for each file stem with no extension added.

Discovery engine

These settings control the engine used for making HTTP requests when discovering content, and interaction with the suite site map. The following options are available:

• Case sensitivity - This setting controls whether Burp will handle filenames case sensitively. If "Auto-detect" is selected, then Burp will start by handling filenames case sensitively, and on discovering the first new item, will test the server's treatment of case variations. Depending on that treatment, Burp may revert to handling filenames case insensitively.
• Add discovered content to suite site map - If this option is selected, then new items identified in the current discovery session will be automatically added to the main suite site map.
• Copy content from suite site map - If this option is selected, then the discovery session will copy any existing relevant content from the main suite site map into the discovery site map, to provide a stronger starting basis for discovering new content.
• Spider from discovered content - If this option is selected, then the discovery session will perform conventional web crawling, and will process the responses to discovery requests looking for links to additional new content.
• Number of discovery threads - This option controls the number of concurrent requests the discovery engine is able to make.
• Number of spider threads - This option controls the number of concurrent requests the crawling function is able to make, if enabled.

Site map

The discovery session employs its own site map, showing all of the content which has been discovered within the defined scope. If you have configured Burp to do so, newly discovered items will also be added to Burp's main site map.