BBOT 2.0 - Release Announcement

The recursive internet scanner gets an upgrade

Today, just in time for DEF CON 32, we are thrilled to announce the release of BBOT 2.0.

BBOT’s new features make it easier to use, while significantly speeding up scans.

Above: A chord graph of the relationships between BBOT's modules and the data types they produce/consume. Click the image to explore it interactively.

How did we get here?

Two years ago we released BBOT (Bighuge BLS OSINT Tool), an open-source scanner inspired by Spiderfoot. Its initial claim to fame was its ability to find more subdomains than any other tool. Since then, it's been steadily gaining users, and as of today, it's been downloaded 400K times. It's always wonderful to hear how people are using it in the bug bounty space. Whenever we hear that BBOT got someone a new payout by finding an outlier subdomain, or a critical RCE, it warms our hearts!

BBOT's success is a result of the countless contributions from the community (thank you!), which include many of the powerful new modules and features in 2.0. Development has been happening at a fast pace. To give you an idea, BBOT has already passed 4,000 commits, surpassing even Spiderfoot (with ~3,700), a tool that has been in active development for ten years! That is how much work has been going into BBOT -- both by us at BLS, and by the community -- and how we've already arrived at version 2.0!

New Features in 2.0

BBOT 2.0 keeps BBOT's original recursive design, while adding some powerful new features and optimizations.

Note: For full release notes, see Upgrading to BBOT 2.0.

Highlights

Here are the three main feature highlights for BBOT 2.0:

• Presets: An alternative to command-line flags that let you conveniently store your entire scan config in a single YAML file.

• BadDNS: Find subdomain hijacks and other DNS-related vulns.

• Speed Optimizations

  • YARA integration by @paulmmueller == insane boost in regex performance!

  • New DNS/HTTP Engines by @thetechr0mancer == leverage all your CPU cores!

Presets

Presets are one of the biggest features in BBOT 2.0. They were born out of necessity, to save you from having to construct giant BBOT commands. This was something we discovered early on: that due to BBOT's extreme customizability and the fact that it had over 100 modules, commands could get out of hand pretty quickly:

Presets solve this by consolidating all your scan settings into a single YAML config. You can create your own, or choose from a list of built-in presets.

You can list them all with -lp:

# list BBOT presets on the command-line
bbot -lp

And enable them with -p:

# enumerate subdomains on evilcorp.com
bbot -t evilcorp.com -p subdomain-enum

You can also mix and match an unlimited number of presets:

# combine subdomains + web spider
bbot -t evilcorp.com -p subdomain-enum spider

You can also create your own custom preset that includes other presets:

target:
  - evilcorp.com
  - 1.2.3.0/24

blacklist:
  - test.evilcorp.com

# include other presets
include:
  - subdomain-enum
  - spider

config:
  web:
    http_proxy: http://127.0.0.1:8080
  modules:
    github:
      api_key: 258e88dcbd3cd44d8e7ab43f6ecb6af0

Run BBOT with your custom preset:

bbot -p ./my_preset.yml

For a full list of built-in presets, see Full List of Presets.

For details on Presets, see the Documentation.

BadDNS

BadDNS is a slick DNS-hijacking tool written by @paulmmueller that's integrated into BBOT 2.0. It replaces BBOT's old subdomain_hijack module, and detects a myriad of vulnerabilities including dangling records.

For details, see the BadDNS Blog Post.

Speed Optimizations

BBOT 2.0 includes several very significant performance improvements, along with numerous small ones. These have combined together to make BBOT 2.0 close to 10x faster than its predecessor.

The two most significant performance-boosting features are YARA integration and new DNS + HTTP engines.

YARA Integration

Initially, we used Python's built-in regex library to mine useful goodies (emails, URLs, subdomains, etc.) from various sources like HTTP responses. This was effective, but not very efficient. Lots of regexes multiplied against lots of data resulted in serious slowdowns for the scan.

In BBOT 2.0, @paulmmueller has completely overhauled the excavate module to use YARA. This not only provides an insane speed boost (YARA has some wicked algorithms for this), it allows you to add on your custom YARA rules. Pair this with the work @Domwhewell has done to download Git repos and docker images, and pair that again with his module that extracts text from practically every file format known to man, what you effectively have is a grep -R for your target's entire web presence. Oh, and he also made a Trufflehog module to search all of that for secrets.

Yeah, we’ve all been busy. Stay tuned for new developments on these features. It's only going to get crazier!

New DNS / HTTP Engines

Before (BBOT v1)

After (BBOT v2)

Early on in BBOT's development, we transitioned to using asyncio. This simplified the code, and resulted in better stability and performance.

However, we are constantly looking for new ways to speed up scans, and the next bottleneck we encountered was in asyncio itself. Specifically, BBOT was issuing so many DNS and HTTP requests that it reached the max capacity of an asyncio loop within a single CPU core.

To address this, we've introduced an optimization to the way BBOT interacts with DNS and HTTP, which gives DNS and HTTP each their own dedicated Python process and asyncio event loop. To achieve this without the dreaded overhead of multiprocessing, we use ZeroMQ in a ROUTER/DEALER configuration. ZeroMQ enables extremely fast and efficient communication between the processes.

Community Shout-outs

Finally, we want to give special thanks to four specific members of the community, who have been most active in contributing to BBOT:

• @Domwhewell for continuing to create loads of powerful modules for secrets-looting and more.

• @Shadow for testing every new feature ruthlessly, and showering us with awesome ideas! (and congrats on writing his first module).

• @colin-stubbs for bringing his DNS expertise to BBOT by adding CAA-record capabilities (and many more to come!).

• @nicpenning and @CarsonHrusovsky for integrating BBOT with Elasticsearch.

Thanks guys, you’ve been awesome to work with! Let’s keep building this thing!