User Guide

CBC Binary Toolkit User's Guide

Who is this for?

Those who want to get up and running with the built-in functionality provided with the toolkit. This includes how to use the example YARA Analysis Engine, as well as details of the built-in SQLite State Management database, and other components.

Where do I go if I have questions?

Review the troubleshooting section if something isn't working as expected. If you want to modify the toolkit, see the Developer Guide.

Index

Configuring API Access
Getting Started Out of the Box
Configuring YARA rules
How to create a Feed
Troubleshooting

Configuring API Access

This section includes

1. How to create an API Key

2. Adding your API Key to your configuration file

If you have already set up CBAPI, skip ahead to Getting Started Out of the Box.

API Key

The toolkit uses CBAPI to communicate with the Carbon Black Cloud. CBAPI requires an API Key. This toolkit communicates with the Unified Binary Store (UBS) and ThreatHunter Feeds. Therefore, your API Key must have access to UBS and Feeds.

This can be achieved by creating a key with API Access Level Type, or with a Custom (RBAC) Access Level with threathunter.feeds , ubs.org.sha256 and ubs.org.file permissions.

Option 1: Create an API Key with `API` Access Level

Use the preset API Access Level Type to create a new API Key.

`Settings -> API Access -> API Keys Tab -> + Add API Key -> Access Level Type Dropdown [API]`

Creating an API Key in the console

Option 2: Create a Custom (RBAC) Access Level

Create a Custom Access Level.

`Settings -> API Access -> Access Levels Tab -> + Add Access Level`

Select the following permissions:

Dot Notation Name for Permission	Access Levels
`threathunter.feeds`	`READ` and `UPDATE`
`ubs.org.sha256`	`READ`
`ubs.org.file`	`READ`

Creating a custom RBAC Access Level

Option 2 (continued): Create an API Key with a Custom (RBAC) Access Level

Use the Custom (RBAC) Access Level created above to make a new API Key.

`Settings -> API Access -> API Keys Tab -> + Add API Key -> Access Level Type Dropdown [Custom] -> Custom access level dropdown [level with Feeds and UBS permissions]`

Creating a custom RBAC Access Level

Once an API Key with the required permissions has been created, it can be used in the toolkit.

Credentials in Configuration File

After creating an API Key with UBS and Feeds access, you must put the key in your configuration file. The toolkit expects an api_token, which is a concatenation of an API Key's Secret Key and Key ID. The values below can be found in your Carbon Black Cloud console, and are unique to your organization and API Key.

Name	Example Value
Org Key	`P1Q2R3S4`
API Secret Key	ABCDEFGH
API Key ID	1234
`api_token`	`ABCDEFGH/1234`

In this example, the carbonblackcloud section of the toolkit configuration file would look similar to this:

carbonblackcloud:
  url: https://defense-prod05.conferdeploy.net
  api_token: ABCDEFGH/1234
  org_key: P1Q2R3S4
  ssl_verify: True
  expiration_seconds: 3600

The toolkit configuration file is covered in detail further down in the guide.

Additional Authentication Information

For more information on API Keys, Access Level Types, and RBAC Permissions, visit the Developer Network Authentication Guide.

Getting Started Out of the Box

This section includes

1. Components Included Out of the Box

Analysis Utility
Built-in State Management Database
YARA analysis engine

2. How to use Out of the Box Functionality

Editing your configuration file
Running the analysis utility

Assumptions

These are the assumptions we make when using this system as-is:

The Toolkit has been installed via pip

pip install cbc-binary-toolkit

You have configured CBAPI
The built-in examples are being used for execution, including the Analysis Utility, Built-in State Management Database, and YARA Analysis Engine.

Components Included Out of the Box

The toolkit can be run with minimal configuration out of the box. We've included an example Analysis Utility that uses a Built-in State Management Database and YARA Analysis Engine to show what is possible.

Analysis Utility

The analysis utility is your entry point to the toolkit. Through command line arguments, you can analyze hashes, clear hashes from the state management database that have already been analyzed, or restart analyzing hashes if an error occurs during runtime and some hashes aren't fully analyzed.

See the How to use Out of the Box Functionality section below for instructions on running the utility.

How the Analysis Utility example steps through execution

The example analysis utility

Loads your configuration file (put link to configuring the file)
Initializes the necessary components (put link to components/ maybe diagram), then
Executes your desired functionality: analyze, clear, or restart.

If you choose to analyze hashes, the toolkit then

Communicates with the Unified Binary Store (UBS)
Gives the metadata retrieved from the UBS to the analysis engine
Accepts analysis results from the analysis engine
Stores then sends the analysis results to a Feed

The feed that analysis results are sent to is specified by Feed ID in your configuration file.

Data Flow for Analysis Utility's`analyze` command

Data Flow within the Binary Analysis Toolkit

Built-in State Management Database

We've included a built-in example for managing the state of hashes you've input for analysis. The example uses SQLite3 with two tables:

run_state: tracks checkpoints as hashes travel through the toolkit
report_item: stores Reports queued for dispatch to the Carbon Black Cloud.

As a user, you will not interact with the state management database.

Example YARA Analysis Engine

The YARA analysis engine is using the yara Python package. YARA is a rule-based static analysis method. With string matching, you can categorize binaries and assign different severities to different categories or rules.

The engine accepts binary metadata, which contains a link to download the binary from AWS S3. After downloading, the binary is checked against the rules in the rules_file. The location of the rules file is specified in the engine: section of your configuration file.

Data Flow within the Binary Analysis Toolkit

How to use the example YARA Analysis Engine

The example YARA engine is located at cbc_binary_toolkit_examples/engine/yara_local/yara_engine.py. It uses one rule, called CompanyMatch.

rule CompanyMatch
{
	meta:
		sev = 2
	strings:
		$microsoft = "microsoft" nocase
		$google = "google" nocase

	condition:
		$microsoft or $google
}

This rule looks in a binary for the case-insensitive strings "microsoft" or "google". The rule has a low severity value of 2, because these are not signs of malicious behavior.

To use this example engine, copy the values from cbc_binary_toolkit_examples/engine/yara_local/conf.yaml.example to your toolkit configuration file and replace feed_id with a real Feed ID. For reference, the example YARA configuration is below.

engine:
  name: Yara
  feed_id: example-feed-id
  type: local
  _provider: cbc_binary_toolkit_examples.engine.yara_local.yara_engine.YaraFactory
  rules_file: __file__/example_rule.yara

Many open and closed source repositories of YARA rules are available online. To make the example more useful, edit the example_rule.yara file and add more rules. You may also edit the rules_file value in the engine: section of your configuration file to use a different rules file. See Configuring YARA Rules for detailed instructions.

How to use Out of the Box Functionality

After installing the toolkit through pip, you must edit your configuration file.

Editing your configuration file

The analysis utility can be supplied with the --config parameter or -c flag to specify where to find your configuration file. If you do not specify a path to a configuration file, the analysis utility defaults to cbc-binary-toolkit/config/binary-analysis-config.yaml.example

At the very least, you must modify the carbonblackcloud and engine sections to make the toolkit functional.

Configuration fields

`carbonblackcloud`

Parameter	Definition	Format	Example Value
`url`	Base URL of your Carbon Black Cloud server	https://defense-<environment>.conferdeploy.net	`https://defense-prod05.conferdeploy.net`
`api_token`	Concatenated API Secret Key and API Key ID	[API Secret Key] / [API Key ID]	`ABCDEFGHIJKLMNOPQRSTUVWX/ABCDE12345`
`org_key`	Organization's Org Key, found in console	8 alphanumeric characters	`ABCDEFGH`
`ssl_verify`	Use SSL to verify connection to Carbon Black Cloud	True or False	`True`
`expiration_seconds`	How long links to download binaries from AWS S3 should be valid for, in seconds	Integer	`3600`

`carbonblackcloud` parameter explanations

url will be in the form https://defense-<environment>.conferdeploy.net/, where <environment> is prod02, prod03, prod05, etc.

api_token will be a concatenation of an API Secret Key and API Key ID. See Credentials in Configuration File for more detail.

org_key can be found in your Carbon Black Cloud Console, under Settings -> API Access -> API Keys.

ssl_verify can be either True or False, depending on if you want to verify your connection with the Carbon Black Cloud via SSL.

expiration_seconds is how long the links to download binaries from AWS S3 will be valid for. This may need to be adjusted depending on how quickly your analysis engine can process binaries. With the included YARA analysis engine, 3600 seconds (60 minutes) is more than sufficient.

`database`

Parameter	Definition	Format
`_provider`	Class name of your chosen persistence provider. Default is the built-in `cbc_binary_toolkit.state.builtin.Persistor`	Fully-qualified class name
`location`	Pathname of the database file to be opened. Default is ":memory", where the database is stored in RAM	path-like object

To get up and running quickly, use the built-in cbc_binary_toolkit.state.builtin.Persistor persistence provider.

`engine`

Parameter	Definition	Format	Example Value
`name`	Name of your analysis engine	String	`"YARA Analysis Engine"`
`feed_id`	The Feed ID where the toolkit should send analysis results to. Can be retrieved from Carbon Black Cloud Feed API	21 alphanumeric characters	`578FTEQBNWSHCPGIXD4R3`
`type`	Whether the analysis engine is running locally or remotely. Only `local` analysis is currently supported.	`local`	`local`
`_provider`	Class name of your chosen analysis engine	Fully-qualified class name	`cbc_binary_toolkit_examples.engine.yara_local.yara_engine.YaraFactory`

The engine section of your configuration file is extensible -- you may add or remove parameters and adjust the analysis utility code accordingly. See the Developer Guide for more information.

For example, the YARA analysis engine requires a rules_file parameter to tell the engine where to look for YARA rules.

Parameter for YARA	Definition	Format	Example Value
`rules_file`	Location of file containing YARA rules	path-like object	`__file__/example_rule.yara`

Running the Analysis Utility

Analyze

user@machine:~$ cbc-binary-analysis analyze --file path/to/hashes.csv

Here, we tell the toolkit we want to analyze a --file, and give the path to our file containing hashes. In this case, our hashes were stored in a CSV file, but we can also use a JSON string containing hashes.

user@machine:~$ cbc-binary-analysis analyze --list ["hashone64charslong", "hashtwo64charslong"]

Clear

Clear all Reports from the persistor:

user@machine:~$ cbc-binary-analysis clear

Clear Reports from the persistor completed after a specific timestamp. Timestamp must be in ISO 8601 date format (YYYY-MM-DD HH:MM:SS.SSS):

user@machine:~$ cbc-binary-analysis clear --timestamp 2020-01-02 10:30:00.000

Restart

Resume analysis on hashes that have already been ingested, but did not finish analysis and sending the results to the Carbon Black Cloud:

user@machine:~$ cbc-binary-analysis restart

For all command line options, see the Running Binary Analysis tool section in the README for detailed usage, or run cbc-binary-analysis --help from your terminal.

Roughly expected performance

Insufficient info to write anything atm

Configuring YARA rules

To use custom YARA rules, use the example rules file or create your own.

Using the Example Rules File

The file is located at src/cbc_binary_toolkit_examples/engine/yara_local/example_rule.yara.

Rules should be listed consecutively, and need to have a meta section with a sev property between 1 and 10.

rule MyFirstRule
{
	meta:
		sev = 7
	strings:
		$string1 = "malware"
		$string2 = "bad_app"
		$string3 = "malicious"
	condition:
		any of them
}

rule MySecondRule
{
	meta:
		sev = 3
	strings:
		$string1 = "rm -rf /"
		$string2 = "malicious"
		$string3 = { 48 45 4C 4C 4F 0A  }
	condition:
		all of them
}

See Writing YARA Rules for more information.

Making your own file

You can use your own file instead of the example file.

If your file is located in the same directory as yara_engine.py, append __file__ to the name of your file in the engine section of your configuration file.

engine:
  name: Yara
  feed_id: Seql2oSTkO8rZrTbBkxnQ
  type: local
  _provider: cbc_binary_toolkit_examples.engine.yara_local.yara_engine.YaraFactory
  rules_file: __file__/my_new_rule_file.yara

Otherwise, if your rules file is not located in the same directory as yara_engine.py, put the full path name to the file in your configuration file.

engine:
  name: Yara
  feed_id: Seql2oSTkO8rZrTbBkxnQ
  type: local
  _provider: cbc_binary_toolkit_examples.engine.yara_local.yara_engine.YaraFactory
  rules_file: /Users/Developer/rules_file.yara

How to Create a Feed

With CBAPI Example Script

CBAPI-Python includes an example to create a feed through the command line.

user@machine:~$ python cbapi-python/examples/threathunter/create_feed.py --name="Binary Analysis Feed" --owner="Developer Relations" --url="https://www.carbonblack.com/" --summary="Binary Analysis YARA Results" --category="Analysis"

> Feed object, bound to https://defense-<environment>.conferdeploy.net.
-------------------------------------------------------------------------------

                  access: private
                category: Analysis
                      id: jVxpUZ32SGKidfTNNDFuVr
                    name: Binary Analysis Feed
                   owner: ABCDEFGH
            provider_url: https://www.carbonblack.com/
            source_label: None
                 summary: Binary Analysis YARA Results

The id is listed as output of the example script. Copy this value into feed_id in your configuration file.

With API Requests

Postman is a useful tool for sending API requests through a GUI. You can import this ThreatHunter Feed Operations Collection via url.

Importing a Collection into Postman

In Postman, use the "Create new private feed" request to make a new feed. Be sure to update the X-Auth-Token header to use your API token, and the Body of the request with details you want.

With cURL

user@machine:~$ curl --location --request POST 'https://defense-<environment>.conferdeploy.net/threathunter/feedmgr/v2/orgs/MYORGID/feeds' \
--header 'X-Auth-Token: MYAPISECRETKEY/MYAPIKEYID' \
--header 'Content-type: application/json' \
--header 'Content-Type: text/plain' \
--data-raw '{"feedinfo": {"name": "Binary Analysis Feed", "owner": "Developer Relations", "provider_url": "https://www.carbonblack.com", "summary": "Binary Analysis YARA Results", "category": "Analysis"},
 "reports": []}
'

> Feed object, bound to https://defense-<environment>.conferdeploy.net.
-------------------------------------------------------------------------------

                  access: private
                category: Analysis
                      id: jVxpUZ32SGKidfTNNDFuVr
                    name: Binary Analysis Feed
                   owner: ABCDEFGH
            provider_url: https://www.carbonblack.com/
            source_label: None
                 summary: Binary Analysis YARA Results

Troubleshooting

Received error code 403 from API:

If you receive an API response similar to:

{"error_code": "FORBIDDEN", "message": "Connector ID A1B2C3D4E5 does not have permission ubs.org.file and action READ on org key ABCDEFGH"}

Ensure your API Key has the required permissions.

yara_engine: Attempted to init engine with non matching engine config

Ensure your engine name is Yara in your configuration file.

engine:
	name: Yara
	feed_id: example-feed-id
	type: local
    _provider: cbc_binary_toolkit_examples.engine.yara_local.yara_engine.YaraFactory

Exiting as default example config file could not be found and no alternative was specified

Make sure the default binary-analysis-config.yaml.example configuration file is in the config folder.

Error downloading hashes from Unified Binary Store: Received a network connection error...

Check your internet connection and retry.

Information here is current as of Carbon Black Cloud Binary Toolkit 1.0.0

CBC Binary Toolkit

Home
Performance Metrics
User Guide
Developer Guide
- Replace Components
  - Analysis Engine
  - Persistence System
- Architecture Overview
- Schemas
- Interfaces
Glossary