Data Export Configuration

Article author
Brian Reynolds
  • Updated

The Data Export feature offers the ability for single organizations and MSP users to export data at regularly timed intervals, and will allow users to configure exports to Amazon S3 or Splunk. Once configured, the data available in the CSV version of the Query Log will transmit in near real-time.

Data Export allows users to combine their Query Log data with other data for monitoring/action/alerting. The Data Export feature can integrate directly with most SIEMs including Datadog, Humio, LogRythm, QRadar, Splunk, and Sumo Logic. Data Export can also connect to S3 compatible services such as Wasabi and Backblaze.

Prerequisites

  • Data Export add-on enabled.
  • Access to configure with one of the supported roles: Owner, Admin, or Super User roles.
    Users with Edit Policies or Read-Only roles will not have access to configure the this feature. 
  • Access to Data Export IP addresses. Review the URL's and IP Ranges documentation for specific information. 

Data Exports are configured at a single organization level or within each sub-organization (MSPs).
Only a single export service can be used per organization or sub-organization.

Data Export Configuration Demonstration Video

 

Amazon S3 Data Export Configuration

  1. Navigate to Tools > Data Export
  2. Select CONFIGURE DATA EXPORT
  3. Select the amazon S3 service
  4. Select CONTINUE
    DE2.png
  5. Enter your Amazon S3 Bucket name.
    This is where you will be hosting your information which is a unique name for your account. Refer to Amazon's guide for more information on where to locate this. 
    DE3.png
  6. Input optional parameters if utilizing
    1. Key Prefix value
      Used to organize the data that is stored in Amazon S3 buckets.
    2. Endpoint fully qualified URL
      Refer to the Amazon S3 guide for more information.
  7. Enter the Region your S3 bucket resides in (e.g., us-east-1)
    For more information, refer to Amazon's guidance on creating a bucket.
  8. Enter the Access Key ID and Secret Access Key values
    Refer to the AWS security credentials page for more details on how to generate an Access Key.
  9. Select VERIFY & TEST ACCOUNT to test the connection
    DE_4.png
    The message displayed below confirms that the account has been successfully configured.
    DE_5.png
  10. Select FINALIZE to complete the process.
    The configuration is now complete.

Splunk Data Export Configuration

Splunk's HTTP Event Collector (HEC) API is utilized, which uses a well-recognized protocol for transferring data. It is scalable, secure, token-based for convenience, and easy to maintain.

The protocol is often implemented by SIEMs and data tools apart from Splunk, and may work out of the box with your preferred data tool as well. For example, Humio implements a one-to-one HEC API which is already confirmed to work with this Data Export feature.

  1. Navigate to Tools > Data Export
  2. Select CONFIGURE DATA EXPORT
  3. Select the splunk> service
  4. Select CONTINUE
    DE5.png
  5. Turn off Use compressed data, if necessary (default is enabled)
  6. Enter your HTTP Event Collector URL and your Active Event Collector Token
    Refer to Splunk's Getting Data In guide for information on how to generate an HTTP Event Collector URL and Token.
    Splunk-DataExport2.png
  7. Select VERIFY & TEST ACCOUNT to test the connection
    DE_4.png
    The message displayed below confirms that the account has been successfully configured.
    DE_5.png
  8. Select FINALIZE to complete the process.
    The configuration is now complete.

Troubleshooting Data Export Configuration Errors

There was an error processing your export

"There was an error processing your export" indicates that either the access credentials have changed or the region setting has changed, which would cause the export setup to fail.  

Displayed when

  • Attempting to verify and test the account configuration
  • If an export fails

An email is sent after 20 errors have occurred. 
CleanShot 2023-04-04 at 16.13.24@2x.png

Resolving

  1. Select EDIT
  2. Check to ensure that the current settings match what you have configured for AWS or Splunk
  3. Make necessary updates to the data export configuration
  4. Select VERIFY & TEST ACCOUNT to ensure that there is no longer an error.

There was an issue verifying & testing your account

Very similar to the prior error, "There was an issue verifying & testing your account" appears when the configuration settings are incorrect, missing, or have been updated during or after the configuration.

DE_10.png

Resolving

  1. Select EDIT
  2. Check to ensure that the current settings match what you have configured for AWS or Splunk
  3. Make necessary updates to the data export configuration
  4. Select VERIFY & TEST ACCOUNT to ensure that there is no longer an error.

Log Formats

Examples

Data Descriptions

Sequence DNSF Name DNSF Description
1 Time When this request was made in UTC.
2 FQDN Fully qualified domain that was requested.
3 Domain Domain that was requested.
4 Protocol Internet protocol used to make the request
5 Username Username of the dashboard user who made the request
6 UserID ID of the dashboard user who made the request
7 QuestionType Type of the DNS request that was made
8 Code DNS return code for the request
9 OriginalCode DNS original return code for the request
10 RequestAddress The external IP that made the request
11 Client Roaming client name that made the request
12 ClientID Roaming client ID that made the request
13 ClientType Roaming client type that made the request.
14 ClientMac Roaming client mac address that made the request. Field may vary based on the roaming client type.
15 IP4 Internal IP that made the request
16 IP6 Internal IP that made the request
17 Region The geographical region where the request was made
18 Network Network or site name where the request was made
19 NetworkID Network or site ID where the request was made
20 Collection Collection name where the request was made
21 CollectionID Collection ID where the request was made
22 Policy Policy name that processed the request
23 PolicyID Policy ID that processed the request
24 ScheduledPolicy Scheduled policy name that made the request
25 ScheduledPolicyID Scheduled policy ID that made the request
26 Seccats One or multiple categories associated with the blocked request
27 Secallowcats One or multiple categories associated with the allowed request
28 Blockcats One or multiple categories associated with the blocked request
29 Blockallowcats One or multiple categories associated with the allowed request
30 Allowed Boolean value indicating whether the request was allowed
31 Threat Boolean value indicating whether the request is categorized as a threat
32 Method Method used to process the request
33 Organization Organization name associated with the request
34 OrganizationID Organization ID associated with the request
35 ApplicationID Application ID associated with the request
36 ApplicationName Application name associated with the request
37 ApplicationCategoryID Application category ID associated with the request
38 ApplicationCategoryName Application category name associated with the request

 

Enable Data Export

Data Export needs to be Activated by an Owner, Admin, or Super User.

DataExportBilling.png

Single Organization Activation

  1. Go to Organization > Billing (defaults to the Subscription tab)
  2. Select Activated to activate Data Export
  3. Select Save

MSP Activation

The Data Export feature cannot be turned on for a single organization — Data Export is applied at the plan level. If you want to enable Data Export for Enterprise plan levels, it will be applied across all organizations with a plan type of Enterprise.

  1. Select on MSP > Billing (defaults to the Subscription tab)
  2. Select Activated to activate Data Export
  3. Select Save

When the Data Export option is activated, it's applied to all sub-organizations (e.g., all plan tiers). With that, if users see the UPGRADE YOUR PLAN screen (Tools > Data Export), Data Export still needs to be Activated.Screen Shot 2023-12-21 at 2.25.13 PM.png

 

 

Was this article helpful?

3 out of 3 found this helpful

Have more questions? Submit a request

Comments

0 comments

Please sign in to leave a comment.