Skip to main content

How can we help you?

Search

help Knowledge Base
forum Community
checklist Release Notes
event Events

Large CSV data export not sending – How to retrieve full logs?

Dave Cattapan

If I export a small quantity of data to a .CSV file, I get a link via email which works just fine. However, if I attempt to export an entire day's worth of my company's web traffic (maybe one million records) I never get an email, even if I wait overnight. How do I get the data I need?

Comments

4 comments

Date Votes
  • Official comment
    Bailey Taylor Community Manager DNSFilter Staff
    • Edited
    Community Regular

    Hi Dave Cattapan, thanks for reaching out!

    After looking into this further, our system has limits in place for large query log exports to ensure stability. If there is a very high volume of queries, the export may be unsuccessful, which is likely why you are not receiving the email for a full day's worth of data.

    To successfully retrieve your data, we recommend exporting logs in smaller time frames rather than attempting to bulk export all logs at once. This should allow you to gather the necessary data without hitting system limitations.

    If you need to export large volumes of data on an ongoing basis, you might find our Data Export add-on useful. This feature allows Single Org & MSP users to configure automated exports to Amazon S3 or Splunk at regularly timed intervals, ensuring access to real-time query log data. You can find more details on setup and functionality here: 🔗 Data Export Configuration Guide.

    Let me know if you have any questions or if I can assist further!

  • Dave Cattapan

    Thanks, Bailey.  This is very helpful! Two questions:

    1. Do you have a sense of how many records can be exported at one time?  Because if it's only about 1,000 records at a time I'm going to develop a different solution than if it's capable of around 500,000 at a time.

    2. It would be nice if the system sent an email indicating that there are too many records to process, instead of me just wondering why I never got an email.

    Thanks,

    Dave

     

    0
  • Bailey Taylor Community Manager DNSFilter Staff
    Community Regular

    Dave Cattapan, great questions! Our export threshold is 100,000 records or less at a time. If the requested record count exceeds this limit, our system should automatically send a failure notification to inform you that the export was unsuccessful.

    If you didn’t receive an email when attempting an export, I’d be happy to look into this further. Could you please provide the date of your last export attempt where no notification was received? This will help us investigate the issue.

    Let me know, and I’ll be glad to assist!

    0
  • Dave Cattapan

    Thanks again, Bailey! I just sent another query request now.  I'm not sure when the last one was done.

    0

Please sign in to leave a comment.

New post

Featured posts