How to Export to BigQuery

How to Export to BigQuery

Once you have a report ready in Data Explorer, you are able to export it to BigQuery. This export option is included in the Enterprise plan. Follow the instructions in this article to get started.

Grant Required Roles and Permissions

A successful connection to BigQuery requires the user to have one of the following roles:
  1. BigQuery Admin
  2. BigQuery Data Editor
  3. BigQuery Owner
The BigQuery Job User role must be added to Data Editor and Owner roles.

Add the Job User Role

In the Google account used for exports, go to IAM and Admin from the main menu in Google Cloud Console. Select Permissions and view by Principals.



Find the account used for exports and click the pencil icon to Edit principal.



Under Role, select BigQuery, then BigQuery Job User.



Create a Dataset

A dataset must be available before proceeding to create an export. Follow the instructions in Google's guide to create a dataset.

The user creating an export must have Write permissions to the dataset.

Set Up Your Initial Backfill and Rolling Updates

Initial backfill

You can backfill data for up to 2 years for most PMA connectors. In your first report in Data Explorer, set the date range to the entire range of data you want to export and click Set Range.



Click Save to save the report.


Under Exports, go to BigQuery and find your export. Click Run Export Now in the row of your export.

 

Rolling updates

In BigQuery, click Edit Export. Refresh period can be set to monthly, weekly, daily, or hourly. 



Make sure your report's date range in Data Explorer is at least as long as the refresh period in BigQuery. Failing to do so may result in data missing from your backfill.



For example:
  1. If your report's date range is one week but your export refreshes monthly, your export will fail to include data from before the last week of the month.
  2. If your report's date range is one week and your export refreshes daily, your export will overwrite the past seven days, every day.
A range of 30 days with a daily or weekly refresh period can be an ideal choice for e-commerce to settle orders that are returned. However, excessively frequent backfills with large date ranges, such as a year's data refreshed every hour, can cause BigQuery to hang. Frequent large requests can also increase your BigQuery charges.

Export Limitations

If you are using connectors other than Facebook Ads or GA4, make sure the report you are trying to export contains fewer than 40,000 rows of data. If you are backfilling a large amount of your historical data into BigQuery, you may need to take additional steps to make sure your exports are under BigQuery's limits. This limitation does not apply to Facebook Ads or GA4.

Your export should run in under 10 minutes. If the export runs for 30 minutes or more, it has hung and needs to be cancelled. To stop the hung export, click Edit Export.


Do not make any changes to the export. Click Update.


The hung export is now stopped.

How to accommodate large datasets

If your data has caused the export to hang, try segmenting the overall date range into two smaller date ranges. Edit the report in Data Explorer to use the first date range, then go to BigQuery and run your export.

If your export completes successfully: 
  1. Return to Data Explorer.
  2. Edit your report and set Date Range to the second date range.
  3. Go to BigQuery and run your export again.
If your export is unsuccessful:

You may need to segment the overall date range into four or more date ranges.
  1. Reduce the size of the date range until the export runs successfully, indicating this range is within BigQuery's export limits.
  2. Run the export for each date range according to the instructions above.




    Still having issues?
    Send us a ticket and we will get back to you.
    Submit a ticket

      • Related Articles

      • BigQuery Export Limitations

        Your data exports to BigQuery may encounter errors in certain situations such as large export sizes. This article will show you how to identify and correct errors with your BigQuery exports. Export Limitations If you are using connectors other than ...

      • Troubleshooting BigQuery

        When troubleshooting BigQuery, you may need to view your BigQuery export logs. This article will show you how to check the logs and see details about any errors. Viewing Export Logs Your BigQuery export logs show a timestamped record of your exports ...

      • BigQuery Error: Code 403; Message: Access Denied

        This article will assist you with the following errors: Code: 403; Message: Access Denied: BigQuery BigQuery: Streaming insert is not allowed in the free tier (accessDenied) Code: 403; Message: Access Denied: Dataset: Permission denied on dataset (or ...

      • How to Create an FTP Export

        FTP can be a valuable way for Enterprise users to export their Data Explorer reports. In this article, we will walk you through creating an FTP export, including selecting an FTP destination, scheduling data refreshes, and managing your exports to ...

      • How to Create a Report in Data Explorer

        Data Explorer is a valuable tool that can be used to create reports for export to BigQuery or FTP , to test out a connection, or even to create a CSV file. This tool is available to all Power My Analytics customers and is included in the cost of your ...