A successful connection to BigQuery requires the user to have one of the following roles:
- BigQuery Admin
- BigQuery Data Editor
- BigQuery Owner
The BigQuery Job User role must be added to Data Editor and Owner roles.
Add the Job User Role
In the Google account used for exports, go to IAM and Admin from the main menu in Google Cloud Console. Select Permissions and view by Principals.
Set Up Your Initial Backfill and Rolling Updates
You can backfill data for up to 2 years for most PMA connectors. In your first report in Data Explorer, set the date range to the entire range of data you want to export and click Set Range.
Click Save to save the report.
Under Exports, go to BigQuery and find your export. Click Run Export Now in the row of your export.
In BigQuery, click Edit Export. Refresh period can be set to monthly, weekly, daily, or hourly.
Make sure your report's date range in Data Explorer is at least as long as the refresh period in BigQuery. Failing to do so may result in data missing from your backfill.
- If your report's date range is one week but your export refreshes monthly, your export will fail to include data from before the last week of the month.
- If your report's date range is one week and your export refreshes daily, your export will overwrite the past seven days, every day.
A range of 30 days with a daily or weekly refresh period can be an ideal choice for e-commerce to settle orders that are returned. However, excessively frequent backfills with large date ranges, such as a year's data refreshed every hour, can cause BigQuery to hang. Frequent large requests can also increase your BigQuery charges.
If you are using connectors other than Facebook Ads or GA4, make sure the report you are trying to export contains fewer than 40,000 rows of data. If you are backfilling a large amount of your historical data into BigQuery, you may need to take additional steps to make sure your exports are under BigQuery's limits. This limitation does not apply to Facebook Ads or GA4.
Your export should run in under 10 minutes. If the export runs for 30 minutes or more, it has hung and needs to be cancelled. To stop the hung export, click Edit Export.
Do not make any changes to the export. Click Update.
The hung export is now stopped.
How to accommodate large datasets
If your data has caused the export to hang, try segmenting the overall date range into two smaller date ranges. Edit the report in Data Explorer to use the first date range, then go to BigQuery and run your export.
If your export completes successfully:
- Return to Data Explorer.
- Edit your report and set Date Range to the second date range.
- Go to BigQuery and run your export again.
If your export is unsuccessful:
You may need to segment the overall date range into four or more date ranges.
- Reduce the size of the date range until the export runs successfully, indicating this range is within BigQuery's export limits.
- Run the export for each date range according to the instructions above.