- TRUNCATE + PARTITION BY DATE: The table is partitioned by a date column and data is only replaced based on date (this is the recommended option, needs to be used with Local Data Retention > Unique By Date).
- APPEND: Each extract is added to the existing table (advanced, should only be used for certain append-only types of data).
- TRUNCATE: Each time import happens, the entire table is deleted and replaced by the most recent extract. (used in combination with bundle stream).
- ERROR (NO ACTION): The Datastream will only create a table if it does not exist, and will not update it afterwards. If an import is attempted, BigQuery will return an error message.
All load jobs to BigQuery through Datatap are subject to the existing BigQuery quotas. Please be mindful of these and contact your BigQuery Account Manager to discuss the limit in case of questions.
When authorizing Datatap with Big Query, the Google Account used must have the following the following permissions:
This translates into a pre-configured role:
To create the correct data entries in the BigQuery table, you have to use Schema Mapping. You can use the defaults or create your own, further information can be found in Datatap Basics - Data Schema.
- Unlike a relational database, you cannot define a set of key columns.
- Whenever the structure of the table has changed it’s necessary to drop or alter the table. The table structure changes whenever the column datatype is changed.
For further details on how to start with Google BigQuery we recommend to read the Google BigQuery Documentation.
- Go to Manage Organizations and select the Destination for the required Organization.
- To configure a custom Datastream table name, select the Datastream in the left column and put a value in the right column.
- Navigate to your Destination's settings.
- If your setup was successful, you will be able to choose a dataset from the drop-down shown below.
- Ensure that there is a dedicated dataset created in your Google Big Query account (Datatap creates tables within this dataset). The name of this dataset(s) will populate the drop-down menu.
- Recheck the user roles of the account that granted the connection.
- If both of the above are definitely correct but it is still not working, try using a service account.