The tables arrive in BigQuery, but the finance dashboard still points at Redshift. A translated query returns a different total, and nobody has agreed which warehouse owns the next scheduled refresh. Moving warehouse data is only part of the job. A Redshift to BigQuery migration also needs a defined reporting boundary, repeatable validation and an operator who can stop the cutover when numbers disagree.
Google provides a BigQuery Data Transfer Service path for Redshift schema and data. The reconciliation, repeat-load controls and cutover checklist below are recommended engineering practices. They are not claims that a transfer automatically migrates every workload or guarantees matching reports.
Start with tables and the people who consume them
Create an inventory of source databases, schemas, tables and reporting dependencies. For every important table, record its owner, business key, update pattern, retention expectations and downstream queries. Include scheduled transformations and exports that may not appear in the dashboard catalog.
Choose the first migration unit around a business result. A useful unit contains the tables, transformations and reports needed to answer one operational question. Copying an isolated table is easy to demonstrate but does not prove the team can work from the new warehouse. For broader planning, use a data migration scope that includes consumers as well as storage.
Understand the documented transfer path
For its public IP migration route, BigQuery Data Transfer Service uses migration agents in GKE and triggers an unload from Redshift to a staging area in Amazon S3. The service then transfers data from that staging area to BigQuery. Google documents a separate private IP route using a VPC. Choose the access model with the security owner before configuring network rules.
The setup requires a destination BigQuery dataset. Google says the destination tables do not need to be created first. This is useful for initial loading, but generated structure is only a starting point. Inspect the resulting types and values against the source before declaring the table ready for production queries.
| Migration layer | Define before the test | Evidence to retain |
|---|---|---|
| Source scope | Redshift schema and selected table names | Reviewed table inventory |
| Transfer access | Database credentials, staging access and network route | Approved access configuration |
| Target structure | Destination dataset and intended business meaning | Schema comparison and exception list |
| Reporting | Queries, refresh order and consumer switch | Matched results and owner sign-off |
Configure access and staging deliberately
The transfer setup collects a Redshift JDBC URL, database username and password, an AWS access key pair, an S3 URI and the source schema. The Google principal creating the transfer also needs the documented transfer and destination dataset permissions. Keep these secrets out of mapping documents, screenshots and routine application logs.
Use a dedicated staging location and define who can read it. Record when files may be removed and how operators prove a transfer is complete before cleanup. A lifecycle policy should reflect the tested recovery window. Deleting staging data before a failed run can be investigated turns a retry into another export.
For the public route, check the cluster port and the documented Google-owned IP ranges for the destination location. Do not replace a precise network rule with broad public access to make a test pass. If the approved route is private, follow that route's setup instead of mixing public and private configuration assumptions.
Make source selection explicit
The transfer configuration selects a Redshift schema and can restrict table names using patterns. Leaving the table pattern empty transfers all tables from the specified schema. Review the resulting inventory before execution, especially when a schema contains scratch tables, restricted data or old reporting copies.
Record the exact selected tables in a migration manifest. Associate each with its target and validation query. A green transfer status without a coverage manifest cannot tell an operator whether an omitted table was intentionally excluded or simply missed. Test selection against the actual catalog rather than trusting a pattern that merely looks correct.
Inspect data before relying on the CSV route
Google documents a specific limitation for the CSV transfer path: files containing the ASCII 0 character cannot be loaded into BigQuery. The documented alternative is to export the data to S3 as Parquet and use an Amazon S3 transfer route. Treat that as a separately tested route, not as a setting that can be changed without checking the resulting files.
Build representative tests around long text, empty values, non-English text, decimals and timestamps present in your own tables. Preserve the source value when investigating a failed conversion. Do not silently replace an unexpected character or round a monetary field just to finish a load. Record every accepted transformation in the mapping.
Review SQL separately from data movement
Google's Redshift migration overview identifies batch and interactive SQL translation services. Translation helps move query text, but the migration team still needs to test the meaning of the result. Run representative queries on an aligned source period and record the expected output before changing production schedules.
Check joins, null handling, date boundaries, aggregation grain and duplicate records using examples your report owners understand. An identical row count can coexist with an incorrect revenue total. Compare totals by business period and category, then inspect outliers at record level. Treat permissions and report access as separate acceptance checks.
Decide how repeat loads and final changes work
Do not assume a batch transfer is continuous change capture. Before the first production rehearsal, define what happens when a load is repeated and how the final source changes will reach BigQuery. Test inserts, updates, deletes and records that arrive after the reporting period closes.
Use an isolated destination for rehearsals so a second test cannot contaminate accepted reporting tables. Assign a run identifier and retain the table selection, source boundary and validation result for each attempt. Agree whether the final process reloads a complete unit or applies a separately engineered change process. Verify the chosen behavior rather than inferring it from one successful load.
Run a reporting cutover checklist
- Confirm the source boundary and stop or coordinate writes for the agreed cutover process.
- Load the final approved data and reconcile every selected table.
- Run critical business queries against the same reporting period.
- Check access with the actual report users and scheduled service identities.
- Switch transformations and dashboards in a reviewed dependency order.
- Monitor results and keep a defined path back to the prior reporting source.
Rollback also needs a data boundary. If new transformations or writes occur only in BigQuery, pointing dashboards back at Redshift may lose that work. Document what must be replayed or reconciled before a reversal. Retain source access until owners accept the business results and the recovery procedure.
How Remqa helps with Redshift to BigQuery migration
Remqa can map the transfer scope, build validation checks and coordinate the reporting cutover with your operators. The work includes the tables people depend on, the queries that define their numbers and the access rules around staging and reporting.
Bring a source schema, an important dashboard and the current refresh process to a migration review. Our migration work starts with those dependencies and the evidence needed to accept the move. The target is a verified reporting workflow, not just another populated dataset.