A file opens correctly in the new bucket, but the application still fails. It may be using an old object URL, expecting a tag that did not move, or relying on an access rule that was never recreated. An Amazon S3 to Google Cloud Storage migration needs separate plans for bytes, metadata, access and the workloads that read or write those objects.
Storage Transfer Service provides the transfer path. The migration team still owns the scope and acceptance checks. This guide focuses on ordinary object migration and the application cutover around it. Treat specialized storage features and historical version requirements as explicit work items rather than assuming a bucket copy includes them.
Inventory buckets and objects before choosing a job
Create a bucket inventory and an object inventory. For each bucket, record the owning workload, source location, encryption setup, access configuration, lifecycle rules and versioning requirements. Identify retention and lock settings that need a separate design review. A transfer of objects is not evidence that those bucket-level controls have been recreated.
For the object inventory, record keys, sizes, required metadata and the versions the business needs. Also identify consumers of object tags, source timestamps and existing URLs. Separate current application files from historical material. Write down whether old versions are required, and verify the chosen transfer mechanism against that scope before scheduling the move.
Use the inventory to define exclusions deliberately. A temporary export or obsolete prefix should not disappear from the plan simply because someone forgot to list it. Record its exclusion and the approving owner. This gives operators a way to distinguish an expected gap from a missed file after transfer.
Make a metadata mapping that names the losses
Google documents preservation of fixed-key metadata such as Cache-Control, Content-Disposition and Content-Type for Amazon S3 transfers. User-defined metadata is preserved as custom metadata. The source ETag is preserved as custom metadata with the key x-goog-source-etag, and object size is preserved as size.
| Source item | Documented transfer behavior | Migration action |
|---|---|---|
| Content-Type and similar fixed-key metadata | Preserved as fixed-key metadata | Test real downloads and browser behavior |
| User-defined metadata | Preserved as custom metadata | Check application lookups and expected values |
| ETag | Stored as x-goog-source-etag custom metadata | Do not confuse it with a universal checksum |
| S3 object ACLs and object tags | Not preserved | Design destination access and a separate tag mapping |
| Source timestamps | Not preserved as destination creation or update time | Carry required business dates separately |
Destination timeCreated reflects when the object is created in Cloud Storage. Destination updated reflects metadata modification there. If an application uses an upload date for customer reporting, define a dedicated business field or manifest entry instead of silently substituting the new creation time. Decide the storage class deliberately too, rather than assuming source classes map automatically.
Set up access on both sides
The S3 transfer setup requires user permissions to configure the job, source access through credentials or federated identity, and destination bucket permissions. Review those as separate identities and responsibilities. An administrator who can create a job is not automatically proof that the transfer can read every scoped object or write to the destination.
Rebuild destination access according to intended application behavior. Because source object ACLs are not preserved, copying files does not demonstrate equivalent permissions. Test an authorized reader, an unauthorized reader and the application’s writer identity. Keep destination data private until the required access tests pass. These tests are recommended engineering controls, not an automatic permission translation feature.
Choose how to handle changes during the transfer
Google documents run-once jobs, scheduled jobs and event-driven jobs. Event-driven transfers can respond to Amazon S3 Event Notifications sent to Amazon SQS. Its migration guidance describes an initial copy, validation, event-driven transfer, a workload stop, application refactoring, synchronization and restart.
Choose a change strategy that matches your workload. For a bounded migration, an agreed write pause may simplify reconciliation. For an active source, define how new and updated objects will be captured and how the final boundary will be checked. Handle source deletions as a specific requirement. Do not assume that a process described for additions and updates supplies a complete deletion history.
Keep destructive cleanup out of the first transfer plan. Record overwrite and removal choices before running the job. Preserve the source as a fallback until acceptance is complete. Also review transfer costs for the chosen network path and source operations, without using a generic cost estimate as a project quote.
Validate content beyond the green status
When source checksum metadata is available, Storage Transfer Service uses end-to-end checksum validation. If source checksums are unavailable and agents cannot run near the source, it uses available metadata such as file size and computes a checksum during transfer to compare with the final Cloud Storage checksum. Google lists multipart S3 objects and S3 objects using SSE-KMS among sources that may not provide checksums for that path.
Do not turn this into a blanket claim that every S3 ETag is a content hash. Google recommends additional checks after transfer for the correct file versions, object set and count, and metadata such as checksums and sizes. Reconcile by prefix and workload so an omitted customer folder cannot hide behind an aggregate total.
Maintain an exception list for missing, changed and failed objects. Give each exception an owner and an explanation. Retry only the intended scope, then repeat the relevant checks. Save the inventory boundary and transfer settings with the validation results so a later operator can reproduce what was accepted.
Cut over the application, not just the bucket
- Confirm that scoped objects and required metadata pass reconciliation.
- Pause or otherwise control source writes at the agreed boundary.
- Capture and validate outstanding changes before switching readers and writers.
- Test uploads, downloads, required headers and access through the real application.
- Declare one authoritative write destination and monitor failures after release.
- Retire source access only after the fallback is no longer needed.
Define rollback before the switch. If new writes have already reached the destination, returning to the source requires a reconciliation plan for those writes. Reverting a configuration value alone does not establish that both systems contain the same data. Our acceptance testing guide uses the same principle: inspect the final state, not just the reported completion.
How Remqa helps with cloud data migrations
Remqa provides engineering support for data migrations, including inventories, mapping, transfer validation and application handoff. Bring a representative bucket, the workloads that use it and the business rules attached to its files. The first useful output is an agreed migration boundary with explicit metadata losses and acceptance checks. Discuss that scope before choosing the final transfer and cutover design.