I needed to download a bunch of files from a GCS bucket to process them locally. gcloud storage rsync does the job, and it’s noticeably faster than the old gsutil for large syncs.

Sync a bucket to local directory:

gcloud storage rsync -r gs://my-bucket/path/ ./local/

Exclude certain files:

gcloud storage rsync -r -x ".*\.tmp$" gs://my-bucket/path/ ./local/

Copy specific files:

gcloud storage cp gs://my-bucket/path/*.csv ./local/

Copy recursively:

gcloud storage cp -r gs://my-bucket/path/ ./local/

The old gsutil still works but gcloud storage is the newer interface:

gsutil -m rsync -r gs://my-bucket/path/ ./local/

The -m flag enables parallel transfers which is faster for lots of files.

Speed comparison: gcloud storage vs gsutil Link to heading

I synced about 10,000 small files (total 2GB) from GCS to local:

  • gsutil -m rsync: ~4 minutes
  • gcloud storage rsync: ~2.5 minutes

The difference is more noticeable with many small files because gcloud storage uses better parallelisation by default. For a few large files, the speed is roughly the same.

For uploads, the gap is even bigger. gcloud storage seems to handle concurrent uploads better, especially with fast internet connections.

Permission gotchas Link to heading

Issue 1: Local file permissions

When syncing from GCS to local, files are created with your default umask. If you then rsync them back to GCS, the permissions metadata might not match what you expect. This matters if you’re serving files directly from GCS (like a static website).

Use --preserve-posix if you care about permissions:

gcloud storage rsync -r --preserve-posix gs://my-bucket/path/ ./local/

Issue 2: Service account permissions

If you’re syncing from CI/CD, make sure your service account has:

  • storage.objects.list (to see what needs syncing)
  • storage.objects.get (to download)
  • storage.objects.create (to upload)

Just Storage Object Viewer isn’t enough for bidirectional sync. You need at least Storage Object Admin on the bucket, or create a custom role with the exact permissions you need.

Issue 3: Deleted files

By default, rsync doesn’t delete files in the destination that don’t exist in the source. Use --delete-unmatched-destination-objects to make the destination match exactly:

gcloud storage rsync -r --delete-unmatched-destination-objects gs://source/ gs://destination/

But be careful - I’ve accidentally deleted files I wanted to keep by forgetting which direction I was syncing.

Further reading Link to heading