technology

s3cmd Revisited x2

I’ve refactored my s3cmd script. I think this should do it for a while.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh
STAGING=${S3_STAGING:-/Projects/sites/ideoplex/ideoplex.com}
BUCKET=${S3_BUCKET:-s3://ideoplex.com}
# 1 hour
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=3600" \
    $STAGING/index.html $BUCKET/index.html
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=3600" \
    $STAGING/atom.xml $BUCKET/atom.xml
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=3600" \
    $STAGING/categories/ $BUCKET/categories/
# 1 day
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=86400" \
    $STAGING/tags/ $BUCKET/tags/
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=86400" \
    $STAGING/archives/ $BUCKET/archives/
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=86400" \
    $STAGING/page/ $BUCKET/page/
# 30 days
s3cmd sync --no-preserve --cf-invalidate --add-header="Cache-Control: public, max-age=2592000" \
    $STAGING/ $BUCKET/

Previously, I synced everything and then updated the Cache-Control headers in subsequent passes. Now, I use multiple sync commands that explicitly set the desired Cache-Control headers each time. I also added the ability to specify the staging directory on the local filesystem and the S3 bucket with environment variables (defaulting to my personal values).

As a reminder, the arguments are:

  • --nopreserve: disable save of filesystem attributes in s3 metadata
  • --cf-invalidate: invalidate the uploaded file[s] in Cloudfront
  • --add-header=…: explicitly set the cache control headers on the uploaded files

Finally, everything now lives at ideoplex/my-scripts on GitHub.