To understand the problem at hand, it is best to set up the use case as it was intended (and before mentioning anything about failover). The Ippon Podcast exploratory project is possible through a combination of CloudFront and S3. The application's index.html, styles, scripts, and media files are all located in us-east-1 S3 buckets with the property for "Static website hosting" enabled. The site is served via a CloudFront distribution concerning these S3 buckets as Origins. So what happens to the site if a bucket is deleted or the files are removed? What about if the entire us-east-1 region goes down?
Failure to retrieve these files will result in 4xx error codes since they will either be considered not found or forbidden. Enabling the "Versioning" property or "Cross-Region Replication" management of these buckets can help ensure the buckets and their files survive; but, the CloudFront distribution still needs a way to retrieve those files in the event of a region shutdown.
In the event of a cache miss at an edge location, CloudFront goes to retrieve the file and replenish the cache. So then how does it work if the origin is unavailable at the time CloudFront goes looking for it? Fortunately, support for Origin Failover was introduced in November 2018, which was another important advancement in making AWS-based web applications highly available. Origin Failover is achievable through the ability to specify a primary and a secondary origin into what is called an Origin Group. When CloudFront is unsuccessful in connecting to the primary origin, an error status code is returned which prompts the failover action. CloudFront will then attempt the same request with the secondary origin. This is customizable in that any combination of the following status codes can be selected: 500, 502, 503, 504, 403, or 404.
In its current state, the Ippon Podcast site relies on S3 buckets located in the us-east-1 region. Proper failover cannot be implemented without redundant S3 buckets in another region like us-west-2. Setup a cross-regional bucket in either of the following ways:
us-east-1 bucket and specify the creation of a new S3 bucket. This will force the "Versioning" property to be enabled on both buckets, and also requires the objects to be copied or synced between the original us-east-1 bucket and the new us-west-2 bucket, but going forward the latest files will automatically be replicated.us-west-2 and copy or sync the objects over from the original us-east-1 bucket. Then, enable "Cross-Region Replication" on the us-east-1 bucket and specify the new S3 bucket which was just created in us-west-2. This will still force the "Versioning" property to be enabled on both buckets and automatically going forward the latest files will be replicated.Note that "Cross-Region Replication" is not required but, without enabling, object consistency between the two buckets will require manual intervention. Using the AWS CLI for better control over S3 operations and an infrequently accessed storage class for a new, cost-effective bucket is recommended. Possible commands might include:
aws s3 mb s3://mybucket_west --region us-west-2
aws s3 cp s3://mybucket s3://mybucket_west --recursive
aws s3 sync s3://mybucket s3://mybucket_west
Returning to the CloudFront distribution, set up the behavior with a new Origin Group through the AWS console with the following steps:
us-west-2's information through the Create Origin interface. Make sure to specify "Restrict Bucket Access" as "Yes" and allow the grant read permission to the desired Origin Access Identity. There should now be at least two origins listed.us-east-1 for the new origin group.Or, set up the behavior with a new Origin Group programmatically:
"DistributionConfig" member using the get-distribution-config command. Save the "DistributionConfig" object to its own file and store the ETag attribute's value for reference as both will be necessary later to update the distribution.aws cloudfront get-distribution-config --region us-east-1 --id EDFDVBD632BHDS5
DistributionConfig file and modify its "Origins" object with the new S3 bucket in us-west-2's information to create a new "Origin"."Origins": {
"Quantity": 2,
"Items": [
{
"Id": "originalHostingS3Bucket",
"DomainName": "mybucket.s3.amazonaws.com",
"OriginPath": "",
"CustomHeaders": {
"Quantity": 0
},
"S3OriginConfig": {
"OriginAccessIdentity": "origin-access-identity/cloudfront/_ID-of-origin-access-identity_"
}
},
{
"Id": "newHostingS3Bucket_west",
"DomainName": "mybucket_west.s3.amazonaws.com",
"OriginPath": "",
"CustomHeaders": {
"Quantity": 0
},
"S3OriginConfig": {
"OriginAccessIdentity": "origin-access-identity/cloudfront/_ID-of-origin-access-identity_"
}
}
]
}
"OriginGroups" attribute if there is not one. Add an "OriginGroup" with the origins that were added previously. Specify any combination of 403, 404, 500, 502, 503, or 504 "StatusCodes" which CloudFront will attempt to connect with the secondary origin."OriginGroups": {
"Quantity": 1,
"Items": [
{
"Id": "OriginGroup-hostingS3Bucket",
"FailoverCriteria": {
"StatusCodes": {
"Quantity": 6,
"Items": [
403,
404,
500,
502,
503,
504
]
}
},
"Members": {
"Quantity": 2,
"Items": [
{
"OriginId": "originalHostingS3Bucket"
},
{
"OriginId": "newHostingS3Bucket_west"
}
]
}
}
]
}
"DefaultCacheBehavior" and/or "CacheBehaviors" members with the file. When using Origin Groups for Behaviors, the "AllowedMethods" are only "HEAD", "GET", and "OPTIONS"."DefaultCacheBehavior": {
"TargetOriginId": "OriginGroup-hostingS3Bucket",
"ForwardedValues": {
"QueryString": false,
"Cookies": {
"Forward": "none"
},
"Headers": {
"Quantity": 0
},
"QueryStringCacheKeys": {
"Quantity": 0
}
},
"TrustedSigners": {
"Enabled": false,
"Quantity": 0
},
"ViewerProtocolPolicy": "redirect-to-https",
"MinTTL": 60,
"AllowedMethods": {
"Quantity": 3,
"Items": [
"HEAD",
"GET",
"OPTIONS"
],
"CachedMethods": {
"Quantity": 2,
"Items": [
"HEAD",
"GET"
]
}
},
"SmoothStreaming": false,
"DefaultTTL": 86400,
"MaxTTL": 31536000,
"Compress": true,
"LambdaFunctionAssociations": {
"Quantity": 0
},
"FieldLevelEncryptionId": ""
}
DistributionConfig JSON file is saved with the changes just made. Then run the update-distribution command using that file's location. Make sure to provide the --if-match flag with the "ETag" value returned in the first step's command. The updated DistributionConfig JSON should be echoed back if successful.aws cloudfront update-distribution --if-match E2QWRUHEXAMPLE --id EDFDVBD632BHDS5 --distribution-config file://origin-failover.json --region us-east-1
And with that, the application should now be better equipped for handling failures! So how can that be tested?
At least in the case of the Ippon Podcast, there are two buckets utilized in its functionality. One is for holding the index.html, styles, and scripts of the application. The other contains the podcast audio recordings. This allows for two different opportunities to test the origin group's failover capability. Just a note, if any of the following described does not reflect immediately, it is because it takes time for the changes to propagate. Invalidating the cache using /* will allow differences to take effect more quickly but may have a cost involved if performing over 1,000 in a month.
Removing the index.html is an easy one to test with because the site will break completely. To see this, perform the following steps:
index.html file from this primary bucket (have no fear about losing the file since it will still be versioned within the bucket and also located in the alternate)index.html fileA similar test can be performed on the media files. While this will not break the site completely, reverting the associated behavior to the single origin and then deleting one or any of the files should cause failure to load/play. Changing the behavior over to use the origin group will allow the backup file to be loaded and played as intended.
Remember to restore removed primary files either by deleting the "Delete marker" under show versions or re-upload the file. Ensure that any changed behaviors use the origin groups.
The Ippon Podcast site is still very much a work-in-progress. Since it is more exploratory, high availability is not as critical. This makes it a good application to test proper failover through CloudFront's origin groups. There are more subtle details to utilizing origin groups which require some exploration. Yet, hopefully, this provided an easy first step towards building and testing redundancy in a CloudFront/S3-hosted application.
Sources: