---
title: Sync Two S3 Buckets Using CDK and a Lambda Layer Containing the AWS CLI
description: "The AWS Command Line Interface (CLI) [https://aws.amazon.com/cli/] is a great\ntool that can be used in your scripts to manage all your AWS infrastructure.\nThere is a lot of useful commands, my favorite one is AWS S3 Sync\n[https://docs.aws.amazon.com/cli/latest/reference/"
image: https://blog.ippon.tech/hubfs/Imported_Blog_Media/Sync-Two-S3-buckets-Using-CDK-and-a-Lambda-Layer-Containing-the-AWS-CLI-3.png
---
[Skip to main content](https://blog.ippon.tech/sync-two-s3-buckets-using-cdk-and-a-lamdba-layer-containing-the-aws-cli#main)
[   ](https://ipponusa.com/)
- [Home](https://ipponusa.com)
- [Show submenu for About About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- [Show submenu for Our Partners Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- [Show submenu for Services Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- [Show submenu for Resources Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
Search
Open main navigation
Close main navigation
- [Home](https://ipponusa.com)
- Show submenu for About About
- About
- [About](https://ipponusa.com/about-us)
- [Who We Are](https://ipponusa.com/who-we-are)
- [Careers](https://ipponusa.com/careers)
- Show submenu for Our Partners Our Partners
- Our Partners
- [Our Partners](https://ipponusa.com/our-partners/)
- [AWS](https://ipponusa.com/our-partners/aws/)
- [Snowflake](https://ipponusa.com/our-partners/snowflake/)
- [Databricks](https://ipponusa.com/our-partners/databricks/)
- [Microsoft](https://ipponusa.com/our-partners/microsoft/)
- Show submenu for Services Services
- Services
- [Services](https://ipponusa.com/services)
- [Snowflake](https://ipponusa.com/service/snowflake-concierge/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Blogs](https://blog.ippon.tech/)
- [Success Stories](https://ipponusa.com/success-stories/)
- Show submenu for Resources Resources
- Resources
- [Resources](https://ipponusa.com/resources/)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour)
- [Videos & Webinars](https://info.ippon.tech/videos-and-webinars)
- [Media Center](https://ipponusa.com/media-center/)
- Search
- [Contact Us](https://ipponusa.com/contact/)
[Contact Us](https://ipponusa.com/contact/)
# Sync Two S3 Buckets Using CDK and a Lambda Layer Containing the AWS CLI
by [Theo LEBRUN](https://blog.ippon.tech/author/theo-lebrun)
April 8, 2021
[Cloud](https://blog.ippon.tech/tag/cloud/) [AWS](https://blog.ippon.tech/tag/aws/) [AWS CDK](https://blog.ippon.tech/tag/aws-cdk/) [AWS S3](https://blog.ippon.tech/tag/aws-s3/) [AWS Lambda](https://blog.ippon.tech/tag/aws-lambda/) [Python](https://blog.ippon.tech/tag/python/)
*Published April 2021*
The [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/?ref=blog.ippon.tech) is a great tool that can be used in your scripts to manage all your AWS infrastructure. There is a lot of useful commands, my favorite one is [AWS S3 Sync](https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html?ref=blog.ippon.tech) which will synchronize two S3 buckets. This is pretty useful when for example you want to copy a bucket from a different AWS account.
Depending on your use case, you might want to synchronize the two buckets based on an event or on a daily basis. In order to do that, the easiest option is just to go serverless and have a Lambda that will take care of the syncing process.
## Lambda Layer to the Rescue
### AWS CLI in a Lambda?
By default, the Lambda runtime includes additional libraries like the AWS SDK so you can interact with your AWS infrastructure. For more details about the Lambda runtimes and which libraries are included, check the [official page here](https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html?ref=blog.ippon.tech).
The problem with the AWS SDK and Boto3 is that the `s3 sync` command is not available, which means that you *will* have to write some custom code. Reinventing the wheel is not really something recommended in software development because it might take a while to write something stable and reliable. In addition, the `sync` command from the CLI includes several handy options, such as `include` and `exclude,` so let's just step back a little and figure out a better solution.
### Lambda Layers
To have additional features and libraries available for your Lambda, you simply have to build a [Lambda layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html?ref=blog.ippon.tech) and have your Lambda use it. By having a layer that includes the AWS CLI, your Lambda will be able to call the CLI and then run the sync process like you would do from your terminal.
There are multiple ways to build your own layer but in our case. Since we are going to use [AWS CDK](https://aws.amazon.com/cdk/?ref=blog.ippon.tech) to deploy our Lambda, we can use the [lambda-layer-awscli](https://docs.aws.amazon.com/cdk/api/latest/docs/lambda-layer-awscli-readme.html?ref=blog.ippon.tech) module. This module is easy to use and installs the CLI under `/opt/awscli/aws` in the Lambda.
### Lambda
I decided to use Python to write the Lambda as the sync process is pretty straightforward. The `while` loop streams the outputs from the sync command to CloudWatch.
```
import logging
import os
import subprocess
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def sync(event, context):
# Lambda to sync two buckets using the aws cli
s3_sync = subprocess.Popen(
"/opt/awscli/aws s3 sync s3://{} s3://{}".format(os.environ['SOURCE'], os.environ['DESTINATION']).split(),
stdout=subprocess.PIPE)
# Stream logs from the cmd
while True:
output = s3_sync.stdout.readline()
if s3_sync.poll() is not None:
break
if output:
logger.info(output.strip())
s3_sync.poll()
```
The two environment variables `SOURCE` and `DESTINATION` contain the buckets' name and are populated by CDK during the stack deployment.
## Deployment and Test
### Architecture and Diagram
The architecture *must* be capable of running the syncing process daily by using the Lambda:
### AWS CDK
I have been a huge fan and advocate of AWS CDK since its first release, as it abstracts concepts like permissions and service interactions. Writing code with your favorite language that ends up deploying components on the Cloud can be quite addictive!
Below is the code that defines our infrastructure:
```
from aws_cdk import (
aws_events as events_,
aws_lambda as lambda_,
aws_s3 as s3_,
aws_s3_deployment as s3_deployment_,
aws_events_targets as targets_,
lambda_layer_awscli as awscli_,
core as cdk
)
class BlogLambdaAwscliStack(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
# Create S3 buckets
bucket_source = s3_.Bucket(self, "Bucket Source")
bucket_destination = s3_.Bucket(self, "Bucket Destination")
# Create Lambda
fn = lambda_.Function(self, "S3_Sync",
function_name="S3_Sync",
runtime=lambda_.Runtime.PYTHON_2_7,
handler="index.sync",
code=lambda_.Code.asset('lambda'),
environment={
"SOURCE": bucket_source.bucket_name,
"DESTINATION": bucket_destination.bucket_name
},
timeout=cdk.Duration.minutes(15))
# Add Lambda layer containing the aws cli
fn.add_layers(awscli_.AwsCliLayer(self, "AwsCliLayer"))
# Grant Lambda access to S3 buckets
bucket_source.grant_read(fn)
bucket_destination.grant_read_write(fn)
# Deploy data folder to source bucket
s3_deployment_.BucketDeployment(self, "Deploy files to source bucket",
sources=[s3_deployment_.Source.asset("./data")],
destination_bucket=bucket_source)
# Trigger lambda every day
rule = events_.Rule(self, "Rule",
schedule=events_.Schedule.rate(cdk.Duration.days(1)),
)
rule.add_target(targets_.LambdaFunction(fn))
```
I decided to use the [Official COVID-19 Data Lake](https://registry.opendata.aws/aws-covid19-lake/?ref=blog.ippon.tech) and the folder `static-datasets` for the source bucket. CDK will take care of uploading the data folder to the source bucket using `BucketDeployment.`
### Testing
The project is available at this [GitHub Repository](https://github.com/Falydoor/blog-lambda-awscli?ref=blog.ippon.tech) in case you want to clone it. You can follow the instructions to deploy / destroy the stack:
```
# deploy the stack
cdk deploy
# destroy the stack
cdk destroy
```
Your console should output something similar once the stack is successfully deployed:
You can manually trigger the Lambda in order to test and then check the CloudWatch logs (we can see the progress of the sync):
Check the destination bucket to make sure that the COVID data was correctly synced:
## Caveats
The main "issue" with AWS Lambda is the "15-minute timeout," which essentially means that the Lambda can timeout before the end of the syncing process in case of a large bucket. The good news is that the Lambda can just re-run and continue the syncing process until it's over because files are created on the fly.
It might be a good idea to do the first sync **manually**, as a large amount of data will take multiple hours. Then, depending on your use case, you can just reduce the schedule to run the Lambda every 30 minutes, for example.
Need help using the latest AWS technologies? Need help modernizing your legacy systems to implement these new tools? Ippon can help! Send us a line at [contact@ippon.tech](mailto:contact@ippon.tech).
## Related Articles
##### [ Python • June 24, 2023 Python in Production (Part 5 of 5) 8 min read](https://blog.ippon.tech/python-in-production-part-5-of-5)
##### [ Python • May 17, 2023 Why CDOs should be investing in Snowpark 2 min read](https://blog.ippon.tech/why-cdos-should-be-investing-in-snowpark-2)
##### [ Python • March 30, 2023 Python in Production (Part 4 of 5) 9 min read](https://blog.ippon.tech/python-in-production-part)
### Comments
### Subscribe to Our Blog!
Stay informed with the latest insights and updates by signing up for our weekly blog newsletter – delivered straight to your inbox!
[Back to blog homepage »](https://blog.ippon.tech/)
Ippon is a consulting and expertise firm, who is convinced that technology is a source of progress for society. We help our clients leverage their digital assets to design an appropriate strategy and deploy their transformation roadmap at scale.
#### Navigation
- [Home](https://ipponusa.com/)
- [About](https://ipponusa.com/about-us/)
- [Services](https://ipponusa.com/services/)
- [Blogs](https://blog.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [eBooks](https://ipponusa.com/ebooks/)
- [The Data Pour](https://info.ippon.tech/the-data-pour?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
[Contact Us](https://ipponusa.com/contact/)
[Join Us](https://ipponusa.com/careers/)
#### Services
- [Data & Analytics](https://ipponusa.com/service/data-analytics/)
- [Cloud Strategy](https://ipponusa.com/service/cloud-strategy/)
- [Artificial Intelligence](https://ipponusa.com/service/artificial-intelligence/)
- [Operating Model](https://ipponusa.com/service/operating-model/)
- [Platform Modernization](https://ipponusa.com/service/platform-moderization/)
- [Product Innovation](https://ipponusa.com/service/product-innovation/)
#### Ippon International
- [France](https://fr.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- [Australia](https://au.ippon.tech/?__hstc=223043268.1036d82247623b329dfefe4e92697801.1714729179243.1714729179243.1714729179243.1&__hssc=223043268.2.1714729179244&__hsfp=803678701)
- #### Contact
- Ippon Technologies
[3431 West Leigh Street Richmond, VA 23230, USA](https://maps.app.goo.gl/RySm6SAwZnsxeiLz9)
- [(844) 477- 6687](tel:8444776687)
- [Sales@ipponusa.com](mailto:Sales@ipponusa.com)
©Copyright 2024 Ippon USA. All Rights Reserved. | [Terms and Conditions](https://ipponusa.com/privacy-policy/) | [Privacy Policy](https://ipponusa.com/privacy-policy/) | [Website by Skol Marketing](https://skolmarketing.com/)
```json
{
"@context" : "https://schema.org",
"@type" : "BlogPosting",
"author" : {
"@type" : "Person",
"name" : "Theo LEBRUN",
"url" : "https://blog.ippon.tech/author/theo-lebrun"
},
"dateModified" : "2024-01-29T15:47:42.806Z",
"datePublished" : "2021-04-08T14:12:00.000Z",
"headline" : "Sync Two S3 Buckets Using CDK and a Lambda Layer Containing the AWS CLI",
"image" : [ "https://blog.ippon.tech/hubfs/Imported_Blog_Media/Sync-Two-S3-buckets-Using-CDK-and-a-Lambda-Layer-Containing-the-AWS-CLI-3.png" ],
"mainEntityOfPage" : {
"@id" : "https://blog.ippon.tech/sync-two-s3-buckets-using-cdk-and-a-lamdba-layer-containing-the-aws-cli",
"@type" : "WebPage"
},
"publisher" : {
"@type" : "Organization",
"logo" : {
"@type" : "ImageObject",
"url" : "https://blog.ippon.tech/hubfs/logo_SVG.svg"
},
"name" : "Ippon Technologies"
}
}
```