AWS S3 + Dynomadb solved concurrent writes warning
In code you only change lancedb.connect
lancedb.connect(
uri=f"s3+ddb://<bucket_name>?ddbTableName=<table_name>",
storage_options={
"region": os.environ.get("AWS_REGION", "us-east-1"),
"aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
"aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"]
},
)
bucket_name
is the s3 bucket name (e.g.aws-s3-v2
)table_name
is the dynomadb table name (e.g.nail-dynamodb-v1
)
Next you will configure in AWS
Create Table
Search DynamoDB
in AWS console. Navigate to Table
section. Click Create table
. Give a table name for table_name
. Next, very important
Partition key: enter base_uri
and set String type
Sort key: enter version
and set number type
Note: I know you confusing, but yes just put the exact base_url
and version
.
Then use Default settings
Click Create table
button.
Permission
Next, search Policies
keyword and navigate to the Policies under the IAM feature
. Click Create policy
Choose JSON
format.
Replace with here:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-s3-bucket",
"arn:aws:s3:::your-s3-bucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:GetItem",
"dynamodb:UpdateItem",
"dynamodb:Scan",
"dynamodb:Query"
],
"Resource": "arn:aws:dynamodb:us-east-1:your-aws-account-id:table/nail-dynamodb-v1"
}
]
}
- Replace your-s3-bucket with your actual S3 bucket name.
- Replace your-aws-account-id with your actual AWS account ID.
- The policy allows:
- Read & write access to S3 for storing LanceDB data.
- Full read/write access to DynamoDB (nail-dynamodb-v1).
Then, you are safe to create policy.
Add Permission
Now navigate to Users
section, you will find a Permissions policies list. Click Add permissions, choose Attach policies directly, then search your policy name and add it.
Done. Now you are ready to go.