AWS S3 + Dynomadb solved concurrent writes warning
In code you only change lancedb.connect
lancedb.connect(
uri=f"s3+ddb://<bucket_name>?ddbTableName=<table_name>",
storage_options={
"region": os.environ.get("AWS_REGION", "us-east-1"),
"aws_access_key_id": os.environ["AWS_ACCESS_KEY_ID"],
"aws_secret_access_key": os.environ["AWS_SECRET_ACCESS_KEY"]
},
)
bucket_nameis the s3 bucket name (e.g.aws-s3-v2)table_nameis the dynomadb table name (e.g.nail-dynamodb-v1)
Next you will configure in AWS
Create Table
Search DynamoDB in AWS console. Navigate to Table section. Click Create table. Give a table name for table_name. Next, very important
Partition key: enter base_uri and set String type
Sort key: enter version and set number type
Note: I know you confusing, but yes just put the exact base_url and version.
Then use Default settings
Click Create table button.
Permission
Next, search Policies keyword and navigate to the Policies under the IAM feature. Click Create policy
Choose JSON format.
Replace with here:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::your-s3-bucket",
"arn:aws:s3:::your-s3-bucket/*"
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:GetItem",
"dynamodb:UpdateItem",
"dynamodb:Scan",
"dynamodb:Query"
],
"Resource": "arn:aws:dynamodb:us-east-1:your-aws-account-id:table/nail-dynamodb-v1"
}
]
}
- Replace your-s3-bucket with your actual S3 bucket name.
- Replace your-aws-account-id with your actual AWS account ID.
- The policy allows:
- Read & write access to S3 for storing LanceDB data.
- Full read/write access to DynamoDB (nail-dynamodb-v1).
Then, you are safe to create policy.
Add Permission
Now navigate to Users section, you will find a Permissions policies list. Click Add permissions, choose Attach policies directly, then search your policy name and add it.
Done. Now you are ready to go.