Scalability and rapid read/write speeds of DynamoDB, combined with full text search by AWS ElasticSearch.
DynamoDB is one of the most efficient database services, provided by AWS, but it lacks one important feature – quick search and filtering of data.
That’s where ElasticSearch comes into play.
One common approach is to “write” to DynamoDB and “read” from ElasticSearch. The thing that is needed in this cases is the “synchronization” of data from DynamoDB to ElasticSearch.
Let’s try to use AWS CDK constructs to build this streaming mechanism.
We’ll start by defining a simple DynamoDB table that holds the data:
Also, let’s create the ElasticSearch Domain where data from the Table will be streamed:
So far so good. Now, how do we get DynamoDB to automatically synchronize to ElasticSearch? Using DynamoDB Streams.
You can’t just create a DynamoDB Stream though. You need to also attach something as a “listener”. In our example, this will be a Lambda function:
We‘ll define the body of the Lambda inside lambda.ts later. For now, let’s just “attach” the Lambda to the DynamoDB stream:
What the above does it is this – it instructs the DynamoDB service to automatically invoke the Lambda defined in functionForStreaming, every time there is a create, update or delete operation of any object within the DynamoDB table. Note that for batch operations (e.g. BatchWriteItem), the Lambda will be invoked in batch mode – one invokation with up to 100 items as payload.
Let’s also allow the Lambda to read-write to the ElasticSearch domain. We will need this later.
Additionally, I’d like to pass some information to the Lambda that will handle the DynamoDB stream’s events: namely the URL of the ElasticSearch domain and the ElasticSearch index where the table should be replicated. We will use this later in the Lambda handler.
And to make it possible for the Lambda to read/write from ElasticSearch securely:
Now, let’s focus on the Lambda that will receive the created, updated, deleted DynamoDB rows and synchronize them to ElasticSearch. This is where most of the logic lives: lambda.ts.
We should be good to go in terms of AWS CDK constructs.
Try deploying these constructs and adding an item to the DynamoDB table. If you browse the ElasticSearch Kibana web interface, you should see the item synchronized.