How to use DynamoDB streams to feed Algolia search

Use AWS Lambda and DynamoDB streams to feed auxiliary services

Photo by Gustavo Rezende from Pixabay

Every tool fits a specific purpose. You do not use a saw to hammer down a nail. Likewise, if you are using DynamoDB as a datastore it is better to use another tool to provide search functionality. To achieve this you need to get your data to the search tool. The easy way to do this is by enabling DynamoDB streams on your DynamoDB table.

A DynamoDB stream is an ordered flow of information about changes to items in a DynamoDB table. When you enable a stream on a table, DynamoDB captures information about every modification to data items in the table.

In this example, we are going to feed the Algolia search service from the changes in the datastore captured by a DynamoDB stream and processed by an AWS Lambda function. You can use this search service to enable search functionality on your website as depicted by the diagram below.

Web application with search function

This is Part 1 of two articles covering using Serverless with DynamoDB Streams. This article details how you can can use AWS Lambda and DynamoDB streams to feed auxiliary services.

★ Part 1 of the series is about how you can use AWS Lambda and DynamoDB streams to feed auxiliary services.

★ Part 2 of the series looks at how you can direct a specific event from a DynamoDB stream to your Lambda function.

Prerequisites:

1. Serverless CLI installed.
2. Basic knowledge of Typescript.
3. AWS account.
4. Algolia account

Searching books 📚

We are going to expand on a previous story where we build a REST API to retrieve books from a DynamoDB table for the online library service. Please consult the previous story on how to set up the DynamoDB table through the AWS Console. We are going to use the following data representing books:

DynamoDB Table data

In this story, we are enabling DynamoDB streams for this table and use an AWS Lambda function to listen to this stream to forward changes to the books to the Algolia search service to be able to search books.

Feed search service with DynamoDB streams

Let’s start building 👷‍♂️

Now we know what we are aiming for, let’s start building. The first step is enabling Streams on the DynamoDB table. Navigate in the AWS console to your DynamoDB table, select the ‘Export and streams’ tab and enable DynamoDB streams. Choose ‘New Image’ on the next screen to get the latest data and confirm.

The complete project for this article can be found on Github https://github.com/cyberworkz/examples in the online-library-stream folder.

Export and streams DynamoDB table

Upon confirming a DynamoDB stream is created that captures changes to the records in the DynamoDB table.

DynamoDB stream

The identifier for the stream contains the date and time the stream is created.

Setup Algolia 🔍

Algolia is a SaaS offering where you can use an API to search for data you have provided. They offer a free account for 10.000 records and 10.000 search requests per month. You can sign up here.

Upon signing up you will be guided to create an Index, let’s fill in ‘dev_online_library’. In the second step, you can import records or add them manually. Let’s go for the latter option. Use the JSON below to add a record.

[
 {
  "PK": "BOOK#23456",
  "Title": "Harry Potter and the Chamber of Secrets",
  "Author": "JK Rowling",
  "objectID": "BOOK#23456"
 }
]

After adding the record, you should see the index with a record.

Search index

By adding the record we are now able to configure searchable attributes. Navigate to the ‘configuration’ tab and add title and author as searchable attributes. There are many more options but for brevity, we keep it to these settings.

searchable attributes

Review and save the settings.

Next, we are going to look up the API keys to use. Navigate back to the overview page. Under the welcome message is a link to API keys.

API Keys

There is a key for using the Search API and a key for using the Admin API. The last one can be used to add records to the index. Note it down for usage in the AWS Lambda function.

Create Lambda Function 🪄

The next step is to create the AWS Lambda function to read the stream. We will use a predefined template to get a head start. Create a new project with the Serverless CLI using the following command:

sls create --template-url https://github.com/cyberworkz/serverless-templates/tree/main/aws-nodejs-typescript-dynamodb-streams --path online-library-stream

This will create a skeleton Serverless typescript project to handle DynamoDB Stream events.

serverless.yml

The Lambda function is configured to listen to DynamoDB streams. The main class in the project is the processor.ts file already set up to process DynamoDB Stream events.

processor.ts

We are going to expand the project to process the Stream events and copy it to the search index.

First, we need to install some extra dependencies to do the heavy lifting for us. Install the following packages:

• algoliasearch
• typedi

npm i algoliasearch --save
npm i typedi --save

Algoliasearch is used to connect to the Algolia service. TypeDI allows injecting dependencies into a class without tight coupling.

Next, we are going to build a model to capture the changed book in the DynamoDB table. Add the following file, book.ts.

book.ts

The ‘objectID’ attribute is an identifier used by Algolia. When adding an object with the same ‘objectID’ the previous one is overwritten. This means updates are also covered.

Now, we need to get the data to Algolia. We are going to process the DynamoDB record and use the Algolia API to push the data to the index. Add the following file, export-search.service.ts for this purpose.

The ‘searchClient’ is constructed with the Algolia application id and Admin API key you noted down earlier. The class processes a DynamoDBRecord by converting it to a Book and then adding it to the index.

The only thing left is to alter the processor.ts file to call the ExportService class. You can look in the Github project for an example of how to do this.

Deploy 🚀

Now we are all set to deploy the Lambda function. Use the following command to deploy it to your AWS account.

serverless deploy

Upon deployment, the AWS Lambda function will be created. Once created, you can set the environment variables to put the application ID and Admin API key values for Algolia.

AWS Lambda environment variables.

To test the function you can edit a ‘Book’ record in your DynamoDB table. The change will be sent to the stream and by the Lambda function add to the Algolia index. You should see the following in your Algolia index:

Algolia index with records.

Done 🙏

So, this wraps it up. I hope this helps you in your Serverless journeys. Again, the code for this project can be found on Github👇 https://github.com/cyberworkz/examples in the online-library-stream folder.

Haiko van der Schaaf

• If you enjoyed this, please follow Serverlesscorner.com on Medium.
• Love ❤️ reading my stories and other ones on Medium? Become a member if you ain’t already.
• Want to read more Serverless? Sign up for my monthly newsletter 📬 about inspiring and insightful stories on Serverless technology and use cases.