Amazon OpenSearch Service is a service of AWS which is use to deploy operate and scale OpenSearch it makes everything very easy. It supports OpenSerach and legacy Elasticsearch OSS its last open source software version is 7.10. In it you have the open that which search engine you want to use.
Amazon OpenSearch Service
- Amazon OpenSearch is also known as Amazon ElasticSearch
- In DynamoDB, queries only exist by primary key or indexes …
- With the help of OpenSearch, we can search any field, even partially matches as well
- It’s normal to use OpenSearch as a complement to another database
- Two modes: managed cluster or serverless cluster
- It do not natively support the SQL (can be enabled via a plugin)
- Ingestion from Kinesis Data Firehose, AWS loT, and CloudWatch Logs
- Security through Cognito & IAM, KMS encryption, TLS
- Comes with OpenSearch Dashboards (visualization)
An Amazon OpenSearch is also know as by the name of Amazon ElasticSearch. So the name change was due to some licensing issues. DynamoDB is just to do a comparison, we can only query the data by primary key or if we want to have indexes on database. But with OpenSearch, we can actually, as the name indicates, search any fields, even for partial matches. So it’s common to use OpenSearch to provide search to application. And so we would use OpenSearch to another database. So OpenSearch can easily used for search, but also as the name doesn’t indicate, we can also do analytic queries on OpenSearch.
You have two modes to provision an OpenSearch Cluster. Either we use the managed cluster option and then actual physical instances will be provisioned. Or you can go the serverless route and have a serverless cluster, where everything from scaling to operations is handled by AWS. And OpenSearch has own query language, it do not support SQL, but we can enable SQL compatibility via a plugin. we can ingest data from different places, such as Kinesis Data Firehose, CloudWatch Logs, IoT, or any of your custom-built application.
You have security provided through integration with IAM, Cognito, you will get at rest encryption and in-flight encryption. you can do analytics on top of the OpenSearch Service, in it you can use something called OpenSearch Dashboards.
OpenSearch patterns DynamoDB
Here are some common patterns to use OpenSearch. So you would have DynamoDB and it will contain your data. This is where your users will insert and delete and update data. And then you send all the streams in a DynamoDB Stream, which is then picked up by a Lambda Function. And that Lambda Function will insert the data into Amazon OpenSearch in real time. And through this process, what happens that your application now has the ability to search for a specific item.
For example, to do a partial search with the item name and then find the item ID out of it. And then once the item ID is obtained, then it will call DynamoDB to actually retrieve the full item from your DynamoDB Table. So that’s a common pattern in which OpenSearch provides the search capability, whereas your main source of data still remains your DynamoDB Table.
OpenSearch patterns CloudWatch Logs
There’s also other ways you can ingest CloudWatch Logs into OpenSearch. So the first one is to use what’s called a CloudWatch Log Subscription Filter, sending data in real time to a Lambda Function that is managed by AWS. And then the Lambda Function in real time sends all the data into Amazon OpenSearch.
The second way is you can also use a CloudWatch Logs and then Subscription Filter. But this time Kinesis Data Firehose can read it from the Subscription Filter. And then near real time, because this is Data Firehose, data will be inserted in Amazon OpenSearch.
