Kinesis Freestyle2 V3 - keyboard accessories kit
$27.92
Mfg # AC730-BLK
CDW # 2335785
| UNSPSC 43211706
Know your gear
The V3 accessory can attach easily to the base of the solo keyboard allowing quick and reproducible slope settings of 5, 10 and 15 degrees without the use of integrated palm supports.
Add to Compare
Enhance your purchase
Kinesis Freestyle V3 is rated
4.30 out of
5 by
25.
Rated 5 out of
5 by
Priyesh Govind from
Good scalability and tech support
What is our primary use case?
We do data acquisition based on what is pumped from the remote data and process it centrally so that we may present to our customers meaningful reports, charts, additional layers of support, or alerts.
What is most valuable?
At the moment, I am not using Amazon Kinesis, but Azure Event Hub, which I have found to be more meaningful and easier to use.
I like the event bubbling feature of Amazon Kinesis, although I ultimately switched to Azure Event Hub. Both solutions have similar features, but the latter offers us certain operational advantages.
What needs improvement?
Amazon Kinesis is not a bad product, but Azure Event Hub provides us with certain operational advantages, as our focus is on Microsoft related coding. This is why .NET is what we use at the backend. While we can use both Azure Event Hub and Amazon Kinesis towards this end, I feel the latter to be less customized or developed for use in connection with the server-less programming.
Amazon Kinesis has a less meaningful and easy use than Azure Event Hub.
Amazon Kinesis involved a more complex setup and configuration than Azure Event Hub.
For how long have I used the solution?
I have been using Amazon Kinesis for the past year, although I have since switched to Azure Event Hub.
What do I think about the scalability of the solution?
The scalability is pretty good. One can have any number of nodes spawned or replicated on the primary. Any load can be handled, perhaps a few terabytes with ease in around 15 seconds. One can scale up to this.
How are customer service and technical support?
While we have not had occasion to contact Amazon tech support concerning the solution, we have in relation to other matters. We felt it to be good.
How was the initial setup?
The initial setup and configuration of Amazon Kinesis was more involved than that of Azure Event Hub.
What's my experience with pricing, setup cost, and licensing?
The solution's pricing is fair. The trick lies in Amazon's pricing. They charge according to the different layers of or types of data that is transfered.
Which other solutions did I evaluate?
In addition to Azure Event Hub, we also have experience with Apache Kafka, which I feel to offer greater power but more complex configuration. This solution has more features for a variety of purposes.
What other advice do I have?
The question of whether I would recommend Amazon Kinesis over Azure Event Hub is tricky. While both have their advantages and I consider them to be almost equal, we feel the latter to be better suited to our environment, which is why we went with it. The data transferring policies and associated costs of Amazon were the deciding factors for me.
I rate Amazon Kinesis as an eight or nine out of ten.
Which deployment model are you using for this solution?
Private Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Microsoft Azure
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2021-09-02T00:00:00-04:00
Rated 5 out of
5 by
Shubham-Joshi from
Helps to stream events but needs improvement in limit
What is our primary use case?
I work in a gaming company that builds games for the global market. We use Amazon Kinesis to stream events.
How has it helped my organization?
I have worked in companies that build tools in-house. They face scaling challenges.
What needs improvement?
Amazon Kinesis should improve its limits.
For how long have I used the solution?
I have been using the product for a month.
What do I think about the stability of the solution?
I rate the tool's stability a ten out of ten.
What do I think about the scalability of the solution?
My company has two to three users for Amazon Kinesis.
How was the initial setup?
I rate the tool's deployment a nine out of ten. Deployment takes one day to complete.
What's my experience with pricing, setup cost, and licensing?
The tool's entry price is cheap. However, pricing increases with data volume.
What other advice do I have?
I rate Amazon Kinesis a seven out of ten.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2023-11-10T00:00:00-05:00
Rated 5 out of
5 by
Faisal Umer from
Provides near real-time data streaming at a consistent rate, but its cost is too high
What is our primary use case?
Amazon Kinesis is a queuing or buffering system that we use as a central place to buffer the incoming data we receive from the source. The actual destination is open-faced. Amazon Kinesis is used as a buffer in between to decouple the workload.
What is most valuable?
Amazon Kinesis's main purpose is to provide near real-time data streaming at a consistent 2Mbps rate, which is really impressive.
What needs improvement?
The solution currently provides an option to retrieve data in the stream or the queue, but it's not that helpful. We have to write some custom scripts to fetch data from there. An option to search for data in the queue can really help us in our day-to-day operations.
Since the solution is a buffer system, you write to it and read from it. The readers are called consumers. If you want to run multiple consumers reading from the queue, you have to enable the enhanced fan-out feature on Amazon Kinesis. This enhanced fan-out feature is quite costly.
There was a point when we had a huge budget increase in one week just because of the enhanced fan-out feature. This feature does not provide any special out-of-the-box functionality. Hence, we struggle to optimize multiple consumers reading from a single queue. We were charged high costs for the solution’s enhanced fan-out feature.
For how long have I used the solution?
I have been using Amazon Kinesis for more than two years.
What do I think about the scalability of the solution?
The solution is pretty good in terms of scaling. Amazon Kinesis has shards, which are the instances or units that the solution spins up for you. Depending upon your account quota, you can spin up as many shards as you want. You can even raise a request to increase that quota, which will be done sooner. Overall, Amazon Kinesis is a really scalable solution.
Our team, consisting of four to five people, uses the solution extensively in our organization.
How are customer service and support?
We really struggle to get better support for Amazon Kinesis.
What's my experience with pricing, setup cost, and licensing?
Amazon Kinesis is an expensive solution.
What other advice do I have?
Amazon Kinesis is an AWS-managed service, just like S3 or EC. We don't have to deploy it; it is just there, and we spin it up. You must go to AWS' service page and click on Kinesis. Then, you can create it by clicking on Create and entering the name.
I would not recommend Amazon Kinesis to other users. Users can choose a cheaper alternative. They can use any other queuing system or in-house Kafka if they have a Kafka team. Amazon Kinesis provides near real-time read-and-write, but its cost is too high. Users can choose another option that provides the same functionality at less cost.
With Amazon Kinesis, you have to run a consumer who sees from Amazon Kinesis. AWS provides the Kinesis Client Library (KCL), which reads from the Kinesis stream. That library is also used in DynamoDB for data checkpointing. For example, if you have one day of data in Amazon Kinesis and started reading from 12 AM yesterday. The Kinesis Client Library (KCL) will check on the data in the DynamoDB. You get charged for the DynamoDB table out-of-the-box, along with Amazon Kinesis.
The DynamoDB table also costs a lot, which should not be the case. It is just read-and-write and is downloaded from the Kinesis Client Library (KCL). The DynamoDB table's cost should be very minimal, but that's not the case. The consumer is not optimal for efficient read-and-write, which further increases the cost. Both Amazon Kinesis and DynamoDB come into the picture.
Overall, I rate the solution a five or six out of ten.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2024-04-28T00:00:00-04:00
Rated 5 out of
5 by
Venkata Maniteja Alapati from
Easy to activate and easy to test and set up
What is our primary use case?
It's a kind of data stream tool that actually does kind of external serverless data streaming to different services at Amazon. Especially for the usage I've done, it's basically for getting data to Redshift.
Kinesis can be one of the sources, and at the same time, Amazon Kinesis can be used to get the data out to other systems through other APIs. Kinesis is one of the tools I have used for typical data streaming.
I've utilized Kinesis Data Streams along with Kinesis Video Streams for different use cases. Again, these are required for working with data as part of Amazon Connect configurations.
What is most valuable?
Kinesis is fairly easy to use; it's kind of a service that is there and then fairly easy to configure. And it gathers lots of data while having proper authentication mechanisms. It's easy to activate and easy to test and set up as well.
Typically, all the contact tracing codes that are there as part of Amazon Connect and also some of the real-time data can be passed with different sources of streaming. Like, Genesys also has data streaming and video streaming, which actually passes the real-time data as well.
So it's pretty straightforward to configure, easy to understand. It's a typical service that is required in Connect integrations, specifically.
What needs improvement?
For me, especially with video streams, there's sometimes a kind of delay when the data has to be pumped to other services. This delay could be improved in Kinesis, or especially the Kinesis Video Streams, which is being used for different use cases for Amazon Connect. With that improvement, a lot of other use cases of Amazon Connect integrating with third-party analytic tools would be easier.
For how long have I used the solution?
What do I think about the stability of the solution?
It's a stable product overall. So far, I haven't seen any issues wherever I have used it.
What do I think about the scalability of the solution?
This is kind of more like a backend tool used by our developers and the engineering team, including myself, to improve configurations and connectivity. Typically, it depends on the use cases and company requirements, but for my company, three people generally do this work.
Which solution did I use previously and why did I switch?
How was the initial setup?
The initial setup is pretty easy and straightforward. It can be done pretty quickly with the direct documentation we have from AWS for Amazon Connect and Amazon Kinesis. That helps things to be configured very quickly.
What's my experience with pricing, setup cost, and licensing?
The pricing depends on the use cases and the level of usage.
If you wanted to use Kinesis for different use cases, there's definitely a cheaper base cost involved. However, it's not entirely cheap, as different use cases might require different levels of Kinesis usage.
But, from an Amazon Connect perspective, where I have used this, it's quite good. It's pretty much cheaper compared to alternative methods.
What other advice do I have?
I would recommend using Kinesis. Again, it depends on the use cases. One specific advantage of Kinesis is how straightforward it is, and it offers a lot of integration possibilities. So, for those wanting to use it with Amazon Connect, this is a good solution.
Overall, I would rate it between an eight out of ten because of its ease of use. I like the ease of use and how we can quickly get the configurations done, making it pretty straightforward and stable.
Disclaimer: My company has a business relationship with this vendor other than being a customer:
Date published: 2024-04-28T00:00:00-04:00
Rated 5 out of
5 by
Wojciech Doganowski from
Has a positive impact on operational efficiency and helps save costs
What is our primary use case?
My company primarily uses Amazon Kinesis within our data processing workflow to push the data into our data lake.
What needs improvement?
AI processing or cleaning up data would be nice since I don't think it is a feature in Amazon Kinesis right now. Amazon Kinesis has a specific purpose, and it may not fit into use cases anymore if it is loaded with too many features.
For how long have I used the solution?
I have been using Amazon Kinesis for a year. My company has a partnership with Amazon. My company is also a customer of Amazon.
What do I think about the stability of the solution?
Stability-wise, I rate the solution a nine out of ten. I have not encountered any problems with the product's stability.
What do I think about the scalability of the solution?
I have not encountered any problems with the product's scalability. Scalability-wise, I rate the solution a nine out of ten.
My company mostly deals with enterprise-sized businesses.
Whether I recommend the product to others depends on the applications, but moving a lot of data in real-time is a very good solution.
How are customer service and support?
The technical support a user receives from Amazon depends on the kind of relationship that they have with AWS. If you are not known by Amazon, support would be low from AWS. If you have a specific relationship, then the support is okay. I rate the technical support an eight out of ten.
How would you rate customer service and support?
Positive
How was the initial setup?
I rate the setup phase a seven on a scale of one to ten, where one is difficult, and ten is easy.
The solution is deployed on a public cloud.
The time required to deploy the solution is something that depends. A quick deployment can be done in a few minutes. If it is a production deployment where all the scripts and other related areas must be managed, it takes a couple of weeks. It is not the deployment itself of the tool alone that is involved since there is a need to prepare everything around it.
What's my experience with pricing, setup cost, and licensing?
I rate the product price a five on a scale of one to ten, where one is cheap, and ten is expensive.
What other advice do I have?
In terms of the most useful part of the tool for our company's data analytics, I think that because we decided to use AWS, Amazon Kinesis is present in our organization. It is easy to configure, highly available and offers high throughput. Amazon Kinesis is one of our company's main resources.
The scalability hasn't impacted our real-time data processing capabilities. Actually, users can scale up, and it doesn't impact our company. There might be a positive impact on scalability, so there are no issues.
The product has had a positive impact on operational efficiency because it actually allows our company to filter out some data and save costs.
The integration capabilities of the product are good. My company has integrated Amazon Kinesis with some APIs, AWS Lambda, S3, and Amazon EKS.
The adoption of data processing and data lakes is the purpose for which Amazon Kinesis is designed.
I rate the tool a seven to eight out of ten.
Which deployment model are you using for this solution?
Public Cloud
Disclaimer: My company has a business relationship with this vendor other than being a customer:Partner
Date published: 2024-05-03T00:00:00-04:00
Rated 5 out of
5 by
Ajay Vashishtha from
Useful for tracking silent events and capturing events
What is our primary use case?
The solution provides real-time event streaming.
What is most valuable?
From my experience, one of the most valuable features is the ability to track silent events on endpoints. Previously, these events might have gone unnoticed, but now we can access them within the product range. For example, if a customer reports that their calls are not reaching the portal files, we can use this feature to troubleshoot and optimize the system.
What needs improvement?
I suggest integrating additional features, such as incorporating Amazon Pinpoint or Amazon Connect as bundled offerings, rather than deploying them as separate services.
For how long have I used the solution?
I have been using the product for a year.
What do I think about the scalability of the solution?
My company has four to five customers. The scalability of Amazon Kinesis has improved various data processing capabilities.
How are customer service and support?
I am happy with the tool's customer support.
How was the initial setup?
Amazon Kinesis' deployment is easy.
What's my experience with pricing, setup cost, and licensing?
The tool's pricing is cheap.
What other advice do I have?
If you're considering using Amazon Kinesis for the first time, I would advise exploring their services. It can be useful for capturing real-time events, greatly benefiting your solution. I rate it a seven out of ten.
Disclaimer: My company has a business relationship with this vendor other than being a customer:customer/partner
Date published: 2024-04-10T00:00:00-04:00
Rated 5 out of
5 by
ChetanGoyal from
Efficient Data Ingestion and good performance
What is our primary use case?
Amazon Kinesis is a service in AWS used for data ingestion. We pull data into Kinesis streams from various sources like OCS and then consume it for analysis and reporting.
What is most valuable?
I find almost all features valuable, especially the timing and fast pace movement. The best part is that there is no loss of data.
What needs improvement?
The price is not much cheaper. So, there is room for improvement in the pricing.
For how long have I used the solution?
I have been working with Amazon Kinesis for about 18 months. I have hands-on experience using and providing it.
What do I think about the stability of the solution?
The performance is quite good, and it's stable. Stability is okay. We didn't face any issues with stability.
What do I think about the scalability of the solution?
It is a scalable solution. It has been in continuous use for two years. So, there are over 20,000 end-users of this solution.
How are customer service and support?
The customer service and support are great. They have provided great assistance when needed.
How was the initial setup?
The initial setup is not straightforward; it has medium complexity. It took almost two weeks to complete the implementation and coding.
What about the implementation team?
Developers are required, along with release managers and other roles. There is a big team that works on the maintenance of the solution.
What's my experience with pricing, setup cost, and licensing?
The price is not much cheaper, but I can't say it's too expensive either. However, the customers do not have to pay any additional costs.
What other advice do I have?
I would definitely recommend using the solution. It's a great service, and it can be used wherever it's applicable in their model and architecture.
Overall, I would rate the solution an eight out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: My company has a business relationship with this vendor other than being a customer:Partner
Date published: 2023-08-05T00:00:00-04:00
Rated 5 out of
5 by
Sushil Pandey from
Available as a managed service that can be shared automatically
What is our primary use case?
I wanted to use the solution in my company for topics, but in the beginning, we did some EOC to use the data analytics part, but then we didn't go with that. We are currently using Amazon Kinesis Data Streams.
What is most valuable?
The best thing about the tool is that I use it as a normal service. Similar to Kafka topic, we use Amazon Kinesis in our company. It is a managed service that can be shared automatically based on consumption and consumer needs, so we don't have to worry. Sometimes, we also get the throughput error, and there are data contaminants. Multiple jobs try to help the queue.
What needs improvement?
There are some kind of hard limits on Amazon Kinesis, and if you hit that, then you will get the throughput exceed error. We have to deal with and reduce how many consumers are hitting Amazon Kinesis Data Streams. We combined multiple jobs into a single job, so that there are no multiple jobs in the mainstream.
For how long have I used the solution?
I have been using Amazon Kinesis for two years. My company has a partnership with the tool.
What do I think about the stability of the solution?
There is no problem with the tool's stability. I think with the tool, you cannot send or store more than one MB of data in one go. You have to split the two MB of data that you want to store. You have to split it into two messages, and then you can store it. Later on, on the consumer side, you have to deal with it and see how you can merge the data. For most of the devices, the data which we get is not more than one MB. In some use cases, we find the need to do something from our end.
What do I think about the scalability of the solution?
Scalability-wise, I rate the solution a five out of ten since we faced some errors in the tool.
What other advice do I have?
In terms of integrating Amazon Kinesis with other tools, I would say that we don't directly do anything. We have our jobs that consume the data. You have multiple ways to read Amazon Kinesis. If you have a traditional job, then you can use some SDK-based libraries to store data or send data. If you want to consume data, there are ways that you can directly do it, such as using JAR files and reading the data from them. If you have Redshift or Amazon Kinesis Data Firehose, you can directly send data to S3, and then later on, you can consume it. There are multiple options to consume data, but we are currently using a via script, and we are not directly sending it to S3 or some other database.
I recommend the tool to others.
If you have to deal with more than one MB, then you have to deal with your process of how you want to send data to places. I just put the data, and then on the job, you have to combine that, which would be tricky. Another way is to store the metadata and then store the actual data somewhere, like in SQL or somewhere else, and later on, you can put the locations. You need to make sure not to have multiple consumers creating the same data stream. Otherwise, you can get the throughput errors.
I rate the tool a nine out of ten.
Disclaimer: My company has a business relationship with this vendor other than being a customer:
Date published: 2024-09-12T00:00:00-04:00
Rated 5 out of
5 by
Prabin Silwal from
Pipeline setup is very simple
What is our primary use case?
In our project, my company uses Amazon Kinesis and some other products from AWS. There are also products from Azure, like static web applications or when we need good storage. In my new job, my company is completely dependent on Azure.
The tool is mostly for the logging part and temporary buffer. In our company, we generate lots of logs and some background events, and we want to send an aggregate of those to our data lake. We use the tool for a data stream or for processing things.
What is most valuable?
The most valuable feature of the solution is that we do not have to worry about other things. After setting the pipeline, we can just get any of the logs, and that will be deposited to the data lake easily.
What needs improvement?
I am not exactly sure about where improvements are needed in the tool. When I was working on the tool, it was very scalable, and the only thing we needed in our company was temporary streaming stuff that could work well. We didn't want to set up our own Kafka, other queues, or processing systems. As it is a cloud tool, it is easy for us to use the tool, and it satisfies all our requirements. Maybe for the other cases, if we need, then it may need some improvements. The tool satisfies our particular needs.
Currently, the pipeline setup is very simple. For our particular use cases, it is because we just want to get the data and send it to the different data lakes or some logging system. Previously, we also used Amazon Kinesis to log those to Splunk, and later on, we removed Splunk and transferred that to Datadog. For our use cases, I don't want any new features in the tool. Amazon Kinesis' use case is for collecting, processing, and analyzing. If anything can be added to the tool, then I feel one should be able to use the same kind of tool so that everything is there in the product, like an alert system, and so that one can analyze, make a query, and do sourcing from the solution itself rather than using other logging and monitoring systems. The tool should focus on having an alert system rather than having to use a third-party solution. We can just get the data over Amazon Kinesis, and we can directly use all the benefits of current analytical tools, like in the areas involving BI, Looker, and Tableau. One would not need to buy the aforementioned tools, and we can just get started with Amazon Kinesis.
For how long have I used the solution?
I have been using Amazon Kinesis for three and a half years. I am a user of the tool.
What do I think about the stability of the solution?
It is a stable solution.
What do I think about the scalability of the solution?
It is a scalable solution. If I include my team, data engineer, and the data centers, I would say that the data center is more focused on data lake only, so Amazon Kinesis is used by around 24 people.
Which solution did I use previously and why did I switch?
Before Amazon Kinesis, my company used to write those directly through the API call to the endpoint of the data warehouse, and there was no mediation, which was a problem for us. We needed some fast writing for some temporary storage where we could apply it as a data stream, and while we searched for a good solution, we came across Amazon Kinesis.
How was the initial setup?
The product's initial setup phase is not difficult because we are using the tool on the cloud. In our company, we choose the tool's cloud version option because it will be easier rather than setting up our own instances.
The solution is deployed on AWS Cloud.
What's my experience with pricing, setup cost, and licensing?
I think for us, with Amazon Kinesis, if we have to set up our own Kafka or cluster, it will be very time-consuming. If one considers the aforementioned aspect, Amazon Kinesis is a cheap tool.
What other advice do I have?
For the use cases, the only thing with Amazon Kinesis is that you already have another queue system or storage like Kafka or any other already-built streaming systems, so you don't have to invest in Amazon Kinesis. If you don't have anything, you can easily start with Amazon Kinesis.
For the logs, handling the streaming part, and if you already have many AWS products while also needing to send data to S3 or any other storage or cloud tool, it will be very optimal to use. I rate the tool a nine out of ten.
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2024-07-08T00:00:00-04:00
Rated 5 out of
5 by
Rajni Kumar Jha from
Used for media streaming and live-streaming data
What is our primary use case?
We use Amazon Kinesis for media streaming and live-streaming data.
What is most valuable?
The most valuable feature of Amazon Kinesis is real-time data streaming. If a customer calls the contact center and gets connected to an agent, the call is simultaneously streamed and stored in an Amazon S3 bucket. We can simultaneously enable the transcript through Amazon Kinesis on a live call and send that data to any team. We can use the solution for the live monitoring, live review, and live transcript of those calls.
We can use the solution's features to check whether all the data is replicating and stored in Amazon S3.
What needs improvement?
The solution has a two-minute maximum time delay for live streaming, which could be reduced.
For how long have I used the solution?
I have been using Amazon Kinesis for ten years.
What do I think about the stability of the solution?
Amazon Kinesis is a stable solution, and I haven't faced any issues with its stability.
What do I think about the scalability of the solution?
Over 5,000 users are using the solution in our organization. The solution's scalability has impacted 99.9% of our data processing capability because Amazon manages it.
How was the initial setup?
You need not install Amazon Kinesis. Once you create an instance, it will automatically integrate with Amazon Kinesis. After that, you only need to define the S3 bucket path created by Amazon S3. Amazon Connect automatically streams through Amazon Kinesis. In the Amazon Kinesis stream, you need to define the URL. After that, it will go to Amazon S3. We need not do it from our side as Amazon totally manages it.
What other advice do I have?
It is not compulsory to use Amazon Kinesis. If you don't want to use the data streaming, you can use just the Kinesis data firehose. Using the Kinesis data firehose is compulsory because we can't store all chats and recordings in Amazon S3 without it.
When a call comes in the Amazon Kinesis instance, it will go to Data Streams if we use it. Otherwise, it will go to the Kinesis data firehose, where we need to define the S3 bucket path, and it will go to Amazon S3. So, without the Kinesis data firehose, we can't store all the chats and recordings in Amazon S3.
Using Amazon Kinesis totally depends upon the user's requirements. If you want to use live streaming for the data lake or data analyst team, you need to use Amazon Kinesis. If you don't want to use it, you can directly use the Kinesis data firehose, which will be stored in Amazon S3.
Overall, I rate the solution an eight out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2024-04-13T00:00:00-04:00
Rated 5 out of
5 by
reviewer1476249 from
Easily replay your streaming data with this reliable solution
What is our primary use case?
In the simpler use case, we were just pumping in some data. We wanted a product, an AWS service, that would accept data in bursts. We were pushing in, for example, 500 records every 300 milliseconds. What I'm trying to say is per second we were trying to pump in around 1,500 records into some streaming services what we were looking at. That type of streaming information would then go into another source, for example Lambda. Then Lambda would consume the data and ultimately we would process and store it in DynamoDB.
This was the basic flow that we had. We were looking for a service. And at that point in time in our organization, the architects were asking us to leverage Kinesis to see how it performed. They wanted to see how it performs, so they were encouraging us to use it. Although we were looking at something as simple as SQS and SNS, they were encouraging us to use Kinesis and that is what we did.
There were a few considerations when we moved Kinesis. What is the reliability? When I say reliability, I mean resilience, or the failure mechanism we thought was required for that use case because we did not want to lose data. Also, we wanted to have the ability to replay from a certain point because we were pumping in reports from a data source and we were always keeping track of the point at which we had stopped. So if we wanted to replay something from the prior data which was already processed by Kinesis, and it failed in the Lambda, we wanted to have the ability to retry and replay the previously processed stream.
That prompted us to use Kinesis because it has the really good feature of being able to replay for 24 hours whatever you've already processed and this allows us to replay it. That was one key feature that we thought we would need. In fact, performance-wise, it performed really well. We also understood that it is actually meant for streaming, video streaming and stuff like that. Even data streaming. It does a good job with it. But mostly, we saw that it is a more suitable service for video streaming simply because when we actually pump data into Kinesis, we don't know how to test it other than waiting for the data to come out of it from the other end and hook into Lambda and extract data out of it and process it.
That's the only way we can test it. That was a drawback but it did not matter too much. But it did matter in the next project, and for the bigger use cases where we used Kinesis. But this project was a simple use case and it served really well, so we kept it as-is. We moved on to the next project, which was bigger. It was an event-driven architecture that we were trying out on one of the features. When we went event-driven, at that time a few of the new features and new services from Amazon which are available right now, were not available.
We thought of using Kinesis again to stream the data from one microservice to another in a proper microservice architecture. We were using this as a communication medium between microservices. This is where the testing was a little complicated for us. Ultimately, what we realized out of the entire exercise was that Kinesis may not have been the right choice of service for us for our use case. But what we discovered were the benefits of using Kinesis and also the limitations in certain use cases.
The biggest lesson learned for us was even before you take up anything like Kinesis, which is a big AWS service, there has to be a POC, proof of concept, done. To see whether it really suits that use case or not. That is what we ultimately realized. Before that, there were a few other reasons why we chose Kinesis over DynamoDB streaming. Ultimately it was from one microservice to another, and each microservice had its own DynamoDB data store.
We were thinking of using the DynamoDB Stream and Kinesis to keep things simple. But it turned out that DynamoDB Streams have a limitation that whatever a stream comes out of DynamoDB it can be consumed only by a single client. But with Kinesis it doesn't matter. Any number of data sources can come in and whatever Kinesis publishes can be consumed by any number of clients. That is why we went with Kinesis in order to see how it performed. Because even performance-wise, we found that we need a crazy load server because we are part of the wagering industry, which needs peak performance. Online betting. In Australia, it's a regulated market and one of the most happening businesses. Here, performance is really important, because there are quite a few competitors, around 10 to 15 prominent competitors and if we have to stand out, our performance has to be beyond the customer's expectation.
So, with that in mind, they knew our performance had to scale up. That is where we found the advantage of using Kinesis. It's been reliable. It has not failed to publish. It actually did fail, but the failure was simply because of pumping in too much data than what Kinesis can take in.
There is a limit that we discovered. I don't remember the numbers there. But we did manage to break Kinesis by pumping in too much data.
How has it helped my organization?
The major advantage with Amazon Kinesis is the availability. Additionally, the reliability is awesome when it comes to Kinesis. Kinesis also offers the replay.
It is incredibly fast. The ingesting of data, the buffering, and processing the data are really fast. With AWS you always get the the dashboard for monitoring. That is a really good aspect for us to see how Kinesis is performing. Otherwise there is no other way for us to know what's happening within Kinesis other than the Lambda kicking in and processing. So the Lambda logs were indirectly necessary for us to look into Kinesis.
The dashboarding AWS provides out of the box for monitoring the performance of benefits is quite nice. Also, it is a self-managed service so we don't need to worry about what happens behind Kinesis. That was another big win for us. We did not have to worry about how to maintain or manage Kinesis in general. That was a consideration. It is kind of server-less.
The scalability was quite acceptable. It can handle a large amount of data as well. It can take in a large amount of data, but there is a limit. It can take a huge amount of data and process it from many sources. We can have any number of data sources coming in, and it can ingest all of them and publish it to wherever you want.
You can design your code in such a way that the Lambda that actually processes whatever is published by Kinesis can kind of segregate the data coming in from multiple data sources, based on the logic that is implemented there. That is a nice feature. Ingesting data from multiple sources, and being able to publish it to multiple destinations.
What is most valuable?
The feature that I've found most valuable is the replay. That is one of the most valuable in our business. We are business-to-business so replay was an important feature - being able to replay for 24 hours. That's an important feature.
In our use case Kinesis was able to handle the rate at which we were pumping in data and it could publish the data to whatever destination, be it Lambda or any other consumer.
We were seeing that there was a delay in the amount of processing time of the Lambda and the subsequent storing into DynamoDB. There was a delay in that process. So, at the rate at which we were pumping in the data, it was obvious we had ensured that this should work. This rate at which we were pumping it is the rate at which the data is published and processed, as well. But we saw that it was not working. Not the Kinesis data nor the subsequent parts of our application, they tended to not be up to the mark with Kinesis. So the business asked us for the ability to be able to get back to a certain point in time and replay the entire thing. That way there is a record if there is an error when it is being processed.
The ordering is another big thing for us. Kinesis is known for maintaining the order in which the data is ingested. We can tweak that and can configure Kinesis to ensure that the ordering is maintained. The order in which the data is actually being published is also important for us. That is why the business was ok even if a thousand record failed to process, because they were okay to start from 500 again, and again reach a thousand. They wanted to ensure that there was no scope for failure there. That is why the replay feature was useful for us. That is why both performance and replay are important. When I say performance, I mean the reliability. Kinesis has an inbuilt replay mechanism that also came in handy for us.
These were the crucial things that we were looking at, and it worked quite well.
What needs improvement?
In general, the pain point for us was that once the data gets into Kinesis there is no way for us to understand what's happening because Kinesis divides everything into shards. So if we wanted to understand what's happening with a particular shard, whether it is published or not, we could not. Even with the logs, if we want to have some kind of logging it is in the shard. That is something that we thought we needed then, but later we realized that Kinesis was not built for that. They must have already improved by now, because I have not been in touch with AWS for the last five, six months since I joined this organization which uses Azure. I did not get to experiment with AWS Kinesis too much after that.
It was built for something else, but we used Kinesis for one purpose and we were expecting a feature out of it that may not have really been the design of the service when they built Kinesis. It was almost like a black box for us, because once the data comes in we need to rely on the Lambda itself to let us know. Because if some Kinesis code is coming in, it processes that we wi...
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-12-31T00:00:00-05:00
Rated 5 out of
5 by
Akinola McLean from
Enables us to respond in real time; great auto-scaling feature
What is our primary use case?
Our primary use case of this solution is as an intricate part of our data pipeline to deal with all of our big data problems. The traffic in our industry is highly volatile. At any given time we could have 10,000 users, and five minutes later it could be 100,000. We need systems fast enough to deal with that elasticity of demand, and the ability to deal with all the big data problems. Volume, velocity, ferocity, things like that. That's where we use the Kinesis platform. They have different iterations of it. The normal Kinesis Stream, is a little bit more manual, but we use that for our legacy technology, and for the more recent ones, we use Kinesis Firehose.
How has it helped my organization?
We dynamically change some of our product offerings based on user interaction. We can respond faster to user behavior, rather than waiting for the data to be at rest. We run some analytics models, and can then react in real time.
What is most valuable?
When it comes to Kinesis Firehose, the most valuable feature is the auto-scaling. It does auto-sharing, auto-correction, things like that and responds dynamically. Secondly, it innately has all the features of our reliable data pipeline, allowing you to store raw documents and transform data on the fly. When data comes into the stream through Firehose, we can see it and analyze every single object, keep the raw objects, carry out some transformations on it in flight, and then put it at rest. It allows us to do some real time analytics using Kinesis Analytics. We do anomaly detection in flight as well. We receive any changes with regards to user patterns and behaviors, in real time because Kinesis allows that.
What needs improvement?
They recently expanded the feature sets, but when we were implementing it, it could only deliver to one platform. I'm not sure where it's at now but multiple platforms would be beneficial. I'd also like to have some ability to do first in, first out queuing. If I put several messages into Firehose, there's no guarantee that everything will be processed in the order it was sent.
What do I think about the stability of the solution?
We've had no problems with stability and we implemented well over a year ago.
What do I think about the scalability of the solution?
The scalability of this solution is good. We are using it extensively with pretty much every single one of the flows.
How are customer service and technical support?
The technical support could be improved. They tend to send you back to the documentation.
Which solution did I use previously and why did I switch?
We switched to Kinesis because of the technical complexity of the previous solution. In the previous solution, Ops would write feeds on the SQS queue, and then it required physical machines to connect, pull the data, transform it and write. That required three or four different technologies. Kinesis has removed a lot of technical complexity to the architecture.
How was the initial setup?
The initial setup was straightforward. Both the user interface and the programmatic access is very intuitive. And again, it's not difficult, even non-technical people would be able to set it up. It took two people to implement. I was responsible for data architecture and we had a developer to transform the data inside. Deployment took less than an hour. The documentation was very helpful.
What was our ROI?
We've been able to drop our costs for ingesting data by about 60 to 70%.
Which other solutions did I evaluate?
We didn't evaluate anything else because no other product offered that type of fast solution at the time. Whatever we looked at added technical complexity to the architecture.
What other advice do I have?
It's important to think about how you are going to fix the end points that connect to your Kinesis files.
I would rate this solution a nine out of 10.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-11-22T00:00:00-05:00
Rated 5 out of
5 by
Roberto_Queiróz from
Great for large environments and has good configuration but needs and experienced person to set it up
What is our primary use case?
We use this solution for quite large environments.
We use it to capture and process a lot of data. We use it, for example for data analytics and query and analyze a stream's data.
How has it helped my organization?
We are a sizable organization and as such, have a lot of data. The solution allows for real-time analysis and you can use a scaler to handle data flows.
What is most valuable?
The solution is very flexible and allows for a lot of configuration. It just offers up a lot of possibilities.
I'm using Amazon S3 and Redshift using Amazon server. I can make large configurations and update in near real-time, so that we have real-time use for batch intervals.
The solution is great for scanning in order to handle environmental data.
The data stream feature on offer is excellent. We use it quite extensively.
The solution works well in rather sizable environments. We deal with a lot of data and it handles it very well.
The solution has a very good alerts system to allow us to respond in real-time.
The dashboards are excellent.
The solution offers very good data capture and integrates well with Power BI and Tableau, for example.
The product makes it very easy to create jobs.
What needs improvement?
The automation could be better. The solution needs to be better at information capture.
Some jobs have limitations which can make the process a bit challenging.
In order to do a successful setup, the person handling the implementation needs to know the solution very well. You can't just come into it blind and with little to no experience.
For how long have I used the solution?
I've used the solution for six or seven years or so.
What do I think about the scalability of the solution?
We work with very large environments and haven't had any issues with feeling constricted by the solution.
How was the initial setup?
Personally, based on my past experience and my long history with the solution, the initial setup was not complex. It was pretty straightforward. I find it very easy to use these tools.
A user will need to understand how to create analytics using processing a large amount of information. There may be legacy solutions in the mix as well. A new user will need to understand the environment and all of the requirements before really digging in.
What I will need, basically, is a data map, where I can find any legacy data. From there I can do the setup and it goes pretty smoothly.
What about the implementation team?
I handle the implementation myself.
Which other solutions did I evaluate?
You can compare this solution to Data Factory and Hadoop. They have a few overlapping characteristics. However, for my industry, Hadoop, for example, wouldn't work as it was lacking some characteristics and parameters and some understanding of the industry itself.
What other advice do I have?
I have a lot of experience in Kinesis and data analytics including in networking in the Amazon AWS environment. My experience is as a big data architect. I draw all environments in AWS.
On a scale from one to ten, I would rate the solution at a six. It's pretty good, and great for big environments, however, you do need to be well versed in the product to set it up.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2021-01-11T00:00:00-05:00
Rated 5 out of
5 by
Haja Najumudeen from
Processes data from hundreds to thousands of processes with very low latency
What is our primary use case?
We use it for:
* Security
* DDoS attacks
* Server application firewall
* Kinesis makes it easy to collect the process
* Real-time analysis
* Streaming data
We can get insight and react quickly to new information.
What is most valuable?
Kinesis is in real-time. It enables you to process stream data in real-time. You can drive it in seconds or minutes instead of hours or days.
Kinesis is a fully managed program streaming application. You can manage any infrastructure. It is also scalable. Kinesis can handle any amount of data streaming and process data from hundreds to thousands of processes in every source with very low latency.
What needs improvement?
Kinesis is good for Amazon Cloud but not as suitable for other cloud vendors.
For how long have I used the solution?
I have been using Amazon Kinesis for two years.
How are customer service and technical support?
We have used their support before. They are quick to help. Support is very good. They can resolve an issue within ten minutes.
How was the initial setup?
The initial setup was straightforward. The deployment took two months.
There are 15 employees who work on it in my company. Some roles include myself as the team lead, support, and solution architects.
What was our ROI?
I have seen ROI.
What other advice do I have?
Kinesis has the best of Amazon: data streaming, building processes, data analytics, data in real-time are very good. The output and monitoring are easy. It has good performance.
I would rate it a nine out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-10-30T00:00:00-04:00
Rated 5 out of
5 by
Pagidal Venkata Subba Reddy from
User friendly and feature rich solution
What is our primary use case?
One use case is consuming sales data and then writing it back into the S3. That's one small use case that we have; from data Shields to data Firehose, from data Firehose to Amazon S3.
There are OneClick data streams that are coming in. For click streams data we established Kinesis data streams and then from Kinesis data streams, we dump data into the S3 using Kinesis Data Firehose. This is the main use case that we have. We did many POC's on Kinesis, as well. Also, one more live project using the DynamoDB database is running in Amazon. From DynamoDB we have triggers that automatically trigger to Lambda, and then from Lambda we call Kinesis then Kinesis writes back into the S3. This is another use case.
Another thing that we did is called Kinesis data analytics where you can directly stream data. For that, we use a Kinesis data producer. From that, we establish a connection to the data stream and then from the data streams to the SQL, which is the Kinesis data analytics tool. From Kinesis analytics, we again establish a connection to the data Firehose and then drive data back into the S3. These are the main use cases that we have for working on Amazon Kinesis.
How has it helped my organization?
In my client's company, there is one live database that comes into the DynamoDB. They want to replicate that in Amazon S3 for their data analytics and they do not want data to be refreshed every second. They want their data to be refreshed at a particular size, like five MBs. Kinesis provides data for that. That's the main improvement that we give to the client.
What is most valuable?
The features that I have found most valuable depend on the use case. I find data Firehose and data streams are much more intelligent than other streaming solutions.
There is a time provision as well as data size. Let's suppose you want to store data within 60 seconds, you can. Let's suppose you want to store data up to a certain size, you can, too. And then you can it write back to the S3. That's the beauty of the solution.
What needs improvement?
Kinesis Data Analytics needs to be improved somewhat. It's SQL based data but it is not as user friendly as MySQL or Athena tools. That's the one improvement that I'm expecting from Amazon. Apart from that everything is fine.
For how long have I used the solution?
I have two years of project experience on AWS, and around six months with Kinesis.
What do I think about the stability of the solution?
I am satisfied with Amazon Kinesis. It is pretty exiting to work on.
What do I think about the scalability of the solution?
Its scalability is very high. There is no maintenance and there is no throughput latency. I think data scalability is high, too. You can ingest gigabytes of data within seconds or milliseconds.
We are a team of five members using Amazon Kinesis. Two are working onshore and three of us are working offshore.
We are all developers implementing, developing, and designing the data pipeline, as well. The thing is we work in a startup company so we have to do all the things from our end on this.
How are customer service and technical support?
As of now we have not had any contact with customer support because we didn't face any complex types of problems while we were implementing our use cases.
How was the initial setup?
The initial setup is very straightforward. It is very well documented and anyone with simple knowledge or common sense can do it.
Implementing is very simple. You can just do it with your fingertips. There might be some improvements that can be made according to the requirements. For that, we do versioning. First we establish the pipeline from the data stream to the S3. That's very easy. You can do it within hours or within minutes. I can say the process is very simple and it's not as complex as it looks.
One more beauty is that Kinesis data Firehose will directly write to S3 in a partitioned way. Based on the timestamp it can directly write in the year, month, day and hour. That's the good thing I found about Amazon Kinesis.
We follow an implementation. We do the deployment directly on Dev. Once we get our results and our processes, and go through Q&A, we implement it directly throughout.
What was our ROI?
Our clients definitely see a return on their investment with Amazon Kinesis.
What's my experience with pricing, setup cost, and licensing?
The pricing depends on the number of shards that we are providing and the time the application is running.
We reduced the cost of the pipeline that we built. We built a generic type of pipeline so that two more times can use same data pipeline.
What other advice do I have?
My advice to anyone thinking about Amazon Kinesis, is that if they have ClickStream or any streaming data which varies from megabytes to gigabytes, they can definitely go for Amazon Kinesis. If they want to do data processing, or batch or streaming analytics, they can choose Amazon Kinesis. And if you want to enable database stream events in Amazon DynamoDB, then you can definitely go for Amazon Kinesis. I don't see any better option for these other than Amazon Kinesis. You can use Amazon Kinesis Data Analytics Tool to detect an anomaly before you process the data. That's one more beauty. The first things we need to determine are the source and the throughput of the data and the latency you want.
On a scale of one to ten I would rate Amazon Kinesis a nine.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-10-28T00:00:00-04:00
Rated 5 out of
5 by
Pablo Giner from
The ability to have one single flow of inputting data from multiple consumers simplified our architecture
What is our primary use case?
In terms of use cases, it depends of which component we're talking about, as we use three of the 4 components. The only one we don't use is the Video Streams.
Kinesis Data Stream is the module that we have been using the longest, essentially we use it to hold data which will be processed by multiple consumers. We have multiple data sources and we use Kinesis to funnel that data which is then consumed by multiple other consumers. We gather data coming from IoT devices, user phones, databases and a variety of other sources and then, as we have multiple consumers, we use Kinesis to actually gather the data and then we process it directly in Lambda, in Firehose, or in other applications.
How has it helped my organization?
Amazon Kinesis has absolutely improved our organization. Before Data Streams, we were using a couple of other solutions, including Talend and Pentaho, to move data around. Each of them were their own silos. So the ability to have one single flow of data from multiple consumers simplified our architecture a lot because you didn't need to copy or read the data multiple times, you just pull that data and then use multiple consumers. It actually simplified our architecture. It will also help us in the future when we have to build additional applications based on the same input data. We already have that data available. It will just be a matter of building the application itself. So it saves us a lot of time.
For Firehose, we perceive time-savings as a result of its incorporation. It takes you a couple of minutes to configure and it saves quite a lot of time in trying to get our information into the data lake.
Regarding Kinesis Analytics, we have real-time alarms and real-time data flows to populate other systems. For example, we populate Salesforce using a tumbling window implemented with Kinesis Data Analytics and Lambda. We also have alarms for things like knowing when someone is affecting our assets and we need to warn the operators in real-time. So Kinesis Analytics has actually given us the ability to track things in real-time that before we didn't have the ability to track.
Because we couldn't do that in the database we needed a component that had the ability to get the last window of data super quickly and if something was wrong, to notify and identify the failing record or the information that we wanted to trigger and with Lambda to notify the user. At certain points, when we had operational issues, we implemented alarms that have the key indicators to help us attack those issues before they grew and it was too late to attack them. So that has been essential for us.
What is most valuable?
I think that all Kinesis components have their own features and their own value. Starting from Data Streams, you have to have it as the data queue or else you would need to go to Kafka or another message broker (with higher implementation effort if your ecosystem is fully hosted in AWS already). I think that the solution they have put together in Kinesis is fairly easy to use. It is definitely a core component in any data architecture.
On the other hand, I find Firehose super simple and super useful for certain use cases. I wouldn't say it is as essential as Data Streams, but it is very handy if you want to just dump data. The connection between Data Streams and Firehose allows you to do that without worrying too much about performance and configuration. I find Firehose super simple to use for a very specific use case, but that use case is very common.
Kinesis Analytics is definitely more cutting edge. Out of Kinesis this is the most innovative part. We have used it for some alarms and for some batch processing in time windows. If we are talking about massive amounts of data, then you need to move to other solutions such as EMR or Glue for big data. If the amount of data is manageable and you want something to analyze on the fly, Kinesis Analytics is very appropriate and it gives you the ability to interact via SQL. So it makes your life easier if you want to develop a relatively self-contained application to do analytics on the fly.
I would say that Data Streams, in a matter of weeks, created a massive time-saving. Something that we haven't factored in is cost savings because we don't need to repeat the same data flow multiple times since each of those data flows are actually cost associated. We're talking about a couple of $100's per month, which is significant. In terms of time-savings here, we are in the scale of weeks.
What needs improvement?
In terms of what can be improved, I would say that within Data Streams, you have a variety of ways to interact with the data; you have the Kinesis client library, the KCL, and you have the Kinesis agent. When we were developing our architecture a couple years back, all the libraries to aggregate the data were very problematic. So the Kinesis Aggregator, which essentially improves the performance and cost by aggregating individual records into bigger one, is something that I found had a lot of room for improvement to make it a lot more refined. At the time I found a couple of limitations that I had to work around. So definitely on that side I found room for improvement.
Something else to mention is that we use Kinesis with Lambda a lot and the fact that you can only connect one Stream to one Lambda, I find is a limiting factor. I would definitely recommend to remove that constraint.
For how long have I used the solution?
I have been using Amazon Kinesis for over 2 years.
What do I think about the stability of the solution?
Kinesis is super stable. This is one of the only few components in AWS for which we have never had any issues with the stability.
What do I think about the scalability of the solution?
Regarding scalability, you wouldn't use Kinesis Analytics for huge, vast amounts of data or for complex processing. It's for relatively simple processing with not too much data. So I wouldn't say that it is infinitely scalable, it really depends on your application and the volume of data.
Right now I don't see us using more of Kinesis. It has a very clear role in our architecture and satisfies that perfectly well. This is one of the initial components that you build. In a roadmap that would be the first 10%. All our work is spent in different actions right now, but we don't have any plans to grow Kinesis further. We used to do some specific real-time analysis with Kinesis Analytics on a case by case basis.It's more on a per need basis.
In other companies we use Kafka, but we didn't replace it with Kinesis.
How was the initial setup?
The initial setup is relatively straight forward.
In terms of the initial setup of Kinesis Streams, is no big deal, you just choose the number of streams and assign a name to your application and that's pretty much it. The effort is in the applications that talk to Kinesis. I would say implementation took around six weeks. Deployment just took two people.
We have our own internal strategy which we started from scratch. So obviously we knew which components we would be deploying first. At the time we didn't use either CloudFormation or CodeBuild. So when we started, we didn't have these tools which we now use all the time for managing the architecture and CICD. But we didn't have it in the initial deployment.
What was our ROI?
Amazon Kinesis has improved our ROI. We obviously pay monthly for Kinesis but for us it is an enabler. We wouldn't have an architecture, or we'd have a terrible architecture, if Kinesis wasn't there.
For the data analytics component, we definitely saw that our ROI clearly improved. The alarms are something that we have actually implemented in very critical tasks when we had a company issue and that we have given visibility and a prompt response to the issues thanks to Kinesis Analytics. So that has definitely proven its ROI.
What's my experience with pricing, setup cost, and licensing?
In terms of the prices, I think it is a fair price. Kinesis Data Stream has a very fair price relative to the value that it provides. Same for Firehose. As for Kinesis Analytics, I find it on the more expensive side because it's a newer component, something fewer people use, and something more innovative, cutting edge, and more specific. I would say Analytics is more on the expensive side of the spectrum. I would say that Kinesis Analytics is the only one that I may complain about if you like low pricing.
Which other solutions did I evaluate?
Kafka is comparable to Data Streams, not to Kinesis Analytics. For Analytics on the fly, I can talk about doing Spark streaming, which is a lot more complex and you need to spend a lot more time setting it up, but it also has more capability in terms of the scaling, so I wouldn't say it's a one-to-one comparison.
I also used StreamSets in the past, where you can gather data and you can also do some transformations on the fly. But again it's not comparable one-to-one so I wouldn't use it for the same use cases.
What other advice do I have?
My recommendation for Data Streams is to do a deep dive into the documentation before implementing to avoid what we did at the beginning. You try to process record by record or push record by record into Kinesis and then realize that it is not cost effective or even efficient. So you need to know that you need to aggregate your data before you push it into Kinesis. So documenting yourself about the best practices in using Kinesis is definitely something I would recomm...
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-10-26T00:00:00-04:00
Rated 5 out of
5 by
Matt Newman from
A great managed service that's simple and easy to maintain
What is our primary use case?
Our primary use case of this solution is for a streaming bus architecture, we get events and they come in through Kinesis like a Jason event. It's usually a change to a database, but it can be any event such as in our application, which feeds into the Kinesis and then we have a Lambda that consumes them and then finally it puts those into a data warehouse which is the ultimate goal. So it's a near real-time data warehouse.
How has it helped my organization?
Instead of doing batch jobs of an ETL, like moving data into a data warehouse, we're now able to do it all through a continuous stream in Kinesis. It means that the data is more up-to-date in our data warehouse, it's more real time.
What is most valuable?
I've used Kafka in the past and Kinesis is a lot simpler. It's all hosted, it's nice it's really good. There aren't too many knobs and things to turn and ways to screw up. It's a pretty simple product and a lot easier to manage because it's hosted by AWS and it accomplishes what we need it to. The other nice thing is that we can make it available to external customers if they want to get a Kinesis feed of our things.
What needs improvement?
I would say that the solution probably has the capability to do sharding so that you can do a lot of things in parallel. I think that the way the sharding works could be simplified and include features that make it easier to scale in a parallel way.
For how long have I used the solution?
I've been using this solution for five years.
What do I think about the stability of the solution?
It's much easier to maintain than a Kafka cluster, but there were some nuances with it. I'd say maybe once a month or once every couple of months we'd get some weird things like AWS getting overwhelmed or the service was degraded for a few hours in a day. I guess that's a drawback of going with the fully managed service, it's just that it depends on the company keeping everything up. To me it's pretty stable, we would just get kind of slow once every month or once every couple of months. Overall I would say it's not perfect, but it was totally great, acceptable.
What do I think about the scalability of the solution?
You can do a bunch of shards, we only use one or a few shards and you can scale way up. It's way more scalable than we ever needed. And we were doing massive, millions of updates per minute. This is a backend service, so it's mostly used by developers and data engineers that were using Kinesis in our company, but our customers all benefited. Customers were sending send data through it, but they don't know that they are using Kinesis. We built the system and then they use it, so they don't know that under the covers it was consistent. We have five to 10 internal employees using it, but probably around 5,000 customers. Where I worked before, we used it basically everywhere. Probably 70% of all of our data was falling through Kinesis. Where I am now, we're just starting, so it's not yet being widely used. We plan to increase usage.
How are customer service and technical support?
When we had some of those slow downs, we used AWS support and I can't say that we had a great experience and they resolved the issues, but they looked into some of the flow downs and ultimately we just decided there was nothing we could do. It left me feeling there was something lacking on the technical support side. They didn't get to the bottom of all my issues, but the issues weren't bad enough to be unhappy about the product overall.
Which solution did I use previously and why did I switch?
We previously used Apache Kafka. We switched because we were already using Amazon for everything else so it made sense, and it was a nice managed solution that would be a lot easier to deal with. It also integrated well with Lambdas. The whole AWS ecosystem is nice to work with because everything integrates with each other.
How was the initial setup?
The initial setup is definitely straightforward because it's a managed service and you only get a few options when you set up a Kinesis stream. It's a lot less overwhelming than setting up a whole Kafka cluster or even if you've managed Kafka, there's still a lot of configuration required to get it up and running and all the choices you can make about topics and things. Kinesis is just much simpler. It only lets you configure what you need to configure. I'd say that kind of POC took about a week and then real production probably a month. We used Terraform for our implementation strategy, but I used CloudFormation in my past job to do that. The deployment was essentially running the CloudFormation template.
What was our ROI?
Compared to what we were doing with Kinesis or with Kafka, which was taking about 30% just to keep things together, with Kinesis I think we're probably saving tens of thousands, if not $100,000 per year.
What's my experience with pricing, setup cost, and licensing?
I would say pricing is really great. If pricing is an issue, I'd definitely recommend Kinesis because our Kinesis costs are under $1,000 a month. The product is super cost effective and it's the same with the licensing. Compared to Google Cloud and Azure, they're probably pretty similarly priced. I wouldn't say you're going to get a huge benefit going to Kinesis, but if you're considering using Kafka or another solution that's not hosted, it's not really worth all the effort when you could just go with a managed solution. It's a lot better cost-wise.
Which other solutions did I evaluate?
We took a look at Google cloud and Azure, they're Pub/Sub solutions, but not really in depth. Because we were already using Amazon, it just didn't make sense to use any other cloud provider.
What other advice do I have?
It's nice to deploy this with the Amazon goodness of Cloud Formation and Terraform, to have it all deployed in a repeatable way. I know that it's easy to go into the console and do it manually, but it's best to do infrastructure as code, in particular with Kinesis.
I would rate this solution a nine out of 10.
Which deployment model are you using for this solution?
Public Cloud
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-10-18T00:00:00-05:00
Rated 5 out of
5 by
Ankeet Chauhan from
Good for data streaming and can be easily implemented
What is our primary use case?
I use the solution in my company for streaming purposes, considering that my company has an AI-based camera for streaming.
What is most valuable?
The most valuable features of the solution are data streaming and the real-time data of our screen, which helps provide the analytics our company needs.
What needs improvement?
There are certain shortcomings in the machine learning capacity offered by the product, making it an area where improvements are required. There is a need to introduce something more into the machine learning area because it helps users learn and get newer things in their day-to-day lives. I think Amazon Kinesis should update machine learning and be up to the mark.
For how long have I used the solution?
I have been using Amazon Kinesis for a year. I am a customer of the product.
What do I think about the stability of the solution?
Stability-wise, I rate the solution a nine out of ten.
What do I think about the scalability of the solution?
Scalability-wise, I rate the solution a ten out of ten.
There are around three users of the product in my company.
How are customer service and support?
The solution's technical support is flawless. I rate the technical support a ten out of ten.
How would you rate customer service and support?
Positive
How was the initial setup?
One of the plus points of the product that makes it so much better is that it is very easy to set up, especially since it is managed by AWS, making it a smooth process.
What's my experience with pricing, setup cost, and licensing?
The product falls on a bit of an expensive side.
What other advice do I have?
There is a separate team in my company that looks after the real-time data analytics in our organization, which I don't know much about. As a DevOps engineer, I take care of the cloud, and because of this, I know why my company uses the product.
The most valuable feature of the product for our company's data processing needs stems from the fact that it operates in real-time. The simplicity of the services offered by the product and the way I can use them is very smooth and easy to understand. The tool also provides good storage and an increase in on-demand capacity, which I think is the best for our company.
Integrating Amazon Kinesis with other AWS services has smoothly helped our analytics workflow. It is also very easy to integrate with other AWS services. The analytics part is also very good with Amazon Kinesis.
I recommend the product to those who plan to use it. Those who are new to data streaming and want to start with a new product can go for Amazon Kinesis, as it is very easy to set up, especially the installation part, which is very easy to handle. Amazon Kinesis is the first option others should consider since it is easy to set it up.
I rate the tool an eight out of ten.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2024-04-02T00:00:00-04:00
Rated 5 out of
5 by
Agustin Calderon from
A cost-effectively processes and analyzes streaming data at any scale as a fully managed service
What is our primary use case?
To recover data and send it to the cloud. A few of our clients have Amazon Web Services and we use Kinesis to deploy the data to their mobiles and to their data processing system. Also to do data analytics.
What is most valuable?
The management and analytics are valuable features.
What needs improvement?
A snapshot from the stream of the data analytics I already have on the cloud. do a snapshot to stop the process and restart a few weeks later when I have more data or more availability of the client teams.
For how long have I used the solution?
I have been using Amazon Kinesis for two years.
What do I think about the stability of the solution?
The stability is a ten out of ten.
How was the initial setup?
We use cloud automation for deployment. We deploy the tags in minutes, and we can also use confirmation to test each part and test end-to-end use cases before we deploy them to the client. So we do everything with cloud automation, and it takes a few minutes to deploy a production environment.
What's my experience with pricing, setup cost, and licensing?
The solution is cheap.
What other advice do I have?
Overall, I rate the solution an eight out of ten.
Which deployment model are you using for this solution?
Public Cloud
Disclaimer: My company has a business relationship with this vendor other than being a customer:Partner
Date published: 2024-02-26T00:00:00-05:00
Rated 5 out of
5 by
T O from
Data is available when the solution is down, but the timeframe of retention support is too short
What is our primary use case?
We are using Kinesis' third-party streaming engine. We are using the AWS cloud and are moving to Azure.
What is most valuable?
One of the best features of Amazon Kinesis is the multi-partition.
Another valuable feature of Kinesis is that when it is down, and in the backup stage, the data is still available.
What needs improvement?
Currently, Kinesis provides only seven days of retention support. It would be beneficial if this could be extended to upwards of 40 days or more.
In the next future release, I would like to see a library that is Java-compliant. It would be beneficial if Amazon Kinesis provided document-based support on the internet to be able to read the data from the Kinesis site.
For how long have I used the solution?
I have been using Amazon Kinesis for almost two years.
What do I think about the stability of the solution?
Amazon Kinesis is stable. I do not see any issues.
What do I think about the scalability of the solution?
The solution is scalable. We have 20 team members using Kinesis.
How are customer service and support?
We have not required support from Amazon.
How was the initial setup?
The initial setup of Amazon Kinesis is easy.
Which other solutions did I evaluate?
We have been looking for a streaming tool. We looked into Kafka, E-Hub, and Kinesis. Kafta is better than Kinesis as it has multiple cloud connectors. More features are available by default with Kafta.
What other advice do I have?
Overall, I would rate Amazon Kinesis a seven out of ten.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Other
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2022-12-07T00:00:00-05:00
Rated 5 out of
5 by
Muhammad Qasim Yasin from
Effective for small businesses, easy to use, and has excellent reporting, but only supports limited file size, batch size, and throughput
What is our primary use case?
We collect data from AWS IoT Core and then capture the stream in Amazon Kinesis. The data is then stored in S3 and shifted to Snowflake for analysis.
What is most valuable?
What I like about Amazon Kinesis is that it's very effective for small businesses. It's a well-managed solution with excellent reporting. Amazon Kinesis is also easy to use, and even a novice developer can work with it, versus Apache Kafka, which requires expertise.
What needs improvement?
My company found some Amazon Kinesis discrepancies, so it's looking forward to a more modernized solution from Apache Kafka.
One area for improvement in the solution is the file size limitation of 10 Mb. My company works with files with a larger file size.
The batch size and throughput also need improvement in Amazon Kinesis. The solution needs to be more open regarding the type of files for streaming and the streaming size. Amazon should not limit those aspects. It should be unlimited. If a company is ready to pay, why not make it unlimited?
What I want to add to Amazon Kinesis is modernization based on the container environment, where I can add containers and more workers. I also expect some human resources to be added and an SLA agreement with Amazon, if possible.
For how long have I used the solution?
I've been using Amazon Kinesis for about one year, and I'm still using it.
What do I think about the stability of the solution?
Amazon Kinesis could be more stable. One of my clients rejected it, while some clients find it okay, stability-wise. I'd rate Amazon Kinesis stability as five out of ten.
What do I think about the scalability of the solution?
I can rate Amazon Kinesis scalability according to the organization size and data load. For a small organization using the solution and Lambda with some transformation through AWS Glue, Amazon Kinesis is the best, scalability-wise. However, if you're dealing with a billion tuples, for example, the solution isn't as scalable, so I would go for Apache Spark or Apache Kafka to handle the load.
When I see that the processing takes longer than fifteen minutes with Lambda and the tenants fail, I use Apache Spark for processing, but that could take up to three or four days to be comparable to big data technologies.
I'd rate the scalability of Amazon Kinesis as four out of ten.
How are customer service and support?
My company contacted some premium partners and technicians of Amazon Kinesis and found the technical support good, but with some limitations. I'd rate support a seven out of ten. Though it had limitations, the interaction with support was pleasant.
How would you rate customer service and support?
Neutral
Which solution did I use previously and why did I switch?
In the future, my company plans to switch to Apache Kafka because it's very flexible and easier to manage. It's also easier to control and manage limits about topics. On the other hand, Amazon Kinesis has some limitations to its charts. It also has a 10 Mb limit to its file size, so if you have a 20 Mb file, you have to make it 10 Mb.
How was the initial setup?
Amazon Kinesis is easy to set up, and it's a ten out of ten for me. Setting it up is a straightforward process.
What about the implementation team?
My company set up Amazon Kinesis for the client.
What's my experience with pricing, setup cost, and licensing?
If you ask a client about Amazon Kinesis pricing, the client usually says it's high. If you ask a business owner, the business owner would tell you that pricing for Amazon Kinesis is a little bit high. For each region, it's a little bit high.
There is a particular concern regarding Amazon Kinesis here in Pakistan because there's no zone in Pakistan. Amazon needs to develop zones here because Pakistan is the biggest country in the region after India. Amazon is losing a lot of business in Pakistan because there's no AWS zone here.
AWS also didn't accept my Pakistan credit card when I was trying to register with AWS. AWS should develop trust here in Pakistan and excellent AWS zones, so Pakistan businesses that want to purchase Amazon Kinesis won't need to depend on Singapore or India.
When I'm closing a deal with a new client, the client would ask, "Why do you need to sign up with a zone in India or Singapore to save data?" I don't have an answer to that question, so a workaround would be to develop on-premise environments for clients to save data.
Amazon Kinesis pricing is sometimes reasonable and sometimes could be better, depending on the planning, so it's a five out of ten for me.
What other advice do I have?
Nowadays, my company works with AWS, Snowflake, Redshift, Amazon Kinesis, Firehose, Aurora, and Athena. In the future, my company plans to work with SAP HANA.
My rating for Amazon Kinesis is six out of ten.
My company is a user of Amazon Kinesis.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2022-12-26T00:00:00-05:00
Rated 5 out of
5 by
Sunil Morya from
The solution is easy to deploy, scalable, and stable
What is our primary use case?
We have utilized the solution for ingesting the data from different applications. For example, when people use a web server they send their weblogs, and clickstreams and the service provider wants to know how many users are currently using the site and what their areas of interest are, we use Amazon Kinesis which has the capability to enable the analytics and provide the information to them.
We also use Amazon Kinesis Data Firehose for putting the IoT data on the Kinesis Data Streams because the data has to be brought from on-prem to the cloud in order to perform the analysis.
How has it helped my organization?
The solution has improved our organization by saving time. Before launching the solution, the only thing we have to do is know the volume of the data and what is the frequency of the data we are going to receive. Based on that, we can configure the capacity of the Kinesis Data Streams, so it divides our requirements in terms of the shards or partitions. Unlike Kafka which works in partitions, Amazon Kinesis works in shards. This means the solution can handle thousands of requests per second for writing and reading. The writing capacity is one KB per request and the reading capacity is four KB per request.
What is most valuable?
The solution has the capacity to store the data anywhere from one day to a week and provides limitless storage for us. The data is time-stamped, and the data is in sequence, so we don't need to maintain the sequence or order of the data, and multiple consumers can reference the data in the Kinesis Data Streams simultaneously. Using the same data one application can perform Task A, and another application can perform Task B.
The Kinesis Data Stream is integrated with Amazon CloudWatch where we can monitor all the requests of who wants to read information, what kind of APIs are being requested, what errors there are, who is producing them, and who are the data consumers. All the information is logged in one spot and this allows us to identify the problematic points easily.
The solution allows us to apply the security so the consumers can have access, based on the subscription they have.
Amazon Kinesis has a fan-out feature that allows us to increase the throughput when the number of consumers increases, instead of having to pull the data from the consumer side the information is posted by the solution itself.
What needs improvement?
The services which are described in the documentation could use some visual presentation because for someone who is new to the solution the documentation is not easy to follow or beginner friendly and can leave a person feeling helpless.
For how long have I used the solution?
I have been using the solution for five years.
What do I think about the stability of the solution?
The stability is good as long as no configuration mistakes are made and the data source works properly.
What do I think about the scalability of the solution?
The scalability is good but if for example, we only created one stream, then wanted to scale it further, we would have to continuously monitor using CloudWatch to ensure the shards are not getting overloaded, otherwise, we have to split the shards. We would have to merge the shards if it is under load. The implementation is only required once with some monitoring and after that, it is very easy to scale. The scalability has some limitations but there is also an on-demand option with a script so when we need to scale up the capacity we can and if we need to decrease the capacity, we can. We can also stop the service completely by scheduling it based on event triggers.
How are customer service and support?
The technical support was not able to resolve the one issue we had.
How would you rate customer service and support?
Positive
How was the initial setup?
The initial setup is straightforward and I give it an eight out of ten. The deployment takes around five minutes.
We have to create our service using CLI Streams, or we can create using the console because we have to provide the configuration, including, what throughput we want, and how many reads and writes we want to be supported. Once we launch, we'll be charged monthly. Setup is very easy, we have to specify whether we want encryption of the data or not if it is accessible to everyone or not, and what kind of services are going to interact with the solution. In AWS, we have to specify, and then we have to provide the roles and policies based on the consumers.
What about the implementation team?
The implementation is completed in-house.
What was our ROI?
The solution is worth the money because it is easier to set up, and since it is integrated with all the AWS services, we can manage the security, the monitoring requirements, and manage the audit information, much easier.
What's my experience with pricing, setup cost, and licensing?
The solution is costly. There is a fee to activate the service and even if it is not being used there is a monthly fee because they continue to maintain the service. If we want to retain the data for longer durations then we are charged the equivalent of Amazon S2 or S3 services. The fee is based on the number of hours the service is running.
What other advice do I have?
I give the solution a nine out of ten.
No maintenance is required for the solution because it is cloud-based.
The solution is available in all three zones, Amazon Kinesis is a good solution if a person wants scalability, availability, and durability of data. I recommend the solution to anyone already using AWS Amazon-managed Kafka service.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: My company has a business relationship with this vendor other than being a customer:Partner
Date published: 2022-12-11T00:00:00-05:00
Rated 5 out of
5 by
Paul Hoc from
Easy to use, easy to configure, and stable
What is our primary use case?
We use the solution for streaming data, in simpler terms. For example, there is a backend application; we need to make that data available for analysis. On the backend side, we don't store the history. We get all the events regarding changes incrementally. If something changes, an event is generated. This is a convenient way to keep track of all the changes.
What is most valuable?
Amazon Kinesis is similar to Kafka, another type of streaming technology, which can be referred to as a queue service to exchange data. Setting Amazon Kinesis up is quick and easy; it only takes a few minutes to configure the necessary settings and start using it. In comparison, Kafka requires setting up a cluster, even if it is available in the cloud, which can be time-consuming. Amazon Kinesis has a user-friendly interface, making it easy to adjust and scale up the number of shards if needed. The cloud is especially useful when starting something new and not needing a lot of resources initially, but with the potential to upgrade later when there is a larger load. Although there is a cost associated with using the cloud, Amazon Kinesis is very flexible and can be easily adjusted when necessary, making it a great advantage.
What needs improvement?
Kinesis can be expensive, especially when dealing with large volumes of data.
For how long have I used the solution?
I have been using the solution for two years.
What do I think about the stability of the solution?
The solution is stable and I don't recall any issues. Once we set the solution up, it usually works and we only investigate if we encounter a problem. However, if there is a large number of events to process, due to limited capacity for example with the shards, then some events may be delayed. This can be easily resolved by adjusting the configuration to provide more capacity.
What do I think about the scalability of the solution?
The solution is scalable, but this also comes with a financial cost. If we want to increase throughput, we can simply increase the number of shards or adjust some config parameters, which can be done in a matter of minutes if we know how to do it. We can scale the solution almost without limitation.
How was the initial setup?
There are a lot of details involved with the initial setup, so if we need something at the outset, we can set up the solution easily. However, the details are important since they are related to how much money we pay and we need to tailor the solution to our needs. If we want to do something more sophisticated, then we need to spend more time comprehending all the details. Initially, we can easily set something up, but eventually, we need to understand it better and adjust it more to our needs.
What's my experience with pricing, setup cost, and licensing?
Cloud services are often cheaper in the beginning, but when the amount of data and needed resources grows, they cost more and more. In my opinion, it is sometimes simpler to use an existing service rather than having to maintain our own internal infrastructure. This way, we can focus on the things we are good at and can make money from, rather than having to employ people to support the infrastructure. In general, cloud services are very convenient to use, even if we have to pay a bit more, as we know what we are paying for and can focus on other tasks. However, if the scale is large, I would consider making changes depending on the situation.
What other advice do I have?
I give the solution a nine out of ten. Amazon Kinesis is easy to use and configure, especially in the beginning. The solution is stable and I have not encountered any issues with it, nor am I aware of any. The solution is effective.
I don't see any missing features in Amazon Kinesis. I haven't spent a lot of time with this interface, as I have only configured it once. If any changes need to be made, I simply adjust Amazon Kinesis and it works. I only go into Amazon Kinesis if there is a need for a new data stream to be included or if the throughput needs to be increased. This doesn't happen very often.
Depending on the requirements, if there is a need to stream data and access it in real time, then I would consider Amazon Kinesis. However, if there is no need for real-time data access, then I will look for some other cheaper options. Companies such as Redshift, Snowflake, and BigQuery are developing databases with built-in streaming functionality. Depending on the case, this may be an option to consider. It also depends on the target; sometimes it is better to use the mechanisms available in the target tool. If we want to have the data on a stream or some hot stories, then I would consider Amazon Kinesis in that case.
Which deployment model are you using for this solution?
Public Cloud
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2023-03-05T00:00:00-05:00
Rated 5 out of
5 by
DouglasStein from
Integrates with Lambda functions, can process a very large amount of data, and comes with good support
What is our primary use case?
We had real-time streaming of data and a very large volume of user activity. We applied machine learning to the data streams. So, Kinesis basically made sure that we got the data, and we didn't lose the data.
How has it helped my organization?
I'm not using it at my current organization. I used it at the last company. Kinesis replaced a whole tier of servers. So, we didn't need to have a server to catch the data and then send the data somewhere else. Kinesis was the input port for very large amounts of data.
What is most valuable?
What turns out to be most valuable is its integration with Lambda functions because you can process the data as it comes in. As soon as data comes, you'll fire a Lambda function to process a trench of data.
What needs improvement?
One thing that would be nice would be a policy for increasing the number of Kinesis streams because that's the one thing that's constant. You can change it in real time, but somebody has to change it, or you have to set some kind of meter. So, auto-scaling of adding and removing streams would be nice.
I'd like to see the size of a Kinesis message go to at least one megabyte per message. That would be nice, but that's an extreme case.
For how long have I used the solution?
I have been using it since it came out in 2014.
What do I think about the stability of the solution?
It is stable.
What do I think about the scalability of the solution?
It is scalable. In the previous company, it was customer-facing. So, there were hundreds of thousands of users. There were very large data volumes.
In this company, we'll probably use Kinesis. We haven't used it yet. We have some more projects that'll come along, and then we'll need it.
How are customer service and support?
I was there when it was beta. They were pretty good then, and they're still pretty good. So, they were five out of five. They were right on top of it.
How would you rate customer service and support?
Positive
Which solution did I use previously and why did I switch?
We didn't use any other solution for streaming data.
How was the initial setup?
It was pretty straightforward. I had one person who did all DevOps. It did not need a dedicated person.
What was our ROI?
We had seen an ROI. We wouldn't have been able to build the product without it.
What's my experience with pricing, setup cost, and licensing?
It was actually a fairly high volume we were spending. We were spending about 150 a month.
Which other solutions did I evaluate?
There were some that we considered, but it was a little cruder.
What other advice do I have?
To someone who would like to implement it, I would simply tell not to shove giant bricks in. Data has to be reasonably sized. A single Kinesis message is measured in K, not megabytes. It's not meant for gigantic things. There is a different strategy for streaming data.
I'd rate it at least a nine out of ten. It was very close to perfect.
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2022-10-22T00:00:00-04:00
Rated 5 out of
5 by
Jesus Martin from
Fast solution that saves us a lot of time
What is our primary use case?
I work as a senior software engineer in eCommerce analytics company, we have to process a huge amount of data.
Only a few people within our organization use Kinesis. My team, which includes three backend developers, simply wanted to test out different approaches.
We are now in the middle of migrating our existing databases in MySQL and Postgres, to Snowflake. We use Kinesis Firehose to ingest data in Snowflake at the same time that we ingest data in MySQL, without it impacting any performance.
If you ingest two databases in a synchronous way, then the performance is very slow. We wanted to avoid that so we came up with this solution to ingest the data in the stream.
We use Kinesis Firehose to send the data to the stream, which then buffers the data for roughly two minutes. Afterwards, it places the files in an S3 bucket, which is then loaded automatically, via an integration with Snowflake that's called Snowpipe. Snowpipe reads and ingests every message and every file that's in the S3 bucket. This stage doesn't bother us because we don't need to wait for it. We just stream the data — fire and forget. Sometimes, if the record is not ingested successfully, we have to retry. Apart from that, it's great because we don't need to wait and the performance is great.
There are some caveats there, but overall, the performance and the reality of it all has been great. This year, 100% of the time when there was an issue in production, it was due to a bug in our code rather than a bug in Kinesis.
How has it helped my organization?
We save a lot of time with Kinesis, but it's difficult to measure just how much. We actually have something similar regarding some other processes. We have developed somewhere else a tool that takes note of the contents of the stream, places them into a file, manually uploads them to the S3, and copies the files into Snowflake. That could be done with Kinesis, but it could take two weeks or 1 month less to get it production-ready.
What is most valuable?
The first would be the one found in the AWS SDK using the asynchronous client: put Record batch function which allows you to put a list of records in one put record request, which saves time and it's more efficient. Also, by using the asynchronous client, the records are sent in the background using an internal thread pool that can be configurable for your needs. In our performance testing, we came across this setting was the fastest solution. It didn't impact anything in the performance of the system process.
The second one would be the ability to link the stream to other places other than S3 via configuration of the stream and without changing a line of code.
Lastly, you can also link a lambda function to the stream to transform the data as it arrives in before writing it in S3, which is great to perform some aggregations or enrich the data with other data sources.
What needs improvement?
The default limit that they have, which at the moment is 5,000 records per second (I'm talking about Kinesis Firehose which is a specialized form of the Amazon Kinesis service) seems too low. Actually, on the first week that we deployed it into production, we had to roll it back and ask Amazon to increase the default limits.
It's mentioned in the documentation, but I think the default settings are far too low. The first week it was extremely slow because the records were not properly ingested in the stream, so we had to try it again. This happened the first week that we deployed it into production, but after talking with Amazon, they increased their throttling limits up to 10,000 records. Now it works fine.
For how long have I used the solution?
We've been using this solution since September 2019.
What do I think about the stability of the solution?
The stability is great. I'd say that maybe we have it running 99% of the time, and nothing stops it.
What do I think about the scalability of the solution?
Amazon Kinesis is definitely scalable. We have huge spikes of data that get processed around midnight and Kinesis handles it fine.
It automatically scales up and down, We don't need to compute it for that. It's great.
How are customer service and technical support?
The only time that we needed to contact Amazon was to ask them to increase the throttling limit. They replied to us very quickly and did what we asked.
Which solution did I use previously and why did I switch?
Initially, we were evaluating Kafka. I think Kafka is faster, but it's less reliable in terms of maintenance; however, when Kafka works, and you have it properly configured, it's much better than Kinesis, to be honest.
On the other hand, Kinesis provides us with better maintenance. Our DevOps team is already oversaturated, so we didn't want to increase the maintenance cost of the production environment. That's why we decided to go with Kinesis; because performance-wise, it's easy to configure and maintain.
How was the initial setup?
I found this solution to be really easy to configure. The essential parts of the configuration include naming the stream and also configuring the buffering time that it takes for a record to get ingested into S3 (how long it will be in the stream until it's put into an S3). You also need to link the Amazon S3 buckets with the Amazon Kinesis stream. After you've completed these configurations, it's pretty much production-ready. It's very, very easy. That's a huge advantage of using this service.
What about the implementation team?
Deployment took a few minutes.
You don't need a deployment plan or an implementation strategy because once you configure it, you can just use a stream. It's not an obligatory version that needs a library, etc. This stream is completely abstract in that way. You only need to configure it once, that's it.
What was our ROI?
We have seen a return on our investment with Amazon Kinesis. We are able to process data without any issue. It's our solution for ingesting data in other databases, such as Snowflake.
Which other solutions did I evaluate?
Developing the stream process manually or using Kafka
What other advice do I have?
If you want to use a stream solution you need to evaluate your needs. If your needs are really performance-based, maybe you should go with Kafka, but for near, real-time performance, I would recommend Amazon Kinesis.
If you need more than one destination for the data that you are ingesting in the stream, you will need to use Amazon Kinesis Data Streams rather than Firehose. If you only want to integrate from one point to another, then Kinesis Firehose is a considerably cheaper option and is much easier to configure.
From using Kinesis, I have learned a lot about the synchronous way of processing data. We always had a more sequential way of doing things.
On a scale from one to ten, I would give this solution a rating of eight.
Which deployment model are you using for this solution?
Public Cloud
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)
Disclaimer: I am a real user, and this review is based on my own experience and opinions.
Date published: 2020-10-21T00:00:00-05:00