AWS Kinesis Data Streams — a Tiny CLI Demo

Kinesis Data Streams is a part of the AWS Kinesis streaming data platform, along with Kinesis Data Firehose, Kinesis Video Streams, and Kinesis Data Analytics. For more information please checkout https://docs.aws.amazon.com/streams/latest/dev/key-concepts.html

This demo assumes that AWS CLI is installed and configured.

Step-1 Create a Single Shard Stream

aws kinesis create-stream --stream-name sree-stream --shard-count 1
aws kinesis describe-stream --stream-name sree-stream
{
"StreamDescription": {
"Shards": [
{
"ShardId": "shardId-000000000000",
"HashKeyRange": {
"StartingHashKey": "0",
"EndingHashKey": "340282366920938463463374607431768211455"
},
"SequenceNumberRange": {
"StartingSequenceNumber": "49589573094609814771801998836967108340238239686222413826"
}
}
],
"StreamARN": "arn:aws:kinesis:ap-south-1:nnn:stream/sree-stream",
"StreamName": "sree-stream",
"StreamStatus": "ACTIVE",
"RetentionPeriodHours": 24,
"EnhancedMonitoring": [
{
"ShardLevelMetrics": []
}
],
"EncryptionType": "NONE",
"KeyId": null,
"StreamCreationTimestamp": 1540631779.0
}
}
aws kinesis list-streams
{
"StreamNames": [
"sree-stream"
]
}
aws kinesis list-shards --stream-name sree-stream
{
"Shards": [
{
"ShardId": "shardId-000000000000",
"HashKeyRange": {
"StartingHashKey": "0",
"EndingHashKey": "340282366920938463463374607431768211455"
},
"SequenceNumberRange": {
"StartingSequenceNumber": "49589573094609814771801998836967108340238239686222413826"
}
}
]
}

Step-2 Add a Record to the Stream

aws kinesis put-record --stream-name sree-stream --partition-key 123 --data "Abcd1"
{
"ShardId": "shardId-000000000000",
"SequenceNumber": "49589573094609814771801998868001443055565414161004363778"
}

Step-3 Read a Record from the Stream

Get the shard iterator for our (single) shard, this represents the position of the stream in a shard to read.

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name sree-stream
{
"ShardIterator": "AAAAAAAAAAFRKm50DH4bd8qHedvJKDYXo1UjG5AEDJ6TFJ+zcM4L5qGobQqLp5d4VblQBRSO5PnEhqPtwXPqVzI4JcHikKjXGgA3EFaele0lAPW1EWiuixQY7x+P4QhLBFelyaMIpsFeWBOQiiloqEWzrKh+rtyj7+TJgjtzuI6P4acYrSd5lIwk94Sai/TMVQJxdC6jRyeroF44qDVqgf8huaWQO3el"
}

Shard iterators have a lifetime of 300 seconds.

aws kinesis get-records --shard-iterator AAAAAAAAAAEltAvJVHCej+HHgJ4ceNr7WAOMvQkaNQrvvPfJ8cj6JzRaClyYgalGoOrQXkgy4ZswqsQt6/im0osJw/atQTrTIfbCLjXDlqrhyKtESMS0EvLUkti+v0s7yEkid0Qe5Bc3zd7X8NqdNiVb33R3Tym0drjD6Y2bWrJhHbXLjDlXhS4PHqpoEuQCcjk8+KGiexWcMjJG7pm7f43HD/yajXHE
{
"Records": [
{
"SequenceNumber": "49589573094609814771801998868001443055565414161004363778",
"ApproximateArrivalTimestamp": 1540632198.342,
"Data": "QWJjZDE=",
"PartitionKey": "123"
}
],
"NextShardIterator": "AAAAAAAAAAGsxcSDkqSF6001I35sawRZbRpl64HpQk3/SpVP1T6Zm2gNXmEtpdBixCfVcj1KDmNxuqi3lSBqH5ThPNVNChKvGJQgbx7ENPvihIAq98/N699xhe9yNjTgqYQpDn6Q23w2Bgknu+NELOLrhJdCjkG2I9gjv91j6bLDyIJXfunlDls4IiBPOnFXWWCBUUvVsgtcz42xAE8YqqfgDfO312UF",
"MillisBehindLatest": 0
}

Let us decode the data.

echo "QWJjZDE="| base64 -D
Abcd1

We can also simplify the passing of the shard iterator in bash.

SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name sree-stream --query 'ShardIterator')aws kinesis get-records --shard-iterator $SHARD_ITERATOR

Step-4 Time to Clean Up

aws kinesis delete-stream --stream-name sree-stream
aws kinesis list-streams
{
"StreamNames": []
}

Thank you for your time! For more such tiny snippets please follow me. These snippets are from my collection of articles and links I have picked up for my trainings.

Sreeprakash Neelakantan

Written by

AWS Certified DevOps Engineer & Solutions Architect Professional — Docker | Kubernetes | DevOps — Trainer | Running | Swimming | Cycling

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade