Amazon Kinesis vs Amazon SQS

When creating a cloud application you may want to follow a distributed architecture, and when it comes to creating a message-based service for your application, AWS offers two solutions, the Kinesis stream and the SQS Queue. In this blog post I explain the cases and the choices we made in order to create a decoupled environment for our cloud based subscription management system.

The main difference between SQS and Kinesis is that the first is a FIFO queue, whereas the latter is a real time stream that allows
processing data posted with minimal delay.


With that been said let us examine the cases. An online web shop and an Order Management System is a case where sensitive data is transferred, for example the order information. To the problem of securing the information sent by the web shop, and ensuring that it is processed from the OMS, a decoupled queue like SQS provides a solid solution. Even in down time the order information would be stored, until requested, in SQS, and later processed by the system when it is restored, by periodically checking for new insertions in the queue.

Adding data to SQS with Node.js

After creating your SQS Queue you can follow the code below to simply add and receive a message:

Add to SQS
var AWS = require('aws-sdk');
AWS.config.loadFromPath(__dirname+'/config.json');
var sqs = new AWS.SQS();

  //adding a message to Queue
 function addToSQS(){
  return new Promise(function(resolve,reject){
    //set up the order as parameters for adding to queue
    var params = {
      DelaySeconds: 10,
      MessageBody: JSON.stringify(myData),
      QueueUrl: "the_Url_to_My_SQS_queue"
    };
    sqs.sendMessage(params, function(error, data) {
      if (error) {
        console.log("Error", error);
        reject(error)
      } else {
        resolve(data.MessageId);
      }
    });
  });
}


//receive a message
function lookUpSQS(){
  var params = {
    QueueUrl: queueURL,
    VisibilityTimeout: 0,
    WaitTimeSeconds: 0
  };

  sqs.receiveMessage(params, function(err, data) {
    if (err) {
      console.log("Receive Error", err);
      return;
    } else {
      //if empty return
      if(data.Messages==undefined){
        return;
      }
      for (i = 0; i < data.Messages.length; i++) {
      //handle your messages
      }
    }
  })
}

 

Notifying third party systems of the new incoming order would also be a responsibility of our OMS. In this case, storing in SQS and later processing and sending the information to these third party systems is not an option. Real time processing is achieved by passing data in a Kinesis stream. Amazon includes in its services the Lambda functions. A Lambda function is an event driven highly scalable code, which perfectly fits the real time nature of a stream.

Add data to Kinesis stream with Node.js

After having created the Kinesis stream and the Lambda function, configured to receive events from Kinesis, adding Data to the stream is done by pushing "Records" to it. An example of sending a single Record and handling the generated event is the following:

Insert to Kinesis
//Insert to kinesis

var AWS = require('aws-sdk');
AWS.config.loadFromPath('./config.json');
var kinesis = new AWS.Kinesis();

function addToKinesis(){
  var params = {
      Data: JSON.stringify(myData),
      PartitionKey: "my_partition_key",
      StreamName:"my_stream_name"
  }
  kinesis.putRecord(params, function(error, data){
    if(error){
      throw(new Error(error))
    }else {
      return;
    }
  });
}


//Handling the event in Lambda

exports.handler = (event, context, callback) => {

  const output = event.Records.map((record) => {
    // Kinesis data is base64 encoded so decode here
    var payload = new Buffer(record.kinesis.data, 'base64').toString('ascii');
    payload = JSON.parse(payload);

    //your payload data handling

    });
callback(null, { records: output });
};

 

Conclusion

After taking into consideration the cases I mentioned above,  SQS is an option that provides high scalability and reliability to your application. But if real time data processing is needed for your message queue, then I would suggest an event driven architecture based on Kinesis.

Say hello@snowball.digital and let’s make your product dream come true

Address line test

Facebook link. Instagram link.