Optimizing GraphQL querying using Redis and Node.js
GraphQL is a good solution for data fetching, but applying complex server side logic, or querying the database for receiving the requested content may cause performance issues. In the previous example of Optimizing and testing GraphQL querying using Node.js and Postman we used GraphQL to fetch data in the form of articles from a decoupled system. That way we avoided the delay of querying the database. There is a drawback in this implementation though. For example a web store with a lot of items stored in categories, or stored in prioritisation lists can not manage efficiently the load when storing all this information locally.
In this blog post I demonstrate our solution to this problem, by making the application scalable, using Redis. As for testing, using GraphQL we request a list of items, that are either stored locally or requested by a Redis server and compare the average response time between the two methods. The articles will be sorted in a random order, which in the case of a decoupled system, is stored in a JSON file using the id of the article, or in the case of the Redis server a sorted set, managing their order, is used.
Installing the modules
The two introduced modules for this example are redis and bluebird. Bluebird allows us to promisify all of redis functions. You can install them by running the commands bellow.
npm install redis bluebirdStoring data in Redis Server
Although storing the articles locally follows the same pattern as the previous blogpost, storing data on the Redis server is done by adding the item in a hash using the article id as the key, and then adding the reference of that item in a sorted set.
function addItem(object, listName, fileNumber, randomPosition) {
return new Promise(function(resolve, reject) {
client.hmset(
[object.id, 'id', object.id, 'title', object.title, 'body', object.body],
function(error, response) {
if (error) { console.log(error); }
client.zadd(
[listName, randomPosition, object.id],
function (err, response) {
if (err) { reject(err); }
else resolve();
}
);
}
);
});
}Declaring the GraphQL queries
For this example I introduce two GraphQL queries. The first fetches a list of articles from the local storage and the second requests them from the Redis server. In the code below notice the usage of the "batch" command that allows sending an array of commands to Redis, reducing the network delay.
fetchMultiArticles: {
type: new graphql.GraphQLList(servedObjectType),
args:{
list: {type: graphql.GraphQLString},
from: {type: graphql.GraphQLInt},
to: {type: graphql.GraphQLInt}
},
resolve: async function(_, {list, from, to}) {
var items = JSON.parse(fs.readFileSync('./' + list+".json", 'utf8'));
var results = [];
for(let i=from; i<=to; i++){
results.push(JSON.parse(fs.readFileSync(JsonFiles + '/' + items[i].id+".json", 'utf8')));
}
return results;
}
},
fetchMultiRedisArticles: {
type: new graphql.GraphQLList(servedObjectType),
args: {
list: {type: graphql.GraphQLString},
from: {type: graphql.GraphQLInt},
to: {type: graphql.GraphQLInt}
},
resolve: async function(_, {list, from, to}) {
var commandArray = [];
var response = await redis.client.zrangeAsync([list, from, to])
for(let i=0; i<response.length; i++){
commandArray.push ( ["hgetall", response[i] ] );
}
var results = await redis.client.batch(commandArray).execAsync();
return results;
}
}GraphQL querying example
A simple example of requesting data from the Node.js server is presented. The first argument of the query indicates the sorted set stored in redis, and also the ordered list name where the files are stored locally, followed by the "from" and "to" variables, indicating the starting and ending item in the list requested.
fetchMultiRedisArticles(list:"randomOrder",from:7, to:9){
id
title
body
}Testing with Postman and Newman
Storing and fetching a small number of articles will have no actual difference in the previously used decoupled system and redis storing system, and so the response time will be similar in both occasions. Scalability will be noticeable in high number of items, so for this post I did two tests, one generates 500,000 articles and the second one 1,000,000.
Of course 1,000,000 items have no impact in redis' performance as it was designed for scalability, but the local application's heap cannot take the load of a list of that many items and the heap stack fails. So the second test will provide us only with the redis results.
Testing with: 500,000 items and running 200 requests
*******************************
Test run JSON: 322.82
Test run Redis: 6.135
*******************************Testing with: 1,000,000 items and running 200 requests
*******************************
Test run JSON: 0
Test run Redis: 5.5
*******************************