How two MongoDB database developers cured performance bottlenecks by offloading transactions to Redis.
TL;DR: Not all real-world data-management problems can be addressed by a single DBMS. These two database developers addressed performance bottlenecks in their MongoDB databases, and avoided expensive hardware upgrades, by offloading some time-sensitive transactions to Redis in-memory databases that were subsequently synched with their corresponding MongoDB documents.
Henry Ford is famous for telling the people buying his Model T that they could have any color they wanted, so long as it was black. For years the database industry was not much different: You could have any database you wanted, so long as it was SQL.
Today it seems there are new databases appearing every time you turn around, few of which bear much resemblance to the traditional relational model. At the same time, organizations are collecting and using more data — and more data types — than ever before. Timely, accurate data analysis is critical to every company’s success.
Rarely is one database going to fit the bill. That makes the ability to integrate with other databases a key feature of every DBMS on the market. More and more organizations are meeting their data-management needs by combining MongoDB’s ability to accommodate unstructured data with Redis’s in-memory performance.
A MongoDB database undone by a simple document counter
A January 30, 2014, post by DJ Walker-Morgan on the Compose blog describes a company that was facing the prospect of an expensive server upgrade to accommodate an increase in database writes. Each time the application ran a task, counters in the documents increased by one, which caused a write operation.
The system was maxing out at 1500 updates per second. Sharding the data across multiple MongoDB instances would allow 1500 more updates per second for each shard, but doing so would require the time and expense of more CPUs. Instead, the company reworked the application to send increment requests to a Redis in-memory database, mapped to the correct key and field, using the “hincrby” command. Redis’s key/value store offers a high transaction rate and low data storage that’s perfect for such incremental operations. The Redis store is ultimately harvested, aggregated, and saved in a MongoDB document.
MongoDB-Redis combo takes the sting out of keeping score
Karl Seguin, the creator of an open-source leaderboard component for game developers, improved the performance of score-keeping by combining a MongoDB data store with Redis’s sorted sets. Seguin describes how he did so on the Practical NoSQL blog.
When you enter a key, a score, and a value in Redis, it supplies simple and efficient ranking methods. The key is the leaderboard id, the scope, and the scope+leaderboard start time. Daily scores for each leaderboard id are added to an ordered set. The “member” parameter ensures that when values are updated, only the member’s highest score is recorded.
Redis’s sorted sets simplifies the process of ranking players in game apps. Source: Practical NoSQL
Seguin estimates that the time required to complete a read operation on a leaderboard with 5 million records was reduced by the new technique from approximately 5 minutes to only 147 milliseconds. Still, a score update could require as many as 8 operations: 1 read, 4 MongoDB writes, and 3 Redis writes. However, there are far fewer writes than reads overall, and the performance boost didn’t require any new hardware, although it did create the need to manage a second data store.
With the new Morpheus Virtual Appliance you can provision, deploy, and monitor your MonoDB, Redis, MySQL, and ElasticSearch databases from a single point-and-click console. The Morpheus database-as-a-service (DBaaS) lets you work with SQL, NoSQL, and in-memory databases across hybrid clouds in just minutes. Each database instance you create includes a free full replica set for built-in fault tolerance and fail over.
Morpheus lets you migrate existing databases from a private cloud to the public cloud, or from public to private. A new instance of the same database type is created in the other cloud, and real-time replication keeps the two databases in sync.Visit the Morpheus site for pricing information and to create a free account.
