Hooking Cachegoose to Redis on NodeJS

On my last post I outlined the steps required to enable caching using the Cachegoose NodeJS module. I thought it would be a fun exercise to add a Redis caching to the node-billz application. Previously I had implemented an in-memory cache which works fine for just me, but in an actual production environment memory could become an issue.

Cachegoose makes it very easy to add the Redis storage to the application since it uses the NodeJS cacheman plugin under the hood. Slight modifications to my main app.js file were required to pass the host, port and password to Redis. I also added the USE_REDIS flag to my .env file so Redis support can easily be toggled on or off without any code changes. Redis integration complete.

Another plus with using Redis is being able to explicitly set the cache eviction policy. The cacheman-memory module follows LRU (least recently used) eviction policy. Redis allows multiple eviction policiesso it allows some flexibility.

Once deployed to the production environment I needed to be able to verify the cache was working correctly. I love using the cli but some things are just easier to visualize, uh, visually. Doing some quick searches I found a few open-source projects out there that were pretty nice, but Redsmin really stood out from the rest. I am using their free plan which allows monitoring of one Redis instance which is all I need.

Going forward I may revert back to the in-memory cache but it was a fun exercise adding Redis support with a few lines of code.


Caching with NodeJS/Express

Working with payees and payments in node-billz I basically end up loading and reloading the same data on every page. The list of people paid each month will rarely change. The list of payments will grow over time but generally only a few payments are made per week. Basically, these are perfect candidates for caching so I thought I would see how difficult it would be to get this implemented in Node.

Searching GitHub I quickly found a few cache providers that would work with MongoDB and more specifically Mongoose since that is what I am using to work with backend data. There were a few false starts with downloading orphaned projects that were no longer supported, but finally I settled with cachegoose. It has been updated recently plus it gave me the option of an in-memory cache as well as caching with Redis and others.

The cache provider module selected, the next step was to get the app configured to use the cache. Two lines of code later everything was working. It does not get much easier. I opted to use the in-memory cache for the time being, but you can easily point cachegoose to a Redis server at initialization.

The final step was to wire the cache to the queries. You can specify the number of seconds you would like to cache a query or you can cache it indefinitely with a custom cache key. If a cache needs to be purged, call the clearCache method with the custom cache key to clear it. I opted to go the second route since the data only changes occasionally. When a payee is modified or a payment is made I clear the cache keys so the data can be re-cached the next time it is queried.

It was incredibly easy to setup caching, aside from the false steps I had at first. Given the size of the ecosystem though I suppose I should not be surprised to find some dead projects out there.

I would be interested in learning about any other libraries or tools you guys are using. Feel free to comment or drop me a line.

Writing a bill calendar with Node and Express

Everyone has their “go to” application when it comes to learning a new software stack. I believe every programmer out there has written the venerable “Hello World” application at least once in their lifetime. Todo apps were all the rage when Angular came on the scene. Some folks I know like writing blog applications. My app of choice is a bill calendar.

It may seem like a strange starting point when it comes to building a learning application, however date manipulation is one of those things that seems very simple on the surface but it is deceptive. You get that down and you can tackle almost anything.

The first time I wrote the bill calendar was with ColdFusion(Lucee) and ColdBox, but I thought it would be really cool to write a single page application using AngularJS. Alas, Angular is great on the client side but if you would like data to persist it requires a back end server. A bill calendar that requires you to enter everyone you pay every time the page is loaded would get monotonous quickly. I could easily write an API in ColdFusion to handle this, but it did give me an excuse to play around with NodeJS.

So here we are

The very first thing I did was to find a starting point. I could have very easily started with a bare-bones implementation of Express and built on top of that, but the NodeJS community is awesome so I did not have to reinvent the wheel. I found the hackathon-starter boiler plate app and was able to quickly get up and running. It almost feels like cheating since Express and much of the middleware has already been done for you, but at the same time it follows best practices and gives you user authentication with social sign-in from the gate. It is possible to jump right in and starting coding with some of the community driven boiler plate apps. If this one was not your cup of tea, simply pick another one!

Now that the foundation has been laid it is time to start to actually build something. The hackathon-starter app breaks the layout into several logical folders: config for your Passport configuration to handle user authentication, the controllers folder that hold the actual logic for your application routes, the models folder that contain the mongoose schema for items and finally a views folder that will hold our PugJS templates. App.js in the root of the folder structure that is the main file for the application and pulls everything together.

The next step is figuring out the structure of the data to retain. Since this is a bill calendar the name, a short description, due date, amount and the website URL would be fantastic to track . There are several other fields related to payees that are retained, but those are the important ones.

I also created a schema for payments so a payee can be marked as paid in the forecast view. If the payment amount changes each month we can also show the end user what the average payment amount is for that payee. That will require us to track the id of the payee, the id of the owner, the date the payment is due as well as the amount.

Once the data has been setup the real fun begins. The forecast is what really ties all these pieces together. Forecast.js in the controllers folder only has one route handler, the getIndex() function. This method will calculates the start date, gets a list of bills for the current week and will then proceed to the next three weeks.

Here is the overall flow of how the whole process works:

  • Load the user’s current list of payees and build a schedule for the current month. Also get a sum of the total amount due each month.
  • Load a user’s payments to see if it corresponds to a current bill and mark it paid
  • Using MomentJS, determine the first day of the current week based on the user’s settings. By default the first day of the week is Sunday, but allow the user to define that value if they would prefer a different start day.
  • Now it is just a matter of using MomentJS to determine if the due date falls in between the start date and end date. If so, add it to the weekly list and add the amount to the weekly total. Also look in the payment information to see if I can match a payment to the bill due date so it can be marked paid.
  • This process continues for every week displayed on the forecast view.

Wrapping it all up

I am not the strongest JavaScript developer, but the time I invested playing around with AngularJS really helped. Being able to write ES6 without having to worry about transpiler configuration to compile it back to ES5 is fantastic. Everything just works.

Overall my first real experience with Node/Express was very positive. The community around NodeJS in general is fantastic and if I hit a road block I could easily find an answer to my questions. I am looking forward to polishing up my bill calendar app, it has been a pleasure so far.

(Originally posted on Medium)


Ode to Node

ColdFusion has just turned 20 years old and I have been developing with it for 16 of those 20 years; almost half my life. I still use ColdFusion every day in my professional life, but I do not feel the excitement for the language as I used to. I can remember when I actually looked forward to the new features introduced with each new release, but those days have past.

I have recently been looking in to other languages to get some sense of excitement back. First I have been looking into Node. The great part about Node is it is easy to pick up and run with if you know javascript.

A few weeks back we had a hackathon at work and this was my first real application hands on with Node. The project was an end-user level interface that allowed creating an issue and having Node interface with Jira through web services to create issues and assign it to the right project bucket. In retrospect, there definitely are easier projects to choose for your first Node project (especially when you are dealing with the learning curve and you have finite amount of time to get the project done). However, I did pull it off with about 15 minutes to spare. I didn’t win the hackathon, but I had a working Node project and a lot of insight in how to get things done in Node.

I started a little side project to continue learning Node, a Diablo 3 character comparison tool which allows you to compare two Diablo 3 characters side by side. This project focuses on a lot of middleware and making things work as well as hooking into the Battle.net API service. The project is structured much like a ColdBox app with routes, handlers and services. It is a work in progress and I am adding new things as I learn them. I have a demo up and running at http://diablo.kisdigital.com and the full source up at https://github.com/robertz/chardiff.

Next up is a look at Go (http://golang.org)