Decoupling Ruby Applications with Amazon SNS & SQS

This article originally appeared on Kapost Engineering. It was also featured in Ruby Weekly.

As the Kapost development team has grown, we found that we’ve reached a threshold where it has become necessary to break our application apart into smaller, more manageable pieces. This includes customer-facing applications that match components of our product vision, APIs that have specific tasks or domains, and internal services responding to application events. Building smaller applications and APIs are topics that are relatively easy insofar as the amount of help and documentation you can find online. However, handling application events within new internal services is something that took a bit more architecting for us before settling on an approach.

There were four primary considerations in what we wanted in an eventing system:

  1. Decoupling: There should be a clear separation of concern between our applications. Whatever app is publishing an event shouldn’t care what other apps are subscribed. There should never be a need to update the code relating to publishing an event such that the application needs to specify subscribers. We simply want to fire-and-forget.
  2. Fan-out: One event could have any number of subscribers. There should not be a one-to-one mapping between a published event and a subscriber.
  3. Speed: We want to be able to publish an event without waiting for it to be published. Related to the first point, we also don’t want to have to publish more than once (i.e.: one publish vs. one publish per subscriber).
  4. Reliability: Systems require downtime, and as we build more of them, the timing of migrations and updates for any given system will vary. We don’t want any published messages to be missed by any particular system due to unavailability.

After some research and consideration, we decided that using Amazon SNS in conjunction with Amazon SQS would allow for all these conditions to be met. Amazon SQS is “a fast, reliable, scalable, fully managed message queuing service” that can continue to enqueue messages regardless of internal system availability. Amazon SNS is “a fast, flexible, fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients” that integrates with SQS out of the box.

We Have the Technology!

With our research complete, we set out to build circuitry, a simple Ruby gem that encapsulates publishing to SNS and subscribing to SQS.

With circuitry, any object can published; it is simply serialized via its to_json method prior to delivery. This means that helpful serialization tools such as active_model_serializers can be used to wrap an object, only publishing data that is appropriate for your use case. Conversely, subscribers will receive the JSON payload along with the topic name that was given when publishing.

An example of how this might be used to track changes to a document is as follows:

# publisher
class Document < ActiveRecord::Base before_update :publish_changes def publish_changes serializer = DocumentUpdateSerializer.new(self) Circuitry.publish('document-update', serializer, async: true) end end # subscriber Circuitry.subscribe('https://sqs.region.amazonaws.com/account-id/queue-name') do |message, topic| HANDLERS[topic].handle(message) end HANDLERS = { 'document-update' => Handlers::DocumentUpdate
}.freeze
 
module Handlers
  module DocumentUpdate
    def handle(message)
      DocumentChangeDeserializer.from_json(message).save
    end
  end
end

It should also be pointed out that publishing accepted an async option. This means that we’re able to publish in the background while our process continues working normally, preventing any additional network latency during user requests.

AWS Configuration

We still need to configure SNS and SQS to work properly before this code will work. Fortunately, AWS gives us the necessary tools to do this.

The first step is to create our SQS queues. A queue should be application-specific, and in order to meet our reliability criteria, we should set up a failure queue as well. AWS requires that we set up our failure queue first.

Create SQS Failure Queue

Next, we have to create our normal event queue. This should be set up with the “Redrive Policy” enabled, which is what allows us to take advantage of the failure queue from the previous step.

Create SQS Event Queue

If done correctly, you should now see both queues in your SQS management console.

SQS Permissions

Before we can check the SQS setup off our list, we need to set permissions on each of these queues. Select either queue from your list, and click the “Add Permission” button as seen above. Ensure that the “SendMessage” permission is enabled. Although I enabled this for “Everybody” in the demo below, a more secure solution would be to only permit it for the AWS account number(s) that are associated with your SNS topics. Be sure to repeat this step for both the event and failure queues.

Add SQS Permission

After our application queues are set up, we need to create any SNS topics that we plan to publish to. Although we’ll only create one topic for this example, the number and names of topics actually created should reflect the events for your apps will be publishing. The topic name can be anything, as long as it matches the topic name used in your code.

Create SNS Topic

Once your topic is created, the last step is to create a subscription between it and the SQS event queue we created earlier. Ensure that the Protocol is set to “Amazon SQS”, while the Endpoint matches the ARN generated for your SQS event queue.

Create SNS Subscription

Results

We’ve just begun using circuitry, so our mileage remains to be seen. With that said, we’re happy with the architecture conceptually and are excited to continue working with it internally. Given that we’ve recently taken Flux head-on for our JavaScript development, it should come as no surprise that we’re fans of the dispatching paradigm.

The circuitry gem is completely open-source, so feel free to share and contribute. We’re excited to see its continued development!

Leave a Reply

Your email address will not be published. Required fields are marked *