Fork me on GitHub

Getting started

Okay, let's see Tochtli in action. In the first exercise we will implement a server that creates website snapshots using screencap gem. But before we go any further, let's check the prerequisites (that are not covered by this tutorial). I assume that you:

  • Have RabbitMQ set up and running with default configuration
  • Have Ruby >= 2.0 installed
  • Played with screencap at least once (on my system, even though I had PhantomJS installed, screencap decided to install its own version of it on the first run, which slowed down everything significantly)

OK, now we are ready to go. Tochtli plays nice with Ruby on Rails, however you can use it pretty much anywhere. In this first tutorial we are going to operate on simple Ruby files, with no frameworks or libraries. First, let's create a Gemfile:

source 'https://rubygems.org'
gem 'screencap'
gem 'tochtli', github: 'puzzleflow/tochtli'

Not much to see here, so we may proceed to creating some logic. In Tochtli we use Ruby classes to define messaging API. Besides an abundance of benefits of that, there is one major downside – you have to share the message definitions between client and server in some fashion. In our simple example we can just put it in a file, which will be required by both client and a server.

class CreateScreenMessage < Tochtli::Message
    route_to 'screener.create'
    
    attribute :url, String 
    attribute :file, String
end

This is a definition of a message that client sends to the server. We want to create a shapshot of a webpage, so we are going to need its URL and a name of the output file. This is pretty obvious with attribute method. Any Tochtli message is a Virtus model. route_to may be less obvious. You can think of it as the route by which the message is sent. One message can be bound to one topic and to avoid confusing and cumbersome code, you should also use only one message with a topic. Technically a topic is RabbitMQ's feature and you can read more about it in one of their excellent tutorials.

Our one message would be enough to get our little system working, but we want more. Snapshotting can take quite some time and we (or at least I) are interested in how long exactly it takes. To do that, lets implement a reply message:

class CreateScreenReplyMessage < Tochtli::Message
    attribute :time, Float
end

Conventionally, we name replies the same way as 'original' messages, with a Reply word inserted as penultimate part. Note that reply message does not have any topic bound. Tochtli creates a reply queue, unique for each client, and the message goes through it. Its name is autogenerated and unique.

Before we go into implementing actual client and server there is one more thing left. Tochtli wants you to define the logger (it will use Rails logger if available). With this addition out common.rb file looks like that:

require 'bundler'
Bundler.require

Tochtli.logger = Logger.new('tochtli.log')

class CreateScreenMessage < Tochtli::Message
    route_to 'screener.create'

    attribute :url, String
    attribute :file, String
end

class CreateScreenReplyMessage < Tochtli::Message
    attribute :time, Float
end

Server

Server in Tochtli uses concept of controllers. So let's define one:

require_relative 'common'

class ScreenerController < Tochtli::BaseController
    bind_to 'screener.*'

    on CreateScreenMessage, :create

    def create
        start_time = Time.now
        f = Screencap::Fetcher.new(message.url)
        f.fetch output: File.join(__dir__, 'images', message.file)
        total_time = Time.now - start_time
        reply CreateScreenReplyMessage.new(time: total_time)
    end
end

So, what's goin on here? First of all, we bind the controller to a set of topics with a wildcard. Then we define a routing for message CreateScreenMessage which is processed by the method create. This method does its capturing stuff, measures time and sends a reply. Hopefully, the convention here is obvious: a part of the topic matching the wildcard is a method name. And our CreateScreenMessage is routed to screener.create, so after receiving it, the controller calls the correct method. The received message is available with a accessor named message. reply comes from Tochtli::BaseController and sends our CreateScreenReplyMessage instance to reply queue I mentioned above.

Now, when we run the file with bundle exec ruby server.rb... Nothing happens! That's because we need to start the controller. Tochtli::ControllerManager will help here. At first we need to setup RabbitMQ connection.

Tochtli::ControllerManager.setup

See setup parameters description for details how to pass your own connection. If no parameters are specified the default Bunny connection is used. For tests you can simply use RABBITMQ_URL environment variable to specify connection string.

The setup should be done only once. After connection is made you can start all loaded controllers (start method accepts the list of controller classes that should be started instead of all preloaded controllers). 

Tochtli::ControllerManager.start

After we add this line, we run the file again and... still nothing. That's because a controller is spawned within another thread, the program goes on, reaches the end and terminates before we can do anything. For now let's cheat with a infinite sleep. We may take a last (for now) look at our server file and start writing the client.

require_relative 'common'

class ScreenerController < Tochtli::BaseController
    bind 'screener.*'

    on CreateScreenMessage, :create

    def create
        start_time = Time.now
        f = Screencap::Fetcher.new(message.url)
        f.fetch output: File.join(__dir__, 'images', message.file)
        total_time = Time.now - start_time
        reply CreateScreenReplyMessage.new(time: total_time)
    end
end

Tochtli::ControllerManager.setup
Tochtli::ControllerManager.start

trap('SIGINT') { exit }
at_exit { Tochtli::ControllerManager.stop }

puts 'Press Ctrl-C to stop worker...'
sleep

Client

Writing client is really simple. So let's get to the code right away:

require_relative 'common'

class ScreenerClient < Tochtli::BaseClient
    def create_screen(url, file_name)
        publish CreateScreenMessage.new(url: url, file: file_name)
    end
end

ScreenerClient.new.create_screen(ARGV[0], ARGV[1])

Not much magic here. The publish method comes from Tochtli::BaseClient and it's responsible for, well, publishing the message to RabbitMQ. With server fired up, run the file with bundle exec ruby client.rb http://google.com google.png and you probably already suspect that I'm just messing with you and that won't work.

That's actually not true.

Even though there was no result in the console, a lot of things happened. First of all, check the images directory. If there were no network problems, the file with screenshot should be there. Then, have a look inside the log file we defined in our first steps.

I, [2015-08-04T15:22:14.377184 #60562]  INFO -- SERVER: Starting ScreenerController...
D, [2015-08-04T15:22:17.503833 #60567] DEBUG -- CLIENT: [2015-08-04 15:22:17 +0200 AMQP] Publishing message 62b59db2-49e9-433a-abcd-6964a36899e7 to screener.create
D, [2015-08-04T15:22:17.510678 #60562] DEBUG -- SERVER: 

AMQP Message CreateScreenMessage at 2015-08-04 15:22:17 +0200
D, [2015-08-04T15:22:17.510738 #60562] DEBUG -- SERVER: Processing by ScreenerController#create [Thread: 70228789879900]
D, [2015-08-04T15:22:17.510825 #60562] DEBUG -- SERVER:     Message: {:url=>"http://dilbert:15672/#/queues", :file=>"rabbitmq.png"}.
D, [2015-08-04T15:22:17.510876 #60562] DEBUG -- SERVER:     Properties: {:content_type=>"application/json", :delivery_mode=>2, :priority=>0, :reply_to=>"amq.gen--xh3k8w7UkgxlmUOjtPnyQ", :message_id=>"62b59db2-49e9-433a-abcd-6964a36899e7", :timestamp=>2015-08-04 15:22:17 +0200, :type=>"create_screen_message"}.
D, [2015-08-04T15:22:17.510936 #60562] DEBUG -- SERVER:     Delivery info: exchange: puzzleflow.services, routing_key: screener.create.
D, [2015-08-04T15:22:29.032863 #60562] DEBUG -- SERVER:     Sending  reply on 62b59db2-49e9-433a-abcd-6964a36899e7 to amq.gen--xh3k8w7UkgxlmUOjtPnyQ: #<CreateScreenReplyMessage:0x007fbed42141f8 @time=11.520917, @properties=nil, @id="964f407f-a57e-41a2-86f8-40ed9ccf5d40", @extra_attributes={}>.
D, [2015-08-04T15:22:29.036243 #60562] DEBUG -- SERVER: Message 62b59db2-49e9-433a-abcd-6964a36899e7 processed in 11525.6ms.
D, [2015-08-04T15:22:29.036788 #60562] DEBUG -- SERVER: Reply on message 62b59db2-49e9-433a-abcd-6964a36899e7 dropped: NO_ROUTE [312]
I, [2015-08-04T15:22:46.767049 #60562]  INFO -- SERVER: Stopping ScreenerController...

Let me explain what we are looking at. The second line is from the client. It published a message to the correct topic (screener.create) and terminated. Publishing is asynchronous so it did not wait for anything. But the server received the message and processed it. It even tried to send a reply that everything is correct, but failed to deliver it because of NO_ROUTE error (transient client reply queue has been already deleted).

Our next task is to maintain the reply queue and use the time from the response to do anything (like, print it on STDOUT). First we want to hold the main program thread until the response comes and only then exit (and sleep 10 at the end of the file is not a solution). The Tochtli-way of doing it is to use the handler. Normally, handlers are asynchronous and you would have to meddle with condition variables and mutexes. Fortunately there is a simpler way. Enter SyncMessageHandler:

def create_screen(url, file_name)
    handler = SyncMessageHandler.new
    message = CreateScreenMessage.new(url: url, file: file_name)
    rabbit_client.publish message, handler: handler
    handler.wait!(20)
    puts "Done in #{handler.reply.time} seconds"
end

You probably already see what I did there. We create a new SyncMessageHandler and upon sending the message bind it to it. Then with handler.wait!(timeout) we block the thread until message arrival. If it does not happen withing specified number of seconds, an exception is raised. After receiving the response, we may access it via handler.reply and write our elapsed time to the console.

Et voilà! We have completed our RPC webpage capturing system. Now let's learn how to scale it. You can find the client and server code in the first tochtli example.