İbrahim Gündüz
Sep 17, 2017 · 4 min read

Message queues are an indispensable part of a system which is under high load. They protect the applications from the destructive impact of the load and help to scale up the applications easily also.

If a system consists of many application instances and/or some devices it means that you have many log producers too. So you should scale out the logging system as you scaled the application. Otherwise, the logging mechanism might be effected from the load that comes from outside, be irresponsible and makes down the entire system easily.

You can scale logging systems using message queues too. Today I’m gonna try to explain how to configure RabbitMQ and Graylog for sending logs to Graylog from a Symfony application through RabbitMQ.

Installing Composer Dependencies:
Install the RabbitMQBundle to access the RabbitMQ instance from the application.

$ composer require php-amqplib/rabbitmq-bundle

Define the RabbitMQ connection into the config.yml

# app/config/config.yml
...
old_sound_rabbit_mq:
connections:
logging:
host: '%rabbitmq_host%'
user: '%rabbitmq_user%'
password: '%rabbitmq_password%'
vhost: '%rabbitmq_logging_vhost%'
producers:
logging:
connection: logging
exchange_options:
name: logging
type: fanout

Define the credentials for the RabbitMQ connection into parameters.yml that described in config.yml

# app/config/parameters.yml
parameters:
...
rabbitmq_host: <your_rabbitmq_host_address>
rabbitmq_user: <your_rabbitmq_user>
rabbitmq_password: <your_rabbitmq_password>
rabbitmq_logging_vhost: logging

Define a new service for a RabbitMQ channel to be used by monolog.

# app/config/services.ymlservices:
...
monolog_mq_channel:
class: PhpAmqpLib\Channel\AMQPChannel
arguments:
- "@old_sound_rabbit_mq.connection.logging"

Define a monolog handler in the Symfony configuration file like the following. For the demostration, we will use config_dev.yml You can define handlers specifically for each environment.

# app/config/config_dev.ymlmonolog:
handlers:
console:
type: amqp
exchange: monolog_mq_channel
exchange_name: logging
level: warning

Now the application is ready to send the logs to RabbitMQ. Let’s set up the RabbitMQ server instance for receiving messages.

Setup RabbitMQ

In the example, I preferred to create a specific virtual server in order to separate RabbitMQ specific dependencies related application and log. You can ignore this step if you plan to use a separate RabbitMQ instance.

$ rabbitmqctl add_vhost logging
$ rabbitmqctl set_permissions -p logging guest “.*” “.*” “.*”

Create a new exchange on RabbitMQ with the name you defined in the configuration as a producer.

$ ./bin/console rabbitmq:setup-fabric

Now the RabbitMQ server is ready to receive logs from the application.

Setup A Log Producer For The Demo

You can simply create a Symfony command like the following, in order to create some log entries on the message queue.

<?php
namespace AppBundle\Command;
use Psr\Log\LoggerInterface;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
class LoggingTestCommand extends ContainerAwareCommand
{
protected function configure()
{
$this->setName('logging:test')
->setDescription('Tester command for sending log over');
}
protected function execute(InputInterface $input, OutputInterface $output)
{
/** @var LoggerInterface $logger */
$logger = $this->getContainer()->get('logger');
$output->writeln('Test messages are sending to message queue now. Please press CTRL+C to break this process.');
while(true) {
$logger->error('Some error occurred while doing sometihng.', [
'some_value' => 'ABC',
'another_value' => 1532,
]);
usleep(500);
}
}
}

Demo

Call the following command to create some logs on the RabbitMQ server.

$ ./bin/console logging:test

As you can see from the following picture; the application is started sending logs to the RabbitMQ server.

Setup Graylog

Sign in to your Graylog as a administrator and go to Inputs section from System menu.

Choose the “GELF AMQP” from the dropdown list on the head of the page and click to Launch new input button (The green button near the dropdown list).

Fill the inputs in the opened pop-up with the following values:

Title: errors
Gobal: (Checked) (This means all nodes in your graylog cluster will use this input as a source)
Exchange: logging
Broker virtual host: logging
Queue: testapp-logs
Broker hostname: (Your rabbitmq host address)
Allow throttling this input: (checked) (This options usally used when you readed logs from file or message queue.)
Bind to exchange: (Checked) (This means graylog will bind the testapp-logs queue to your exchange automatically.)
Routing Key: #

As you can see the logs are started coming from RabbitMQ to Graylog.

You can find this example and all the Graylog/RabbitMQ setup from the following repository.

Thanks for reading!

Developer Space

Tech Blog

İbrahim Gündüz

Written by

Nothing more than developer. Linkedin: https://www.linkedin.com/in/ibrahimgunduz Github: http://github.com/ibrahimgunduz34/

Developer Space

Tech Blog

Welcome to a place where words matter. On Medium, smart voices and original ideas take center stage - with no ads in sight. Watch
Follow all the topics you care about, and we’ll deliver the best stories for you to your homepage and inbox. Explore
Get unlimited access to the best stories on Medium — and support writers while you’re at it. Just $5/month. Upgrade