Sending Logs From Symfony Applications To GrayLog Through RabbitMQ

Message Queues are indispensable tools of the systems which is under high load. You can split your systems to small pieces and provide communication between each other through message queue services.

If your system happens by many pieces it means you have so many log entries which is coming from multiple application instances in same time. So you should scale your logging system like how you scaled your application also. Otherwise, your application may go down becaouse of your logging system might be bottleneck.

You can use message queue systems to avoid this trouble. Now, I will explain how you can implement sending log entries to GrayLog through rabbitMQ in a symfony application.

Installing Composer Dependencies:
Let’s install the RabbitMQBundle to access rabbitmq service on our symfony application.

$ composer require php-amqplib/rabbitmq-bundle

Define your rabbitmq connection into your config.yml

# app/config/config.yml
...
old_sound_rabbit_mq:
connections:
logging:
host: '%rabbitmq_host%'
user: '%rabbitmq_user%'
password: '%rabbitmq_password%'
vhost: '%rabbitmq_logging_vhost%'
producers:
logging:
connection: logging
exchange_options:
name: logging
type: fanout

Define your rabbitmq credentials to parameters.yml which you described in config.yml as a variable.

# app/config/parameters.yml
parameters:
...
rabbitmq_host: <your_rabbitmq_host_address>
rabbitmq_user: <your_rabbitmq_user>
rabbitmq_password: <your_rabbitmq_password>
rabbitmq_logging_vhost: logging

Define a new channel as a service to using in monolog configuration.

# app/config/services.yml
services:
...
monolog_mq_channel:
class: PhpAmqpLib\Channel\AMQPChannel
arguments:
- "@old_sound_rabbit_mq.connection.logging"

Define a handler to your monolog configuration in related environment configuration file. I’m going to make definition in config_dev.yml for testing now.

# app/config/config_dev.yml
monolog:
handlers:
console:
type: amqp
exchange: monolog_mq_channel
exchange_name: logging
level: warning

Now our application is ready to sending logs to rabbitmq. Let’s prepare our rabbitmq server to accepting logs from our application.

Create a virtual rabbitmq host and set needed permissions to izolate your queue and exchange which you will use for logging from your existing application dependencies on rabbitmq.

$ rabbitmqctl add_vhost logging
$ rabbitmqctl set_permissions -p logging guest “.*” “.*” “.*”

And finally create the exchange on rabbitmq which we declared in configuration as a producer.

$ ./bin/console rabbitmq:setup-fabric

Now our rabbitmq server is ready to accept logs that sent by our application.

Let’s create a symfony command to see how log entries goes to rabbitmq as a message.

<?php
namespace AppBundle\Command;
use Psr\Log\LoggerInterface;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
class LoggingTestCommand extends ContainerAwareCommand
{
protected function configure()
{
$this->setName('logging:test')
->setDescription('Tester command for sending log over');
}
protected function execute(InputInterface $input, OutputInterface $output)
{
/** @var LoggerInterface $logger */
$logger = $this->getContainer()->get('logger');
$output->writeln('Test messages are sending to message queue now. Please press CTRL+C to break this process.');
while(true) {
$logger->error('Some error occurred while doing sometihng.', [
'some_value' => 'ABC',
'another_value' => 1532,
]);
usleep(500);
}
}
}

Let’s call the following command and see how logs come to rabbitmq

$ ./bin/console logging:test

Our application managed sending logs to rabbitmq as you can see. Let’s configure our graylog to consume messages from rabbitmq.

Let you sign in to your graylog application as a administrator and goto Inputs section from System menu. You will see a page like the following.

Choose the “GELF AMQP” choice and click to launch new input (which is green one) button. Fill the inputs in the opened popup like the following

Title: errors
Gobal: (Checked) (This means all nodes in your graylog cluster will use this input as a source)
Exchange: logging
Broker virtual host: logging
Queue: testapp-logs
Broker hostname: (Your rabbitmq host address)
Allow throttling this input: (checked) (This options usally used when you readed logs from file or message queue.)
Bind to exchange: (Checked) (This means graylog will bind the testapp-logs queue to your exchange automatically.)
Routing Key: #

We are ready!

You can download and review my example application which can running on docker-compose. All services can be up automatically with a single docker-compose up command.