How to Speed Up Bulk Email Sending in PHP with Queues, Workers, and Cron Jobs

Sending bulk emails directly in a PHP script can be a slow and error-prone process. If you’ve ever tried sending hundreds or thousands of emails in a single request, you’ve likely encountered timeout issues, slow response times, and a generally poor user experience. In this post, we’ll explore how to handle bulk email sending efficiently using queues, workers, and cron jobs.

Problem: Slow Email Sending

Imagine you have a feature where users can send a newsletter to a large list of subscribers. If you try to send these emails directly within the request:

• Slow Response Times: Sending even one email can take 2–3 seconds. With hundreds of recipients, it becomes painfully slow.
• Timeout Issues: PHP scripts often have execution time limits (e.g., 30 seconds). If the process takes longer, it fails.
• Poor User Experience: The user has to wait for the entire process to finish before getting a response, making the application feel unresponsive.

Solution: Background Processing with Queues, Workers, and Cron Jobs

To solve this problem, we can break down the email-sending process into three main components:

1. Queue: A place to store tasks temporarily before they are processed.
2. Worker: A background script that processes tasks from the queue.
3. Cron Job: A scheduled task that triggers the worker script at regular intervals.

1. What is a Queue?

A queue is like a to-do list. When a user requests to send a newsletter, instead of sending emails immediately, the tasks (emails) are added to the queue. The queue can be a database table that stores information about each email (e.g., recipient email address, subject, message).

Why Use a Queue?

• It decouples the email-sending process from the main request.
• It allows the application to respond quickly to the user without waiting for emails to be sent.

2. What is a Worker?

A worker is a script or process that runs in the background, separate from your main web application. It picks up tasks from the queue and processes them (in this case, sending emails).

Why Use a Worker?

• It handles time-consuming tasks without blocking the main application.
• It can run independently, even if the user closes the browser.

3. What is a Cron Job?

A cron job is a scheduled task in Linux/Unix-based systems. It automatically runs a script at specific intervals (e.g., every minute, every hour).

Why Use a Cron Job?

• It ensures the worker script runs regularly to process tasks from the queue.
• It provides a simple way to automate background tasks without manual intervention.

How the Solution Works

User Action:

• The user initiates sending the newsletter (e.g., clicks “Send Newsletter”).
• Instead of sending the emails directly, the application adds the email tasks to a queue table in the database.

Worker Script:

• A PHP script (send_newsletters.php) acts as a worker that reads the queue and sends the emails.

Cron Job:

• A cron job is set up on the server to run the worker script every minute.
• This ensures the emails are sent regularly, even if there are many to process.

Example Implementation

Let’s create a simple newsletter-sending feature using this approach.

Step 1: Create a Queue Table

First, create a table in your MySQL database to store the email tasks:

CREATE TABLE email_queue (
    id INT AUTO_INCREMENT PRIMARY KEY,
    recipient_email VARCHAR(255),
    subject VARCHAR(255),
    message TEXT,
    status ENUM('pending', 'sent', 'failed') DEFAULT 'pending',
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Step 2: Add Email Tasks to the Queue

When the user clicks “Send Newsletter,” add the email tasks to the queue instead of sending them directly:

<?php

function addEmailsToQueue(array $recipients, string $subject, string $message)
{
    $db = new PDO('mysql:host=localhost;dbname=myapp', 'username', 'password');

    $stmt = $db->prepare("INSERT INTO email_queue (recipient_email, subject, message, status) VALUES (:email, :subject, :message, 'pending')");

    foreach ($recipients as $email) {
        $stmt->execute([
            ':email' => $email,
            ':subject' => $subject,
            ':message' => $message,
        ]);
    }

    echo "Emails added to queue successfully!";
}

Step 3: Create the Worker Script (send_newsletters.php)

This script will process the tasks from the queue and send the emails:

<?php

$db = new PDO('mysql:host=localhost;dbname=myapp', 'username', 'password');
$emailService = new PHPMailer\PHPMailer\PHPMailer();

$stmt = $db->query("SELECT * FROM email_queue WHERE status = 'pending' LIMIT 20");
$emails = $stmt->fetchAll(PDO::FETCH_OBJ);

foreach ($emails as $emailTask) {
    $emailService->setFrom('noreply@myapp.com', 'MyApp Newsletter');
    $emailService->addAddress($emailTask->recipient_email);
    $emailService->Subject = $emailTask->subject;
    $emailService->Body = $emailTask->message;
    $emailService->isHTML(true);

    if ($emailService->send()) {
        $db->query("UPDATE email_queue SET status = 'sent' WHERE id = {$emailTask->id}");
    } else {
        $db->query("UPDATE email_queue SET status = 'failed' WHERE id = {$emailTask->id}");
    }
}

echo "Worker script finished processing.";

Step 4: Set Up a Cron Job

To run the worker script periodically, set up a cron job:

1. Open the crontab editor:

crontab -e

2. Add the following line to run the worker script every minute:

* * * * * /usr/bin/php /path/to/your/project/send_newsletters.php > /dev/null 2>&1

Benefits of This Approach

• Non-Blocking: The user doesn’t have to wait for the emails to be sent. The request is processed quickly.
• Scalability: The system can handle a large number of emails without timing out.
• Reliability: Failed emails can be retried since the queue stores tasks until they are processed successfully.
• Better User Experience: The application remains responsive, and the user is not left waiting.

Conclusion

By using queues, workers, and cron jobs, you can significantly improve the performance of bulk email sending in your PHP application. This approach is scalable, reliable, and provides a better experience for your users.

If you found this tutorial helpful, share it with others who might be struggling with similar issues. Happy coding!