Enhancing PHP to Kafka Message Sending Speed: The Proxy Approach

Photo by Veri Ivanova on Unsplash

Introduction: Sending messages from PHP to Kafka can sometimes result in slow performance due to the lack of persistent connection support in PHP. However, there are solutions available to address this issue and improve the speed of message transmission. In this article, we will explore one such solution: using a proxy.

The Challenge: PHP’s Lack of Persistent Connection Support By default, PHP does not provide native support for persistent connections, leading to increased latency when sending messages to Kafka. This can result in message-sending speeds ranging from 300+ms with occasional deviations exceeding 1 second.

The Solution: Utilizing a Proxy To overcome the performance limitations caused by PHP’s lack of persistent connection support, utilizing a proxy server is an effective approach. The proxy acts as an intermediary between PHP and Kafka, establishing and maintaining a persistent connection with Kafka on behalf of PHP.

One Recommended Proxy: Confluence Apache Kafka REST Proxy One reliable and widely-used proxy option is the Confluence Apache Kafka REST Proxy. It is a stable and free instrument with active community support. The proxy allows PHP applications to send messages to Kafka using a persistent connection, significantly reducing latency and improving overall message transmission speed.

Example Implementation: Go Program as an Alternative Proxy Alternatively, you can develop a custom proxy using any language that supports persistent connections with Kafka, such as Go, Node.js, or others. This approach provides more flexibility and customization options. For instance, I wrote a small program using Go, which achieved comparable speeds to the Confluence proxy but consumed only 2MB of RAM, compared to the Confluence proxy’s 700MB.

Confluence example

Here’s an example of a Docker Compose file that sets up Confluent’s Apache Kafka REST Proxy along with ZooKeeper and Kafka:

version: '3'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.0
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      - ZOOKEEPER_CLIENT_PORT=2181
      - ZOOKEEPER_TICK_TIME=2000

  kafka:
    image: confluentinc/cp-kafka:6.0.0
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092
      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1
      - KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS=0

  rest-proxy:
    image: confluentinc/cp-kafka-rest:6.0.0
    container_name: rest-proxy
    depends_on:
      - kafka
    ports:
      - "8082:8082"
    environment:
      - KAFKA_REST_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_REST_LISTENERS=http://0.0.0.0:8082
      - KAFKA_REST_SCHEMA_REGISTRY_URL=http://schema-registry:8081

  schema-registry:
    image: confluentinc/cp-schema-registry:6.0.0
    container_name: schema-registry
    depends_on:
      - kafka
    ports:
      - "8081:8081"
    environment:
      - SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL=zookeeper:2181
      - SCHEMA_REGISTRY_HOST_NAME=schema-registry

In this example, the Docker Compose file sets up the following services:

• ZooKeeper: Used by Kafka for distributed coordination.
• Kafka: The Apache Kafka broker for message storage and processing.
• REST Proxy: The Confluent Kafka REST Proxy for providing HTTP-based access to Kafka.
• Schema Registry: The Confluent Schema Registry for managing Avro schemas.

Now, you should have Confluence’s Apache Kafka REST Proxy, ZooKeeper, Kafka, and Schema Registry up and running. You can communicate with the REST Proxy at http://localhost:8082 to send messages to Kafka using a persistent connection.

Please note that the version numbers (6.0.0) used in the Docker Compose file are just examples. You can adjust the version numbers to match the desired version of the Confluent Platform.

GoLang Example

Here’s an example of a simple Go program that acts as a proxy for sending messages from PHP to Kafka with persistent connection support:

package main

import (
 "log"
 "net/http"

 "github.com/confluentinc/confluent-kafka-go/kafka"
)

var producer *kafka.Producer

func main() {
 // Configure Kafka producer
 p, err := kafka.NewProducer(&kafka.ConfigMap{"bootstrap.servers": "localhost:9092"})
 if err != nil {
  log.Fatal("Failed to create Kafka producer: ", err)
 }
 producer = p

 // Start HTTP server
 http.HandleFunc("/send", sendMessageHandler)
 log.Fatal(http.ListenAndServe(":8080", nil))
}

func sendMessageHandler(w http.ResponseWriter, r *http.Request) {
 message := r.FormValue("message")

 // Produce message to Kafka
 deliveryChan := make(chan kafka.Event)
 err := producer.Produce(&kafka.Message{
  TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
  Value:          []byte(message),
 }, deliveryChan)

 if err != nil {
  log.Println("Failed to produce message to Kafka: ", err)
  w.WriteHeader(http.StatusInternalServerError)
  return
 }

 e := <-deliveryChan
 m := e.(*kafka.Message)
 if m.TopicPartition.Error != nil {
  log.Println("Delivery failed: ", m.TopicPartition.Error)
  w.WriteHeader(http.StatusInternalServerError)
  return
 }

 log.Printf("Message sent to Kafka: topic=%s, partition=%d, offset=%d", *m.TopicPartition.Topic, m.TopicPartition.Partition, m.TopicPartition.Offset)

 w.WriteHeader(http.StatusOK)
 w.Write([]byte("Message sent successfully to Kafka"))
}

In this example, the Go program acts as an HTTP server that listens on port 8080. It exposes an endpoint “/send” that accepts POST requests with a parameter called “message” containing the message to be sent to Kafka. The program uses the Confluent Kafka Golang client library to interact with Kafka.

To run this program, make sure you have the confluent-kafka-go library installed (go get -u github.com/confluentinc/confluent-kafka-go/kafka). Also, update the Kafka broker configuration (bootstrap.servers) in the main function to match your Kafka setup.

Remember to handle any necessary error handling and further customize the program as per your requirements.

Please note that this is a simplified example and may not include all the necessary error handling or production-level features.

Conclusion: By employing a proxy server, like the Confluence Apache Kafka REST Proxy or developing a custom solution, PHP applications can overcome the inherent limitations in PHP’s lack of persistent connection support. This enables faster and more efficient message transmission to Kafka, reducing latency and improving overall system performance. Choose the proxy option that best suits your requirements and enjoy accelerated message sending from PHP to Kafka.

Remember to monitor and optimize your proxy configuration based on your specific use case to ensure optimal performance and reliability.