Introduction

Managing logs is a crucial aspect of any application. Logs provide vital information about the application’s behavior, errors that occurred, and assist in troubleshooting. Two popular tools for log management in Go applications are Logrus and Lumberjack. Logrus is a feature-rich logging library, while Lumberjack handles log file rotation and compression. In this article, we will explore how these tools can be used together to create a robust and scalable logging system.

Why Use Logrus with Lumberjack?

Logrus:

• Structured Logging: Logrus allows you to create structured logs, which are easier to search and analyze.
• Hooks: You can add hooks to send logs to various destinations like log monitoring services, files, etc.
• Formatting: Logrus supports multiple formats, including JSON, making it flexible for different logging requirements.

Lumberjack:

• Log Rotation: Lumberjack automatically handles log file rotation based on size, time, or number of backups.
• Compression: It can compress old log files to save disk space.
• Retention Policy: Lumberjack allows you to set policies for retaining log files, ensuring that old logs are archived or deleted as per your configuration.

Setting Up Logrus with Lumberjack

Here’s a step-by-step guide on how to set up Logrus with Lumberjack in a Go application:

1 — Install Dependencies:

go get github.com/sirupsen/logrus
go get gopkg.in/natefinch/lumberjack.v2

2 — Configure Logrus with Lumberjack:

package main

import (
    "github.com/sirupsen/logrus"
    "gopkg.in/natefinch/lumberjack.v2"
)

func main() {
    log := logrus.New()
    log.SetOutput(&lumberjack.Logger{
        Filename:   "./logs/application.log",
        MaxSize:    10, // Max size in MB
        MaxBackups: 3,  // Max number of old log files to keep
        MaxAge:     28, // Max age in days to keep a log file
        Compress:   true, // Compress old log files
    })

    log.Info("This is an info log message")
    log.Warn("This is a warning log message")
    log.Error("This is an error log message")
}

Logging Examples

Here are some examples demonstrating how to use Logrus for different log levels and structured logging:

1 — Basic Logging:

log.Info("Application started")
log.Warn("This is a warning")
log.Error("An error occurred")

2 — Structured Logging:

log.WithFields(logrus.Fields{
    "event": "user_signup",
    "user":  "john_doe",
}).Info("New user signup")

3 — Logging with Hooks:

hook, err := logrus_syslog.NewSyslogHook("udp", "localhost:514", syslog.LOG_INFO, "")
if err == nil {
    log.Hooks.Add(hook)
}

Advantages of Using Logrus with Lumberjack

• Scalability: As your application grows, log files can become large. Lumberjack manages log file size and rotation, preventing disk space issues.
• Flexibility: Logrus’s structured logging and multiple format support make it adaptable to various logging requirements.
• Efficiency: Automated log rotation and compression reduce manual intervention, saving time and effort in log management.

Log Storage Modes with Lumberjack

Lumberjack offers various modes for storing logs:

• Size-Based Rotation: Rotate log files when they reach a certain size.
• Time-Based Rotation: Rotate logs based on a specific time interval.
• Backup Retention: Keep a specified number of old log files.
• Compression: Compress old logs to save disk space.

Conclusion

Using Logrus with Lumberjack provides a powerful logging solution for Go applications. Logrus offers rich features for creating structured logs, while Lumberjack ensures efficient log file management through rotation and compression. Together, they help maintain a scalable, flexible, and efficient logging system that meets the needs of growing applications.

Pulumi is a modern infrastructure as code platform that enables developers to define, deploy, and manage cloud infrastructure using familiar programming languages like TypeScript, Python, Go, and C#. Unlike traditional templating or DSL-based approaches, Pulumi allows you to leverage the full power of general-purpose languages, including their ecosystems, tools, and best practices, to manage your infrastructure.

Adopting best practices when using Pulumi is crucial to ensure that your infrastructure is reliable, secure, and maintainable. These best practices help you avoid common pitfalls, enhance collaboration among team members, and streamline the deployment process. By following these guidelines, you can build robust, scalable infrastructure that meets the needs of your applications and services while minimizing risks and operational overhead.

1 — Version Control Infrastructure as Code (IaC) Store your Pulumi code in a version control system like Git. This enables team collaboration and tracks changes over time.

2 — Organize Your Project Structure Keep your Pulumi project organized by separating configuration files, code, and resources. Use directories and modules to logically group resources.

3 — Use Configurations for Environment-Specific Values Use Pulumi configuration files to manage environment-specific values (e.g., dev, staging, production). This ensures that your code is environment-agnostic.

4 — Secure Sensitive Information Store sensitive information, such as API keys and passwords, in Pulumi’s secrets management system. Avoid hardcoding sensitive data in your codebase.

5 — Automate Deployments Integrate Pulumi with your CI/CD pipeline to automate deployments. This ensures consistency and reduces the risk of human error during manual deployments.

6 — Write Idempotent Code Ensure your Pulumi code is idempotent, meaning multiple runs should produce the same result. This helps in achieving predictable infrastructure states.

7 — Use Proper Resource Naming Conventions Adopt a consistent naming convention for resources to make them easily identifiable and manageable.

8 — Manage State Securely Use a secure and reliable backend for Pulumi state management, such as the Pulumi Service, AWS S3, or Azure Blob Storage. Ensure state files are encrypted and access-controlled.

9 — Test Your Infrastructure Code Write unit and integration tests for your Pulumi code. Use tools like pulumi/pulumictl to test your infrastructure changes before applying them.

10 — Document Your Code and Processes Maintain clear and comprehensive documentation for your Pulumi code and deployment processes. This aids team onboarding and knowledge sharing.

11 — Leverage Pulumi’s Multi-Language Support Use the language you’re most comfortable with, whether it’s TypeScript, Python, Go, or C#. Pulumi supports multiple languages, allowing you to leverage existing skills.

12 — Monitor and Manage Resource Drift Regularly check for drift between your Pulumi code and the actual deployed resources. Use tools and processes to detect and correct drifts.

13 — Implement Rollbacks and Recovery Plans Plan for rollbacks and recovery. Ensure you have strategies to revert changes in case of failures during deployments.

14 — Keep Dependencies Up-to-Date Regularly update your Pulumi CLI, libraries, and dependencies to benefit from the latest features, improvements, and security patches.

15 — Collaborate with Stacks Use Pulumi stacks to manage different environments and share configurations across teams. Stacks allow you to isolate environments and collaborate effectively.

Following these best practices helps ensure your Pulumi projects are maintainable, secure, and scalable, providing a robust foundation for your infrastructure as code endeavors.

Go, also known as Golang, is a powerful and efficient programming language that has gained immense popularity in recent years. Whether you’re building web applications, microservices, or command-line tools, Go has a rich ecosystem of libraries that can enhance your productivity and streamline your development process. Here are ten essential libraries to consider for your next Go project:

Gin

• Description: Gin is a lightweight and high-performance web framework for Go. It is known for its speed and minimalism, making it ideal for building RESTful APIs.
• Features: Middleware support, JSON validation, routing, error management, and rendering

import "github.com/gin-gonic/gin"

func main() {
    r := gin.Default()
    r.GET("/ping", func(c *gin.Context) {
        c.JSON(200, gin.H{
            "message": "pong",
        })
    })
    r.Run()
}

Echo

• Description: Echo is another fast and minimalist web framework for Go, focusing on simplicity and performance.
• Features: Routing, middleware, data binding, and validation.

import (
    "net/http"
    "github.com/labstack/echo/v4"
)

func main() {
    e := echo.New()
    e.GET("/", func(c echo.Context) error {
        return c.String(http.StatusOK, "Hello, Echo!")
    })
    e.Start(":8080")
}

GORM

• Description: GORM is an ORM (Object-Relational Mapping) library for Go that simplifies database interactions.
• Features: Associations, hooks, preloading, transactions, and more.

import (
    "gorm.io/driver/sqlite"
    "gorm.io/gorm"
)

type Product struct {
    gorm.Model
    Code  string
    Price uint
}

func main() {
    db, err := gorm.Open(sqlite.Open("test.db"), &gorm.Config{})
    if err != nil {
        panic("failed to connect database")
    }
    db.AutoMigrate(&Product{})
    db.Create(&Product{Code: "D42", Price: 100})
}

Viper

• Description: Viper is a comprehensive library for configuration management in Go applications.
• Features: Reading from JSON, TOML, YAML, HCL, envfile and Java properties config files, remote configuration systems, and more.

import (
    "fmt"
    "github.com/spf13/viper"
)

func main() {
    viper.SetConfigName("config")
    viper.AddConfigPath(".")
    err := viper.ReadInConfig()
    if err != nil {
        panic(fmt.Errorf("Fatal error config file: %w \n", err))
    }
    fmt.Println("Config value:", viper.GetString("key"))
}

Cobra

• Description: Cobra is a library for creating powerful and modern CLI applications.
• Features: Command line flag parsing, command aliases, and extensive documentation support.

import (
    "github.com/spf13/cobra"
)

func main() {
    var rootCmd = &cobra.Command{
        Use:   "app",
        Short: "A brief description of your application",
        Run: func(cmd *cobra.Command, args []string) {
            fmt.Println("Hello, Cobra!")
        },
    }
    rootCmd.Execute()
}

Go Kit

• Description: Go Kit provides a comprehensive toolkit for building microservices.
• Features: Transport-agnostic RPC, service discovery, load balancing, tracing, and more.

import (
    "github.com/go-kit/kit/endpoint"
    "github.com/go-kit/kit/log"
    "github.com/go-kit/kit/transport/http"
)

func main() {
    logger := log.NewLogfmtLogger(os.Stderr)
    endpoint := func(ctx context.Context, request interface{}) (interface{}, error) {
        return "Hello, Go Kit!", nil
    }
    handler := http.NewServer(endpoint, decodeRequest, encodeResponse)
    http.ListenAndServe(":8080", handler)
}

func decodeRequest(_ context.Context, r *http.Request) (interface{}, error) {
    return nil, nil
}

func encodeResponse(ctx context.Context, w http.ResponseWriter, response interface{}) error {
    w.Header().Set("Content-Type", "application/json")
    return json.NewEncoder(w).Encode(response)
}

Logrus

• Description: Logrus is a structured logging library for Go that provides leveled logging.
• Features: Hooks, formatters (JSON, text), and extensive field logging.

import (
    "github.com/sirupsen/logrus"
)

func main() {
    var log = logrus.New()
    log.Out = os.Stdout
    log.WithFields(logrus.Fields{
        "animal": "walrus",
        "size":   10,
    }).Info("A walrus appears")
}

Migrate

• Description: Migrate provides a simple way to manage database migrations.
• Features: CLI support, migration files in various formats, and database driver support.

import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

func main() {
    m, err := migrate.New(
        "file://path/to/migrations",
        "postgres://localhost:5432/database?sslmode=disable")
    if err != nil {
        log.Fatal(err)
    }
    m.Up()
}

JWT

• Description: JWT is a Go library for working with JSON Web Tokens.
• Features: Token creation, parsing, and validation.

import (
    "github.com/dgrijalva/jwt-go"
    "time"
)

func main() {
    token := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.MapClaims{
        "name": "John Doe",
        "exp":  time.Now().Add(time.Hour * 72).Unix(),
    })
    tokenString, err := token.SignedString([]byte("secret"))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(tokenString)
}

Swagger

• Description: Swagger is a tool to automatically generate and serve documentation for your Go API.
• Features: Integration with Go HTTP servers, customizable documentation, and OpenAPI support.

import (
    "github.com/swaggo/swag/example/basic/docs"
    "github.com/gin-gonic/gin"
)

func main() {
    r := gin.New()
    docs.SwaggerInfo.BasePath = "/api/v1"
    r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
    r.Run()
}

These libraries cover a wide range of functionalities, from web frameworks and ORMs to configuration management and logging. By incorporating these tools into your Go projects, you can significantly enhance your development workflow and create robust, scalable applications.

Introduction

In the rapidly evolving landscape of Machine Learning (ML) and Artificial Intelligence (AI), the ability to efficiently develop, deploy, and maintain ML models is crucial. This process, known as MLOps (Machine Learning Operations), combines the principles of DevOps with ML to streamline the end-to-end lifecycle of ML models. Google Cloud Platform (GCP) offers a suite of services that facilitate each stage of this lifecycle. In this article, we will explore a detailed MLOps workflow using GCP services, illustrated by a comprehensive diagram.

1. Data Ingestion

Services: Cloud Storage, BigQuery, Pub/Sub
Description: The first step in any ML project is to collect and store raw data from various sources. GCP provides several tools to facilitate this process:

• Cloud Storage: Used for storing large volumes of unstructured data.
• BigQuery: A highly scalable data warehouse for storing structured data.
• Pub/Sub: A messaging service for ingesting streaming data in real-time.

2. Data Processing and Preparation

Services: Dataflow, Dataproc, BigQuery
Description: Once the data is ingested, it needs to be cleaned and transformed into a suitable format for analysis and model training. GCP offers robust tools for data processing:

• Dataflow: A fully managed service for stream and batch processing.
• Dataproc: A fast, easy-to-use, fully managed cloud service for running Apache Spark and Apache Hadoop clusters.
• BigQuery: Used for executing complex queries on large datasets.

3. Data Exploration and Visualization

Services: Datalab, Data Studio, AI Platform Notebooks
Description: Data exploration and visualization are essential for understanding the data and generating insights. GCP provides the following tools:

• Datalab: An interactive tool for exploring, analyzing, and visualizing data.
• Data Studio: A powerful dashboarding and reporting tool.
• AI Platform Notebooks: Jupyter notebooks that are managed, scalable, and integrate seamlessly with other GCP services.

4. Model Training

Services: AI Platform Training, Compute Engine, TensorFlow
Description: Training the ML model is a critical step that involves developing algorithms and feeding the prepared data into these algorithms to create predictive models. GCP supports this with:

• AI Platform Training: A managed service for training ML models at scale.
• Compute Engine: Provides the computational resources needed for intensive training tasks.
• TensorFlow: An open-source library for developing ML models.

5. Model Evaluation and Tuning

Services: AI Platform Training, TensorBoard
Description: Evaluating the model’s performance and tuning its parameters are necessary to ensure accuracy and efficiency. GCP facilitates this with:

• AI Platform Training: Used for running and managing training jobs.
• TensorBoard: A suite of visualization tools to inspect and understand the training process.

6. Model Deployment

Services: AI Platform Prediction, Kubernetes Engine (GKE), Cloud Functions
Description: Deploying the trained model to a production environment where it can make real-time predictions is the next step. GCP offers several deployment options:

• AI Platform Prediction: A managed service for serving ML models in production.
• Kubernetes Engine (GKE): A managed Kubernetes service for deploying containerized applications.
• Cloud Functions: A serverless environment to execute functions in response to events.

7. Model Monitoring and Management

Services: Stackdriver, AI Platform, BigQuery
Description: Continuous monitoring and management of the deployed model are crucial for maintaining performance and making necessary adjustments. GCP provides tools for:

• Stackdriver: Monitoring, logging, and diagnostics for applications on GCP.
• AI Platform: Provides model versioning and management capabilities.
• BigQuery: Used for analyzing logs and performance data.

8. Continuous Integration and Continuous Deployment (CI/CD)

Services: Cloud Build, Cloud Source Repositories, Cloud Scheduler
Description: Automating the lifecycle of ML model development and deployment is key to ensuring efficiency and consistency. GCP supports CI/CD with:

• Cloud Build: A continuous integration service that automates the build and test process.
• Cloud Source Repositories: A managed source control service for Git repositories.
• Cloud Scheduler: A fully managed cron job service for scheduling tasks.

Conclusion

Implementing an MLOps workflow using GCP can significantly enhance the efficiency and reliability of ML model development and deployment. By leveraging GCP’s comprehensive suite of services, organizations can streamline their ML pipelines, from data ingestion to continuous deployment and monitoring. This holistic approach ensures that ML models are not only developed quickly but also maintained and updated effectively, leading to better performance and more accurate predictions.

Embrace the power of MLOps with GCP and transform your ML projects into robust, scalable solutions that drive real business value.

Introduction

In the era of big data, recommendation systems have become a cornerstone for enhancing user experience by providing personalized content and suggestions. Deep learning models, known for their ability to capture complex patterns in data, are increasingly being used to power these systems. However, deploying such models in a production environment requires a robust and efficient backend. This is where Go (Golang) shines, providing performance and scalability.

In this article, we will explore how to integrate a deep learning-based recommendation system, trained with TensorFlow in Python, into a Go application. We will walk through the steps of exporting the trained model, setting up a Go service to serve the model via API, and calling the Python script from Go to make predictions.

Key Steps and Implementation

1. Exporting the Trained Model in Python

First, we train a recommendation system using TensorFlow and save the trained model. This step involves typical machine learning workflow steps such as data preprocessing, model training, and evaluation.

Here’s a snippet showing how to save the trained model:

import tensorflow as tf

# Assuming `model` is your trained model
model.save('recommendation_model.h5')

2. Setting Up the Go Service

We will create a Go service that can handle API requests and interact with the Python model for predictions.

Installing Dependencies

We use the Gorilla Mux package for routing and the exec package to run the Python script:

go get -u github.com/gorilla/mux

Go Service Code

Below is the Go code for setting up a basic HTTP server that interacts with our Python model:

package main

import (
    "encoding/json"
    "log"
    "net/http"
    "os/exec"
    "strconv"

    "github.com/gorilla/mux"
)

type PredictionRequest struct {
    UserID int `json:"user_id"`
    TopN   int `json:"top_n"`
}

type PredictionResponse struct {
    Recommendations []string `json:"recommendations"`
}

func getRecommendations(userID int, topN int) ([]string, error) {
    cmd := exec.Command("python3", "predict.py", strconv.Itoa(userID), strconv.Itoa(topN))
    output, err := cmd.Output()
    if err != nil {
        return nil, err
    }

    var recommendations []string
    err = json.Unmarshal(output, &recommendations)
    if err != nil {
        return nil, err
    }

    return recommendations, nil
}

func recommendHandler(w http.ResponseWriter, r *http.Request) {
    var req PredictionRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    recommendations, err := getRecommendations(req.UserID, req.TopN)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    res := PredictionResponse{Recommendations: recommendations}
    json.NewEncoder(w).Encode(res)
}

func main() {
    r := mux.NewRouter()
    r.HandleFunc("/recommend", recommendHandler).Methods("POST")
    log.Fatal(http.ListenAndServe(":8000", r))
}

3. Python Script for Predictions

The Python script (predict.py) will load the saved model and make predictions based on the input received from the Go service.

import sys
import json
import pandas as pd
import numpy as np
from tensorflow.keras.models import load_model
from sklearn.preprocessing import LabelEncoder, StandardScaler

# Load the model
model = load_model('recommendation_model.h5')

# Load and preprocess the data
df = pd.read_csv('dataset.csv')

label_encoders = {}
for column in ['Branch', 'City', 'Customer type', 'Gender', 'Payment']:
    le = LabelEncoder()
    df[column] = le.fit_transform(df[column])
    label_encoders[column] = le

scaler = StandardScaler()
df[['Unit price', 'Quantity', 'Tax 5%', 'Total', 'cogs', 'gross margin percentage', 'gross income', 'Rating']] = scaler.fit_transform(
    df[['Unit price', 'Quantity', 'Tax 5%', 'Total', 'cogs', 'gross margin percentage', 'gross income', 'Rating']])

le_product_line = LabelEncoder()
df['Product line'] = le_product_line.fit_transform(df['Product line'])
label_encoders['Product line'] = le_product_line

df = df.drop(['Invoice ID', 'Date', 'Time'], axis=1)

def recommend_items(user_id, top_n):
    user_data = df.iloc[[user_id]]
    predictions = model.predict(user_data.drop('Product line', axis=1).astype(float))
    top_items = predictions.argsort()[0][-top_n:][::-1]
    return [label_encoders['Product line'].inverse_transform([i])[0] for i in top_items]

user_id = int(sys.argv[1])
top_n = int(sys.argv[2])
recommendations = recommend_items(user_id, top_n)
print(json.dumps(recommendations))

Running the Service

Ensure the dataset and the saved model are in the correct directory, and run the Go service:

go run main.go

Testing the API

To test the API, you can use a tool like curl:

curl -X POST -H "Content-Type: application/json" -d '{"user_id": 0, "top_n": 3}' http://localhost:8000/recommend

Benefits of Using Go for Machine Learning APIs

• Performance: Go is known for its performance and efficiency. It compiles to native code and has low overhead, making it ideal for building high-performance APIs.
• Concurrency: Go’s goroutines make it easy to handle multiple requests concurrently, which is crucial for a recommendation system serving many users simultaneously.
• Simplicity: Go’s simplicity and clean syntax reduce the complexity of building and maintaining APIs, ensuring that developers can quickly deploy and scale their applications.
• Scalability: Go is designed with scalability in mind. It can handle a large number of connections, making it suitable for high-traffic applications.

Conclusion

Integrating a deep learning-based recommendation system with a Go backend leverages the strengths of both technologies. While TensorFlow provides the power and flexibility of deep learning, Go offers the performance and scalability needed for production-grade APIs. This combination ensures that your recommendation system is both intelligent and efficient, capable of delivering personalized experiences at scale.

By following the steps outlined in this article, you can create a robust recommendation system that combines the best of machine learning and backend engineering. Whether you’re working on a small project or a large-scale application, this approach will help you deliver high-quality recommendations to your users.

Next Steps

In the next steps we will aproach the deploy stage with Pulumi and GCP. Stay close to see the news.

Part One — Building a Deep Learning-Based Recommendation System with TensorFlow and Go — Part 1 | by Daniel Gaias Malagurti | Jun, 2024 | Medium

Introduction

In recent years, deep learning has revolutionized various fields, including natural language processing, image recognition, and recommendation systems. Unlike traditional machine learning algorithms, deep learning models can automatically extract features from raw data, making them highly effective for complex tasks. In this article, we will explore how to build a deep learning-based recommendation system using TensorFlow. We will walk through the process of loading and pre-processing the dataset, creating and training a deep learning model, and making item recommendations.

Dataset Overview

The dataset used in this project contains transaction records from a retail store. Each record includes information such as the invoice ID, branch, city, customer type, gender, product line, unit price, quantity, tax, total amount, date, time, payment method, cost of goods sold (cogs), gross margin percentage, gross income, and customer rating. Here is a sample of the dataset:

| Invoice ID | Branch | City      | Customer type | Gender | Product line          | Unit price | Quantity | Tax 5% | Total  | Date     | Time  | Payment    | cogs  | gross margin percentage | gross income | Rating |
|------------|--------|-----------|---------------|--------|-----------------------|------------|----------|--------|--------|----------|-------|------------|-------|-------------------------|---------------|--------|
| 750-67-8428| A      | Yangon    | Member        | Female | Health and beauty     | 74.69      | 7        | 26.1415| 548.9715| 1/5/2019 | 13:08 | Ewallet    | 522.83| 4.761905                | 26.1415       | 9.1    |
| 226-31-3081| C      | Naypyitaw | Normal        | Female | Electronic accessories| 15.28      | 5        | 3.8200 | 80.2200 | 3/8/2019 | 10:29 | Cash       | 76.40 | 4.761905                | 3.8200        | 9.6    |
| 631-41-3108| A      | Yangon    | Normal        | Male   | Home and lifestyle    | 46.33      | 7        | 16.2155| 340.5255| 3/3/2019 | 13:23 | Credit card| 324.31| 4.761905                | 16.2155       | 7.4    |
| 123-19-1176| A      | Yangon    | Member        | Male   | Health and beauty     | 58.22      | 8        | 23.2880| 489.0480| 1/27/2019| 20:33 | Ewallet    | 465.76| 4.761905                | 23.2880       | 8.4    |
| 373-73-7910| A      | Yangon    | Normal        | Male   | Sports and travel     | 86.31      | 7        | 30.2085| 634.3785| 2/8/2019 | 10:37 | Ewallet    | 604.17| 4.761905                | 30.2085       | 5.3    |

Pre-processing the Data

Before training the model, we need to pre-process the dataset to ensure it is in the correct format. This involves encoding categorical variables, normalizing numerical features, and splitting the data into training and testing sets.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import LabelEncoder, StandardScaler

# Load the dataset
df = pd.read_csv('dataset.csv')

# Encode categorical columns
label_encoders = {}
for column in ['Branch', 'City', 'Customer type', 'Gender', 'Payment']:
    le = LabelEncoder()
    df[column] = le.fit_transform(df[column])
    label_encoders[column] = le

# Normalize numerical columns
scaler = StandardScaler()
df[['Unit price', 'Quantity', 'Tax 5%', 'Total', 'cogs', 'gross margin percentage', 'gross income', 'Rating']] = scaler.fit_transform(
    df[['Unit price', 'Quantity', 'Tax 5%', 'Total', 'cogs', 'gross margin percentage', 'gross income', 'Rating']])

# Encode the target column
le_product_line = LabelEncoder()
df['Product line'] = le_product_line.fit_transform(df['Product line'])
label_encoders['Product line'] = le_product_line

# Drop unnecessary columns
df = df.drop(['Invoice ID', 'Date', 'Time'], axis=1)

# Split the data into training and testing sets
train, test = train_test_split(df, test_size=0.2, random_state=42)

Creating the Deep Learning Model

We will create a deep learning model using TensorFlow’s Keras API. The model consists of an input layer, three hidden layers with ReLU activation functions, and an output layer with a softmax activation function.

import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Dense

# Function to create the model
def create_model(input_dim, num_classes):
    inputs = Input(shape=(input_dim,))
    x = Dense(128, activation='relu')(inputs)
    x = Dense(64, activation='relu')(x)
    x = Dense(32, activation='relu')(x)
    outputs = Dense(num_classes, activation='softmax')(x)
    model = Model(inputs, outputs)
    return model

# Define model parameters
input_dim = train.shape[1] - 1  # Number of input features
num_classes = len(df['Product line'].unique())  # Number of classes in the target column

# Create the model
model = create_model(input_dim, num_classes)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Display model summary
model.summary()

Training the Model

Next, we will train the model using the training data. We will also validate the model on the test data to monitor its performance.

# Prepare training and testing data
X_train = train.drop('Product line', axis=1).astype(float)
y_train = train['Product line'].astype(int)
X_test = test.drop('Product line', axis=1).astype(float)
y_test = test['Product line'].astype(int)

# Train the model
history = model.fit(X_train, y_train, epochs=20, batch_size=32, validation_data=(X_test, y_test))

Making Recommendations

After training the model, we can use it to make item recommendations for users. The recommend_items function takes the trained model, user data, and the label encoders, and returns the top N recommended items.

# Function to recommend items
def recommend_items(model, user_data, label_encoders, top_n=3):
    predictions = model.predict(user_data)
    top_items = predictions.argsort()[0][-top_n:][::-1]
    return [label_encoders['Product line'].inverse_transform([i])[0] for i in top_items]

# Example recommendation for a specific user
user_data = X_test.iloc[0:1]  # User data (modify as needed)
recommended_items = recommend_items(model, user_data, label_encoders)
print(f"Recommended items: {recommended_items}")

Conclusion

In this article, we have built a deep learning-based recommendation system using TensorFlow. We started by pre-processing the dataset, creating a deep learning model, training the model, and finally making item recommendations. Deep learning offers powerful tools for building recommendation systems that can capture complex patterns in data, leading to more accurate and personalized recommendations.

Next steps

How to use tensorflow models with go, to improve performance

part 2 — Integrating Deep Learning-Based Recommendation Systems with Go — Part 2 | by Daniel Gaias Malagurti | Jun, 2024 | Medium

Notebook link

Google Colab

In the landscape of modern software development, microservices architecture has gained significant traction. One of the critical components in such architectures is the communication protocol between different services. gRPC, a high-performance RPC framework developed by Google, has become a popular choice for this purpose. Coupled with Go, a language known for its efficiency and simplicity, gRPC offers a robust solution for building scalable and efficient microservices. This article delves into the strengths of gRPC with Go and provides practical examples to get you started.

What is gRPC?

gRPC stands for Google Remote Procedure Call. It is an open-source framework that uses HTTP/2 for transport, Protocol Buffers (protobuf) as the interface description language, and provides features such as authentication, load balancing, and more. gRPC is designed to make communication between services more efficient and straightforward.

Why Use gRPC with Go?

1. Performance and Efficiency: gRPC uses HTTP/2, which provides multiplexing of multiple requests over a single connection, reducing latency and improving overall performance. Go, known for its minimal resource usage and high performance, complements this by providing fast execution and efficient concurrency management.
2. Strongly Typed Contracts: gRPC uses Protocol Buffers for defining service contracts. This ensures that the communication between client and server is type-safe, reducing the chances of runtime errors and making the system more robust.
3. Bi-directional Streaming: gRPC supports streaming in both directions. This is particularly useful for applications that require real-time data exchange, such as chat applications, live data feeds, and IoT.
4. Ease of Use: The combination of Go’s simplicity and gRPC’s straightforward service definitions makes the development process intuitive and efficient.

Setting Up gRPC with Go

Let’s walk through a simple example to illustrate how to set up a gRPC service in Go.

Step 1: Install Dependencies

First, install the necessary tools. You’ll need the Protocol Buffers compiler (protoc) and the Go plugins for gRPC and protobuf.

# Install Protocol Buffers compiler
brew install protobuf

# Install Go plugins for gRPC and Protocol Buffers
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Ensure the binaries are in your PATH
export PATH="$PATH:$(go env GOPATH)/bin"

Step 2: Define the Service

Create a .proto file to define your gRPC service and messages. Let's call it helloworld.proto.

syntax = "proto3";

package helloworld;

service Greeter {
  rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
  string name = 1;
}

message HelloReply {
  string message = 1;
}

Step 3: Generate Go Code

Use the protoc command to generate Go code from the .proto file.

protoc --go_out=. --go-grpc_out=. helloworld.proto

This will generate two files: helloworld.pb.go and helloworld_grpc.pb.go.

Step 4: Implement the Server

Create a Go file to implement the server. Let’s name it server.go.

package main

import (
  "context"
  "log"
  "net"

  "google.golang.org/grpc"
  pb "path/to/helloworld"
)

type server struct {
  pb.UnimplementedGreeterServer
}

func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
  return &pb.HelloReply{Message: "Hello " + in.Name}, nil
}

func main() {
  lis, err := net.Listen("tcp", ":50051")
  if err != nil {
    log.Fatalf("failed to listen: %v", err)
  }
  s := grpc.NewServer()
  pb.RegisterGreeterServer(s, &server{})
  if err := s.Serve(lis); err != nil {
    log.Fatalf("failed to serve: %v", err)
  }
}

Step 5: Implement the Client

Create another Go file to implement the client. Let’s call it client.go.

package main

import (
  "context"
  "log"
  "os"
  "time"

  "google.golang.org/grpc"
  pb "path/to/helloworld"
)

func main() {
  conn, err := grpc.Dial("localhost:50051", grpc.WithInsecure(), grpc.WithBlock())
  if err != nil {
    log.Fatalf("did not connect: %v", err)
  }
  defer conn.Close()
  c := pb.NewGreeterClient(conn)

  name := "world"
  if len(os.Args) > 1 {
    name = os.Args[1]
  }
  ctx, cancel := context.WithTimeout(context.Background(), time.Second)
  defer cancel()

  r, err := c.SayHello(ctx, &pb.HelloRequest{Name: name})
  if err != nil {
    log.Fatalf("could not greet: %v", err)
  }
  log.Printf("Greeting: %s", r.GetMessage())
}

Step 6: Run the Server and Client

Start the server:

go run server.go

In a new terminal, run the client:

go run client.go

You should see the greeting message output in the client terminal.

Conclusion

gRPC, when combined with Go, offers a powerful and efficient solution for building microservices. With features like bi-directional streaming, strong typing, and high performance, gRPC is an excellent choice for scalable and robust service communication. By following the steps outlined in this article, you can set up a basic gRPC service in Go and start leveraging the strengths of this powerful duo in your projects.

Introduction to Deep Learning

Deep learning, a subset of machine learning, has revolutionized numerous fields by enabling machines to learn from vast amounts of data and make predictions or decisions. It involves the use of neural networks with many layers (hence “deep” learning) to model complex patterns in data. This technology powers many modern applications, from voice assistants and image recognition to autonomous vehicles and recommendation systems. In this article, we will explore some of the key algorithms in deep learning, their employability, and the frameworks in which they can be found.

Key Deep Learning Algorithms

1. Convolutional Neural Networks (CNNs)

Employability: CNNs are primarily used in image processing tasks such as image recognition, object detection, and segmentation. They are also employed in video analysis and certain types of signal processing.

Frameworks:

• TensorFlow
• PyTorch
• Keras

2. Recurrent Neural Networks (RNNs)

Employability: RNNs are designed for sequential data and are widely used in natural language processing (NLP) tasks like language modeling, machine translation, and speech recognition. They are also used in time series prediction and music composition.

Frameworks:

• TensorFlow
• PyTorch
• Keras

3. Long Short-Term Memory Networks (LSTMs)

Employability: LSTMs are a special kind of RNN capable of learning long-term dependencies. They are highly effective in NLP tasks, such as text generation, sentiment analysis, and speech recognition, as well as in time series forecasting.

Frameworks:

• TensorFlow
• PyTorch
• Keras

4. Generative Adversarial Networks (GANs)

Employability: GANs are used to generate synthetic data that resembles real data. Applications include image generation, video generation, and data augmentation. GANs are also being explored in fields such as healthcare for generating medical images and in finance for creating synthetic financial data.

Frameworks:

• TensorFlow
• PyTorch

5. Transformer Networks

Employability: Transformers are now the backbone of many state-of-the-art NLP models, including BERT and GPT-3. They excel in tasks such as language translation, text summarization, and question answering.

Frameworks:

• TensorFlow
• PyTorch
• Hugging Face Transformers

6. Autoencoders

Employability: Autoencoders are used for tasks such as data compression, denoising, and dimensionality reduction. They are also employed in anomaly detection and in generating new data samples.

Frameworks:

• TensorFlow
• PyTorch
• Keras

7. Reinforcement Learning (RL)

Employability: RL algorithms are used in decision-making tasks where an agent learns to perform actions in an environment to maximize cumulative reward. Applications include game playing (e.g., AlphaGo), robotics, and autonomous driving.

Frameworks:

• TensorFlow (TF-Agents)
• PyTorch
• OpenAI Gym

Conclusion

Deep learning has transformed the landscape of artificial intelligence by providing powerful tools to learn from data and perform complex tasks. Each algorithm has its unique strengths and applications, making it crucial to choose the right one based on the specific requirements of a project. Frameworks like TensorFlow, PyTorch, and Keras provide robust environments to implement and experiment with these algorithms, enabling both researchers and practitioners to push the boundaries of what is possible with deep learning.

Managing Git hooks can be a tedious task, especially when working on large-scale projects with multiple contributors. Enter Left Hook, a powerful and efficient tool written in Go that simplifies the process of managing Git hooks. Whether you’re looking to enforce coding standards, run tests, or automate tasks, Left Hook has got you covered.

What is Left Hook?

Left Hook is an open-source Git hook manager that provides a fast and flexible way to manage and execute Git hooks. It allows developers to write hooks in any language and execute them efficiently, helping teams maintain code quality and streamline their workflows.

Key Features

• Speed and Efficiency: Left Hook is optimized for performance, ensuring that hooks run quickly without slowing down your workflow.
• Easy Configuration: With simple YAML configuration files, setting up and managing hooks is straightforward and intuitive.
• Language Agnostic: Write hooks in any language you prefer — Left Hook will handle the execution.
• Parallel Execution: Run multiple hooks in parallel to save time during commit and push processes.

Installing Left Hook

Getting started with Left Hook is simple. Follow these steps to install and configure it in your Go project:

1. Install Left Hook

• You can install Left Hook using go get or a package manager like Homebrew (for macOS users):

go install github.com/arkweid/lefthook@latest

2. Initialize Left Hook in Your Repository

• After installation, initialize Left Hook in your Git repository:

lefthook install

3. Configure Git Hooks

• Create a .lefthook directory in your project root and add YAML files to configure your hooks. Here’s an example configuration for running lint checks and tests before a commit:

pre-commit:
  scripts:
    lint:
      runner: golangci-lint run
    test:
      runner: go test ./...

4. Automate Push Hooks

• You can also configure hooks to run before pushing code to the repository. For instance, to ensure lint checks and tests pass before a push, add the following configuration:

pre-push:
  scripts:
    lint:
      runner: golangci-lint run
    test:
      runner: go test ./...

Enhancing Your Go Project with Left Hook

Let’s put Left Hook into action. Suppose you have a Go project and you want to ensure that all code meets the linting standards and passes the tests before pushing to the repository. Here’s a step-by-step guide to setting this up:

1. Install Go Tools

• Ensure you have golangci-lint installed for linting and Go’s testing tools:

go install github.com/golangci/golangci-lint/cmd/golangci-lint@latest

2. Add Configuration Files

• Create the necessary configuration files in the .lefthook directory. For example, create .lefthook/pre-push.yaml with the following content:

pre-push:
  scripts:
    lint:
      runner: golangci-lint run
    test:
      runner: go test ./...

3. Test Your Setup

• Make some changes to your code and try committing and pushing. Left Hook will automatically run the specified scripts, ensuring your code adheres to the defined standards.

Conclusion

By integrating Left Hook into your Go projects, you can automate essential tasks, maintain high code quality, and streamline your development workflow. Its ease of use, speed, and flexibility make it an indispensable tool for any development team.

Start using Left Hook today and experience the benefits of automated Git hooks in your projects.

While Postman is widely recognized as a powerful tool for API development and testing, it also offers significant capabilities for API security testing. By crafting specific HTTP requests, tech leads and developers can probe API endpoints for vulnerabilities, ensuring robust error handling and effective authentication mechanisms. This article will explore how to use Postman for security testing, specifically focusing on a Go API with known vulnerabilities.

Overview of Postman for Security Testing Postman allows users to send various types of HTTP requests to API endpoints and observe the responses. This capability is crucial for security testing as it can help identify how well the API handles unexpected input, manages authentication, and more.

Step-by-Step Guide to Using Postman for Testing the Go Vulnerable API

Step 1: Set Up Postman

1. Download and Install Postman

• Visit the Postman website and download the version suitable for your operating system.
• Install and open Postman.

Step 2: Configure the API Requests

1. Create a New Collection

• In Postman, create a new collection for your API tests. This helps organize your requests and responses.

2. Add Requests to the Collection

• For each endpoint in your Go API, create a corresponding request in Postman. Here’s how to set up a few key requests:

GET /users

• Method: GET
• URL: http://localhost:8080/users
• Purpose: Tests how the API lists users and checks for unauthorized data leakage.

POST /users

• Method: POST
• URL: http://localhost:8080/users
• Body: Set to raw JSON and input {"name": "<script>alert('XSS')</script>", "email": "test@example.com"}
• Purpose: Checks for Cross-Site Scripting (XSS) vulnerabilities and how the API handles unexpected input.

GET /users/:id

• Method: GET
• URL: http://localhost:8080/users/1 OR 1=1
• Purpose: Attempts SQL injection to see if the API improperly handles input leading to potential data breaches.

Step 3: Send Requests and Analyze Responses

1. Execute Each Request

• Run each request in the collection and observe the responses.
• Look for unexpected status codes, unauthorized data exposure, error messages that reveal too much information, or any signs of the API mishandling the request.

2. Document Findings

• Record any vulnerabilities or issues discovered during testing. Note the conditions under which they occur and potential impacts.

Step 4: Remediate and Retest

1. Address Vulnerabilities

• Based on your findings, make necessary code adjustments in your Go API to mitigate the vulnerabilities.
• For example, implement proper input validation to protect against SQL injection and XSS.

2. Retest the Endpoints

• After making changes, retest the endpoints using the same Postman requests to ensure that the vulnerabilities have been effectively resolved.

Conclusion Using Postman for security testing is a practical approach that leverages its request crafting and response handling capabilities to uncover potential security weaknesses in APIs. By methodically testing each endpoint and analyzing the responses, developers can significantly enhance the security of their APIs. This proactive measure is crucial in developing secure applications that can resist common attacks and protect sensitive data.

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Venture | Cubed
• Tired of blogging platforms that force you to deal with algorithmic content? Try Differ
• More content at Stackademic.com

Enhancing Web API Security: Essential Open-Source Tools for Tech Leads: Part 6— Postman was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.