To avoid confusion, it is best to mention a little something about terminology:

Authentication: when user sign in without any privileges
Authorization: we assign the user privileges (write/read)
Client: usually is referred to an API or website that is trying to use the IDP

Keep in mind that my code does not have to be the best of the best, and you may be able to do better. I demonstrated how to easily integrate this technology into your tech stack.

if you are just looking for the full code:
https://github.com/BitR13x/idp_provider

So what is that?

Have you ever heard of Single Sign-On (SSO), which uses Google, GitHub, and other services? That is essentially what OAuth and OpenID Connect are.

OAuth is an industry standard for authorizing a user to access third-party software. In other words, the user does not have to sign in several times with the same password to utilize each service.

This diagram depicts how the login is done in a basic manner; we are attempting to redirect authentication to (identity) provider.

Difference between OAuth vs OIDC

OAuth 2.0 (Authorization): Handles “What can you do?”

• You do not care about the user it self, you only care about what they can do.

OIDC (Authentication): Handles “Who are you?”

• OIDC is extended OAuth, you care about the user, who is it?
• Authentication + Authorization

How it really works?

We will start from the login button:

1) Redirect

• The process begins when a user clicks “Login.”
• Your application (client) doesn’t ask for a password. Instead, it redirects the user’s browser away from site to the OpenID Connect Provider (IdP).
• User will login as usual

2) Callback

• Once the user successfully logs in and approves the request
• The IdP redirects the browser to a specific URL on your server — usually defined as /callback.
• Attached to this URL is a short-lived (This code is not a key)

3) Code for Token

• The application (client) sends code to the IdP and IdP sends back access token.

4) Access Token

• With this token you can access the API’s

Perhaps you are wondering why we do not return the access token right away?

The difficulty is that you need to notify the program (client) that you have completed login, which is why the callback is required.

Why not append the access token directly to the URL? It is possible, but it is considered a security risk because it can be spoof, which is why an AuthCode is issued that is only valid for a certain time.

Options for implementation

A few words on current options:

• SaaS (Auth0, Okta, AWS Cognito)
• Open Source (Keycloak, Ory Hydra)

Access tokens:

Opaque Tokens — “Reference Tokens”

• The string is simply a pointer to a record in the database.

Most providers give JWTs (JSON Web Tokens), which is why we will concentrate on them.

Here is an example of the token_id sent to the client:

{
  "iss": "https://idp.example.com",
  "sub": "user_id",
  "aud": "client",
  "exp": 1735689600,
  "iat": 1735686000,
  "nbf": 1735686000,
  "jti/nonce": "e9c1a2f4"
}

Jti, or nonce, protects against replay attacks (re-use of the same token); nonce is used to identify each token and is commonly used as a “kill switch”.

• exp (expiration)
• iat (issued at)
• nbf (not valid before)

Login Page

One option for smaller infrastructures is to build a login directly on the client and use the IdP as a “Auth provider.” This is less scalable than the standard approach of building a single login page for the IdP and redirecting the client to it.

Implementation of IdP in elysia

We will be focusing mostly on IdP endpoints because there are numerous methods for implementing database systems or deployments, among other things.

To begin, it is good practice to make explanation route:

    .get('/.well-known/openid-configuration', () => {
        const issuer = config.oidc.issuer;
        return {
            issuer: issuer,
            authorization_endpoint: `${issuer}/authorize`,
            token_endpoint: `${issuer}/token`,
            userinfo_endpoint: `${issuer}/userinfo`,
            jwks_uri: `${issuer}/.well-known/jwks.json`,
            id_token_signing_alg_values_supported: ['RS256'],
            scopes_supported: ['openid', 'default']
        };
    })

This will introduce our routes, letting the client know where to authorize and obtain tokens.

Our goal will be implement these routes.

We’ll need to begin with:

Token implementation

JWT tokens use a variety of transportation algorithms, most notably HS256 and RS256.

HS256 is symmetrical and uses a simple secret to verify and decode, making it perfect for application authentication (which we will use later).

RS256, on the other hand, uses public/private keys, which are required for cross-application verification; we simply generate these keys and disclose the public key.

# generating private/public key
openssl genpkey -algorithm RSA -out private.pem -pkeyopt rsa_keygen_bits:2048
openssl rsa -pubout -in private.pem -out public.pem

import * as jose from 'jose';
import { readFileSync } from 'fs';
import { join } from 'path';
import { config } from '../config';

let privateKey: jose.KeyObject;
let publicKey: jose.KeyObject;
let jwks: { keys: jose.JWK[] };

export async const initializeKeys = () => {
    const privateKeyPath = join(import.meta.dir, '../config/private.pem');
    const publicKeyPath = join(import.meta.dir, '../config/public.pem');

    const privateKeyPEM = readFileSync(privateKeyPath, 'utf8');
    const publicKeyPEM = readFileSync(publicKeyPath, 'utf8');

    privateKey = await jose.importPKCS8(privateKeyPEM, 'RS256');
    publicKey = await jose.importSPKI(publicKeyPEM, 'RS256');

    const jwk = await jose.exportJWK(publicKey);
    jwk.kid = config.oidc.jwt.kid;
    jwk.use = 'sig';
    jwk.alg = 'RS256';

    jwks = {
        keys: [jwk]
    };
};

export const getPrivateKey = () => {
    return privateKey;
};

export const getJWKS = () => {
    return jwks;
};

    .get('/.well-known/jwks.json', () => {
        return getJWKS();
    })

With that done we need to implement the authorize route:

Authorize

Here we need to define our database tables:

There isn’t much to surprise here. All we need to do for the client is declare the redirect URLs; otherwise, another vulnerability known as open redirect would occur.

This route must generate the AuthCode and return it to the callback, but we must ensure that everything we receive is correct.

Client:

        const client = await clientRepo.findOneBy({ id: client_id });

        if (!client) {
            return status(400, "Invalid client_id");
        }
        if (!client.redirect_uris.includes(redirect_uri)) {
            return status(400, "Invalid redirect_uri");
        }

User and scope

        const requestedScopes = (scope || '').split(' ');
        const allowed = requestedScopes.every(s => client.allowed_scopes.includes(s));
        if (!allowed) return status(400, "Invalid scope");

        if (!user) {
            const currentUrl = request.url;
            return redirect(`/login?return_to=${encodeURIComponent(currentUrl)}`);
        };

And define model:

    .get('/authorize', async ({ query: { client_id, redirect_uri, scope, state, nonce }, redirect, status, user, request }) => {
        const clientRepo = AppDataSource.getRepository(Client);
        ...

        const code = uuidv4();
        await AuthCode.create({...}).save();
        ...

        // constructing url
        const redirectUrl = new URL(redirect_uri);
        redirectUrl.searchParams.set('code', code);
        if (state) redirectUrl.searchParams.set('state', state);

        return redirect(redirectUrl.toString());
    }, {
        query: t.Object({
            client_id: t.String(),
            redirect_uri: t.String(),
            scope: t.Optional(t.String()),
            state: t.Optional(t.String()),
            nonce: t.Optional(t.String())
        })
    })

Models in Elysia are checkers (specify what we require for that specific route) and can be simply connected to Swagger or Openapi (api documentation).

The state guards against Cross-Site Request Forgery (CSRF) attacks.

The client will transmit it to the IdP and then verify that it is correct.
And now that we have virtually everything, all we need to do is generate tokens.

Token generation

This route will also need to check for the proper client (client id, secret) and code that we received (if it exists, expiration date, and if matching client id).

        // ID Token (OIDC)
        const idToken = await new jose.SignJWT({
            sub: authCode.user.id,
            name: authCode.user.username,
            email: authCode.user.email,
            nonce: authCode.nonce,
            aud: client_id,
            iss: issuer,
        })
            .setProtectedHeader({ alg: 'RS256', kid: config.oidc.jwt.kid })
            .setIssuedAt()
            .setExpirationTime(config.oidc.jwt.idTokenExpiration)
            .sign(privateKey);

        // Access Token (OAuth2)
        const accessToken = await new jose.SignJWT({
            sub: authCode.user.id,
            scope: authCode.scope,
            aud: api_aud,
            iss: issuer
        })
            .setProtectedHeader({ alg: 'RS256', kid: config.oidc.jwt.kid })
            .setIssuedAt()
            .setExpirationTime(config.oidc.jwt.accessTokenExpiration)
            .sign(privateKey);

This code is the main element of the route. It generates the tokens that the API will check using our public key, therefore we must sign it with our private key and set the appropriate parameters.

The ID token is intended for the front-end, whereas the access token grants us access to APIs.

However, you may have noticed that we only received a single aud for the access token, and we would need to repeat the authentication process, you are right.

We have several approaches that we can take:

• Super Token (listed as aud)
• Separate Tokens (using the refresh token)

The verification methodology for a refresh token is the same as for code; the refresh token is generated during the phase in which you get correct code.

if (code) {
   ... // handle code
   remove code
} else if (refresh_token) {
   ... // handle refresh_token
   remove refresh_token
} else {
   return status(400, { error: "unsupported_grant_type" });
};

// generate access_token + token_id + refresh_token

For sake of simplicity we will not focus on the this problem.

And instead use this version:

    .post('/token', async ({ body: { client_id, client_secret, code }, error, request }) => {
        // Validate Client
        ...

        // Validate Code
        const authCode = await AuthCode.findOne({ 
            where: { code },
            relations: ["user", "client"]
        });
        ...

        // Generate Tokens
        ...
        await codeRepo.remove(authCode);

        return {
            access_token: accessToken,
            id_token: idToken,
            token_type: 'Bearer',
            expires_in: 7200
        };
    }, {
        body: t.Object({
            client_id: t.String(),
            client_secret: t.String(),
            code: t.String(),
            api_aud: t.String()
        })
    })

We have now received a token on our client and may interact with the API.

The following path ‘/userinfo’, however it simply provides information about the logged user from the access_token, as the name implies.

There is still so-called “database garbage”. This is addressed via Expiration Management.

DELETE FROM auth_code WHERE expiresAt < NOW

API side of IdP

We do not need to handle login/registration anymore; we can just focus on the tokens. We need to verify that the token is coming from the right source and that the signature is correct:

import { createRemoteJWKSet, jwtVerify } from 'jose'

export async function verifyAccessToken(token: string) {
  const JWKS = createRemoteJWKSet(
    new URL(`${ISSUER}/.well-known/jwks.json`)
  );

  const { payload } = await jwtVerify(token, JWKS, {
    issuer: ISSUER,
    audience: 'API-A'
  })

  return payload
};

We have now available this token:

{
  sub: user.id,
  scope: scopes,
  aud: api_aud,
  iss: issuer
}

Where issuer and audience is validated from “jose” library.

If you do not have access to these libraries, you must check the issuer and audience, and remember to get the public-key for signature verification from:

const iss_url = `${ISSUER}/.well-known/jwks.json`

We also set the protected-header during the token creation. This header contains:

• alg: check protects against “alg: None” algorithm confusion attacks.
• kid: Token is using an anticipated key.
• type: token is the expected format.

We now have the user’s ID, which we may manage as we see fit, as well as scope, which allows us to control what the user can do.

Front-end side of IdP (React example)

You must implement the ‘/callback’ that will receive the code and then send it to IdP via the ‘/token’ route.

With that, the login is completed.

It is quite basic. The only difficulty is that react uses DOM rendering, thus you need to use:

window.location.pathname === '/callback'

To decide if the current page is callback.
Redirect function that will be assigned on SSO button:

export const loginWithRedirect = () => {
    const redirectUri = window.location.origin + '/callback';
    const clientId = client_id;
    const audience = 'test_audience'; // needs to match with API
    const scope = 'default';
    const responseType = 'code';

    const authUrl = `${IDP_VHOST}/authorize?` + 
      `client_id=${encodeURIComponent(clientId)}&` +
      `redirect_uri=${encodeURIComponent(redirectUri)}&` +
      `response_type=${encodeURIComponent(responseType)}&` +
      `scope=${encodeURIComponent(scope)}&` +
      `audience=${encodeURIComponent(audience)}`;

    window.location.href = authUrl;
  }

Main logic in the callback page will be just:

const code = urlParams.get('code');
const response = await axios.post(`${IDP_VHOST}/token`, {
  client_id: client_id,
  client_secret: client_secret,
  code,
  api_aud: 'test_audience'
});
const { access_token, id_token, refresh_token } = response.data;

Benefits

• Scalable infrastructure (you can make several APIs operate together in harmony, leveraging the capabilities of frameworks)
• Single Sign-On (SSO)

Disadvantages

• Permissions require additional maintenance
• Incorrect implementation may lead to vulnerabilities.

Attack surface in OAuth/OIDC

Just a quick word on the topic of attack surface.

This topic is huge at it’s own we gone through a few of options already.

Identifying the attack surface is critical for assessing risk and determining where to hunt for vulnerabilities.

IdP can create a complex system where can be made a mistake pretty easily and it is good to know where to look for them.

Conclusion

Identity Providers built on OpenID standards play a huge role in modern digital ecosystems by delivering secure, interoperable, and user-centric authentication. By decoupling identity from individual applications.

OpenID-based IdPs reduce security risks in larger infrastucture, simplify access management, and enable seamless single sign-on across platforms. As organizations continue to adopt cloud services, APIs, and distributed architectures.

With everything said, you should be able to create your own IdP that can be used to integrate several applications and query them with a single login.

On top of that you know the most common vulnerabilies and why they can occure.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey👋

Behind google is complex system and there is lot of caching:

• Crawl + Index High-PageRank URLs
• Preprocessing via LLMs (e.g., BERT, MUM)
• Candidate Document Retrieval
• Neural Matching & RankBrain Filtering
• Ranking Function (Weighted Scoring)

And I’d like to start at the top and work my way down.

One step at a time

The first question I asked myself was how Google truly acquires the URLs, because crawling requires at least one URL and does not guarantee that you will get all URLs, and what if the missing URL has the solution you were searching for?

Google uses a variety of sources, including seed URLs (gov, edu, and media), recursive crawling (link discovery — google bot), user contribution via API, and direct ingest from partner APIs or curated sources.

If we have pages we can search for information. Keyword matching was previously employed in a variety of ways, but Google now uses AI to determine what you’re looking for, such as RankBrain, Neural Matching, and LLMs.

Crawling

Building your own bot to tour the internet is not a tough effort if you know how to use Python; all you have to do is request a URL and check for an element with a link, then do it recursively. It’s another story if you want it to be efficient and multi-threaded.

A good example of crawler is gospider

PageRank

Google’s success can be attributed to the Markov chain, which is formed into PageRank.

By “formed,” I mean that it derives from a Markov chain and the key idea behind PageRank is to measure the importance of a webpage in simple words.

The difference between them is that transition probability is calculated from links and teleportation step (don’t worry, it’s math, but not that difficult).

Transition probability is declared by outgoing links.
When we look at page “A”, we can see that page has 2 outgoing links so the transition probability row is:

** A - B - C - D - E - F
A: [0, .5, 0.5, 0, 0, 0]

For example: P(B|A) = 1/N = 1/2 = 0.5

why 0s, because the probability of getting for example to the page A is 0.

Full transition probability matrix would look like this:

     A    B    C     D    E    F
A [  0,  0.5, 0.5,   0,   0,   0 ]
B [  0,    0, 0.5, 0.5,   0,   0 ]
C [  1,    0,   0,   0,   0,   0 ]
D [  0,    0, 0.5,   0, 0.5,   0 ]
E [  0,    0, 0.5,   0,   0, 0.5 ]
F [  0,    0, 1.0,   0,   0,   0 ]

Page rank vector (π) is declared by random state, because after while it converges(reach) to a number, size of page rank vector is number of pages.

Where next iteration of rank vector is dot product (previous state and transition probability).

This is calculated until change is lower then tolerable threshold.

Teleportation

With that explained, there is one more tiny thing to perform to solve problems such as a person becoming stranded on the internet or sporadically switching pages.

During the calculating procedure, there is a random state known as the dumping factor.

P’ = αP + (1 — α)E
where:
- E = matrix where elements “1/N” (size of P)
- α = dumping factor usually “0.85”
- P = transition probability

If you are interested in seeing more or just formal explanation about PageRank I recommend this video.

Note a little bit out of context: do not get stuck at the idea of just the webpage, Markov chain is powerful thing and is also used in AI for predicting next word. If you understand PageRank then you understand Markov chain.

RankBrain

RankBrain is one of Google’s top 3 ranking factors (along with backlinks[PageRank] & content relevance,).

In high point of view you can see search like this:

Query → Query Interpretation (vectorization, context) → Search Similar → Rank → Return

RankBrain cooperates with LLM (BART or similar), RankBrain will calculate similarity to reduce number of results and forward it into the LLM to find perfect match.

To understand more and apply this knowledge, I chose to create a custom file search that uses RankBrain and LLM to match correct files or directories.

As our small RankBrain I will be using FAISS (Facebook AI Similarity Search), it is a library for fast nearest-neighbor search in large vector collections.

This library include two approaches on searching the large vector collection:

• IndexFlatIP: Exact search using dot product similarity.
• IndexHNSWFlat: Approximate search using Hierarchical Navigable Small World (HNSW) graphs with flat storage.

The main idea behind is that FlatIP is focused on accuracy and HNSWFlat is focused on speed.

next on the list is LLM, this LLM needs to be focused on ranking (giving score for a match) for this I choose CrossEncoder specially model ms-marco-MiniLM-L-6-v2 .

Applying this knowledge to create custom file search

Whole project can be found on github here.

I started with classic file retrieval and used docs to create script that look like this:

from sentence_transformers import SentenceTransformer, CrossEncoder
import faiss
import numpy as np


# Load sample documents
def read_files_from_dir(dir_path: str):
    for root, _, files in os.walk(dir_path):
        for file in files:
            yield root + "/" + file


dir_path = os.environ['HOME']
documents = [file for file in read_files_from_dir(dir_path)]

# Encode documents
encoder = SentenceTransformer('all-MiniLM-L6-v2')
doc_vectors = encoder.encode(documents, normalize_embeddings=True)

# Create FAISS index
dimension = doc_vectors.shape[1]
index = faiss.IndexFlatIP(dimension)
index.add(doc_vectors)
doc_map = {i: doc for i, doc in enumerate(documents)}


def search_docs(request):
    query_vector = encoder.encode([request.query], normalize_embeddings=True)
    D, I = index.search(np.array(query_vector), request.top_k)

    candidates = [doc_map[i] for i in I[0]]
    pairs = [(request.query, doc) for doc in candidates]
    scores = cross_encoder.predict(pairs)

    reranked_docs = [doc for _, doc in sorted(zip(scores, candidates), reverse=True)]

    return QueryResponse(query=request.query, results=reranked_docs)

print(search_docs({query: "idek attachments"}))

Reading files and directories

Quickly I realized kind of hard or impossible problem to solve “performance”, because looping home directory isn’t that good idea as I thought.

We will be looping around millions of files and we are still in python even in C/++, rust and in other high performance language it still takes around 25 seconds to loop the files and we do not only want to loop it but also encode it into vectors. Well it’s a great start I guess.

So clearly we need to be more smarter then looping though home directory we need relevance, multiprocessing and caching.

So I started with the read_directories function that will loop directories when I look back I should just recreate it in C and import it into python, but end up with a much more faster version (compared to single threaded in python) that is using multiprocessing:

def read_dirs_from_dir(dir_path: str, max_workers=8):
    """Yield directories under dir_path as soon as they are found."""
    base_path = Path(dir_path)
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futures = {executor.submit(scan_dirs, base_path): base_path}

        while futures:
            for future in as_completed(futures.keys()):
                futures.pop(future)
                try:
                    subdirs = future.result()
                except OSError:
                    # future does not exist
                    continue

                for subdir in subdirs:
                    yield str(subdir)
                    futures[executor.submit(scan_dirs, subdir)] = subdir

Then implemented with similar logic read_files function, but I wanted to add some context to them, so I read 200 bytes from the file, and if it’s a programming language file, I get a function list using regex:

def read_n_bytes_from_file(file_path: Path, n: int) -> bytes:
    try:
        with file_path.open("rb") as f:
            return f"{str(file_path)}: {f.read(n).decode('utf-8', errors='ignore')}"
    except OSError:
        # binary file (cannot decode bytes)
        return f"{str(file_path)}: ''"


def process_file(file_path: Path) -> tuple[str, str]:
    """Return (file_path, text)."""
    ext = file_path.suffix.lstrip(".").lower()
    regex = COMPILED_PATTERNS.get(ext)
    if regex:
        return str(file_path), extract_functions_from_file(file_path, regex, ext)
    else:
        return read_n_bytes_from_file(file_path, 200)

Caching, batching and indexing

Okay from this point we got some solid functions for reading files, but we will have memory issues when it comes to large lists of files. So next is caching, I end up creating sqlite3 database.

def create_connection(database_path: str = "documents.db", first_time: bool = False):
    cx = sqlite3.connect(database_path)
    cu = cx.cursor()
    cu.execute("ATTACH DATABASE 'cache.db' AS cache")
    if first_time:
        clear_cache(cu, clear_all=True)
        create_cache(cu)
    return cx, cu

definitely I didn’t planning to open connecting for every file/directory found So the next step is to break the files/directories into batches and then enter them into the database using pickle and base64; during this process, I also began indexing them in the FAISS index.

# init cache
cx, cu = create_connection(first_time=True)
cx.close()

def index_strings(encoder, dir_path: str, table: str, read_func, index, batch_size=3000, max_workers=8):
    """indexing in parallel with FAISS"""
    def save_and_clear():
        vectors = encoder.encode(batch_buffer, normalize_embeddings=True)
        index.add(np.array(vectors, dtype=np.float32))

        # cache - 3000
        cx, cu = create_connection()
        
        # just database query insert to table
        insert_cache(cx, cu, b64e(pickle.dumps(batch_buffer)), table)
        cx.close()

        batch_buffer.clear()

        # run garbage collection after batch
        gc.collect()

    batch_buffer = []

    for string in read_func(dir_path, max_workers=max_workers):
        batch_buffer.append(string)

        if len(batch_buffer) >= batch_size:
            save_and_clear()

    if batch_buffer:
        save_and_clear()

The reason for exiting the database at the start is that we are going to run a multiprocessing function, and if we get a connection in a different thread, we will run into a lot of problems, so we need to create a new connection for each thread.

I also made this function into general purpose in order to swap type of index at the fly.

Only batching

Now we could have used the rest of the code from the beginning, but hold on, we must figure out the batching from database.

It’s small normalization, but painful to be honest:

def batch_normalization(candidates, I):
    indices = []
    for i in I[0]:
        tmp = int((i // BATCH_SIZE) + 1)
        if not tmp in indices:
            indices.append(tmp)

    candidates = get_docs_by_ids(cu, indices, "cache.directories")
    valid_candidates = []
    for c, index in zip(candidates, indices):
        for i in I[0]:
            if int((i // BATCH_SIZE) + 1) == index:
                candidate = pickle.loads(b64d(c.encode("utf-8")))[i % BATCH_SIZE]
                valid_candidates.append(candidate)

    return valid_candidates

The problem here is that database starts from index 1 instead of 0 and we need to find the correct batch using:

# F_index from FAISS starts from 0
F_index // BATCH_SIZE + 1

BATCH_SIZE # just our batch size

# this will get from the n file/directory
i % BATCH_SIZE
# i = index of the array saved that's why not again +1

# F_index = 4532, BATCH_SIZE = 2000
# to query: 4532 // 2000 (+1) = 2 (+ 1) we get correct batch
# then correct position [532] = [4532 % 2000]

At the end I’ve wrapped it into web interface:

Conclusion and final words

Still it’s far from good optimization, because even 2411 directories took 13 seconds, but it was interesting experience.

Some potential upgrades include rewriting into rust or just a few functions into C, and to not loop without relevance for example each folder is known for some purpose (.git, node_moduls, vendor) and git repositories have usually README.md, the name can tell you whether it is worth continuing or not.

I was planning to also use summaries from models to add more context, but even looping directories took a lot of time so it’s just left it there and not used.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey👋.

Code in this article is publicly available right here:

GitHub - BitR13x/AI-WAF: Web Firewall that is powered by AI and signatures

So theory comes first, what is firewall?

It is usually a hardware or software-based system which monitors all incoming and outgoing traffic and, based on a defined set of security rules, it accepts, rejects, or drops that specific traffic. Web application firewall (WAF) is simply a firewall for web application.

In order to secure the internal network from unauthorized traffic, we need a Firewall.

How about the IPS/IDS?

Intrusion Detection System (IDS): Looking for the signature of known attack types or detecting activity that deviates from a prescribed normal and then report it.

Intrusion Prevention System (IPS): Basically the same, but it can block the packet/request.

Example of an open-source IPS/IDS: Suricata

Interesting sources related to IPS/IDS:
Suricata cheetsheet
nDIP — Deep Packet Inspection
Intrusion-Detection-CICIDS2017 — in-depth analysis of the CICIDS2017 dataset

Comparison IPS/IDS and firewall:

The differences are on thin ice, so I will not go deeper, because you could argue that the WAF or iptables can detect attacks and block them, meaning it would be something more complex than just a firewall.

Let’s move on, implementation of iptables?

By combining iptables and Python, you can create a custom intrusion detection and prevention system (IDS/IPS) that can intercept network packets before they reach the target service.

For example, using libraries such as NFQueue or Scapy in Python, you can:

• Analyze packets in real time
• Detect suspicious patterns
• Perform an automated action (e.g. dropping a packet or blocking an IP address).

For example, here is a python script that blocks all IP addresses that are not in whitelist.txt:

def process_packet(packet):
    scapy_packet = IP(packet.get_payload())
    features = extract_features(scapy_packet, packet)
    with open("./whitelist.txt") as file:
        for line in file:
            if line[0] == "" or line[0] == "\n":
                continue

            if scapy_packet.src == line.strip().replace("\n", ""):
                print(f"❌ Dropping Packet: {scapy_packet.summary()}")
                packet.drop()
        
    packet.accept()


# Start Netfilter Queue
nfqueue = NetfilterQueue()
nfqueue.bind(1, process_packet)

print("Monitoring Traffic in Real-Time")
try:
    nfqueue.run()
except KeyboardInterrupt:
    print("\nStopping Firewall...")
    nfqueue.unbind()

It works by using NFQUEUE, which is queueing packets to userspace. In simple terms, you get access to the packet using python. But you need to specify a queue number:

sudo iptables -D INPUT -p tcp --dport 80 -j NFQUEUE --queue-num 1

And you can do pretty neat stuff with this.

Implementation of WAF

We need to intercept HTTP(S) requests before they are delivered to the target server. This gives us the ability to analyze the content of the request and decide if it contains potentially malicious code.

We implement this step using a reverse-proxy component that I have programmed in Python using the library HTTPServer (a possible different approach is with the mitmproxy library).

The reverse-proxy acts as an intermediary between the user and the server. All traffic goes through this proxy first, which then decides whether to forward the request.

(The whole code is in the repository mentioned in the introduction)

This is how we forward the request:

def do_GET(self, body=True):
  url = protocol+f'{hostname}{self.path}'
  resp = requests.get(url, headers=self.headers, verify=False)
  self.wfile.write(resp.text.encode(encoding='UTF-8',errors='strict'))

If we want to check the URL, we simply do something like this:

def do_GET(self, body=True):
  url = protocol+f'{hostname}{self.path}'
  if verify_url(self.path): # we know that hostname is safe (from init)
    resp = requests.get(url, headers=self.headers, verify=False)
    self.wfile.write(resp.text.encode(encoding='UTF-8',errors='strict'))
  else:
    self.send_error(403, "Not allowed")

Signature-based

A signature-based WAF is a security mechanism designed to detect known patterns or signatures of malicious activity.

I’ve collected rules from coreruleset, user-agents and suricata.

Then made a regex search to match if any string is in the request or response.

    def __get_files_in_dir(self, dir_path: str) -> list:
        if os.path.isdir(dir_path):
            return os.listdir(dir_path)
        else:
            return []

    def __search_string_in_file(self, file_path: str, string: str) -> bool:        
        with open(file_path, "r") as f:
            for line in f:
                # empty line or comment
                if line[0] == "#" or line[0] == "\n":
                    continue
                
                # remove newline
                if line[-1]:
                    line = line[:-1]

                if re.search(re.escape(line), string):
                    return True # packet dangerous

        return False

    def __verify_signature(self, string: str, var: str) -> bool:
        files = self.__get_files_in_dir(self.signatures_paths[var])
        if len(files) > 0:
            for file in files:
                file_path = os.path.join(relative_path(self.signatures_paths[var]), file)
                if self.__search_string_in_file(file_path, string):
                    return False
        
        # returning True if path does not exist!
        return True

AI-based

AI-powered Web Application Firewall uses artificial intelligence and machine learning to detect and block web attacks in real time.

I used fwaf-dataset and PayloadAllThethings to train two models using PyTorch.

I didn’t create new model, but fine-tuned distilbert-base-uncased (you can use llama or any different model for classification).

    def verify_url(self, url: str) -> bool:
        inputs = self.tokenizer(url,
            truncation=True,
            padding=True,
            return_tensors="pt",
            max_length=500
        ).to(self.device)

        with torch.no_grad():
            outputs = self.url_model(**inputs)

        probabilities = torch.nn.functional.softmax(outputs.logits, dim=-1)
        predicted_class_index = torch.argmax(probabilities, dim=-1).item()

        predicted_class_name = self.url_labels[predicted_class_index]
        logging.info(f"Probabilities for each class: {probabilities.numpy()}: {predicted_class_name}")

        if max(probabilities[0]) < self.probability_catch:
            # We are not that certain
            return True

        if predicted_class_name == "goodqueries":
            return True
        else:
            return False

Main disadvantages

Complexity: Setting up and keeping up a firewall can be time-consuming and difficult
Limited Flexibility for developers
Limited adaptability: frequently rule-based for signatures and AI can’t be that much trusted

Possible upgrade to this project

One option is anomaly-based detection.
Once we gather enough data to define normal network flow, we can take action on anomalies.

Conclusion

Creating a comprehensive security system for a server environment is not only possible, but also highly effective when using a combination of multiple layers of protection, including WAF and iptables.

A good security architecture can significantly reduce the risk of successful cyber-attacks such as XSS, XXE, CSRF, DDoS or SQL injection.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey👋.

Building your own network protection with Hybrid WAF and iptables was originally published in System Weakness on Medium, where people are continuing the conversation by highlighting and responding to this story.

The original binaries are saved here.

Endlesscycle

After de-compilation and renaming I end-up with this code:

int main(int argc,char **argv) {
  int iVar1;
  char *pcVar2;
  int i;
  int i2;
  
  pcVar2 = (char *)mmap((void *)0,158,7,33,-1,0);
                    /* 3486694491 */
  srand(UINT_001042b8);
  for (_i = 0; _i < 158; _i = _i + 1) {
    for (_i2 = 0; _i2 < (ulong)(long)INT_ARRAY_00104040[_i]; _i2 = _i2 + 1) {
      rand();
    }
    iVar1 = rand();
    pcVar2[_i] = (char)iVar1;
  }
                    /* construct function using rand */
  iVar1 = (*(code *)pcVar2)();
  if (iVar1 == 1) {
    puts("You catch a brief glimpse of the Dragon\'s Heart - the truth has been revealed to you");
  }
  else {
    puts("The mysteries of the universe remain closed to you...");
  }
  return 0;
}

This binary is initialized with srand and creates a function based on the rand function and it should be possible to recreate this function by setting the srand (for example in python), but I choose the easiest way by examining it in GDB.

=> 0x7ffff7fbf000: push   rbp // init
   0x7ffff7fbf001: mov    rbp,rsp
   0x7ffff7fbf004: push   0x101213e
   0x7ffff7fbf009: xor    DWORD PTR [rsp],0x1010101 
                           // 0x101213e ^ 0x101213e 
   0x7ffff7fbf010: movabs rax,0x67616c6620656874
   0x7ffff7fbf01a: push   rax
   0x7ffff7fbf01b: movabs rax,0x2073692074616857
   0x7ffff7fbf025: push   rax
   0x7ffff7fbf026: push   0x1
   0x7ffff7fbf028: pop    rax
   0x7ffff7fbf029: push   0x1
   0x7ffff7fbf02b: pop    rdi
   0x7ffff7fbf02c: push   0x12
   0x7ffff7fbf02e: pop    rdx
   0x7ffff7fbf02f: mov    rsi,rsp
   0x7ffff7fbf032: syscall // write 18 bytes from stack f01b, f010
                              // What is the flag?
   0x7ffff7fbf034: sub    rsp,0x100 // 256 bytes reserved for buffer
   0x7ffff7fbf03b: mov    r12,rsp
   0x7ffff7fbf03e: xor    eax,eax 
   0x7ffff7fbf040: xor    edi,edi
   0x7ffff7fbf042: xor    edx,edx // cleaning
   0x7ffff7fbf044: mov    dh,0x1
   0x7ffff7fbf046: mov    rsi,r12
   0x7ffff7fbf049: syscall         // read into r12
   0x7ffff7fbf04b: test   rax,rax // if read is not empty
   0x7ffff7fbf04e: jle    0x7ffff7fbf082


   // for loop sliding 4 bytes with xor
   0x7ffff7fbf050: push   0x1a // 26
   0x7ffff7fbf052: pop    rax
   0x7ffff7fbf053: mov    rcx,r12 // rcx = r12
   0x7ffff7fbf056: add    rax,rcx // rax += rcx + 26
   0x7ffff7fbf059: xor    DWORD PTR [rcx],0xbeefcafe
   0x7ffff7fbf05f: add    rcx,0x4 // rcx += 4
   0x7ffff7fbf063: cmp    rcx,rax // if rcx < rax
   0x7ffff7fbf066: jb     0x7ffff7fbf059


   0x7ffff7fbf068: mov    rdi,r12
   0x7ffff7fbf06b: lea    rsi,[rip+0x12]        # 0x7ffff7fbf084
   0x7ffff7fbf072: mov    rcx,0x1a
   0x7ffff7fbf079: cld

   // comparing rdi (r12, input) with rsi (0x1a bytes = 26 bytes)
   0x7ffff7fbf07a: repz cmps BYTE PTR ds:[rsi],BYTE PTR es:[rdi]
   0x7ffff7fbf07c: sete   al
   0x7ffff7fbf07f: movzx  eax,al
   0x7ffff7fbf082: leave
   0x7ffff7fbf083: ret

You can notice on the end there is a comparison RDI with RSI, where RDI is input. I dumped the memory:

// 0x7ffff7fbf084: memory dump
0x7ffff7fbf084:   0xb6  0x9e  0xad  0xc5  0x92  0xfa  0xdf  0xd5
0x7ffff7fbf08c:   0xa1  0xa8  0xdc  0xc7  0xce  0xa4  0x8b  0xe1
0x7ffff7fbf094:   0x8a  0xa2  0xdc  0xe1  0x89  0xfa  0x9d  0xd2
0x7ffff7fbf09c:   0x9a  0xb7

When reconstructing the behavior in C it should look something like this:

int main() {
    char buffer[256];
    char message[] = "What is flag?";
    char expected[] = "..."; // 0x7ffff7fbf084
    ssize_t bytes_read;
    
    // Print message
    write(STDOUT_FILENO, message, sizeof(message) - 1);
    
    // Read user input
    bytes_read = read(STDIN_FILENO, buffer, 256);
    if (bytes_read <= 0) {
        return 0;
    }
    
    // XOR first few bytes with 0xBEEFCAFE
    for (int i = 0; i < bytes_read - 4; i += 4) {
        *(int *)(buffer + i) ^= 0xBEEFCAFE;
    }
    
    // Compare with expected string
    if (memcmp(buffer, expected, 26) == 0) {
        puts("You got the flag!");
        return 1;
    }
    
    puts("Try again...");
    return 0;
}

So now we can get the hands on python and reverse it:

import struct

# copied from memory dump
encrypted_data = [
    0xb6, 0x9e, 0xad, 0xc5, 0x92, 0xfa, 0xdf, 0xd5,
    0xa1, 0xa8, 0xdc, 0xc7, 0xce, 0xa4, 0x8b, 0xe1,
    0x8a, 0xa2, 0xdc, 0xe1, 0x89, 0xfa, 0x9d, 0xd2,
    0x9a, 0xb7
]

# the key that is used
xor_key = 0xBEEFCAFE

original_bytes = bytearray()
for i in range(0, len(encrypted_data), 4):
    # XOR is applied to 4-byte chunks
    chunk = encrypted_data[i:i+4]
    while len(chunk) < 4:
        chunk.append(0)
    
    # unpack it into int and back
    # (reverse of XOR is another XOR with same value)
    encrypted_int = struct.unpack("<I", bytes(chunk))[0]
    decrypted_int = encrypted_int ^ xor_key
    original_bytes.extend(struct.pack("<I", decrypted_int))

original_input = original_bytes.decode(errors="ignore")

print("Original input:", original_input)

Next on the list is:

Impossimaze

When trying to make sense in the binary I’ve ended up with this code:

int main(int argc,char **argv) {
  int iVar1;
  uint uVar2;
  uint max_y_2;
  uint max_x_2;
  ulong max_x;
  char char_to_write;
  int x;
  int y;
  int *char_index;
  undefined4 in_register_0000003c;
  long in_FS_OFFSET;
  int half_x;
  int max_y;
  char position_x_y [24];
  long canary;
  
  canary = *(long *)(in_FS_OFFSET + 40);
                    /* newterm(getenv("TERM"), stdout, stdin); */
  initscr(CONCAT44(in_register_0000003c,argc));
  cbreak();
  noecho();
  curs_set(0);
  keypad(window,1);
  max_y = getmaxy(window);
  max_x = getmaxx(window);
                    /* ensures half */
  half_x = (int)(((uint)(max_x >> 31) & 1) + (int)max_x) >> 1;
  max_y = max_y / 2;
  x = 0;
  do {
    max_y_2 = getmaxy(window);
    max_x_2 = getmaxx(window);
    if (x == 260) {
      half_x = half_x - (uint)(1 < half_x);
    }
    else if (x < 261) {
      if (x == 258) {
        max_y = max_y + 1;
      }
      else if (x == 259) {
        max_y = max_y - (uint)(1 < max_y);
      }
    }
    else {
      half_x = half_x + (uint)(x == 261);
    }
    werase(window);
    wattr_on(window,1048576,0);
    wborder(window,0,0,0,0,0,0,0,0);
    if (2 < (int)max_x_2) {
      x = 1;
      do {
        y = 1;
        if (2 < (int)max_y_2) {
          do {
            uVar2 = get_char(x,y);
            if ((int)uVar2 < 61) {
              char_to_write = 'A';
              if ((int)uVar2 < 31) {
                char_to_write = (-(uVar2 < 31) & 133U) + 86;
              }
            }
            else {
              char_to_write = (-(uVar2 - 61 < 120) & 202U) + 86;
            }
            iVar1 = wmove(window,y,x);
            if (iVar1 != -1) {
              waddch(window,(int)char_to_write);
            }
            y = y + 1;
          } while (y != max_y_2 - 1);
        }
        x = x + 1;
      } while (max_x_2 - 1 != x);
    }
    wattr_off(window,1048576,0);
    wattr_on(window,2097152,0);
    x = wmove(window,max_y,half_x);
    if (x != -1) {
      waddch(window,88);
    }
    wattr_off(window,2097152,0);
    snprintf(position_x_y,16,"%d:%d",(ulong)max_y_2,(ulong)max_x_2);
    x = wmove(window,0,0);
    if (x != -1) {
      waddnstr(window,position_x_y,4294967295);
    }
    if ((max_y_2 == 13) && (max_x_2 == 37)) {
      wattr_on(window,524288,0);
      wattr_on(window,2097152,0);
      char_index = &INT_001040c0;
      x = 6;
      do {
        y = x + 1;
        x = wmove(window,6,x);
        if (x != -1) {
                    /* add a character (with attributes) to a curses window and advance cursor  */
          waddch(window,(&DAT_00104120)[*char_index]);
        }
        char_index = char_index + 1;
        x = y;
      } while (y != 30);
      wattr_off(window,2097152,0);
      wattr_off(window,524288,0);
    }
    x = wgetch(window);
  } while (x != 113);
  endwin();
  if (canary != *(long *)(in_FS_OFFSET + 40)) {
                    /* WARNING: Subroutine does not return */
    __stack_chk_fail();
  }
  return 0;
}

This binary is keeping track of window size, position x-y and moving cursor using wmove and waddch.

My approach was to search for printing any method, that may print out our flag. There were a few functions that caught my eye:

uVar2 = get_char(x,y);
if ((int)uVar2 < 61) {
    char_to_write = 'A';
    if ((int)uVar2 < 31) {
      char_to_write = (-(uVar2 < 31) & 133U) + 86;
    }
}
else {
  char_to_write = (-(uVar2 - 61 < 120) & 202U) + 86;
}

waddch(window,(int)char_to_write);

// Write formatted output to sized buffer
snprintf(position_x_y,16,"%d:%d",(ulong)max_y_2,(ulong)max_x_2);
waddnstr(window,position_x_y,4294967295);

char_index = &INT_001040c0;
waddch(window,(&DAT_00104120)[*char_index]);

waddch is used add a character to a curses window and advance cursor.

So we got few options, we will try to eliminate it one by one by thinking what it does, the first is kind of complex shifting of the char depending on get_char function:

int get_char(int x,int y) {
  return *(int *)(&DAT_00102020 +
                 (long)((x + *(int *)(&DAT_00102020 + (long)((y + 1337) % 256) * 4)) % 256) * 4);
}

But I stepped out and was thinking about the “char_to_write = ‘A’;”, it seemed weird to me, so I moved on.

Next we just printing position x-y, that’s clearly not what we are searching for.

Moving on we end-up here:

char_index = &INT_001040c0;
waddch(window,(&DAT_00104120)[*char_index]);

I noticed that the char_index variable is actually an array of ints and if some conditions are met then the char is printed.

So I tried to extract some of the non-null ints from the memory and added it to the DAT_00104120 and what we got here: “HTB{“, beginning of the flag and I knew that I was on the right path.

00104205 H
0010413c T
001041d8 B
001041d2 {
00104160 T
00104188 H
001041f2 3
001041aa _
00104170 c
001041a6 u
0010421c r
00104193 s
001041f2 3
001041aa _
0010420d i
00104193 s
001041aa _
00104165 b
0010421c r
00104176 o
001041eb k
001041f2 3
0010420f n
001041e0 }

# HTB{th3_curs3_is_brok3n}

Final words

Thanks for reading, if someone is interested in collaboration write me on discord (bitr13x) or email me (sw33tbit@protonmail.com).

The original binaries are saved here.

Recommended knowledge

First get familiar with C (often you will end-up in de-compiler), it is very useful to learn little bit of assembly, registers and calling conventions if you need to use GDB.

Most important is to learn how to google build-in functions and more.

Few tips how to approach these types challenges

Find your de-compiler (Ghidra, IDA, Binary ninja, cutter, radare2 and more), there are lot’s of options just find what you are comfortable.

I stick with Ghidra and cutter (switching to the cutter if Ghidra does not decompile built-in functions or cannot find the main, just to check).

Get the bigger picture do not look into one part of the code too long if you look at a function from a different part, you can spot what is it for and decide if you need to understand it.

When looking for the main function, look for common patterns (the main function is usually loaded in the same manner from the entry point of the binary).

Plot graph of the execution flow for bigger binaries, it can help with clarity.

Make comments (the same reason).

Computemagic

When opening a binary usually you would see weird names and types.

After some renaming and figuring out what does what. I end up with this code:

 
int main(int argc,char **argv) {
  int error_res;
  ssize_t read_return;
  long in_FS_OFFSET;
  undefined4 option_value;
  socklen_t address_len;
  int socket;
  int socket_filedesc;
  int check;
  size_t length;
  sockaddr socketAddr;
  char buffer [24];
  undefined8 local_10;
  
  local_10 = *(undefined8 *)(in_FS_OFFSET + 40);
  option_value = 1;
  address_len = 16;
  while( true ) {
                    /* int socket(int domain, int type, int protocol); */
    socket = ::socket(2,1,0);
    if (socket == 0) {
      perror("socket failed");
                    /* WARNING: Subroutine does not return */
      exit(1);
    }
    error_res = setsockopt(socket,1,15,&option_value,4);
    if (error_res != 0) {
      perror("setsockopt failed");
                    /* WARNING: Subroutine does not return */
      exit(1);
    }
    socketAddr.sa_family = 2;
    socketAddr.sa_data[2] = '\0';
    socketAddr.sa_data[3] = '\0';
    socketAddr.sa_data[4] = '\0';
    socketAddr.sa_data[5] = '\0';
    socketAddr.sa_data._0_2_ = htons(9003);
    error_res = bind(socket,&socketAddr,16);
    if (error_res < 0) {
      perror("bind failed");
                    /* WARNING: Subroutine does not return */
      exit(1);
    }
    error_res = listen(socket,3);
    if (error_res < 0) break;
    socket_filedesc = accept(socket,&socketAddr,&address_len);
    if (socket_filedesc < 0) {
      perror("accept failed");
                    /* WARNING: Subroutine does not return */
      exit(1);
    }
    sendBanner(socket_filedesc);
    read_return = read(socket_filedesc,buffer,16);
    if (read_return < 0) {
      perror("read failed");
                    /* WARNING: Subroutine does not return */
      exit(1);
    }
    length = strlen(buffer);
    if ((length != 0) && (socketAddr.sa_data[length + 13] == '\n')) {
      socketAddr.sa_data[length + 13] = '\0';
    }
    check = checkSpell(buffer,socket_filedesc);
    if (check == 1) {
      puts("\nSpell check successful.");
    }
    else {
      puts("\nSpell check failed.");
    }
    close(socket_filedesc);
    close(socket);
  }
  perror("listen failed");
                    /* WARNING: Subroutine does not return */
  exit(1);
}

The binary starting a socket and waiting for connection.

Here is the most interesting parts of the code:

socket_filedesc = accept(socket,&socketAddr,&address_len);
read_return = read(socket_filedesc,buffer,16);
length = strlen(buffer);
if ((length != 0) && (socketAddr.sa_data[length + 13] == '\n')) {
  socketAddr.sa_data[length + 13] = '\0';
}
check = checkSpell(buffer,socket_filedesc);
if (check == 1) {
  puts("\nSpell check successful.");
}
else {
  puts("\nSpell check failed.");
}

Now we know how large the input string must be and we know what function is used to validate if the input is correct.

int checkSpell(char *buffer,int filedesc) {
  int return_val;
  size_t length;
  
  length = strlen(buffer);
  if ((int)length < 1) {
    puts("Empty string.");
    return_val = 0;
  }
  else {
    switch(*buffer) {
    case 'A':
      func_1(buffer,filedesc);
      break;
    ...
    ...
    case 'Z':
      func_26(buffer,filedesc);
    }
    return_val = 1;
  }
  return return_val;
}

Switch case, that is calling different functions depending on the first letter.

It can look complex, but I realized it has to take parameters in order to check the actual string and I end up look into each function that has parameters and found:

void func_13(char *buf,int filedsc) {
  int result;
  
  result = check_other(buf);
  if (result != 0) {
    read_flag(filedsc);
  }
  return;
}

Where the real validation happens:

bool check_other(char *buf) {
  int res_strcmp;
  long in_FS_OFFSET;
  int i;
  char hardcoded [16];
  long canary;
  
  canary = *(long *)(in_FS_OFFSET + 40);
  // all what we need to reverse:
  for (i = 0; i < 16; i = i + 1) {
    hardcoded[i] = buf[i] + 0x4U ^ 13;
  }

  res_strcmp = strcmp(hardcoded,"AhhF1ag1571GHFDS");
  if (res_strcmp != 0) {
    printf("Incorrect: %s\n",buf);
  }
  else {
    printf("Correct: %s\n",buf);
  }
  if (canary != *(long *)(in_FS_OFFSET + 40)) {
                    /* WARNING: Subroutine does not return */
    __stack_chk_fail();
  }
  return res_strcmp == 0;
}

Which I reverse by using python (the 0x4U means unsigned 4):

a = "AhhF1ag1571GHFDS"
buf = [""] * len(a)

for i in range(len(buf)):
    buf[i] = chr(ord(a[i]) + 4 ^ 13)

print("".join(buf)) # HaaG8hf8468FAGEZ

Oldauth

Here is the same process rename everything and type it:

int main(int argc,char **argv,char **envp) {
  bool bVar1;
  int iVar2;
  char *fgets_check;
  size_t newline_index;
  undefined7 extraout_var;
  undefined7 extraout_var_00;
  undefined7 extraout_var_01;
  undefined7 extraout_var_02;
  undefined4 in_register_0000003c;
  long in_FS_OFFSET;
  char username [112];
  char key [1000];
  long canary_bit;
  
  canary_bit = *(long *)(in_FS_OFFSET + 40);
  setup(CONCAT44(in_register_0000003c,argc));
  printf("Enter the key: ");
  fgets_check = fgets(key,1000,stdin);
  if (fgets_check == (char *)0) {
    puts("Error!");
    iVar2 = -1;
  }
  else {
    newline_index = strcspn(key,"\n");
    key[newline_index] = '\0';
                    /* xor by 82 every char
                        */
    change(key,newline_index,82);
    if (newline_index != 16) {
                    /* 
                        */
      fail();
    }
    if (key[2] != 'Q') {
      fail();
    }
    if (key[13] != '4') {
      fail();
    }
                    /* suma of chars and modulo (4: how many chars, 3: modulo) */
    bVar1 = check(key,4,3);
    if ((int)CONCAT71(extraout_var,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 4,5,8);
    if ((int)CONCAT71(extraout_var_00,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 8,4,5);
    if ((int)CONCAT71(extraout_var_01,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 12,4,3);
    if ((int)CONCAT71(extraout_var_02,bVar1) == 0) {
      fail();
    }
    printf("Enter the username: ");
    fgets_check = fgets(username,100,stdin);
    if (fgets_check == (char *)0) {
      puts("Error!");
      iVar2 = -1;
    }
    else {
      newline_index = strcspn(username,"\n");
      iVar2 = compare_with_target(username,newline_index);
      if (iVar2 == 0) {
        fail();
      }
      succeed();
      iVar2 = 0;
    }
  }
  if (canary_bit != *(long *)(in_FS_OFFSET + 40)) {
                    /* WARNING: Subroutine does not return */
    __stack_chk_fail();
  }
  return iVar2;
}

Find interesting parts of code:

(I’ve named the first input as key)

Next there is called four times on an input check function.

    bVar1 = check(key,4,3);
    if ((int)CONCAT71(extraout_var,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 4,5,8);
    if ((int)CONCAT71(extraout_var_00,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 8,4,5);
    if ((int)CONCAT71(extraout_var_01,bVar1) == 0) {
      fail();
    }
    bVar1 = check(key + 12,4,3);
    if ((int)CONCAT71(extraout_var_02,bVar1) == 0) {
      fail();
    }

After some adjustments for the function:

bool check(char *key,int for_end,int modulo) {
  int suma;
  int i;
  
  suma = 0;
  for (i = 0; i < for_end; i = i + 1) {
    suma = suma + key[i];
  }
  return suma % modulo == 0;
}

I also noticed a second fgets meaning there are two parts (password and username). I started with reversing the check function using python:

key = ["", "", "Q", "", "", "", "", "", "", "", "", "", "", "4", "", ""]
def rev_check(start, end, modulo):
    suma = 0
    def_letter = "d"
    for i in range(start, start+end):
        if not key[i]:
            key[i] = def_letter
        
        suma += ord(key[i])

    if suma % modulo != 0:
        tmp = suma % modulo
        key[i] = chr(ord(def_letter) - tmp)

# from the binary
rev_check(0, 4, 3)
rev_check(4, 5, 8)
rev_check(8, 4, 5)
rev_check(12, 4, 3)

print("REV Check:", key)
for c in range(len(key)):
    key[c] = chr(ord(key[c]) ^ 82)

print("XOR:", key)

with open("rev_out", "w") as f:
    f.write("".join(key))

print("password:", "".join(key).encode())

Not the prettiest code, but hey it works.

2nd part is username:

    printf("Enter the username: ");
    fgets_check = fgets(username,100,stdin);
    if (fgets_check == (char *)0) {
      puts("Error!");
      iVar2 = -1;
    }
    else {
      newline_index = strcspn(username,"\n");
      iVar2 = compare_with_target(username,newline_index);
      if (iVar2 == 0) {
        fail();
      }
      succeed();
      iVar2 = 0;
    }

Noticed that there is called the compare_with_target function, to check if the username is correct.

Don’t get confused with the return “!=” it has to return the “1”.

This function is a straight forward checking hard-coded variable with a small ASCII shift (+2).

The reverse of this function in python would look like this:

# username
username = ""
for c in "elb4rt0pwn":
    username += chr(ord(c) - 2)

print("username:", username)

After sending it all together into the binary you would get a flag.

Final words

Thanks for reading, if someone is interested in collaboration write me on discord (bitr13x) or email me (sw33tbit@protonmail.com).

What is Boolean Satisfiability (SAT) and Satisfiability Modulo Theories (SMT)

Boolean Satisfiability abbreviated SAT. Boolean stands for True or False and satisfiability is when it is true under some assignment of values to its variables. Meaning SAT solvers trying to find if the variables can be replaced by the values TRUE or FALSE in such a way that the formula evaluates to TRUE. The standard definition is:

SAT solvers determine the ‘satisfiability’ of boolean set of equations for a set of inputs.

In other hand the SMT practically doing the same thing, but it combines the power of SAT solvers and other types of solvers to solve more complicated formulas for example handling inequalities, uninterpreted functions. SMT can handle formulas that involve multiple theories such as linear arithmetic, bit-vectors, and arrays.

Intro into Z3 Solver

Why we were talking about SMT and SAT, because the Z3 is one of the SMTs. It can be used for a variety of purposes such as formal verification, model checking, static program analysis and lot more.

Often when navigating through this field you will come across the terms like:

Symbols (= variables),
Unsat/sat (= unsatisfiable/satisfiable), (= unsolvable/solvable)
Concrete values (= strings, decimals and integers)
Literals (= SAT variable in short)
Clauses (= combination of literals)

Bonus

P problem (Polynomial problem): Refers to problems that can be solved efficiently (in polynomial time [n² + n, n]) by a deterministic algorithm. Example: Sorting a list.

NP problem (Nondeterministic Polynomial problem): Refers to problems for which a solution can be quickly verified (in polynomial time). Example: Verifying a solution to a Sudoku puzzle.

How the Z3 actually works? Well here you go:

The fastest way how to demonstrate it, is how to solve Facebook quiz:

We can rewrite it into Z3 form, where flowers will be the symbols (variables):

from z3 import *

# flowers => Symbolic [Int] type
rflower = Int("rflower")
bflower = Int("bflower")
yflower = Int("yflower")
# rflower, bflower, yflower = Ints('rflower bflower yflower')

solver = Solver()

# (Constraints) Equations
solver.add(rflower + rflower + rflower == 60)
solver.add(rflower + bflower + bflower == 30)
solver.add(bflower - yflower == 3)
# note: if we know the result is positive, we can add constraint: flower > 0

# Solver.check => CheckSatResult[sat, unsat, unknown]
if solver.check() == sat:
    model = solver.model()
    print("Sat solution:")
    print(
        f"Answer is: {model[yflower]} + {model[rflower]} + {model[bflower]} =", 
        f"{model[yflower].as_long() + model[rflower].as_long() + model[bflower].as_long()}"
    )
else:
    print("unsat: No solution found.")

I think everything is straight forward except one thing why “.as_long()”? That’s because model[variable] is not Int, but class z3.z3.IntNumRef.

When we dive a little bit deeper and look how Z3 is actually solving these problems:

Solving reverse engineering challenge:

What do we even see? It’s binary that checks whatever license format is right or not, the license is hardcoded under some if statement, which has to be true in order to crack the binary.

The argc arguments is the number of arguments supplied we know we need to run the binary with at least one argument and the length is 32 bytes.

There are two approaches about this problem using Z3, we use it to solve as equation or with angr that will control flow of the program and try to brute force it until everything is met.

We will apply Z3 by rewriting the if statement into equation (you don’t need to do it manually, help yourself, for example you can use global replace)

There is a small catch with the conversion, the chars are from -128 to 127 and when they overflow it will move back and forth.

In order to solve this weird unusual acting we will use BitVecs, I thought of using a function to handle the logic, but I end up using BitVecs for the simplicity.

from z3 import *


def check_license_with_z3():
    solver = Solver()
    # Create array of bitvecs with size of 32
    string = [BitVec(f"string_{i}", 8) for i in range(32)]

    solver.add(
        string[29] == string[5] - string[3] + ord("F"),
        string[2] + string[22] == string[13] + ord("{"),
        string[12] + string[4] == string[5] + 0x1c,
        string[25] * string[23] == string[0] + string[17] + 0x17,
        string[27] * string[1] == string[5] + string[22] + -21,
        string[9] * string[13] == string[28] * string[3] + -9,
        string[9] == ord("p"),
        string[19] + string[21] == string[6] + -128,
        string[16] == string[15] - string[11] + ord("0"),
        string[7] * string[27] == string[1] * string[13] + ord("-"),
        string[13] == string[18] + string[13] + -101,
        string[20] - string[8] == string[9] + ord("|"),
        string[31] == string[8] - string[31] + -121,
        string[20] * string[31] == string[20] + 4,
        string[24] - string[17] == string[21] + string[8] + -23,
        string[7] + string[5] == string[5] + string[29] + ord(","),
        string[12] * string[10] == string[1] - string[11] + -36,
        string[31] * string[0] == string[26] + -27,
        string[1] + string[20] == string[10] + -125,
        string[18] == string[27] + string[14] + 2,
        string[30] * string[11] == string[21] + ord("D"),
        string[5] * string[19] == string[1] + -44,
        string[13] - string[26] == string[21] + -127,
        string[23] == string[29] - string[0] + ord("X"),
        string[19] == string[8] * string[13] + -23,
        string[6] + string[22] == string[3] + ord("S"),
        string[12] == string[26] + string[7] + -114,
        string[16] == string[18] - string[5] + ord("3"),
        string[30] - string[8] == string[29] + -77,
        string[20] - string[11] == string[3] + -76,
        string[16] - string[7] == string[17] + ord("f"),
        string[1] + string[21] == string[11] + string[18] + ord("+"),
    )

    if solver.check() == sat:
        model = solver.model()
        print(
            "".join([
                chr(c.as_long())
                for c in [model.eval(string[i]) for i in range(len(string))]
            ])
        )
        print("License Correct")
        return True
    else:
        print("License Invalid")
        return False

(Bonus if we want to just get the string “License Correct”, it’s done by overwriting the if instruction.)

Conclusion

We’ve learnt about Z3, the most fundamental terminology in Z3, and how to solve basic problems with Z3.

With a little knowledge about Z3, you will be able to solve complex problems like rubic cube, sudoku, magic square and lot more.

If you enjoyed this article, clap👋 and follow me! Thanks for reading! Looking forward to seeing you in the future.

More Resources

Exemplar challenges: https://github.com/PwnFunction/learn-z3
Docs: https://github.com/Z3Prover/z3/wiki#background
YouTube explanation: https://www.youtube.com/watch?v=EacYNe7moSs

When starting out learning about XSS there is a lot of stuff, but most of it is just repeating, and you can’t find anything new. This guide should also cover the special cases that are hard to find on the web.

Why my JavaScript is not executing?

• Browser protection against JavaScript execution

1) SOP (Same-Origin Policy)

It is a security rule that keeps data safe by making sure that web pages from one website (origin) can’t access data from another website (origin) without permission.

Anytime when we want to read the response from an API or different website (origin), the website has to answer with a CORS header to allow reading, otherwise we will get a SOP error:

2) CSP (Content Security Policy)

A security feature that allows website owners to control what content (like scripts, images, or styles) can be loaded onto their web pages.

You can set your CSP in html using meta tag (or using header Content-Security-Policy):

<meta
  http-equiv="Content-Security-Policy"
  content="default-src 'self'; img-src https://*; child-src 'none';" />

3) CORS (Cross-Origin Resource Sharing)

Way for a website to request resources (like data) from another website even if they have different origins (domains).

It’s known under these headers:

Access-Control-Allow-Origin: https://foo.example
Access-Control-Allow-Methods: POST, GET, OPTIONS
Access-Control-Allow-Headers: X-PINGOTHER, Content-Type
Access-Control-Max-Age: 86400

It’s used to enable sharing of resources between different websites, but in a controlled way.

4) Sanitized characters

If a site properly sanitizes characters, the characters are rendered as HTML entities and the JavaScript or HTML tags are not executed.

• A double quote (“) as " or "
• A single quote (‘) as ' or '
• An opening angle bracket (<) as &lt; or &#60;
• A closing angle bracket (>) as &gt; or &#62;

Weird JavaScript

• Encoding, function execution and much more is acting very weird when it comes to javascript

Encoding

JavaScript supports various types of encoding, like Hex, Decimal, Unicode, named and HTML5 entities.

The weird thing about this, is that we can turn these encoding directly into chars or numbers without using a decoding function.

Also, it’s useful to know that a single character escape sequence exists:

'\b' // backspace
'\f' // form feed
'\n' // new line
'\r' // carriage return
'\t' // tab
'\v' // vertical tab
'\0' // null
'\'' // single quote
'\"' // double quote
'\\' // backslash

We can break down a word or sentence using this knowledge:

"A\W\E\S\O\M\E" // "AWESOME"

'Multiple \
lines' // "Multiple lines"

Hex:

We can encode using hex, but here is the problem that we can’t declare variables or call functions, because without a double quote it will cause an error.

const a = () => 1;

"\x61" // returns "a"
\x61 // Invalid escape sequence

Unicode:

We can declare functions and variables, we can also half encode word so that it won’t get filtred out.

// unicode character "63" is allowed as a variable
const \u0061 = () => 1;
\u{63} = 1 

\u{61}() //correctly calls the function

"H\u0065LLO" // HELLO

Function Execution

Functions may be quite fascinating in JavaScript, we can invoke them without parentheses:

function printout(n1) {
  return n1;
};

printout("Hello World!"); // output: "Hello World"
printout`Hello World!`; //  output: ["Hello World"]

If we want to execute a function with more parameters, we can utilize template literals. however, the indexing starts at 1 rather than 0.

function sum(n1, n2) {
  console.log(n1, n2); // ["n1", "", ""], n2
  return Number(n1[0]) + n2;
}; 

sum`5${4}`; // output: 9

// Easier way through arguments
function x() {
  console.log(arguments);
};

x`${1}${2}${3}`; // output [["", "", ""], 1, 2, 3]

There are more ways to execute a function. For example:

Interesting ways of executing alert found in book from Gareth Heyes:

window.valueOf=alert;window+1
setTimeout`alert\x281337\x29`
'1337,'.replace`1337${alert}`
Reflect.apply.call`${alert}${window}${[1337]}`/

And more from PayloadsAllTheThings:

window['alert'](0)
parent['alert'](1)
self['alert'](2)
top['alert'](3)
this['alert'](4)
frames['alert'](5)

[7].map(alert);
[8].find(alert);
[9].every(alert);
[10].filter(alert);
[11].findIndex(alert);
[12].forEach(alert);

Iframe and Window

We can dereference window.alert and it will also dereference alert, because the origin of alert function lies on window.

Iframes can be used for JavaScript sandbox execution, suppose we got this code, and we want to execute a XSS to get the secret.

var secret = "MY-SECRET";
function createSandboxedEnvironment(code) {
    const iframe = document.createElement('iframe');
    iframe.style.display = 'none';
    document.body.appendChild(iframe);

    const sandboxWindow = iframe.contentWindow;
    const sandboxDocument = iframe.contentDocument;

    try {
        sandboxWindow.eval(code);
    } catch (error) {
        console.error('Error in sandboxed code:', error);
    } finally {
        document.body.removeChild(iframe);
    };
};

createSandboxedEnvironment(`console.log('Sandboxed:', secret);`); // undefined

If we execute an XSS in the current scope, we will not receive any secrets or cookies. This means that it must be executed in the main window rather than the iframe window.

createSandboxedEnvironment(
  `console.log('Sandboxed:', parent.secret);`
); // MY-SECRET

createSandboxedEnvironment(
  `console.log('Sandboxed:', window.top.secret);`
); // MY-SECRET

createSandboxedEnvironment(
  `console.log('Sandboxed:', window.parent.secret);`
); // MY-SECRET

Another interesting aspect of the window object is that it allows you to launch a new tab and read from it if you are on the same origin. Assume we discovered XSS in a page and there is a /notes endpoint on the same origin. If we wish to read the /notes of another user, we may simply open a new window and read them.

<script>
    window.open("http://localhost/notes.html", "pwn");
    setTimeout(`console.log(window.open("", "pwn").document.body.textContent)`, 1000);
</script>

But there is a catch that browsers usually block new windows. Still, we can get it through iframe .

<script>
    const iframe = document.createElement('iframe');
    iframe.src = "http://localhost/notes.html";
    iframe.style.display = "none";
    document.body.appendChild(iframe);
    setTimeout("console.log(iframe.contentDocument.body.textContent)", 1000)
</script>

When we can’t access a window, the document.defaultView property is simply a way of obtaining the window object. For example, we can usually use a workaround that will allow us to access the “defaultView” property.

let node = document.createElement('div');
node.ownerDocument.defaultView.alert(1337);

Payloads

There are plenty of them, and I mean it. When it comes to XSS, we may get fairly creative, combining every mentioned aspect to overcome filters. The reason XSS is so dangerous is that it includes keyloggers, not to mention general user data and session hijacking.

Most popular sources of XSS payloads:

• PayloadsAllTheThings
• Portswigger.net
• Payloadbox

Weird browser

Document.getelementById()

Did you know that you could get the element by id simply with a variable:

<a href="clobbered:1337" id=x></a>
<script>
  alert(x);//clobbered:1337
  alert(typeof x);//object
</script>

Data URLs

data:,Hello%2C%20World%21

Data URLs are URLs that can render a webpage with scripts and tags in a browser; for example, we can use them to render a webpage directly in a redirect.

Target property

There was a time when element a with parameter _blank could access the original page, but this is no longer the case. However, if the parameter rel is set to opener, there may still be an issue.

<a href="http://localhost" target="_blank" rel="opener">EXAMPLE</a>

The attacker can redirect the original webpage to his own destination or can just execute on the window JavaScript:

if (window.opener) {
  window.opener.location = 'https://you-re-hacked.com';
};

if (window.opener) {
  window.opener.alert("YOUR COOKIES ARE MINE");
};

Defending

Snyk

Automated security tool for scanning and monitoring your software development. It’s incredibly simple to set up and utilize. Sometimes snyk produces false positives, but it is still worthwhile to verify.

Keep your libraries up-to-date

Most vulnerabilities are not detected immediately, and people strive to address them as soon as possible when strange things occur. Keeping your libraries up-to-date is also important and sometimes painful.

Javascript/Markdown sanitilization

Sanitization is a good security practice to use when including or reflecting client-side HTML on your website. There are several open-source libraries. One example is: sanitize-html

CSP (Content Security Policy)

As mentioned, CSP can enhance your security by preventing cross-domain scripts from being loaded. However, it is not sufficient to include only CSP.

Conclusion

XSS is one of the most pervasive and dangerous vulnerabilities in web security, capable of compromising user data, hijacking sessions, and executing malicious code.

Theoretically, any user input might be utilized as an attack vector, potentially resulting in a vulnerability. You should pay special attention to these areas to ensure that they do not cause any strange behavior.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey👋.

After a short break, I’m back with another interesting article, where we look deep into how neural networks work. I will try to explain it in a simple way and hopefully, you will find out that in the end it’s not that hard.

I’m not a professional and I could be wrong if you spot any mistakes, head into comments 🙂.

With that said let’s jump into the darkness…

You will learn

Fundamentals of Neural Networks

• What Are Neural Networks?
• Neurons

Neural Network Layers

• Input Layer
• Hidden Layers
• Output Layer

Common Implementations

• Feedforward Neural Networks (FNN)
• Convolutional Neural Networks (CNN)
• Recurrent Neural Networks (RNN)
• Long Short-Term Memory Networks (LSTM)

What Are Neural Networks?

You could think of them like equivalent of the human brain’s information processing system. They are designed to mimic the brain’s ability to learn, adapt, and process data. These networks consist of artificial neurons, or nodes, interconnected by weighted connections, the whole calls it self: Artificial Neural Network (ANN).

Neural networks are inspired by the workings of the brain. In our brains neurons communicate by transmitting signals through synapses. Artificial neural networks replicate this process using models.

These networks are exceptional, at recognizing patterns making predictions and handling volumes of data, which is crucial for tasks such, as image recognition, language comprehension and recommendation systems.

Neurons

Building Blocks of Neural Networks

In a neural network, each neuron acts as a processing unit.

Artificial neurons take input data, perform calculations on it, and produce an output.

These outputs are then used as inputs for subsequent layers or are the final predictions of the network.

Weights and Biases

Two essential components are weights and biases. These elements determine how strongly an input signal influences the neuron’s output.

• Weights💪: Weights is the strength of connections between neurons. Each connection between neurons has an associated weight. Larger weights amplify the input’s impact on the neuron’s output, while smaller weights diminish it.
• Biases👤: Biases are like an offset or threshold for the neuron. They are shifting the activation function left or right. If you want to read more about biases you can click here.

Weights and biases enable neurons to learn and adjust their behavior during training. The process of determining the appropriate weights and biases, often known as model training or optimization.

They are capable of learning complicated patterns and correlations among data, making them useful for tasks like as image recognition, natural language interpretation, and many others.

Example of neural prediction

Imagine you are training a neural network to predict whether a student will pass a test based on the number of hours they study. Your input data consists of the number of hours a student studied, and the output is representing whether they passed (1) or failed (0).

To make this prediction, the neuron performs the following operation:

Output = Activation Function(Weight × Number of Hours Studied + Bias)

• Activation function⚡️: It determines whether a neuron should “fire” (output a signal) or not based on the input it receives. The most frequent functions are Sigmoid, ReLU (Rectified Linear Unit), Tanh (hyperbolic tangent).

Here the bias can represent how hard is the test, Let’s say the weight is 0.1, and the bias is -5 and you studied for 40hours.

Output = Activation Function(0.1 x 40 x -5) = -1

Since it’s -1 meaning you wouldn’t pass. Of course predicting the test like this is nonce, but you got the point.

Neural Network Layers

Neural networks are made up of layers, each having its own function and characteristics.

Understanding these levels is critical for understanding how information moves through a network and how the decisions are made.

We’ll look at the three main types of layers in a neural network architecture: the Input Layer, Hidden Layers, and Output Layer.

Input Layer

The Entry Point of Data

The first layer of a neural network is the Input Layer. Its major duty is to receive data and forward it to the following layers for processing.

Each neuron in the input layer corresponds to a feature or input variable, it serves as an important link between the external data and the neural network.

• No Computation🚧: Neurons in the input layer don’t perform any computation. They simply transmit the input data to the hidden layers.
• Size Determination🔍: The number of neurons in the input layer is determined by the dimensionality of the input data. For instance, in an image classification task, each pixel might correspond to a neuron in the input layer.

You may encounter situations where input values are constrained to a specific range [0, 1] that’s called standardization and normalization. It can help the neural network converge faster during training and make it more robust to different input scales. However, these techniques are not always necessary and depend on the type of data.

Hidden Layers

The Workhorses of the Network

In a neural network, the true computing takes place in the Hidden Layers. These layers are in charge of extracting characteristics from input data as well as learning complex patterns and representations.

While a network can have numerous hidden layers, the number and size of hidden layers are determined by the network’s architecture and the task’s complexity.

• Weighted Sum➕: The values provided are multiplied by their weights. These weighted sums are computed by taking the dot product, between the input values and the corresponding neuron weights. The outcome is a collection of sums.
• Bias Addition➕: After calculating the weighted sums, each neuron adds a bias term.
• Activation Function⚡️: The result of the weighted sum plus the bias for each neuron is then passed through an activation function.
• Hierarchical Learning🧠: Hidden layers progressively learn more abstract and high-level features as information flows through the network’s layers.

The hidden layers of a neural network act as feature extractors.

Output Layer

Making Predictions

Final layer of the neural network. It produces the network’s predictions or outputs based on the processed information from the hidden layers. The structure and number of neurons in the output layer depend on the nature of the task.

• Loss Calculation📉: Once the network produces its output, the loss or error between the predicted values and the actual target values (ground truth) is calculated. This loss is a measure of how well the network is performing.
• Backpropagation and Training🔄: The computed loss is used to update the network’s weights and biases through a process called backpropagation and optimization algorithms like gradient descent. The network learns to adjust its parameters to minimize the loss, making its predictions more accurate over time.
• Interpretation📖: The values produced by the neurons in the output layer are often interpreted as probabilities or class scores, depending on the problem. The highest-scoring class or classes are the network’s final predictions.

The final predictions or values are created in the output layer based on the network’s learnt weights and biases. Network’s performance is enhanced by training, in which it learns to make better predictions by altering its parameters.

Common Implementations

Neural networks come in a variety of shapes and sizes, each one customized to specific types of data and activities. I’ll introduce you to four common neural network implementations and share insights on where they thrive.

Feedforward Neural Networks (FNN)

The Foundation of Neural Networks

Feedforward Neural Networks, often referred to as multilayer perceptrons (MLPs), serve as the foundational architecture for neural networks. They are versatile and can be used for a wide range of tasks, including classification and regression.

• Architecture🏰: FNNs consist of an input layer, one or more hidden layers, and an output layer. Information flows in one direction, from input to output, without loops or cycles.
• Applications📋: FNNs find applications in image classification, sentiment analysis, and other structured data tasks. Their simplicity and effectiveness make them a valuable choice for many problems.

Convolutional Neural Networks (CNN)

Unraveling Image and Spatial Data

Convolutional Neural Networks are specifically designed for tasks involving grid-like data, such as images and spatial data. They excel at capturing spatial hierarchies and patterns.

• Architecture🏰: CNNs incorporate convolutional layers for feature extraction and pooling layers for dimensionality reduction. These layers allow the network to understand spatial relationships in the data.
• Applications📋: CNNs are the go-to choice for image classification, object detection, facial recognition, and tasks involving grid-like data.

Recurrent Neural Networks (RNN)

Mastering Sequential Data

Recurrent Neural Networks are designed for tasks that involve sequential data, where the order of elements matters. They have memory cells that enable them to retain information from previous time steps.

• Architecture🏰: RNNs have recurrent connections that loop back on themselves, allowing them to maintain a form of memory. This makes them suitable for tasks like natural language processing, time series prediction, and speech recognition.
• Applications📋: RNNs are used in machine translation, text generation, speech synthesis, and any task involving sequences.

Long Short-Term Memory Networks (LSTM)

Overcoming the Shortcomings of RNNs

While RNNs are powerful for sequential data, they suffer from vanishing gradient problems. Long Short-Term Memory Networks (LSTMs) were developed to address these issues by introducing specialized memory cells.

• Architecture🏰: LSTMs include memory cells with gating mechanisms that control the flow of information, allowing them to capture long-range dependencies in sequential data.
• Applications📋: LSTMs are widely used in tasks requiring long-term dependencies, such as speech recognition, language modeling, and sentiment analysis.

These are just a few of the common neural network architectures, each tailored to different data types and tasks. As we explore these implementations further, you’ll gain a deeper understanding of how to choose the right architecture for your specific AI and machine learning projects.

Conclusion

They can be used almost everywhere and they are ground breaking because they can remove the repetitive tasks from daily life, but at their core they cannot replace humans.. So, no worries 😊.

Anyway, If you enjoyed this article, clap👋 and follow me📑! Thanks for reading! Looking forward to seeing you in the future.

If you want a better understanding, I recommend you read the documentation or at least know the differences between server-side and client-side. With that said, let’s go straight into it.

Setup the Network

This is similar to Godot 3.5, but the syntax is a little bit different:

For example, now everything about multiplayer is actually under a global class named “multiplayer” (MultiplayerAPI), meaning when we connect signals, it’s under the multiplayer class.

# GODOT 3.5
get_tree().connect("connected_to_server", self, "_connected_to_server")
# GODOT 4
multiplayer.connected_to_server.connect(self._connected_to_server)

For quick setup, you can use this code:

This code you will add as auto-load and then a create node with lobby and connect the signals, then button to switch scenes. To better understand the code, you can check my previous guide to multiplayer 3.5.

Multiplayer Synchronizer

In this version of Godot, they added the multiplayer synchronizer, which simplifies the programming of multiplayer. Now you need to add it to the player tree, and then just set the properties that you want to synchronize.

This is so helpful and can save you a lot of time.

Multiplayer Spawner

Another new feature that also simplifies the process of spawning and managing networked objects in multiplayer games

But when I was using it, I fell into lots of errors, and then I just implemented my own spawner. If you want to play with it on your own, I recommend you look up the documentation.

RPC Calls

RPCs are essential for implementing networked gameplay and interactions, and that’s why it’s useful to know how to use them properly.

RPC changed syntax to a much simpler version.

For example, now you don’t have any master, puppet, remote, etc., but you have much simpler syntax.

Syntax of RPC Calls in Godot 4:

You will start with “@”, and then you will specify RPC’s mode, sync, transfer_mode.

# mode, sync, transfer_mode (order doesn't matter)
# reliable -> tcp, unreliable -> udp (spamming), unreliable_ordered
@rpc("any_peer", "call_local", "reliable")
func totaly_sync():
  print("Running locally and any_connected_client running this function")

@rpc("call_local")
func local_call():
  print("calling just local function")

# You also can use different channels used mostly for optimizations 
@rpc(any_peer, 1)
func my_chat_func(message):
 print("RPC received on channel 1, message: " + str(message))

# Calling rpc function:
# rpc() == self.rpc
rpc("totaly_sync")
# with arguments:
rpc("my_chat_func", "Hello")

But be careful, you can’t send any node because that could lead to an RCE vulnerability, so you need to work around it. Usually, I declare a list of items that I want to sync and then just send the index through the network.

@onready var items = [
 preload("res://path/to/scene.tscn"),
 preload("res://path/to/scene.tscn"),
]

@rpc("any_peer", "call_local", "reliable")
func sync_items(index: int) -> void:
 spawn_item(items[index])

Controlling the execution flow

The last thing on our list is how to actually control the execution flow. The main way to control the flow is by:

# check whoever executing the code if its the owner of current node
mutiplayer.is_mutliplayer_authority()
# check if its the server
multiplayer.is_server()
# set the ownership of the node
NODE.set_multiplayer_authority()

When you instancing the player, you need to set him as the multiplayer_authority of its node. This will make your life much simpler.

And keep in mind that when using the “_ready()” function, the execution of this function usually happens before the multiplayer_authority is set.

Conclusion

With this article, you should be able to program multiplayer into your game. Sometimes multiplayer can be a pain because something is not syncing properly, and debugging the error is a little bit difficult, but with a cool head, you eventually find the error and gain a little bit more knowledge.

If you enjoyed this article, clap and follow me! Thanks for reading, and I wish you the best luck on your journey.

Common Blockchain Security Threats

Blockchain technology has a lot of potential benefits, but it’s not without its risks. Here are some common blockchain security threats you should be aware of:

51% Attacks

This is when a single entity controls the majority of the network’s computing power. In such a scenario, the attacker can rewrite transaction histories, effectively controlling the network.

Smart Contract Vulnerabilities

Smart contracts are self-executing contracts with the terms of the agreement between buyer and seller being directly written into code. This code can have vulnerabilities that attackers can exploit, which can lead to theft or misuse of assets.

Private Key Theft

Private keys are used to access blockchain wallets and are the gateway to the funds in those wallets. If an attacker can obtain a private key, they can gain access to the wallet and the funds it holds.

Best Practices for Blockchain Security

To mitigate the risks associated with blockchain security threats, there are several best practices you can follow:

Encryption

Use strong encryption to protect your data in transit and at rest.

Multi-factor authentication (MFA)

It’s easy for a password to be stolen, guessed, or hacked, especially if it’s weak or reused across multiple accounts.

Regular Security Audits

Important component of any effective cybersecurity strategy. These audits involve systematically reviewing your organization’s security controls, policies, and procedures to identify vulnerabilities and risks that could lead to security breaches.

Regularly audit your blockchain systems to identify and fix vulnerabilities.

Keep Software and Hardware Up-to-date

It provides security patches to protect against vulnerabilities, improves performance and compatibility, adds new features

Ensure that you are using the latest software and hardware to prevent vulnerabilities.

Emerging Trends in Blockchain Security

The blockchain industry is constantly evolving, and so are the security measures being implemented. Here are some emerging trends in blockchain security:

Decentralized Identity Solutions

Using blockchain technology to create decentralized identity solutions that allow individuals to control their own digital identities.

Blockchain-based Security Tokens

Security tokens are a new type of digital asset that represent ownership in traditional financial assets, such as stocks and bonds. Using blockchain technology, these assets can be made more secure and transparent.

Conclusion

Blockchain technology has the potential to revolutionize the way we store and transfer data, but it’s not without its risks. Understanding the common blockchain security threats and implementing best practices and emerging trends can help protect your data. By following the best practices and keeping up with emerging trends, we can work towards a more secure blockchain ecosystem.

Thank you for taking the time to read this! If you liked this article, please clap and follow me! I hope to see you again in the future.