Yet another year of Google Summer of Code, yay!

It gave me immense pleasure to take forward the project that I had started off in the previous sprint of GSOC’22.

Survey6: In brief

Survey6 is a geo-distributed grid application with C&C Server as its center and the probes as the packet collection application for intercepting IPv6 traffic on linux.

Work done in GSOC’2022

Major parts of the C&C Server and some functionalities of the probe was done in the previous year.

1. Writing protobuf definitions
2. Setting up server database
3. writing connection and disconnection endpoints
4. data capture mechanism at probe
5. CLI for server
6. packaging for server

Work done in GSOC’2023

By the end of this term, we are almost done with an MVP of survey6 such that all the three components are ready to be used. Tasks implemented in this sprint:

1. continuous data collection and rsync transfer
2. connection and disconnection on the probe
3. regular sending of the heartbeats
4. event log at the server
5. probe CLI
6. probe packaging
7. data aggregator — cleaning hourly and daily data

It was a great summer under the guidance of Udesh Kumarasinghe. Thank you for helping me in facing all the road blockers and bring this project to successful end.

Survey6 — Sprint2 @ GSOC’23 was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

GSoC Week 11–12 @SCoRe Lab

gRPC is default for server to server communication these days.

What is RPC?

RPC stands for Remote Procedural Calls. Instead of calling methods on the client, we call methods on the server side. With this, it is easy to change the code on the server rather than changing code for each client. One more advantage on that front is, we can have our server and client in different languages according to our requirements.

How RPC works?

1. serialize the arguments.

Serialization is the process of converting a data object into a series of bytes that saves the state of the object in an easily transmittable form.

2. Send the serialized code to server

3. De-serialize the arguments at server

4. Call the function at server

5. serialize the results and send it to the server

6. deserialize the result at the client.

Protocol Buffers is the default serializer for gRPC. Protobufs are 4 times faster and generates 4 times smaller code than regular json serializtion.

gRPC uses HTTP2.0 which always secure http over TLS.

Timestamp

We are going to import “google/protobuf/timestamp.proto” which is the definition by Google of what is at timestamp. And in our start request we can add a field called time which is of type google.protobuf.timestamp. Generate the code.

Note that the generated type is not Python’s datetime.

Importing Timestamp:

from google.protobuf.timestamp_pb2 import Timestamp

It’s a different type, and you need to do conversion back and forth every time. We set the request time field with fromDatetime(python-datetime) as follows:

If you have a Timestamp you can convert it to datetime using the toDateTime() method:

And there is also a method to get the current time. So you can do Timestamp().GetCurrentTime().

Serializing Protobuf structure to JSON

Json is a very popular serialization format. Sometimes we like to take the types that were generated from protocol buffers and instead of serializing them to protocol buffers serialize them to json. When I want to convert it to json, I’m not going to use the json from the standard library but this json serialization library:

from google.protobuf.json_format import MessageToJson

It knows more about protocol buffers types and will do the right things. We get nice json document.

json = MessageToJson(proto_message)

gRPC Testing

Testing your code is important. gRPC has some utilities for testing. But I find out that only using pytest/unittest, you can test gRPC without the need for these test utilities. Here, it is for server testing:

• Create a request with all the data I need.
• For the context use mock from the mock library
• Create server.
• Call the method with the request and the context.

context = MagicMock()

service = ClientConnectionService()

request = pb2.ClientConnectRequest(host_name = “User Test 2”)

request.request_epoch_time.FromDatetime(datetime.now())

response = service.ClientConnect(request,context)

self.assertEqual(response.connection_status,pb2.SUCCESS)

self.assertIsNotNone(response.uid)

gRPC : tips, tricks and testing was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

How do we distribute a python package that has multiple PyPi dependencies? Any project with debian dependencies is easy enough to package:

• we mention dependencies in the control file of the source package
• have executable files in /usr/bin
• the project files in /usr/share

and we are done with it!

However, when we have pip dependencies, what about the source files of those? Do we get them on the fly on the user machine or should we bundle the source code with our project code? This thread has a nice discussion on the same.

Approach 1

One major approach is to create debian packages for the dependencies, , list them as requirements in your package, and ship all the .debs or source packages. This can be done using stdeb.

• Create setup.py
• Install stdeb

pip install python-stdeb

• This command creates /debian source package

python3 setup.py — command-packages=stdeb.command debianize

Or

If you want to directly create .deb, use this command :

python setup.py --command-packages=stdeb.command bdist_deb

• Add install and postinst files in /debian [as per requirement, In my case:

a. Install script copies .envand the service file

b. Postinst activates the service ]

• create .deb and /DEBIAN (binary package)

dpkg-buildpackage -b -rfakeroot -us -uc

• Installaing the package

sudo apt-get install ./package_name.deb

Approach 2

We could have installed the PyPi dependencies in the user system from postinstall script by doing pip install. It is a serious security risk, definitely a no-no. You wouldn’t even be able to inspect the dependencies by extracting the deb, because they are downloaded and installed at install time. It's an approach that completely bypass the repositories system. No concerned user would be happy with a package that, behind the scenes (and as root, remember!), downloads additional non trusted software from non trusted sources.

So we found a small way out. We create a new user while installing the packages, we install the dependencies in that user and keep our project code in that same user directory. That’s all. So we have have eliminated the problem of security by doing this as the installation does not affect entire system.

You can find the implementation of the above approach here.

We than build the source package using:

dpkg-buildpackage -b -rfakeroot -us -uc

References

1. https://askubuntu.com/questions/327543/how-can-a-debian-package-install-python-modules-from-pypi
2. https://pypi.org/project/stdeb/

That’s all, we are done and dusted with packaging.

BR!

Debian packaging of a python project was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

What is package?

A package is simply an archive that combines all the libraries and binaries ,configuration and all the dependencies required to install the programs (i.e application, software or languages)

Why?

Histrionically in UNIX, every program had to write compiled linked and run. Then UNIX got the ability to use libraries (“shared objects”), ELF executables, etc. To solve the task of building more complicated software easily, make was developed. Make uses an makefile we had to create the makefile on our own so this was also a laborious task.

The main problems with make file’s are

• The specified package is not present on the location.
• Package is depend on another package.
• If the downloaded package is not working properly.

The above problem can be solved by manually downloading the package. But there may be an version problem that downloaded package version is differant than we required or does not support.

This is called an “Dependency Hell” which is solved by package manager. How exactly the package manger works and solve all this problem, we discuss in following section.

Package Managers

1. For a Debian based distribution system i.e. Linux mint , ubuntu , Debian ,pure OS etc. The base package manager is DPKG (Debian package). the frontend package that is generally used is APT its like cherry on the ice-cream top. There are some traditional frontend package managers that is also used with DPKG is APT-GET.
2. For a RPM based distribution system like fedora, centos ,RHEL , we use Red-hat package managers (rpm) . The base package manager for a RPM based distribution system is RPM. The frontend package managers that is used more often with rpm is YUM ( YELLOW DOG Updater, Modified) .The more advanced version that we uses more often as a frontend package manger is DNF.

dpkg

dpkg is a tool to install, build, remove and manage Debian packages. The primary and more user-friendly front-end for dpkg is aptitude(1). dpkg itself is controlled entirely via command line parameters, which consist of exactly one action and zero or more options. The action-parameter tells dpkg what to do and options control the behavior of the action in some way.

dpkg suite

dpkg — Debian’s package maintenance system

This is the dpkg suite of programs that form the foundation of the Debian’s
package management system; on the lower layer there are ‘dpkg-deb’ and
‘dpkg-split’ programs handling the binary formats, and ‘dpkg-source’ program
handling the source formats; there is a collection of tools to handle building
source packages into binary packages; there is the medium-level and less
user-friendly command-line interface (CLI) in the form of the ‘dpkg’ command;
and then there is the terminal user interface (TUI) ‘dselect’ program (which
has gone out of preference in favor of the apt (CLI) and aptitude (TUI)
programs).

The dpkg suite also includes some other programs currently maintained
on external repositories, namely ‘dpkg-repack’, ‘dpkg-www’, ‘dupload’
and ‘debsig-verify’.

.deb

A deb file is an archive that contains data. Marked with the .deb extension, it is used to easily distribute and install programs for Linux Debian and derivatives. Deb files are handy when your app needs to take care of additional dependencies, integrate itself with the desktop, run pre and post install scripts and so on.

Internally, a deb package contains a collection of folders that mimics a typical Linux file system, such as /usr, /usr/bin, /opt and so on. A file put in one of those directories will be copied to the same location in the actual file system during installation. So, for example a binary file put into <.deb>/usr/local/bin/binaryfile will be installed to /usr/local/bin/binaryfile.

Source directory

package contains the following files in its debian/ directory:

• changelog
• compat
• control
• copyright
• docs
• node-libravatar.install
• rules
• source/format
• watch

A very good reference apart from the official documentation is this:

https://feeding.cloud.geek.nz/posts/whats-in-a-debian-directory/

Binary directory

Create a temporary working directory to make your package in. On the outside instead, all deb package files follow a specific naming convention:

<name>_<version>-<revision>_<architecture>

That is:

• <name> – the name of your application;
• <version> – the version number of your application;
• <revision> – the version number of the current deb package;
• <architecture> – the hardware architecture your program will be run on.

Put your program files where they should be installed to on the target system. For example, suppose you want your program to be installed to /usr/local/bin:

mkdir -p hello_1.0-1_arm64/usr/local/bin

The -p flag to the mkdir command will create nested directories. Then copy the executable file in there:

cp ~/YourProjects/Hello/hello hello_1.0-1_arm64/usr/local/bin

The control file lives inside the DEBIAN directory. Mind the uppercase: a similar directory named debian (lowecase) is used to store source code for the so-called source packages. This tutorial is about binary packages, so we don't need it.

Let’s create the DEBIAN folder first:

mkdir helloworld_1.0-1_arm64/DEBIAN

And then create the empty control file:

touch helloworld_1.0-1_arm64/DEBIAN/control

Open the file previously created with your text editor of choice. The control file is just a list of data fields. For binary packages there is a minimum set of mandatory ones:

• Package – the name of your program;
• Version – the version of your program;
• Architecture – the target architecture;
• Maintainer – the name and the email address of the person in charge of the package maintenance;
• Description – a brief description of the program.

For example:

Package: hello
Version: 1.0
Architecture: arm64
Maintainer: Internal Pointers <info@internalpointers.com>
Description: A program that greets you.
 You can add a longer description here. Mind the space at the beginning of this paragraph.

The control file may contain additional useful fields such as the section it belongs to or the dependency list. The latter is extremely important in case your program relies on external libraries to work correctly. You can fill it manually if you wish, but there are helper tools to ease the burden. I will show you how in the next few paragraphs.

Building

This is the last step. Invoke dpkg-deb as following:

dpkg-deb --build --root-owner-group <package-dir>

So in our example:

dpkg-deb --build --root-owner-group <helloworld_1.0-1_arm64>

The --root-owner-group flag makes all deb package content owned by the root user, which is the standard way to go. Without such flag, all files and folders would be owned by your user, which might not exist in the system the deb package would be installed to.

The command above will generate a nice .deb file alongside the working directory or print an error if something is wrong or missing inside the package. If the operation is successful you have a deb package ready for distribution.

Installation

sudo apt-get install package.deb

Final words

Now that we know packaging we will see different ways of packaging of python application in the next blog.

References

1. https://medium.com/@knoldus/working-of-package-manager-in-linux-775ca131c388
2. https://sauravstwt.medium.com/package-management-in-linux-7db916968247
3. https://www.internalpointers.com/post/build-binary-deb-package-practical-guide
4. https://feeding.cloud.geek.nz/posts/whats-in-a-debian-directory/
5. https://www.senties-martinelli.com/articles/debian-packages
6. https://github.com/FooBarWidget/debian-packaging-for-the-modern-developer

Packaging Overview was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

GSoC Week 6–7@SCoRe Lab

Interesting developments this week!! It started small and led to a completely different output.

As stated in the earlier blog, I had to write an installation script. Installation steps at this point of time are as follows:

1. clone the repository
2. create a virtual env
3. install pip requirements
4. copy service file in its directory
5. get the service up and running

User would clone the directory and then run setup.bash. Lets start with scripting now. I would give you a gist of all the concepts used and combine them all to write and execute our script.

Building Blocks🧩

Unix has more than one possible shell: C Shell (csh), Bourne Shell (sh), Korn Shell(ksh), Bourne Again Shell (bash).

Executing v/s Sourcing a bash script

A
> ./myscript

B
> source myscript

The differences are:

When you execute the script you are opening a new shell, type the commands in the new shell, copy the output back to your current shell, then close the new shell. Any changes to environment will take effect only in the new shell and will be lost once the new shell is closed.

When you source the script you are typing the commands in your current shell. Any changes to the environment will take effect and stay in your current shell.

Read here for detailed reference.

.bashrc

The .bashrc file is a script file that’s executed when a user logs in. The file itself contains a series of configurations for the terminal session. This includes setting up or enabling: coloring, completion, shell history, command aliases, and more. The ~/.bashrc file determines the behavior of interactive shells. It gets executed every time a user starts up a fresh terminal session in interactive mode on their Linux system.

Location: ~/.bashrc

(~ Tilde indicates user home directory)

PS: You can find a skeleton of bashrc in /etc/skel/.bashrc. If different users want different bash configurations then you must put a .bashrc file in that users home folder.

Aliasing

A Bash alias is basically a shortcut you can set for a command. If you have to type some long command out often, you can shorten it or make it more memorable with a Bash alias.

$ alias start= “systemctl cnc-server start”

Now, you can just use the command start whenever you want.

However, there’s a catch: this is only in your current shell session. Close your window, and this alias is gone.

If you plan on keeping that alias for the future, you will have to open up either your ~/.bash_profile or ~/.bashrc files with a text editor and save the aliases directly there. You can also organize the aliases in ~/.bash_aliases and run that from ~/.bashrc as follows:

if [ -f ~/.bash_aliases ]; then
    . ~/.bash_aliases
fi

Putting it all together

Compilation was pretty easy, put all the commands that you would have entered while manual installation.

Additionally I used system aliasing to start, stop and view status of the service. I wrote an aliasing script and catted it to ~/.bash_aliases. Aliasing gave better accessibility, look and feel.

I had used function aliasing so I could accept arguments. Here is the code snippet of aliasing.

function cncserver { 
if [ -z “$1” ]; 
then echo “Enter a command: start / stop” 
elif [ “$1” = “start” ]; 
then systemctl start survey6_server 
elif [ “$1” = “stop” ]; 
then systemctl stop survey6_server 
elif [ “$1” = “status” ]; 
then systemctl status survey6_server 
else echo “Command not supported” 
fi 
} 
alias cncserver=cncserver

We’ll be packaging the project in the next blog :)

Happy Coding!

Scripting Installation was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

Well, nothing very fancy this week. Basics and essentials. I managed to disconnect the client with one gRPC call. It disconnects the client from the server. The server basically maintains two tables: one for connected and other for disconnected clients or maybe you can call it archives. The archives contain the hostname and the last active time of the probe. So the server, on disconnection removes the probe info from the connected clients table to put it into the archives.

With this we are almost done with the massive issue that we started in week 3. The PR was reviewed and merged.

Plans for week 6:

1. Automation of installation using bash scripts

2. Bash script aliasing for running the commands: starting and stopping of server, health checks to start with.

3. Unit testing of code written so far.

4. Error handling and logging.

Project: Survey6

See you with next weeks updates! Tc.

Some implementation touches.. was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

We use SQLite for data persistence here. We need the list of registered probes to accept data from valid users and trace the health of each of them. SQLite is pretty easy to use with python3. It comes bundled with python3 so we don’t need to install any separate packages.

Standard SQL queries work with SQLite too. Quoting resources that I refer to for using SQLite in python:

1. https://medium.com/analytics-vidhya/programming-with-databases-in-python-using-sqlite-4cecbef51ab9
2. https://towardsdatascience.com/python-sqlite-tutorial-the-ultimate-guide-fdcb8d7a4f30

& the official docs of course.

Whenever I receive a new request from the probe to connect, I have to save its details in my server’s registry. The fields that I would require for the probe are hostname, registered time, last active time, current status and last time when data was sent.

Then, I was required to run the server as the system service. You might have already guessed it what I might have used: Systemd

Creating services is a good thing when you need to run a certain application every time your linux computer boot for example. Basically, Systemd is an init system. We do have some other init systems like SystemV. There are plenty of resources out on the internet. Please allow me to help you out with a few I like:

1. https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6
2. https://www.golinuxcloud.com/beginners-guide-systemd-tutorial-linux/
3. https://medium.com/the-code-vault/systemd-simple-service-examples-ce586afeed27
4. https://medium.com/@vinicenter/creating-a-service-in-linux-using-systemd-f2ded253cd92
5. http://0pointer.de/blog/projects/systemd.html

Just to refresh your memory, I would like to re-iterate through the difference between BASH & SH.

BASH is Bourne Again SHell. It is the superset of SH & BASH is a lil more powerful than SH. BASH supports more commands. A very significant example is source. source is a command implemented in bash, but not in sh. { Source, as in, source venv/bin/activate }

This is a trivial pointer rather but might save you sometimes :)

Thank you for reading it all.

Happy Coding!! BR!!

System Service & DB was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

This week started off with creating the C&C Server. The server is built using gRPC.

gRPC ?

gRPC is a modern open source high performance Remote Procedure Call (RPC) framework that can run in any environment. This is a good article that might convince you if you gotta ask “why gRPC?”.

That is its main advantage over REST: gRPC, in most cases, is way faster and more robust, as it defines a specific set of rules each request and response should adhere to.

By default, gRPC uses Protocol Buffers, Google’s mature open source mechanism for serializing structured data (although it can be used with other data formats such as JSON).

For most of the message transfers of our project, we too use the protobuf!

Protocol Buffers

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data — think XML, but smaller, faster, and simpler.

Protocol Buffers support a wide variety of languages and data types. The official documentation of protocol buffers is very well maintained & easy to understand.

So, in this week, I sat down to write the protobuf definitions for Survey6. The services mainly include connecting/disconnecting the probes to/from the server and the health check mechanism. The definitions include the function calls and the message formats. I kept in mind this style guide to maintain the quality of my code at the same time.

P.S: Protobufs also support Timestamp datatype :)

Directory structure was also setup during this week. Python3 does no more require __init__.py to initialize a package. A lovely guide to python packaging and modules is here.

I hope this helps :)

Up Next: Setting up system service & connecting to database.

The Server: gRPC & Protobuf was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

The coding period officially began on 13th June’22 with a lot of excitement. Here is an update from week 1 & week 2 ….

Survey6 is a fresh project. The initial push that I required was provided by mentor Ravindu De Silva (Thanks) by creating a GitHub issue. The task at hand was to create first function of the code base : to collect, annotate and store the captured IPv6 packets on the probe. Fairly easy! Scapy helped me with collection and storage. Guess what I did with annotation? After breaking my head all over, I made system calls to annotate file metadata which I later got to know is local to the file system. What a rookie!

When I was pretty much done writing the piece of code, It was a mere 20 line function! I knew something was missing then. Even though size of code that we write is secondary, I had missed big time error handling! Absolutely, feel free to judge me. The point of me writing this blog is not to show how stupid one can be, instead, I intend to help someone from averting such mistakes and maybe understand that this is how we grow by making mistakes and having mentors to guide you out. I learnt a lot from the elaborate reviews by my mentors Ravindu De Silva and Maanas Talwar.

I improved upon error handling, error logging, project structure management and of course metadata annotation. Good coding practices also including usage of environment variables whenever we use variables. At some point during development you will need to think about the environment in which your application will run and the potentially sensitive or environment specific information your application will need to perform its tasks. Where I am required to transfer the files, annotating file’s system meta data won’t help. I had to make something portable. On receiving some advice on the same, we finally stored the metadata in a .json file.

In the second week, I made changes corresponding to all the reviews and the resulting code was a lottt better!!! I was indeed happy with the final code. By then I had also improved upon my writing style and I started figuring out the best practices of writing a python code. But the feeling of my first PR being merged was🔥.

Resources

Here is a list of some good resources that had helped me (just in case) :
1. Robust Error Handling
2. Using environment variables with python
3. Use this .gitignore template by GitHub

Capturing the packets… was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.

When I thought of entering the world of open source, what could be a better platform than GSOC? I browsed through various organizations from the past year GSOC; I went through their projects, tech stack & code base. Honestly, all of it was overwhelming for me. Either the tech stack was completely alien to me or the topics did not appeal my interest. I was ready to give up. Here, I would like to thank my mother, peers and seniors for pushing me and keeping me alive.

I continued my search until in March I came across Survey6 Project under Web Telescope |SCoRe Lab. My interest in cybersec and networking was matched by Web Telescope.

About Web Telescope

Web Telescope is a sub-org of SCoRe Lab that contributes and promotes cybersecurity research.

My Project: Survey6

To aid cybersecurity researchers in their study about IPv6 research and promote its study, people at Web Telescope have come up with a tool called Survey6 that collects all the IPv6 packets via probes and send the collected packets to the server where the packets are processed and stored for further studies. The collection is done both actively and passively.

My Task

My task here is to create a passive packet collector(probe) that would be Linux based CLI application, the server that would collect the packets and set up effective communication and packet files transfer between the probe and server.

I saw this as an immense opportunity to grow and learn. After contacting the mentor, understanding the project, getting my doubts cleared, learning some of the tech stack and making a prototype, I finally typed a proposal.

And, there was a long wait until the selection announcement ( 20th May, 2022 ). I was indeed thrilled.

Well, then started community bonding period: I had a meeting with my mentor where I got to know so much more about the project, aligned my vision of the project with the vision of the organization. We refined certain implementation details and discussed the tech stack. I used the rest of my period to get familiar with other tools from the tech stack and architecture details.

Gratitude

I am grateful to my mentor, Ravindu De Silva for the selection. I also thank my mom and my peers at Society of Robotics & Automation, VJTI who motivated me to make an application. Special thanks to Kush Kothari for helping me review my application. Sincere thanks to all those people whom I have crossed path with.

I am looking forward to a great summer with SCoRe Lab and hope my contributions make some difference.

Thanks for reading all the way here. I’ll keep you updated with my updates. Stay tuned!!

GSOC’22 @SCoRe Lab was originally published in SCoRe Lab on Medium, where people are continuing the conversation by highlighting and responding to this story.