Using the internet nowadays is very intuitive. We type down the URL of the site we want to go to in the web address bar and press enter. All websites are located through long strings of numbers or characters that are otherwise known as IP addresses. This would be a very unwieldy way to navigate as memorizing such longs strings would be tough. Instead, the IP address is mapped to a domain name. A URL incorporates this domain name with other information if needed to direct to a specific web page. So how does the browser work from URL to server? Let’s have a deeper look and see what happens in the background.

How the browser checks for the website

First, the browser will check its own cache if there is an existing copy of the URL stored. The main function of the cache is to speed up internet surfing by keeping an offline copy of the web site. The amount of resources kept within the cache is finite so there is an expiration date on cached files. After the expiration date, these files are in a stale state. These files are sometimes not deleted as the browser can send a request to the target server to check if these files are still fresh and can be re-used. If the browser cache does not have anything, then the browser will make a system call to the OS to check the OS’s DNS cache. If the OS’s DNS cache has no record of the web address, then the browser will check the router cache. If all these checks fail, then the ISP’s DNS records are checked.

Getting the IP address

If the url is not in any of the caches, then the ISP starts a DNS query. A DNS query involves a recursive search of all DNS servers until the IP address to the web address is either found or unable to find it. Most address nowadays contain up to a third level domain. For “https://www.holbertonschool.com", the query will start at the root name server. This server will send the query to the relevant top-level domain server, which in this case is “.com”. After the top level domain is the second level which is “holbertonschool”. Within the server that contains the second level domain, the matching IP for the third level domain “www” is sent back to the browser.

Initiating a connection to the server

Once the browser gets the IP address, the browser will start a TCP connection with the server. Once a connection is established, the browser will initiate a TCp three-way handshake (SYN-SYN-ACK). First, the browser will send a SYN(synchronize) signal to the server to see if the server is open. If it is, the server will send back a SYN-ACK signal that signifies it has acknowledged the initial syn request. Last, the browser will send an ACK signal again to acknowledge the server response and start the actual transfer of data.

Servers like burning walls for protection

For protection against unwanted connections, servers are usually protected with firewalls. Firewalls can either be software or hardware based and have multiple types, such as packet filtering, stateful, and application layer. Firewalls mainly look at the data coming to the server and uses rules to determine if the data should be accepted or rejected. Since it is difficult to think up of every rule possible for all the possible data variations, there is a default policy that consists of the basic accept, reject, and drop actions. Where to put a firewall is preferential as it can be applied to all incoming ports at the same time like both the web server and load balancer.

Talking to each other in a secure way

Once the TCP connection is established, the browser starts by sending a GET request for www.holbertonschool.com using Hypertext Transfer Protocol (HTTP). This request will first be routed to the IP address and can possibly connect to a load balancer depending on the traffic of the web server. The load balancer’s job is to spread the incoming connections to 2 or more servers based on its settings such as round robin, least connected, and etc. A firewall can be placed here to protect the load balancer so data traffic will also be monitored here.

Sending data back

Once the data passes the load balancer and is sent to the right web server, a response will be generated. In order to get any extraneous data that is not static, the web server will connect to a dynamic application server. The dynamic app server’s job is to provide a way to display non-http data back to the browser in such a way that can be read in an http format. Unfortunately, the web server can only work with HTTP and CSS. So, all non HTTP code is used in an application server.

An application that is coded in a language like Python can used from the application server and can even connect to a database to grab data. This database can be relational like MySQL or non relational like Mongo. All the data collected from the application server will all be packed in an http format and sent back to the web server for the web server to finish the data request. Within the server response, there are different instructions on how to handle the http response such as the status code, cache control, content encoding, cookie settings, and etc.

Voilà

Once the browser gets the http response, it will parse through the http code and create the basic web layout first. Then it will check if there are additional resources to get from the server such as images and send GET requests for those. When the loading is done, the entire page will be cached so this entire process will stop at the browser cache check phase. So really, all the caching is very useful when improving the end user’s experience when web surfing.

As of late, IoT devices have been expanding the limits of the internet and data collection. Anything nowadays can be an IoT device from our washing machine to our tvs and even our light bulbs and wearables. This is made possible due to the lowering cost to manufacture microprocessors and wifi antennas. We have even integrated our homes to a degree with devices such as Nest and several home security devices. Nevertheless, while IoTs are a recent hit, the ideas and concepts for it certainly aren’t.

History of IoT

Internet of Things(IoT): “Sensors and actuators embedded in physical objects are linked through wired and wireless networks, often using the same Internet Protocol (IP) that connects the Internet.”

The term IoT itself is not a recent word by a long shot. It was first introduced by Kevin Ashton back in 1999 when he was introducing radio frequency identification (RFID) technology as a way to connect devices together. However, the idea of connecting devices together through the web was around since the 70s. The concept of IoTs did not gain traction until 2010. There are a few reasons why IoTs suddenly boomed in recent years. They include cheaper electronic parts, large-scale data analysis tools, the evolution of networks like 3G and 4G, and cloud storage. Nowadays, companies are using IoTs to collect data to help determine how to improve and innovate products to better fit the market.

Development of IoTs

Since 2010, IoTs have been gaining interest from companies as it can have a noticeable effect on businesses and consumers. IoTs are generally limited to data collection and uploading data to the internet. So, the main purpose of an IoT device generally depends on what purpose the device is for and what kind of data it should record. For businesses, IoT devices can apply to multiple areas depending on what the end goal is. For example, in warehouse settings, IoT devices can be used to track and manage inventory changes to help make operations smoother. However, the most important use would be the accessibility to data when the consumer is interacting with the device. This data can help businesses make devices smarter and offer a better experience to the user while simultaneously interpret the data for improvement and growth.

With the integration of IoT devices in everyday life becoming smoother as more data is being collected, consumers start to realize that IoT devices can afford a plethora of ways to improve their lifestyle. Even Raj Talluri, Qualcomm senior vice president of product management, has stated that “Consumers are already demanding that their various devices connect and sync seamlessly — from their fitness trackers to cars to home security cameras”. Since IoT devices are becoming more streamlined and intuitive with each successive generation, its applications can be applied to different areas such as home security and health monitoring. What also makes these devices more user-friendly is how all the devices are tied together. There are multiple ways to connect IoT devices to the internet and one common method is Bluetooth. With Bluetooth, all data collected can be sent to your phone and have the data redirected to servers on the cloud. After the data is processed, the results will be sent back to your phone and be ready for reviewing. This simple passing of data is what makes IoTs so powerful and it is only made possible thanks to the cloud.

The advent of cloud data storage was a boon to the growth of IoT devices as it provided a cheap and sure method of storing collected data and working with it. This allowed IoT devices to forgo any computing work and leave it up to the servers, which meant IoT devices can be built with more power efficient parts and downsize a lot. With the smartphone acting as a common IoT platform and the computing power left to the cloud, more possibilities opened for IoT devices which can be seen as the “number of global developers building apps for the Internet of Things (IoT) rose 34% year-over-year” due to the increased interest in IoT and its possible applications, developers are going for more niche markets such as the manufacturing sector. Applications of IoT in manufacturing will include machine efficiency surveillance, machine maintenance, and optimizations. With businesses turning to mobile solutions, it has been estimated that by 2020 the number of connected IoT devices will triple from 10 billion to 30 billion and businesses will be the top adopter of IoT devices.

For manufacturers, the influx of data from IoT devices is seen as a way to improve and keep track of multiple things at once. There a couple reasons why data collection is seen as an important resource which are: optimization, digitization of the factory, and tracking the factory environment. Optimization generally resolves around finding and eliminating unnecessary work in the production flow which leads to reducing wasted materials. Digitization of the factory does two things: promotes the management of equipment remotely and alert when parts need to be replaced soon to minimize work interruption. Tracking the factory environment is important as it can enable asset tracking and protection and ensure the safety of the workers. There can be multiple ways IoT can be applied such as health monitoring in real time, air quality management, security enforcement, and constant regulation and observation of dangerous substances and gasses.

AI and IoT

With IoT’s explosive growth, there is a lot of data to process and much more will be coming in the near future. While data analysis tools are available, they are not as valuable as what a functioning AI can bring to the table. With an AI, it can be possible to increase the amount of data processed significantly while fulfilling the purpose of the device. For example, wearable sensors have become a mainstream item and can be applied in many scenarios such as healthcare. Currently, there are health sensors that can detect and send real-time health data to health professionals. With an AI, it can be possible to predict when a heart attack can happen and alert the proper authorities on time.

Current Limitations of IoT devices

One of the biggest problems for IoT devices is the overall cost to make an IoT device. While costs for electronic parts have indeed gone down, the amount of different parts added to, let’s say, a lamp would make it cost way more than a conventional lamp. Even an IoT light bulb is many times more expensive than a simple cfl light bulb. This mainly applies to small and low-cost appliances as the technology to fabricate microprocessors has not progressed enough to either make it cheaper. This is not seen however in appliances that either have empty space to accommodate more electronics like washing machines or have been remodeled from the ground up like smart watches or fitbits.

Another problem is the lack of bandwidth and infrastructure to accommodate for the sudden increase in wireless devices. In the past, the most that would be connected to a router would be a couple pcs, a laptop and a phone. Now, there can be as easily as 10 devices connected within a household and that can put a significant strain on the network infrastructure if this is repeated across a country. Improving or overhauling the infrastructure is an option but it is a very costly one and will also take a huge amount of time and labor. Even if the current infrastructure were to be fixed, there is no guarantee that by the time it is finished, the rate of growth of IoTs will plateau enough or the updated infrastructure is enough at all.

Despite the lack of a growing infrastructure, there is another problem that is stunting the growth of IoT devices and its integration with society which is a common IoT platform. Currently there are no universal standards where multiple IoT apps can seamlessly integrate with each other which can cause a problem when trying to operate multiple devices with a central app. Any current solutions to this problem are either still in development or simply not a fully capable platform for all devices.

But the most pressing issue with IoT devices is security and how data is secured. Since IoT devices record and send data, this opens a slew of privacy concerns. To date, there are numerous data collecting devices that can aggregate a ton of sensitive data such as Amazon echo, WeMo, and other smart house products. These devices can collect data such as where you are in your house, whether or not you’re home, or simply if you’re healthy or not. Consumers generally don’t think about the ruggedness of an IoT’s security but recent events have shown that IoT hacks have become a pressing concern. According to a study from Ponemon Institute, security of an IoT app is repeatedly sidelined due to either budget restraints or simply wasn’t an important goal from the beginning. In on portion of the study aforementioned, out of 600 surveys 26% of them stated that security testing was not done and an average of 38% of IoT apps tested contained vulnerabilities. The lack of security can be mainly attributed to the rushed completion of the products as 75% of the surveys stated pressure on the development team is why the IoT apps contain so many bugs.

Final thoughts

Currently IoT is in its early but fast stage of growth. It has already gained traction with the consumers and there is demand for more products. There are also many incentives to further develop IoT technology not only for consumers but for businesses as well. However, there are a few technological bottlenecks to address such as the network infrastructure and the debate on how our personal data should be handled and stored.

Sources:

• The internet of things: convenience at a price
• IoT Past and Present: The History of IoT, and Where It's Headed Today
• Internet of Things: How Much Does it Cost to Build IoT Solution?
• What is IoT devices (internet of things devices)? - Definition from WhatIs.com
• How AI and IoT must work together
• The Internet of Things in manufacturing: benefits, use cases and trends
• IoT Device Security is Being Seriously Neglected - Aberdeen Essentials

Around the 1940s, AI research was conceptualized but was met with setbacks due to the lack of powerful computer hardware needed to power heavy computational algorithms. Today, AI is a booming field with plenty of practical applications and research topics.

What is an AI?

When AI was first conceptualized during the 1940s, researchers were entertaining the idea of creating a digital brain. The basic biology of a brain can be emulated with simple computer concepts. For example, neurons within the brain only activates once it passes a certain stimulus threshold. Once that threshold is met, then the neuron will fire. This is similar to digital signals where signals are either at 1 or 0 as an example. Another concept used was the theory of computation. This theory deals with how efficiently problems can be solved with an algorithm.

Currently, AIs can be split into two types: narrow and general. Narrow AIs are systems that have been taught to handle specific tasks without being hard coded to do so. The most common types are language recognition and visual recognition systems. Narrow AIs are currently the most widely applicable as they can be used from for video processing to cancer detection. General AIs are systems with the intelligence and mental acuity of humans . They can do anything from simple tasks like making a spreadsheet to abstract operations like holding a conversation using its own database while understanding the subtle undertones of the conversation. Current leaders in AI research are split on how soon this type of AI will come to fruition given the current hardware limitations and limited understanding of the physiology of the brain.

Machine Learning and its relation to AI

In order to pursue making an AI there were many approaches to the problem. One approach was to hard code how the program was to react. This approach never led to a major success and was unwieldy to make. The difficulties of creating a program to react to any sort of input led researchers to believe that AI systems had to be able to acquire their own understanding by extracting rules and such from raw data themselves. This approach enabled computers to work on problems and make decisions that appear subjective. This is now known as machine learning. In a nutshell, Machine Learning is a subfield of AI and contains further subfields like Representation learning and deep learning.

Given how machine learning is done, the main problem is how the algorithm represents the data it is given. For example, if you were to give a machine learning AI a random picture, it will not be able to determine any key information from it. The data needs to be formalized in a way that the algorithm can understand the problem at hand and return an answer. For more advanced machine learning algorithms, the main problem is how the program will extract the proper information properly. For example, a visual recognition system can properly recognize shapes like a circle. If you were to ask it to pick out where is a wheel of a car, then the task becomes exponentially harder. What seems like an autonomous and easy task becomes challenging as it is not simple to describe a wheel digitally. While a wheel has a simple shape, its shape can be obfuscated with shadows, glare, or even another object sitting in front of it. So instead of coding the program on how to look for a wheel, it will be trained to recognize what a wheel looks like.

Different methods of training

There are two common methods of training a machine learning algorithm: supervised and unsupervised learning. In supervised learning, the system is given labeled data sets while unsupervised learning involves unlabeled data sets.

With supervised learning, the system learns by comparing its own output with the labeled output given to it and adjusts accordingly. The end goal is to have the system take in unlabeled data and output whatever the desired result is. For example, using the previous example of a visual recognition system recognizing a wheel as a base it would be given data sets with the images labeled with a wheel or have no wheel. After being trained to recognize a wheel, it should be able to take in unlabeled data sets and point out if there is a wheel or not.

Unsupervised learning involves giving the system large quantities of unlabeled data which the algorithm will sift through and find patterns it can identify. Without being told what is what, it is possible for the algorithm to look at seemingly disconnected data sets and organize it in meaningful ways. This type of learning is commonly used with transactional data or data with customer information. While this type of learning can be used as a way to refine purchase suggestions, this method is also used to catch fraudulent card usage as the system is looking for purchases that don’t follow the usual trend of the consumer.

Example Learning Algorithms

Linear Regression:

This algorithm is one of the most well known and a very popular one for machine learning. This algorithm is used for supervised learning. Linear Regression is a linear model where there is a linear relationship between the input variables and the single output variable. There are multiple ways to train this algorithm like Ordinary Least Squares. We are basically trying to make a line fit through a quantity of data in such a way that the addition of all the distance from the line squared is minimized.

Logistic Regression:

This algorithm is used in supervised learning and is very popular for problems where the output is either a yes or a no. It can take in one ore more input variable and measures the relationship between the output and input variables using probability. These probabilities are then converted with a function to a value between 0 and 1. Like linear regression models, this model works best when attributes unrelated to the output variables are removed.

K-Means Clustering:

K-Means clustering is an example of an unsupervised learning algorithm. This algorithm works by specifying how many points(centroids) you want on a graph which is K. Then all the points are placed randomly in a graph. We then take our data and see which centroid each data point is nearest to and assign this data point to that cluster. After assigning all the data points to it nearest cluster, you would take all the data points that were assigned to a particular cluster and average all the vectors up and that becomes the new centroid. This keeps on happening until data points are no longer being assigned to different centroids.

Deep Learning:

Deep learning is a part of a concept called representation learning. Representation learning is the idea for the algorithm to discover how to interpret data to in order to categorize it. An example would be to classify shapes. For simple 2D shapes, we can easily differentiate shapes based on the number of edges i.e circles have 0, triangles 3, and squares 4. So the representation is the number of edges. This can get more complicated for 3D shapes as the number of edges does not directly correlate to an equivalent 2D shape but our brains can still recognize these differences. It is this process that representation learning is trying to emulate.

The main problem with working with representation learning is how to get the algorithm to process raw data without being confused with variations like color changes and shape obfuscation, and even accents via regional differences. Deep learning approaches this problem by breaking down representations into simpler ones using a neural network. For example, lets say we want the final representation to be a car. Instead of looking for a car straight away from a random picture, we would start with looking at the general color of objects. Then we would break it down further by looking for edges, then contours, then object parts and compile all the results into a result like a car or human.

What is a neural network?

A neural network is a digital model of the our brains. It is emulating how neurons work and tries to mimic biological functions like the firing threshold of a neuron. Of course, these models aren’t meant to be accurate representations of the brain but rather a copy of the computational functionality of the brain. In order to understand the basic functionality of a neural network, we will need to understand its base component, a perceptron.

A perceptron is the digital equivalent of a single neuron. It can take in a couple binary variables and output a single binary variable. This is similar to a biological neuron as a single neuron can take stimulus from various other neurons via synapses and either be activated or not. This difference in activation can be comparable to a digital signal, which only has a 1 or 0.

However, in order to control how a perceptron gives its output all inputs are given a weight, or a real number that ranks how import a particular input is. Once the inputs, adjusted by its respective weight, is greater than a certain threshold value, then the perceptron would give either a 1 just like a biological neuron. By varying the weights and threshold numbers, we can create a completely unique decision-making model that can “assess” different inputs to make a decision. Of course, a single perceptron would not be capable of complex decisions so many are needed and one example of this is a multiplayer perceptron.

A multilayer perceptron is a type of forward-feeding network that takes in a set of input values and returns an output value. The entire model can be seen as a mathematical function that is broken into simpler functions that provides a new representation of the input variables. These simpler functions would be represented in the hidden layers of a mulilayer perceptron. These layers are called hidden layers because the data being outputted from these perceptrons are concepts that the model considers important. A quick analogy would be when you recognize a car. You don’t consciously look for a wheel or the contour of the body but the brain does and these unconscious thoughts would be part of the hidden layers.

With each level within the hidden layers, the input data gets more and more transformed until the output layer. The amount of hidden layers is called depth and is a reason why deep learning has the word “deep” in it. Each layer within the model can be seen as the current state of the computer’s memory after running a set of instructions. The depth of a particular model determines how many instructions a computer can run in sequence. This is powerful because later layers can refer back to the results of earlier layers. There is no universal depth number that qualifies a model as a deep learning model as opposed to a shallow learning model. However, it is hotly contended that machine learning is the most viable approach to build AI systems that can operate in the real world.

Sources:

• Deep Learning via Multilayer Perceptron Classifier - DZone Big Data
• The Hitchhiker’s Guide to Machine Learning in Python
• The Difference Between Artificial Intelligence, Machine Learning, and Deep Learning

In Python, everything is an object. In order to understand what this means we will need to understand how Python is set up. In Python’s documentation:

Objects are Python’s abstraction for data. All data in a Python program is represented by objects or by relations between objects.

This means that everything from strings to even numbers are stored as objects. This also means that everything have a type and unique IDs. An object in python has a type associated with it. This type determines what operations that object can be used with. An ID is an integer or long integer that uniquely identifies that object. In CPython, this id number is the memory address for that object. Why are these 2 points important for mutable and immutable objects? These two identifiers are important to distinguish how Python’s memory allocation works with these two different types of objects and how are they resolved after different operations.

So What Are Types, Classes and Instances?

As mentioned before, all objects in Python have a type. This type is like the parent of the object related to it. For example, the number 1 is an object of the type int. Classes are something that was an issue in older Python versions. Old style classes are generally user-defined while types are built ins. In current versions of Python, this has been changed and there is now no real difference between the two. If anything, class is now a way for users to make user defined types. Instances are pretty much objects. That is to say while 1 is an int object; 1 is an instance of type int.

The id() and type() built-ins

As was briefly mentioned, the id() builtin in CPython gives back an integer that ties to the memory address of a particular object during its lifetime. For example, if I were to type id(1), I will be given a unique integer tied to the object 1. I have run the command twice in the example below to show that the id is indeed unique and 1 is an instance, or object, of type int.

The type() built-in returns what the type of an object is. It should be noted that since everything is an object even types have a type, which is type.

Mutable Objects

Based on namesake alone, it can be inferred that mutable objects are objects capable of being changed while keeping its id. In Python, the types that are mutable are: lists, dictionaries, sets, bytearray, and etc. A quick and simple way to check if an object is mutable is through the use of the id() builtin.

In the example above, I have made an object of type list i with the value of 1 in it. After creating the list, I have appended a 2 at the end of the list. The append built in mutates the list by increasing its length by one and adds in its argument at the end of the list. I have taken the id of object i before and after the change and as you can see, they are the same. This means that the object i has been changed but kept its unique id and subsequently its memory address.

Immutable objects

Immutable objects are the direct opposite of mutable objects as they do not allow changes after its creation. Such object types are: integers, float, complex, strings, tuple, frozensets, and bytes.

In this example I have made an object i of type int. We know that i is immutable so when we add 1 to i, we will have 2 and a different id value. I have also taken the ids of the numbers themselves and we can see that the id of i is a direct copy of the ids of the integers. This is expected behavior because within python, there is an array of integer objects from -5 to 256 preallocated for us. Whenever we reference to a number within this range, we are actually referring to an existing object in the memory instead of creating a new object.

Why Should I Care If It’s Immutable or Mutable?

Since immutable objects are impervious to changes, what happens when we try to concatenate two immutable objects like two strings?

As we can see, the id for object i was changed as we concatenated the string "World" to it. In this operation, the concatenation of i and the string created a new object i with type string that contains the value of both the old object i and the string “World”. This means that “changing” an immutable type is an arduous process as a new copy is needed. This can be a very expensive process for extremely long strings or dictionaries. On the flip side, mutable objects are easier to mutate.

However, the inflexibility of immutable types can come in handy. Since the actual object cannot be changed unlike mutable objects, we can be certain that the object will stay the same unless an operation is used. There is a loophole to this immutable rule though. What happens when you add a mutable object within an immutable object like a tuple?

Here, I have created an object i with the class tuple which holds a reference to a string, which is immutable, and a list, which is mutable.

Since the list is mutable, I can use list built ins to modify the list itself. As we can see, none of the ids have changed meaning the mutable object list within the immutable tuple is the same as before. So what is immutable is the content of the tuple itself meaning the references to the string and the list. So this means that immutable objects within an immutable object cannot be changed as the only way to “change”an immutable object is to make a new object with the updated values.

A Small Intro to == and is

Within Python there are two operators == and is. The == operator checks for values while the is operator checks for identities(id). This means that x is y is similar to id(x) == id(y).

In the first line, I have created a list called dish that contains two strings, spam and eggs. I have created an alias of dish which is container so now container and dish point to the same list. When I do a value comparison check with == , I get back true because they both contain the same two strings. I also do an identity comparison using is and I get back True as well. Next, I have created a new list called bowl with the same two strings in the list. I do a value comparison check with == between bowl and dish and I get back true. However, when I do an identity check with is between bowl and dish, I get False.

How Are Arguments Passed to Functions?

Learning the difference between mutable and immutable objects is important to understand how they are treated when passed to functions(parameter passing) and how memory allocation works in Python. In programming, there are two main ways of passing arguments to functions: call by value and call by reference. Call by value means that only the value of the argument and not the argument itself was sent to the function. This means that the original argument hasn’t changed after the program resolves. Call by reference is when the program gets the memory address of the argument. This means that any changes made within the program can be seen after the program resolves. In the Python documentation, arguments are passed by assignment. This means that arguments are passed to functions as references to the object(argument).

In the example above, I have made a simple program that takes in an input and adds a list [10] to the input. I have passed in i which is the list [1,2] and got back an updated list with the 10 in it. As you have noticed, all of the addresses are the same whether inside or outside the function. Since the object i was a mutable object, Python has resolved the parameter passing of i similar to call by reference.

In this example, I have made a program that will add a string”World” to the input, which will also be a string. Based on the ids printed, we can see that when an immutable value is passed into the function, Python behaves like a call by reference. But since we are “changing” an immutable object, Python’s behavior “switches” to call by value and creates a new string object to hold the newly created object. Of course, the original input was never changed as the new object created within the program was never saved anywhere and was discarded.

A library in C is a group of functions and declarations bundled together for use by other programs. Within the library, the application programming interface (API) of a library is defined in a number of header files (.h).

An API (application programming interface) is a term meaning the functions/methods in a library that you can call to ask it to do things for you — the interface to the library.

The implementation of the programs listed in the header are expressed in a c source file precompiled into an object file (.o) . Standard libraries are usually stored in the /lib or /usr/lib directories. It should be noted though that libaries come in two versions: static and dynamic. There are different naming schemes for these two types. One common naming rule is each library must have the lib prefix appended to the name. The file suffix is dependent on what type of library it is. For static libraries, files are appended with a suffix .a while dynamic libraries are appended with the suffix .so .

For more information on static libraries, go to my past post on it.

What is a Dynamic Library?

A dynamic library works differently from a static library by being loaded into memory at runtime instead of being inserted into the actual program. This makes dynamic libraries easier to share between programs and saves on space as the library only needs to be loaded once.

How to create a Dynamic Library

To create a library in linux, you will need to precompile any c files into object files. This is done through the gcc command with the c flag. The fPIC flag makes the library position-independent and prevents the library to be shared by different processes.

gcc -c -fPIC *.c

To make the actual library, you need to gcc the object files with the share flag. Do not forget to add the lib prefix and end with the .so suffix.

gcc -share -o liblibrary.so *.o

To check what functions are included in the library file, use the nm function. The D flag shows what are in the initialized data section.

nm -D liblibrary.so

How to use a Dynamic Library

Now that you have compiled a dynamic library, you will have to add location of the file to the environment using export or at the standard library folders like ~/lib, /usr/local/lib, /usr/lib. To export to the environment use:

export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

This will export the variable LD_LIBRARY_PATH with the current working directory (.) and keeping the previous LD_LIBRARY_PATH.

Now that you have set the dynamic library to be used, you can just compile whatever c files reference to the programs within the library with gcc. The L flag is used to specify which directory to search for a library. The l flag requires the name of the library file without the lib prefix and .so suffix in order to work.

gcc -Wall -pedantic -Werror -Wextra -L . main.c -l liblibrary

Benefits of using a Dynamic Library versus a Static Library

One obvious benefit is the reduction in space usage compared to static libraries. Static libraries are always copied into the executable file so the end result will always contain all the functions from the header regardless of whether or not it is used. Another benefit of dynamic libraries is the portability between programs. Since dynamic libraries can be called at runtime, the same dynamic library can be used between different programs and can be unloaded from memory when not needed. The added modularity of dynamic libraries also means that the library can be independently upgraded from the rest of the program.

In 2013, Vitalik Buterin introduced Ethereum as an open decentralized platform based on blockchain that allows developers to build applications. The platform uses a distributed public blockchain network as its core to provide a “trustless” network for people from different countries to interact with each other. This network of nodes (computers) always communicate with each other in order to validate each transaction that goes through to the existing blockchain. Each transaction is powered with a cryptocurrency called ether which can be broken down into smaller denominations to make computations more affordable.

What Are The Main Concepts For Ethereum?

Ethereum was made such that developers can make an application that utilizes blockchain and is not limited in scope of what applications can be on the ethereum network. With blockchain, Ethereum can help decentralize a concept utilized under a central establishment such as loans from banks. Apps such as Weifund is a crowdfunding platform uses smart contracts to give investors a broader range of possibilities. Since Ethereum is a “trustless” transactional system, the platform must be immutable, tamper-free, and secure to cut out a third party to oversee transactions.

How Does Ethereum Work?

What is Blockchain?

Blockchain is a digital ledger of any transactions that aim to be incorruptable while being distributed freely. This allows the ledger to be transparent and corruption-free as mutliple nodes would hold a copy of the blockchain and validate any new additions of blocks. All blockchains start with a transaction where 2 or more parties exchange data. In order to validate a transaction, a lot of processing power is needed so one can borrows someone else’s computer to compute the mathematical proof to validate the transaction. This is known as mining. Once the proof has been generated and any payment fulfilled, the tranaction is added to the blockchain as a new block with its own hash and the resulting hash of the blockchain. What makes blockchains secure is the multiple nodes that contain a copy of the blockchain for verification and the hashing strength of the blocks. Thus, an effort is needed to unhash the blockchain and any fraudulant changes on the blockchain will be rejected by the nodes and prevent it from being uploaded into the network.

What Are Accounts and Fees?

In Ethereum, there are two types of accounts: external and contracts. External and contract accounts can start transactions in response to other transactions received. Only external accounts can send messages to other external accounts or to other contract accounts using its private key. Thus, only external accounts can keep adding to Ethereum blockchains.

Whether it is an external or contract account, transactions are usually done through someone else’s computer via mining. Mining is done by detailing the cost of the amount of gas per computational unit and how many units is needed. This means that a higher gas price per unit is highly desirable. A caveat is that if the maximal amount of gas set aside for the computation is not enough, the transaction will come out with an error and will be logged to the blockchain. Any gas used will be lost and sent to the miner for computing whatever part was completed. Otherwise, any unused gas will be returned to the poster. This market for computational power is there for security reasons and to prevent users from maliciously overtaxing the network. In addition to computing, storage fees are charged since adding to the blockchain takes up storage space at each node so there is a need for an incentive to keep the chain as small as possible.

What is a Static Library?

Static libraries are just archive files of object files with a list of addresses for each program. Such files are named with the prefix “lib” and the suffix “.a”. During the linking phase of compilation, the entirety of the library is included in the final executable file. This means that if 3 different programs linked to the same static library then the resultant executable for each program will contain the full copy of the static library. This can lead to rather large file sizes. Of course, this also means that the executable is now completely self-sufficient with no dependencies and the coder can be sure the right library is being used.

On the other hand, there are dynamic libraries. These libraries aren’t included in the final executable but are rather loaded into memory at runtime prior to resolving symbols. This means that many programs can reference to this shared file while keeping the program small and avoid the need to load the library for every program that needs its symbols.

How to create static libraries

In order to create a static library, you will have to start with making object files of the programs to be included in the library.

gcc -c *.c

This command creates all the object files from every c source file within the current directory. Once created, the ar command is needed to make the library file itself.

ar -rc libstat.a *.o

The r flag instructs the command to replace older object files int he library with the new object files. The c flag instructs the command to make the library if it doesn’t already exist.

Now that a new library exists, it must be indexed to help the compiler speed up symbol lookup within the library. This is especially useful with large libraries that can have thousands of symbols. The command for indexing a library is simply ranlib.

ranlib libstat.a

To check the contents of the library, we can use the ar command with the -t flag.

The nm command creates a detailed list of the symbols included in the library.

How to use the static library

Using a static library is very straightforward as it only requires the gcc command. For now, I will use:

gcc main.c -L. -lstat -o main

As you noticed by now, there are two different l flags utilized. These flags help the compiler locate the library. The big L tells the compiler which directories to look in. In this case I listed the current directory using the . but the full path name can also work. The small l flag tells the compiler the name of the library. When using the l flag, be sure to remove the prefix and file extension.

Time to wrap it up

Long story short, static libraries are nice to have as they are not required at run time so distribution of the final program is easier. Since the library is essentially baked into the program, bug fixes will generally be harder to go around as the entire program will have to be recompiled and redistributed as well as any other program that uses the same static library. This is where dynamic libraries shine. Dynamic libraries are loaded into memory at runtime prior to resolving symbols. This means that you can swap out dynamic libraries at any time when the program is closed as long as the symbols are left intact. Dynamic libraries also only link whatever is needed so the main takeaway is to use static libraries when the program absolutely need certain object codes.

GCC ( GNU Compiler Collection) is a collection of compilers for multiple languages such as C, C++, and Java. A compiler is a program that converts C source files into computer readable files which the processor can read and execute. For now, we will focus on the C aspect of the compiler. In order for the compiler to work, the program will need to go through 4 steps in order to convert source code into machine-readable files.

What are C source files?

But before we can talk about the steps of a compiler, we will need to define a couple key terms. C source files are the human readable code created by programmers that will eventually get processed into machine-readable code. Any file that have C syntax is basically a source file. Whether it works or not is a different story. C source files defines what functions are needed and its arguments.

What are header files?

Header files contains C declarations for functions and macros. Header files are generally used across various source files so the resultant executable can access any library needed. In C, header files are usually denoted with the .h filename extension.

The 4 steps of compilation

Step 1: Pre-processing

The first step of the compiling process is pre-processing. This step is essential for the compiler program to continue processing the source code. One of its main functions is removing any comments within the source code as there is no need for the processor to run comments. Its other function is to remove any macros and replace them with the entire content of the macros. Macros are defined by using ‘#define’ and can come in two types, object-like or function-like. Object-like macros are identifiers that will be replaced by a specified parameter.

If the compiler reads this, then any variable mention of apples will automatically be replaced with 4. Function-like macros are similar but instead of simple variable replacements it deals with functions. So you would put parenthesis after the macro name and follow it up with the function it will be replaced with.

With this function-like macro, the compiler will take any statement with multiply in it and replace it with the function on the right and fit in the appropriate arguements.

It should be noted that any lines affected by pre-processing are denoted with the ‘#’ symbol. This means that header files are also replaced by the compiler. The replacement of the header file line is important as header files contain function and macro definitions. Besides the header file, there are conditional statements to define macros and predefined macros already ready to use within the program such as __DATE__ or __FILE__. We can also see what source code would look like after going through the pre-processing phase using the -E option from gcc.

Step 2: Compilation

The compiler at this stage takes the pre-processed source code and compiles it into assembly instructions for a specific processor. This conversion makes it possible for different assemblers to create an object file from the same assembly instructions. In a sense this is the intermediary step where the source code goes from a complete human readable state into a semi-human readable state.

Step 3: Assembly

During this phase, an assembler from the compiler translate the assembly instructions to machine-readable code, or object code. The object code outputted by the assembly is code completely unreadable by humans at this stage and is only for the processor as it is binary.

Step 4: Linking

After creating the assembly code, the compiler is now in the linking stage. The linker part of the compiler puts the binary together with the startup code and required libraries for the functions and anything defined. These are needed to produce a complete executable file. By default, any executable file compiled from gcc will be named a.out.

You can also change the name of the output executable with the -o option.

Summary

There are four steps to compiling a source file to an executable. The first step, pre-processing, involves dropping any comments and replacing any macro with its definition. Basically anything that can be replaced will be replaced. The second step is compilation. In this step, the compiler is prepping the source code into assembly instructions in preparation for the assembler. The third step, assembly, involves reading the assembly instructions to create an object file. The fourth step, linking, takes this object file and links in any needed libraries so functions called within the code can work and the startign code of the executable.

Hard links and symbolic links are two different methods to refer to a file in the hard drive. These methods are part of the filesystem that organizes what file is what and where. A hard link is essentially a synced carbon copy of a file that refers directly to the inode of a file. Symbolic links on the other hand refer directly to the file which refers to the inode, a shortcut. In order to understand how symbolic and hard links work, we will need to go over what are inodes.

What is an inode?

The inode is a database that describes the file/directory attributes such as metadata and the physical location on the hard drive. They are essentially the numerical equivalent of a full address. With an inode, the OS can retrieve information about the file such as permission privileges and the physical location of the data on the hard drive to access the file. Should a file be moved from one folder to another, the file will be moved to a different location on the hard drive and its inode value will change with it automatically. This will be important for hard links. Speaking of hard links….

What is a hard link?

A hard link is a direct reference to a file via its inode. You can also only hardlink files and not directories. By using a hardlink, you can change the original file’s contents or location and the hardlink will still point to the original file because its inode is still pointing to that file. There is no referencing to the original file. In addition, hardlinks can only refer to files within the same volume otherwise symbolic links will be needed. To make a hard link of a file, you will require the ln command and refer to the source file before naming what the hard link will be named. Here is an example of how a hard link named test 2 will be made.

The file test should be completely empty and I will add “Hello” to it via the hard link.

As seen in the photos above, I have changed the original file via the hard link by adding “Hello”. By opening the original file, the word “Hello” is already there. We can further make sure the files are referring to the same inode by using the ls -i command.

Now what will happen if we copy over a similar file called test from a different folder into this folder? For this experiment we will change the folder name from ‘test’ to ‘test folder’.

Here we can see that the cp command does not change the inode value of the original value but mv does. We have copied over a file from the parent directory into ‘testfolder’ and the inode value has not changed. It is only when you move over a file and replace the file that the inode value changes.

What are symbolic links?

Symbolic links are essentially shortcuts that reference to a file instead of its inode value. This method can be applied to directories and can reference across different hard disks/volumes. Since the symbolic link is referring to the original file and not its inode value, then replacing the original file into a different folder will break the symbolic link, or create a dangling link.

Since the symbolic link is a link that directs to the original file, changing the symboliclink should change the original file.

What will break a symbolic link is when the original file is moved to a different file or deleted.

So symbolic links can be seen as a static link to the last known location of the original file. The link should work even if you replace the original file with a different file with the same name.

Ls is a command on linux that lists almost everything within a current directory. I say almost because you will have to modify the command a bit in order to see all the hidden files and directories, which is the -a option. In order to understand what ls does, we will need to understand a bit on what the command ls is running from, the shell.

What is the shell?

A shell is a program that takes commands and runs them through the kernel to perform the commands. The output is either listed as standard output or redirected into a file to store the results. One shell that is widely used and familiar with most users is the bash shell. The bash shell is an improvement of the old sh shell used in the past. Besides bash, there are other shells such as ksh (Korn), Csh (C), and dash (Debian). To open a shell, it is as easy as connecting to Linux via ssh or opening a terminal from the desktop versions of Ubuntu.

What is the ls command and what can it do?

The ls command lists files and directories within the current directory by default. Without any options, the ls command will not list hidden files unless you specify the a option. Hidden files are denoted by having a (.) prefixed to the file name. For example, here is an example of using the ls command without any options.

With the a option, hidden files will be shown.

It should also be noted that the (.) and (..) listed at the top are symbolic links that denote the current directory and the parent directory respectively. Another popular option is the -l option or the long option. The long option prints the file permissions, number of links, owner name, owner group, file size, time of last edit, and file/directory name.

We can also mix different options together to get a very output. For example,we can combine the past two options to print out the file details of all files and directories as well as the hidden ones.

What is a wildcard (*) and its effects with ls searches

The wildcard is a character that can substitute any character in a search. This lends to any search command a lot of power and flexibility. For example, there are 3 files called ‘a.jpg’, ‘aa.jpg’, and ‘aaa.jpg’. If I run the ls command with ‘*.jpg’, then I will get all files that ends with ‘.jpg’.

If I run a command with ‘a*.jpg’, then I will get back files all files as well. This is because a* means all files that start with a lowercase a that ends with ‘.jpg’. Soif I want to look for just ‘.html’ files then I would use ls ‘*.html’. So, if I run a command such as ls *.c, I will end up with all files that end with the .c suffix.