StructLearning: a great help to solve the Civil Engineering exercises
Each Civil Engineering student asked himself, at least once in his life, how to understand the constructions science exercises quickly and if there are practical and simple tools to solve a structure to get the stress’ characteristics.
The Solution to this not uncommon problem, according to our experiences, is to use the Machine Learning for an automatic recognition of the structural schemes, using a commonly used object such as a mobile phone.
Below a small demonstration of how the app will work:
By applying the Machine Learning to Civil Engineering, you can recognise the structural schemas.
Click on the scheme if the recognising is right.
We can get the immediate resolution of simple isostatic schemes, with the Bending Moment and Shear stress values as output, after entering the beam length and the strength or moment intensity applied.
In this article we will describe the technical steps that led us to develop and recognize patterns using automatic recognition.
But first, let’s see in more detail what Machine Learning is.
Machine Learning in StructLearning
Introduction
Machine Learning or ML (in Italian apprendimento automatico) is that branch of Artificial Intelligence (AI) in which computers learn without being explicitly programmed. Learning tools allow the machine to develop and refine algorithms using huge amounts of data as models.
The potential of ML is to be able to order thousands of images identifying which one contains a specific data set in a few minutes; the only “manual” would be thing to tag the data with meta tags, meaning telling to the model that one specific object has a certain name. In the case of our model we had to say, for example, that the figure “arrow” is called “force”.
Frameworks we used
Below the frameworks used to implement our project.
UIKit
The UIKit framework provides the infrastructure of the iOS or tvOS apps. It is essential to define the window and the display structure for the interface implementation, the event management infrastructure for the distribution of Multi-Touch and other types of app inputs and the main execution cycle necessary for manage the interactions between the user, the system and your app. Other features offered by the framework include support for animation, documents, drawing and printing, information on the current device, management and display of text, support for research, accessibility, app extension and resource management.
AVFoundation
The AVFoundation framework combines four main technological areas which together comprise a wide range of activities for the acquisition, processing, synthesis, control, import and export of audiovisual media on Apple platforms. Work with audiovisual resources, control device cameras, process audio and configure system audio interactions.
Vision
The Vision framework performs the detection of reference points of faces and faces, the detection of texts, the recognition of bar codes, the registration of images and the tracing of general functions. Vision also allows the use of custom Core ML models for activities such as classification or object detection.
Apply machine vision algorithms to perform a variety of activities on input images and videos.
Training is everything!
MLObjectDetector
To train an automatic recognition model, we can use an object detector, normally used to detect specific objects within images.
To create the object detection model, you have to train it through images and annotations for each image, each annotation has a label and a region for an object within the image, and when the training is over the reliability is assessed, showing the model a series of testing images, with annotations that the model has never seen. The resulting metrics tell us how accurate the model is and if it is reliable enough as averagePrecision and meanAveragePrecision.
If we notice that there are too many errors, we can add more data and modify the parameters and restart the train, until the model is sufficiently trained; when this happens, we can export it. The resulting file will have the .modelml’s extension and will be ready to be used in our app.
In our case, bacause of the amount of data used for training, we have inserted a .json file, containing informations regarding the position of the object to be recognized for each image.
Here are the details.
Create a model for object recognition
In our case, to create the ML model, we used CreateML, an XCode tool. The model is the result of applying the ML algorithm to set the training data and it is used to make a forecast based on the input data. Core ML is the foundation of frameworks such as Vision, for images’ analysis, Natural Language for texts’ processing , Speech used to convert audio into tests and SoundAnalysis used to identify sounds and audio.
To create a good Machine Learning model we need a good amount of data that will be used to “train” the model and a .JSON file with the annotations regarding the objects’ to recognize position in each image that we will use for training; in our case we asked a good number of people to draw structural models by hand and we had obtain 159 images to train the model.
In the Annotation.json file we wrote all the information of the image regarding the position of the object to recognize, this for each image that we will use for training the model, the process for each image is hard if done by hand, but using IBM Cloud Annotation (https://cloud.annotations.ai/) the process is much faster, in fact all you need to do, after choosing the type of annotation, is inserting the images that you’ll use to train and select the subjects that our all will recognize for you.
Here an example of IBM Cloud Annotation:
Select the object you want to recognize in your image:
Assign a name to each selected object:
Do this for each image you will use for the training session.
Export your model:
The resulting .json file will be this:
As you can be see from the picture, the label and the coordinates are automatically written for each object identified. The annotation.json file will be placed in the Training folder, with all the training images.
How can I create a model file with createML?
Open XCode and choose “Xcode → Open Developer Tool → Create ML”:
Create a new document and choose “Object Detector” and click “Next”:
Write a name for your project and click “Next”; choose the folder where you want save your project or create it and click “Next”; this is the window that will appear:
Add 80% of your images in “Training Data” box and click “Train”
The resulting graph will be something like this for the training:
This is the graph for the validation:
After the training is finished, you can go to the testing phase, adding 20% of the remaining images.
If the evaluation performance isn’t good, you can retrain with more training data.
When your model is completed and satisfactory, you can save your model and use it in your app.
Recognising object in live capture
In our project we used Vision frameworks.
With the Vision framework, we can recognize live capture objects, Vision requests with a Core ML model, return results as VNRecognizedObjectObservation objects, which identify objects found in the acquired scene.
In order to acquire the image via camera, we need to configure it in a slightly different way to the standard configuration to acquire the data from it in the best possible way.
Camera Configuration
The first thing to do is to to configure AVCaptureSession:
It’s very important to choose the right resolution for your app, infact you don’t have to put the highest available one if the app doesn’t require it, actually to the opposite the best choice is to select the lowest one so that Vision can process the results faster and more efficiently.
You can set the resolution as greater or equal to that of the images used in the model.
Add camera to app
Specify pixel format:
WARNING: The camera may stop working if the buffer queue fills the available memory, to avoid this problem, Vision framework blocks the call until the end of the next request, in our case we chose to set the size of the thing equal to 1, so that even if other requests occur in the processing time, they are ignored and only the one following the end of the previous processing will be accepted.
Commit the session configuration:
Set a preview level on the display controller so that the camera can insert its frames in the app’s user interface:
Specify device orientation:
Labeling using CoreML classifier
The CORE ML model that you include in the app also determines which labels are used in the Vision object identifier.
To load the model use VNCoreMLModel:
Create a VNCoreMLRequest with this model:
Parse Recognised Object Observations
The result we will have is a series of observations. With labels and boxes that delimit the recognized object, to analyze these observations, we use the following:
The array of labels lists each identifier in descending order of confidence, the maximum of which is 1, while the rectangle delimits the identified object.
You can use VNRecognizedObjectObservation resulting from the recognition to initialize a tracking of objects such as VNTrackObjectRequest.
Our project, future implementations
Using these steps in our app, we created an application that recognizes simple constructions’ science schemes. As future implementations, we intend to implement and develop all the resolution part of the exercise.
Every time the app recognizes a pattern, after confirmation by the user on the regularity of the found pattern, it will show a data entry page such as the intensity of the force or moment and the length of the beam, after which it resolves the diagram and in output will present the solution in terms of diagrams and numerical values of the shear and the bending moment.
StructLearning will available on TestFlight soon!
Thanks for reading this article.
StructLearning developed during the COVID-19 quarantine in Italy by the ML@home student group at the Apple Developer Academy in Naples (Italy)
ML@home group is composed by
- Antonietta Cirma (https://www.linkedin.com/in/antonella-cirma-814295114/)
- Giovanni Erpete (https://www.linkedin.com/in/giovanni-erpete-51035617b/)
Follow us on LinkedIn!
