Digitalize your engineering assets leveraging AI

Our innovative solution combines traditional computer vision with deep learning to efficiently extract structured data from complex engineering diagrams. Learn more.

Introduction

Over the past two decades, the shift towards digitizing documents and extracting valuable insights has become a cornerstone of effective data management practices. This is particularly true for companies in a sector like engineering, where there is many varied outputs of documents ranging from billing memos to complex diagrams. The challenge of efficiently managing and tracking the contents of these files grows exponentially when they’re not organized systematically.

Traditionally, extracting data from these documents has been a manual, labor-intensive task, consuming time and resources that could be more effectively utilized elsewhere within an organization. However, the emergence of machine learning technologies presents a significant opportunity to streamline these processes, reducing the burden and enhancing efficiency.

ThoughtsWin Systems is at the forefront of this movement, developing cutting-edge solutions to assist organizations in extracting and managing data from their documents. This article explores the innovative strategies employed by ThoughtsWin Systems to revolutionize data management practices, leveraging technology to achieve unprecedented levels of efficiency and productivity.

Challenge

Despite the technological advancements that have enabled various industry practitioners to store files on cloud or on-premise hardware storage, knowing what’s in a file without opening it — and doing so with minimal computational costs — has become a necessity. It also helps keep track of information for business stakeholders. However, extracting such information from both paper-based and digital images involves numerous steps in preprocessing a file, identifying what’s important, and then having someone type these details into a database. To date, converting unstructured data, such as images and videos, into structured formats remains a complex task. This issue is particularly prevalent in industries heavily reliant on visual data and text, such as engineering and architectural design.

Another challenge faced in this laborious task is that while human labor retains advantages, such as nuanced judgment and contextual understanding, it is also prone to frequent errors. When performing repetitive and tedious tasks, individuals are susceptible to mistakes, including typos and misinterpretations. Moreover, these tasks often consume significant amounts of time, which ThoughtsWin aims to reduce by leveraging advancements in the field of machine learning in recent years.

In the engineering industry, digitizing old engineering diagrams is a significant challenge, especially when scaling operations to manage millions of drawings. Essential information such as the title, drawing number, revision number, and tags must be extracted when digitizing engineering diagrams.

For example, in the diagram template below, the title, drawing number, and revision number might be “The Main Title,” “203.045.0678–02,” and “2,” respectively. Despite the apparent simplicity of this task, the global variety of templates, each potentially containing thousands of diagrams, presents a formidable obstacle.

Furthermore, the existence of component tags in such drawings, as seen in the sample Piping and Instrumentation Diagram, requires meticulous analysis, potentially straining the eyes and constituting a heavy task for a single person, not the best use of their time.

In such scenarios, rule-based software alone is insufficient to address this complexity. Moreover, some drawings are handwritten, while others may be damaged by exposure to sunlight or water. To effectively address these challenges, robust solutions are required, as outlined in the subsequent section.

Solution

To effectively extract structured data from unstructured sources like engineering diagrams, leveraging both traditional computer vision (CV) techniques and deep learning is essential, each offering distinct advantages. Traditional CV, grounded in mathematical and geometric principles, is adept at recognizing patterns, edges, and shapes through well-established algorithms. This approach excels in environments with minimal image variability and clearly defined features.

Conversely, deep learning, particularly through Convolutional Neural Networks (CNNs), advances the processing of complex image data. It thrives on identifying intricate patterns by learning from extensive datasets, often outperforming traditional CV in accuracy and versatility. Deep learning’s strength lies in its ability to generalize across different image qualities and complexities, adapting to new scenarios without the need for reprogramming.

However, neither approach alone can fully address the challenges of extracting data from engineering diagrams with the necessary precision and efficiency.

Traditional CV methods, effective for straightforward tasks, often struggle with the complexity and diversity of engineering diagrams, such as overlapping elements or variable line weights. Deep learning, despite its pattern recognition capabilities, is limited by the need for large, annotated training datasets, which are rare or costly in specialized fields like

engineering. Moreover, the opaque nature of deep learning models complicates the understanding of their decision-making processes, a significant concern in fields requiring transparency.

A hybrid approach, combining traditional CV’s immediate precision with deep learning’s adaptability and learning capacity, emerges as a superior solution. This strategy harnesses the strengths of both methodologies, mitigating their limitations and enabling accurate information extraction from engineering diagrams. By integrating the reliable detection capabilities of traditional CV with the sophisticated pattern recognition of deep learning, this approach facilitates a more effective and efficient digitization of engineering diagrams.

Introducing DREX: Our innovative solution combines traditional computer vision with deep learning to efficiently extract structured data from complex engineering diagrams. By leveraging the precision of traditional CV and the adaptability of deep learning, DREX offers unparalleled accuracy and versatility. Say goodbye to the limitations of single-method approaches and embrace a hybrid solution designed to meet the demands of data extraction in specialized fields.

Approach

Our method for data extraction from diagrams employs a multi-stage process that utilizes both Deep Learning techniques and traditional Computer Vision approaches.

Processing Files and Identifying Engineering Diagrams

The initial step involves preprocessing the files. For files in DWG format, a native format for several CAD packages, we convert them to PDFs. Once all files are in PDF format, we transform them into images to leverage various Python libraries for image processing. However, not all images represent engineering diagrams — some are merely text-based PDFs without diagrams or are irrelevant to the project. Therefore, we use a classification model to identify images relevant to our needs. This classification helps us curate a proper dataset, selecting samples for annotation to aid in training our model.

Diagram Annotation

To develop a model capable of extracting the required information from images, we utilize annotation tools. This process involves manually tagging each diagram with more than ten labels necessary for recognition and then exporting these labels into a JSON file. This file is instrumental for model training, containing essential details such as the coordinates for bounding boxes and the class labels established during the annotation phase.

Training an Object Detection Model

Post-annotation, we advance to training the YOLO (You Only Look Once), an object detection model to identify and highlight key areas within the diagrams. Object detection models operate by analyzing the spatial features of images to detect various objects within them. By leveraging a complex network of convolutional neural networks (CNNs), the YOLO model assimilates from the richly detailed examples in our annotated dataset. This enables it to efficiently recognize and localize key features within new diagrams, ensuring precise identification of relevant information with remarkable speed and accuracy.

Digital Character Conversion

Following object detection, the identified diagram areas undergo a meticulous Optical Character Recognition (OCR) process. This transformation is not merely about digitizing visual data; it’s about turning unstructured information into a structured format ripe for initial processing. This pivotal step marks the transition from visual to digital, setting the stage for enhanced data management.

Text Refinement

The OCR phase, while transformative, is not infallible. To elevate the accuracy and clarity of the OCR output, we harness the power of an advanced Large Language Model (LLM). This model doesn’t just refine the data; it enriches it, ensuring that the final output is organized in a manner that seamlessly integrates into our database systems. This sophisticated text refinement process underscores our commitment to data integrity and usability.

Results

In our analysis, we achieved our objective of more than 80% accuracy rate in data extraction from engineering diagrams, which included both handwritten and typed formats. A significant highlight of our efforts is our performance in tag extraction, where drawings ranged from having no tags to having more than 100, depending on their type. In these instances, we reached an 85% accuracy rate. This accomplishment emphasizes the efficiency and precision of our approach in accurately interpreting and digitizing complex diagrammatic information, showcasing our capability to handle scenarios with high tag density effectively.

Benefits

The training and fine-tuning of a model to analyze such a vast number of images requires time and precision. However, the investment in this process is justified when compared to the labor-intensive and time-consuming task of manually entering data for each image into a database. Our solution automates this process, leveraging advanced AI/ML techniques to efficiently handle

images that can scale to millions. Whereas a human might spend approximately 5–10 minutes analyzing and inputting data manually for each image, our system can extract the data in about 10 to 100 seconds, depending on the image size. This automation not only saves significant time but also reduces the potential for human error associated with manual data entry. Investing resources in validating the results is more efficient than manual entry, as it is considerably faster. By streamlining

the data analysis process, we facilitate a more efficient allocation of resources, allowing teams to concentrate on more strategic tasks that require human insight, rather than on repetitive data entry tasks. Our solution’s adaptability, coupled with understanding how to approach such niche use cases, underlines our confidence in its scalability across various domains beyond just engineering.

Take the Next Step with ThoughtsWin Systems

Elevate your data management process with Thoughtswin Systems’ innovative AI/ML enabled solutions. Streamline your engineering diagram analysis and enhance efficiency across your projects. For more information on our services and how we can assist you, reach out to mahesh.shankar@thoughtswinsystems.com. Let’s revolutionize your data management process together.