What is the Sketch2mesh algorithm that the future of Computer Aided Design technology
Greetings everyone, today I am here before you on a very special and exciting topic for me. This article was presented to you as the first article “introduction” of a series to be published about the introduction, implementation, extensibility and future place of the Sketch2mesh algorithm, which I believe in its future potential very much and want to support its development.
Content:
1- Basic Terms
2- Importance of 3D Shape Reconstruction
3- Related Works of Sketch2Mesh
4- Adventages and Disadventages
5- Conclusion
1- Basic Terms
Before we starting the Sketch2mesh algorithm there are some terms that we must know.
CAD (Computer Aided Design): Enables designers, industrial engineers and artists to create, analyze and modify 3D models.
Sketch: It is a 2D depiction of an object. It is often used as the simplest notation. It is an important requirement for 3D shape reconstruction.
3D mesh: A three-dimensional representation of an object or shape. 3D meshes are made up of a series of triangles. 3D meshes are used in a variety of applications such as 3D modeling, computer graphics, and 3D printing.
3D Shape Reconstruction: It is the process of recreating 3D objects from 2D images or drawings.
SVR (Single View Reconstruction): The process of reconstructing an object or shape from a single view. SVR is a subfield of 3D shape reconstruction.
2- Importance of 3D Shape Reconstruction
3D shape reconstruction is a technology that has the potential to revolutionize the way designers, industrial engineers, and artists interact with Computer Aided Design (CAD) systems. It allows designers to shy through the tons of complex applications they need to learn to produce 3D models and the tons of time and energy required to produce them, and to draw and interact with shapes by drawing in 2D, which is natural for practitioners. Current deep learning approaches have been promising in retrieving 3D point clouds and volumetric grids from 2D sketches. However, these approaches often produce roughly 3D surface representations and are difficult to edit.
To address the difficulties of 3D Shape Reconstruction;
Shortcomings of Current Deep Learning Approaches: Although current deep learning approaches are trained on synthetic data, they often provide rough 3D surface representations, meaning they are less efficient at estimating areas not visible in the sketch. These methods require more than one sketch perspective or work with limited views.
Single-View Reconstruction (SVR) Issues: The diversity in artists’ specific drawing methods makes it difficult for local feature pools to detect the product. This difficulty brings a great deal of variability in the educational process, with different people drawing in different ways, and generalization becomes problematic. Also, these architectures do not learn a compact representation of 3D shapes, which makes it difficult to edit the learned models.
The Problem of Generalization: Different drawing styles create a great deal of variability in the educational process, which complicates the capacity to generalize.
Lack of Compact 3D Representation: Current architectures do not provide a compact representation of 3D shapes, making it difficult to use the learned models in applications such as shape editing.3D shape reconstruction is still an emerging field. The biggest benefit to this development is provided by the Sketch2mesh algorithm with the solution methods mentioned below.
3- Sketch2mesh Related Works
First, sketch2mesh applies 2 basic approaches to these challenges. These approaches are called Sketch2Mesh/Render and Sketch2Mesh/Chamfer.
Sketch2Mesh/Render; It is a state-of-the-art image conversion technique trained to synthesize foreground/background images from sketches and then use the resulting images as targets for differentiable rasterization.
Sketch2Mesh/Chamfer; Chamfer refers to the smoothing of sharp surfaces in the real world. The Sketch2mesh algorithm adjusts the position of the strokes of the 3D shape to match the strokes in the input drawing. It accomplishes this by minimizing the 2D Chamfer distance.
Remarkably, Sketch2Mesh/Chamfer, despite being simpler, is almost identical or better than the performance of Sketch2Mesh/Render. The former uses only the outer object outlines for refinement; Most graphic designers draw these outlines in a similar way, which helps with generalization. You also don’t need a helper network to convert sketches to background or foreground images.
Now let’s talk about the rapid explosions in 3D modeling technologies in recent years. A lot has changed when it comes to creating 3D models from images and drawings. First, let’s look at how we represent 3D surfaces. We used a specific template to create surfaces, and from that pattern we created shapes. But this method was somewhat limited. Then, we started to define surfaces with a function, which gave us more freedom. But this new method was a bit more complicated. Fortunately, a new approach has recently been developed that can use this complex method in a simpler and more effective way.
Next, let’s look at how to create 3D models from drawings. This is actually a topic that has been researched for years. In the old days, to turn a drawing into a 3D model, we either made some assumptions or worked with limited templates. But thanks to deep learning, we’ve been able to make this process much better. For example, we can take a drawing and turn it into a 3D model from 12 different vantage points.And finally, let’s say a few words on the topic of creating a 3D model from a photo. In fact, this has improved further thanks to newly developed methods. But some of these methods need extra information to modify the resulting model. So, sometimes another photograph or information is needed to perfect the model obtained from one photographNext, let’s look at how to create 3D models from drawings. This is actually a topic that has been researched for years. In the old days, to turn a drawing into a 3D model, we either made some assumptions or worked with limited templates. But thanks to deep learning, we’ve been able to make this process much better. For example, we can take a drawing and turn it into a 3D model from 12 different vantage points.
And finally, let’s say a few words on the topic of creating a 3D model from a photo. In fact, this has improved further thanks to newly developed methods. But some of these methods need extra information to modify the resulting model. So, sometimes another photograph or information is needed to perfect the model obtained from one photograph.
Finally, there are also some innovations when it comes to making 3D models better with photography. In particular, the improvements made using silhouettes (the outline of an object) are quite promising.
4- Adventages and Disadventages
Adventages of Sketch2mesh:
Suitability for Different Drawing Styles: Sketch2Mesh evaluates drawings well regardless of what style they are drawn. Pix2Vox, unlike other methods like MeshRCNN or DISN, gives great results against different drawing styles. So much so that you can get amazing 3D models even with hand drawing!
No Additional Network Needed: As we know, some methods require special network training. But Sketch2Mesh doesn’t need anything like that, which makes it much easier to use and install.
Interaction with Drawings: Especially in the world of design, being able to interact directly with a drawing is a great feature. Thanks to Sketch2Mesh, designers and artists can not only play with 2D drawings, but also convert those drawings into 3D models.
Superior Performance: If you’re interested in the technical details, this part might be important to you. In tests with hand drawing and synthesized drawings, Sketch2Mesh gives better results than other state-of-the-art methods.
Disadventages of Sketch2Mesh:
Data Set Mismatch: Sketch2Mesh doesn’t work well with some data sets. It has a hard time using data sets, especially Prosketch.
Sparse and Vague Information Problem of Drawings: Drawings sometimes provide little and ambiguous information, making Sketch2Mesh’s job difficult. Different people are able to draw in different ways, which leads to some problems in the educational process, and there is difficulty in generalizing.
5- Conclusion
Sketch2Mesh algorithm stands as a testament to the relentless evolution of Computer Aided Design (CAD) technology, paving the way for seamless transitions from sketches to 3D models.
The algorithm, with its remarkable adaptability to various drawing styles and the elimination of the need for additional network training, promises a more intuitive and efficient modeling process.
Nevertheless, like any innovation, Sketch2Mesh has its limitations, particularly when dealing with certain datasets and the inherent ambiguities in sketches. However, its advantages are significant and indicate a future where 3D modeling from sketches becomes not only viable but commonplace. As the world of design continues to evolve, tools like Sketch2Mesh will undoubtedly play a pivotal role, enabling creators to bridge the gap between imagination and reality with unprecedented ease.
Resource:
1- Benoit Guillard, Edoardo Remelli, Pierre Yvernay and Pascal Fua. Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches, 2021.
