LLM for Optimisation in 3-D Space: A comparison with Deterministic optimisation methods

Recommendations for the present and the future.

Large Language Models (LLMs) are machine learning models created from an immensely large amount of data curated from a wide range of domains. The volume and variety of data they are trained with make them capable of understanding and generating natural language and other types of content to perform a wide range of tasks. LLMs and other associated generative models have been applied to various tasks that involve text, audio, image and video understanding and generation. However, when it comes to spatial data such as maps and building architectures, application of LLMs is not yet common. More challenging could be the use of these models for a placement, routing, and clash resolution problems within a restrained 3-D space.

Prior to the advent of Artificial Intelligence (AI) methods of solving problems, there are deterministic methods that do not rely on historical methods but based on algorithms that are employed to find the optimal placement or movement of an object within a 3-D space. Deterministic 3-D optimization algorithms are used in various fields, particularly in engineering tasks such as placement and routing in chip design, robotic path planning, and facility layout optimization. Here, deterministic means the algorithm provides the same output for a given input every time, unlike stochastic algorithms which might incorporate random processes and thus can yield different results on different runs. Machine learning models employ some level of stochasticity to allow the model to generalise more to diverse problems. Some of the deterministic algorithms currently used in various fields of engineering include: Branch and bound algorithms for VLSI circuit designs, Sequential Quadratic Programming (SQP) for robotic and mechanical design problems, Deterministic Annealing for Sensor Network localisation and facility placement and Layer Assignment algorithm for multi-layer VLSI design, among others. These algorithms play crucial roles in fields requiring precise control and optimization in 3-D environments.

In this blogpost, we compare deterministic optimisation methods with the potentials of using LLM for optimisation problems especially in the 3-D space. We consider data formats, computational complexity, precision of solution and the ability of the algorithms to adapt to changing problems and environments.

3-D Data formats

The 3-D space data is represented in different formats, with points representing the faces, vertices and edges. In 3-D space optimization, data representation is critical. Common 3-D data formats include:

1. Point Clouds: Collections of points in space, typically generated by 3-D scanners. Each point has coordinates (x, y, z) and may include additional attributes like color or intensity. Example of a project implemented with point cloud data is 3D-LLM.
2. Meshes: Composed of vertices, edges, and faces that define the shape of a 3-D object. Common formats include STL, OBJ, and PLY.
3. Voxel Grids: A volumetric representation where space is divided into a regular grid of cubes (voxels). Each voxel can store information such as occupancy, color, or density.
4. Parametric Models: Represent 3-D objects using mathematical functions and parameters. Commonly used in CAD and BIM applications.
5. Polygonal Models: Represent surfaces using polygons, typically triangles or quadrilaterals. Widely used in graphics and simulation.

Understanding these formats is essential for applying optimization techniques, as they influence the choice of algorithms and the efficiency of the optimization process.

Deterministic Optimisation Methods

For every optimisation method, especially the deterministic approaches, a clear objective function must be formulated and the constraints to achieving an optimal solution must also be clearly defined. For some problems, multiple objectives will need to be simultaneously optimised. For example, distance between objects of a given kind will need to be maximised subject to some constraints on what the minimum and maximum distance should be among the object types. Further, all objects of the same kind will need to exist within a defined sub-space within the total space available for placing all objects of all kinds. Deterministic optimization methods rely on well-defined mathematical procedures to find the optimal solution. Common methods include:

1. Linear Programming (LP): Optimizes a linear objective function subject to linear equality and inequality constraints. Suitable for problems where both the objective function and constraints are linear.
2. Integer Programming (IP): Similar to LP but with integer constraints on some or all variables. Used in combinatorial optimization problems.
3. Non-Linear Programming (NLP): Deals with optimization problems where the objective function or constraints are non-linear. Requires more complex algorithms like gradient descent or interior-point methods.
4. Gradient-Based Methods: Iterative techniques that use the gradient of the objective function to find local minima or maxima. Common algorithms include gradient descent, conjugate gradient, and quasi-Newton methods.
5. Dynamic Programming: Breaks down a problem into simpler subproblems and solves them recursively. Useful for problems with overlapping subproblems and optimal substructure properties.

These methods are characterized by their predictability and precision, often providing exact solutions to well-defined problems. However, they can be computationally intensive and may struggle with complex, high-dimensional, or highly non-linear problems.

LLM-based Optimisation in the 3-D space

The application of LLM to the design and optimisation of 3-D space is an emerging area of research. Currently, most applications relate to the analysis of data extracted from the representation of the 3-D space to recommend design and optimisation strategies. However, the emerging research progress currently explores the application of LLM for design and optimisation in the following areas:

• Robotics Planning and Pathing: Smart-LLM has been proposed for generating tasks for multi-agent robots. The model has been trained to take high-level task instructions and generate low-level task plans and task allocation that can be executed by each robot. For physical robots, LLMs can help in planning the movements of robotic arms to perform tasks such as assembly, welding, or painting. Similar LLMs can optimize robot paths to navigate around obstacles in a 3D space efficiently.
• Engineering Design: LLMs can be applied to structural optimisation and aerospace engineering. Can we use an LLM, like GPT-4, to transform human language into a detailed 3D model of a building? This is a current debate among experts in the building design industry. Although, there are current concerns about lack of precision in using LLMs for prompting generative software, it was agreed that LLMs can accelerate the iterative design process and enable non-experts in architecture to contribute innovatively to architectural designs. With the help of prompt engineering, the idea of non-experts can be transformed into more specific instructions that can enable generative architectural software to create more specific 3-D designs of the building. Autodesk, being a leader in the building design industry has developed an LLM-based building design tool called TileGPT. TileGPT from Autodesk combines the Wavefunction Collapse (WFC) algorithm with prompting mechanism of Generative Pretrained Transformers (GPTs). This shows that hybrid mechanisms that combine deterministic and probabilistic optimisation methods often provide better results for 3-D designs and optimisation.
• Medical Imaging: The application of LLM in the 3-D space can be seen in healthcare and Medicine. The 3D Reconstruction from medical imaging data such as CT or MRI scans can be optimised via LLM prompts. LLMs can also help in surgical planning by assisting surgeons in planning complex procedures by optimizing the positioning and orientation of surgical tools. Current research is exploring Visual Language Models (VLM) for reconstruction of medical images through text or for appropriate generation of text analysis given a medical image. Advancing Medical imaging with Language models is an almost exhaustive review of how LLMs such as ChatGPT can be used to enhance medical image reporting. Although LLMs have potential for guiding medical image reconstruction, most of the existing research and applications relate to medical image analysis for diagnosis and consultation.

Use-cases: Application of LLM in Architectural Design and Optimisation

Here we focus on existing and potential application of LLMs for optimizing the layout of building elements to maximize space utilization and to comply with the building codes for various services integrated into a building design. Due to the emerging nature of use of LLM in Design and Manufacture, a systematic approach to determining where LLMs can be applied is currently in place. The following stages have been identified in Large Language Models for Design and Manufacture for analysing the potentials of integrating GenAI and LLMs for automation and semi-automation of Architectural designs and manufacture:

1. Generating a design,
2. Constructing a design space and design variations,
3. Preparing and documenting designs for manufacturing or construction,
4. Evaluating a design’s performance, and
5. Discovering high-performing designs within a given performance metric and design space.

Current experiments show that while there are possibilities in achieving the above automation with LLM, the results still leave out a lot to be desired. Precise instructions that can enable the generating tools to produce the correct design intent has not been achieved by the LLMs. Human intervention and post-processing are required to bring the output of the LLM to the required specification.

Figure 1 shows a general concept for using an ideal LLM that has domain expertise in Building Information Modelling (BIM). Although the LLM is neither a human expert nor a Design Software, it understands human expertise in optimal architecture, engineering, and construction (AEC). It therefore knows how to generate low level instruction for design software, given high-level human instruction about conceptual designs. It can also take design data from an existing engineering or architectural design and recommend the changes that the design software can make to existing designs for optimal layout and cost-effective delivery of the final designs.

A General framework for building LLMs for Architectural Design and Optimisation

Human users can be in the loop to refine the high-level prompts given to the LLM so that it can generate more precise instructions that can deliver the designs and optimisations envisaged by the user. However, the LLM has to be properly trained in order to deliver optimal designs based on data collected from human experts. Various methods that range from low computational to high computational costs are available depending on the optimisation tasks required. These methods are considered next.

To further explore the possibility of using LLMs for Architectural design and optimisations, the techniques of few-shot learning, prompt engineering, Retrieval Augmented Generation (RAG) and then fine-tuning will need to be applied to existing LLMs to adapt them to BIM-specific domain. Specific use cases such as design generation for specific building types and optimisations such as BIM coordination and clash resolution will need to be properly defined and then the relevant training data collected. To achieve very complex design and optimisations, several task-specific models may need to be sequenced in a pipeline, where the output of a model becomes the input to the next. Some time will need to be spent in data extraction from CAD software. These data will need to be properly curated and formed into text formats (such as JSON lines) for both input and target, and then used for the required aspect of LLM enhancement. Hence, domain-specific instructions coupled with relevant LLM enhancement techniques are still required in order to bring existing LLMs to the standard where they can generate outputs with high level of precision and accuracy when executed by CAD software.

From our practical experience with existing LLMs (accessed via API), the max token limitation is a major drawback in providing instructions and geometric information to LLMs for the purposes of of optimizing designs. Gpt 4 has max tokens of 8192 but even gpt-4-turbo with max token length of 128,000 tokens may still struggle to taken in the volume of text input data extracted from very large architectural designs. For large input tokens, the computational time for the optimisation process is quite large. As it stands currently, certain code and performance optimisation method is required to enable the use of existing LLMs for inputs and instructions. In the extreme case, one may consider training or finetuning LLMs using advanced methods that can increase the efficiency of the new architectural optimisation-focused LLM engines.

Comparison of LLMs with Deterministic Optimisation Methods

The table below summarises the findings for comparing deterministic optimisation methods with LLM-based solution.

Comparing deterministic optimisation methods with LLM-based solutions

Recommendations

1. Use LLMs for Complex and Evolving Problems: LLMs are well-suited for problems that involve complex, high-dimensional data and require adaptability to new information.
2. Combine Methods for Best Results: Hybrid approaches that integrate deterministic methods with LLMs can leverage the strengths of both, providing high precision and flexibility.
3. Invest in Training Data: For LLMs to be effective in 3-D optimization, they need access to extensive and diverse datasets that cover various optimization scenarios. Time should be invested in the collection and curation of training data to ensure the success of the LLM solution. Both few-shot and fine tuning approaches should be explored for best results.
4. Monitor and Validate Results: Given the probabilistic nature of LLMs, it’s crucial to validate their outputs using deterministic methods or domain expertise to ensure reliability. Use an iterative approach and validate results on simple problems before moving to more complex implementations. Results should be verifiable and test data should be taken from various scenarios that are representative of the target deployment of the solution.

Summary and Conclusion

Deterministic methods offer precision and predictability but struggle with complexity and adaptability. In contrast, LLMs provide flexibility and learning capabilities, making them suitable for dynamic and complex problems. However, their probabilistic nature requires careful validation.

In conclusion, the choice between these methods depends on the specific requirements of the optimization task. For static and well-defined problems, deterministic methods remain a reliable choice. For evolving and complex scenarios, LLMs offer promising potential, especially when combined with traditional techniques to enhance overall performance and reliability. As technology advances, the integration of LLMs in optimization workflows is likely to become more prevalent, driving innovation and efficiency in 3-D space optimization across various fields. For the time being, more time should be invested in the collection and curation of diverse supervised learning data to enhance the performance of LLMs in 3-D optimisation problems.