Unlocking Technical Contradictions in Patents Using LLMs: A Journey with TRIZ and Prompt Engineering
In the evolving landscape of intellectual property, extracting meaningful insights from patents is a challenging yet critical task. My recent analysis delved into how a large language model (LLM) can be leveraged to extract technical contradictions from patents based on the principles of TRIZ (Theory of Inventive Problem Solving). The twist? I did it using only prompt engineering.
The Experiment: A Blend of Innovation and Validation
To validate the effectiveness of my approach, I utilized raw data from a scientific publication where 1,600 patents had been meticulously labeled manually. This dataset served as the gold standard for comparison. By employing metrics such as BERTScore, BLEU, and ROUGE, I was able to quantify the performance of the LLM against human annotations.
Diverse Results: A Closer Look
The results of this experiment were intriguing and diverse.
While the mean values for all the measures except BERT Recall are below 0.5, indicating a generally low average performance, the maximum values tell a different story. High individual results over 0.9 were achieved, demonstrating that the model is capable of generating highly accurate text in some instances.
BERT Recall, in particular, shows even better performance with a reasonable mean value and numerous results exceeding 0.5. This suggests that the model consistently captures a significant portion of relevant content from the reference texts, highlighting its strength in maintaining the semantic essence of the original inputs.
Let’s explore a couple of examples that highlight the capabilities and limitations of using LLMs for such complex tasks.
A High-Performance Example
One notable success was observed with Patent: US10276191B2.
Manual Generated Contradiction Set:
- First part of contradiction: In this type of voice recognition system, to reduce the communication volume from the client device to the server device, a process has been developed that detects a section of voice hereinafter referred to as a speech section from a signal received by the client side, and only transmits a signal in the detected speech section to the server device.
- Second part of contradiction: In detecting the speech section in the client device, it is difficult to accurately detect the speech section, because resources of the client device are limited compared to those of the server device. Furthermore, because the voice state of the client side differs according to the environment and changes often, there is a possibility of not being able to collect spoken voice.
AI Generated Contradiction Set:
- First part of contradiction: In this type of voice recognition system, to reduce the communication volume from the client device to the server device, a process has been developed that detects a section of voice hereinafter referred to as a speech section from a signal received by the client side, and only transmits a signal in the detected speech section to the server device.
- Second part of contradiction: In detecting the speech section in the client device, it is difficult to accurately detect the speech section, because resources of the client device are limited compared to those of the server device.
In this instance, the LLM accurately replicated the first part of the contradiction and closely mirrored the second part, demonstrating a high degree of semantic understanding and extraction capability. The BERTScore for this extraction was impressively high, indicating a strong semantic match.
A Low-Performance Example
Conversely, the LLM struggled with Patent: US10028242B2.
Manual Generated Contradiction Set:
- First part of contradiction: With the rapid increase in requirement for total network throughput, a network interference problem becomes a key factor hindering further improvement of network capacity. Increasing throughput by eliminating or making use of interference from the perspective of multiple sending ends becomes an important trend, such as methods of cooperative transmission, coordinated beamforming, and interference alignment and the like.
- Second part of contradiction: In the prior art, a technical solution for synchronization, sharing of information such as data, obtaining of channel information, and data transmission has larger overhead, lower work efficiency, and limited application scenarios in an actual application.
AI Generated Contradiction Set:
- First part of contradiction: With the rapid increase in requirement for total network throughput, a network interference problem becomes a key factor hindering further improvement of network capacity.
- Second part of contradiction: Increasing throughput by eliminating or making use of interference from the perspective of multiple sending ends becomes an important trend.
In this case, the LLM failed to capture the full scope of the contradiction. It accurately identified the first part but did not fully grasp the second part’s complexities, leading to a significant deviation from the manual annotation. The BLEU and ROUGE scores reflected this gap, showing a notable difference in performance.
Insights and Future Directions
Despite the mixed results, this experiment underscored the potential of LLMs in extracting complex concepts from patents. The key takeaway is that data plays a pivotal role in training and fine-tuning these models. With sufficient high-quality data, LLMs can indeed unravel intricate technical contradictions.
This realization led to the creation of TRIZ-GPT, a dedicated website aimed at harnessing the power of LLMs for TRIZ analysis. The platform allows users to explore results, contribute data, and collaborate on refining the models. By building a robust dataset and community, we can enhance the capability of LLMs to tackle even the most complex patent analysis tasks.
Join the Journey
I invite you to visit TRIZ-GPT, explore the findings, and join us in this exciting endeavor. Together, we can push the boundaries of what’s possible with AI and patent analysis. Check it out and be part of the innovation revolution!
— -
Feel free to share your thoughts, insights, or questions in the comments below. Let’s drive the future of patent analysis forward, one prompt at a time.
