Introducing Orca: Microsoft’s Revolutionary Open-Source AI Model

Microsoft, in collaboration with OpenAI, has been at the forefront of integrating AI capabilities into its products and services while also developing specialized models for specific use cases. A recent breakthrough from Microsoft Research introduces Orca, an innovative AI model that leverages imitation learning from large language models. Designed to overcome the limitations of smaller models, Orca emulates the reasoning processes of foundational models like GPT-4.

Orca, a language model of 13 billion parameters, offers the advantage of optimization for specific tasks and training using large language models such as GPT-4. Despite its smaller size, Orca requires fewer computing resources to operate, allowing researchers to tailor their models to their specific needs and run them independently, without relying on massive data centers.

In their pursuit of progressive learning, Microsoft utilizes extensive and diverse imitation data to train Orca. This approach has already yielded impressive results, with Orca surpassing its predecessor, Vicuna, by a staggering 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard (BBH). Furthermore, the new model boasts a 42% increase in speed compared to conventional AI models on AGIEval.

Remarkably, Orca demonstrates comparable reasoning capabilities to ChatGPT on benchmarks such as BBH, showcasing its competency in logical thinking. Additionally, it exhibits competitive performance on academic examinations such as SAT, LSAT, GRE, and GMAT, albeit falling short of the capabilities of GPT-4.

The research team at Microsoft emphasizes that Orca can learn effectively through step-by-step explanations provided by humans and more advanced language models. This collaborative learning approach enables Orca to continually enhance its skills and expand its capabilities.

The development of Orca is rooted in recent research focusing on improving the effectiveness of smaller models through imitation learning, utilizing outputs from large foundational models. Challenges arise due to limited imitation signals from shallow LFM outputs, small-scale homogeneous training data, and a lack of rigorous evaluation, often resulting in an overestimation of the small model’s capabilities, as they tend to imitate the style but not the reasoning process of LFMs.

To address these challenges, the Microsoft team has introduced Orca, a groundbreaking 13-billion parameter model that excels in imitating the reasoning processes of LFMs. Drawing on rich signals from GPT-4, including explanation traces, step-by-step thought processes, and complex instructions, Orca benefits from the guidance of ChatGPT in its learning process. Through meticulous sampling and selection of large-scale and diverse imitation data, progressive learning is promoted.

The results are exceptional, with Orca outperforming traditional instruction-tuned models like Vicuna-13B by over 100% in complex zero-shot reasoning benchmarks such as Big-Bench Hard (BBH) and achieving a 42% improvement on AGIEval. Notably, Orca achieves parity with ChatGPT on the BBH benchmark and demonstrates competitive performance, with a mere 4-point gap from the optimized system message, in professional and academic examinations like SAT, LSAT, GRE, and GMAT, even in zero-shot settings without CoT. While trailing behind GPT-4, this research highlights the promise of learning from step-by-step explanations, whether generated by humans or advanced AI models, as a direction for further enhancing model capabilities and skills.

In conclusion: The introduction of Orca marks a significant milestone in the advancement of AI models. Microsoft’s open-source approach and collaboration with OpenAI have yielded a model that excels in reasoning and demonstrates remarkable performance across various benchmarks and examinations. As the research continues to explore the potential of step-by-step explanations, the future holds great promise for further enhancing the capabilities of AI models and revolutionizing the field of artificial intelligence.

Source- Paper