PARIS, France, May 4th, 2021 — Paris-based startup LightOn, maker of photonic co-processors for large-scale AI, unveils today PAGnol, a 1.5 billion parameter language model. PAGnol is a collection of large French language models, geared towards free-form text generation — with PAG standing for pré-apprentissage génératif. With 1.5 billion parameters, PAGnol-XL is the largest model trained to date for the French language. PAGnol is based on the GPT architecture and uses scaling laws predictions for efficient training.
PAGnol is the first language model trained by LightOn, in cooperation with the ALMAnaCH team of Inria Paris, and used the…
Beidi Chen, a Postdoc Researcher at Stanford, was the guest of LightOn’s 13th AI Meetup and presented her work on SLIDE&MONGOOSE: LSH Frameworks for Efficient Neural Networks Training⚡ to appear at ICLR 2021.
We have previously talked about co-designing hardware and software ↔️ to unlock the next generation of machine learning models 🤖, and this talk fits perfectly in that narrative.
The underlying idea is to rely on Locality Sensitive…
Progress usually comes from a steady technology bootstrap…until it doesn’t.
Take for instance the race for the $1,000 genome that started in the early 2000s. Initially, sequencing the human genome meant a race between the well-funded public and private sectors but more importantly, the resources for the first breakthrough ended up costing upwards of $450M. Yet despite all the economic promise of genome sequencing, had Moore’s law been applied, sequencing one full genome would still cost $100,000 today. However, once the goal became clearer to everyone, a diversity of technologies and challengers emerged. This intense competition eventually yielded a growth…
Igor Carron, our CEO and co-founder gave a talk at #mathia2021 conference on March 9th, 2021 where he drew a parallel between the scaling laws that enabled industrialization in the 1920’s and the new scaling laws in AI of the 2020’s.
AI is at its infancy and it needs to have guiding principles (as embedded in these empirical laws) and it also needs to develop new hardware. Igor showed how, in this context, LightOn can help unlock Transformative AI. Enjoy!
Thank you to the organizers (FMJH et FSMP) of the MATH & IA, conference for having us speak at the unique event.
At the 12th LightOn AI Meetup, we hosted Hashem Ghanem, a Ph.D. student at GIPSA-Lab, Grenoble, and IMB, Dijon, who presented his work on🕸️ Fast Graph Kernel with Optical Random Features ⚡ to appear at ICASSP 2021.
Hashem has been one of the first users of our technology on the LightOn Cloud, and we are particularly proud of his work using Optical Processing Units being accepted to a prestigious conference.
A week ago we released LightOn Appliance, the world’s most powerful photonic co-processor for AI and HPC. We originally had our technology available on the cloud with a nascent community, it can now be installed on-premises, as a 2U rackable unit.
Transformative AI such as OpenAI GPT-3 is starting to blossom with huge economic promises while stressing global silicon capabilities. With the Appliance, we are enabling researchers, scientists, and engineers to open pathways to large-scale computations that would just not be feasible with current silicon technology.
We are already accepting pre-orders for the Appliance, available under leasing schemes starting at € 1.900 per month. A product data sheet, as well as pricing information, are available at LightOn.ai/lighton-appliance
Here is the press release.
Software is eating the world, machine learning is eating software, and, well, transformers 🤖 are eating machine learning.
We discussed this in our previous meetup: attention mechanisms are very apt to parallelization ⛓️ and avoid catastrophic forgetting 😶🌫️ , however, their limited scalability is a roadblock 🚧…
For the 10th edition of the LightOn AI Meetup we had Krzysztof Choromanski, Research Scientist at Google Robotics NYC and Adjunct Assistant Professor at Columbia University, presenting his work on Rethinking Attention with Performers, that will appear as oral presentation at ICLR 2021. Congratulations Krzysztof and co-authors!
The leading approaches in language modeling are all obsessed with TV shows of my youth — namely Transformers and Sesame Street. …
We live in interesting times!
A combination of post-Moore’s law era and the advent of very large ML models require all of us to think up new approaches to computing hardware and AI algorithms at the same time. LightOn is one of the few (20) companies in the world publishing in both AI and hardware venues to engage both communities into thinking how theories and workflows may eventually be transformed by the photonic technology we develop.
The 9th LightOn AI Meetup was awesome! This time, we had Sara Hooker, Research Scholar at Google Brain, talking about The Hardware Lottery. The video of the meetup is on our Youtube channel. Subscribe to the channel and subscribe to our Meetup to get notified of the next videos and events. We meet again on the 14th of January to discuss the Performer architecture with Krzysztof Choromanski!
Everything starts from discussing what incentivized the development of hardware, software and machine learning in isolation from each other: Moore’s law 📈 and Dennard scaling 📈 provided a predictable increase in compute…