Prompt: Humanoid Robot with human brain learning at a Library

AI vs Human Intelligence: Do We Want to Compete?

Scientists have studied and measured how much energy the human brain uses for learning and retrieving memories. For example:

• At rest, the brain represents just 2% of a person’s body weight but uses around 20% of the body’s total energy. Most of that energy powers neuron communication required for thought processes.
• During focused learning, the brain’s energy consumption can increase by 12-25%. Scans like PET and fMRIs confirm increased blood flow to regions like the hippocampus during memory formation and learning tasks.
• The energy powers molecular changes at neuron connections (synapses) that store memories. This includes forming stronger synaptic connections between neurons activated in unison during learning.
• Retrieving a memory regenerates activity in some of the same neuronal connections. More brain energy goes toward accessing episodic memories (recalling past experiences) than accessing semantic memories.
• Over a whole day, the additional energy explicitly spent on learning and remembering may represent 10% or less of total brain energy usage. But for focused mental tasks, learning drives significantly higher local neural activity.

Overall, there are some remaining gaps in knowledge, but scientists have quantified clear increases in brain energy consumption directly attributable to the molecular processes of learning and memory retrieval. Advanced imaging confirms distinct parts of the brain ramping up activity as they consolidate and access different forms of memories.

Now, estimating the brain’s energy usage during learning and memory retrieval in absolute energy units like kilowatt-hours involves some approximations, but based on research data, one could propose is a rough breakdown:

• The brain's resting metabolic rate is around 12-15 watts of continuous power consumption.
• During intense learning, power consumption can increase locally in active regions of the brain by 2-3 additional watts sustained over hours of study.
• For shorter-term memory formation measured over minutes, peak estimated power can briefly reach up to 30 watts locally.
• Integrating additional power used over time, one study estimates the brain uses an extra 0.9 calories per minute during intense memorization. That's about 1.5 watts sustained.
• Over a full day of learning new skills and concepts, the total energy used may equal 20-40+ kilojoules or 0.0055 - 0.011 kilowatt-hours higher than a rest day.

So in total kWh numbers - the brain may use only around 0.01 kWh more for learning over hours or a full day. But in specific regions active during memory encoding, power in watts can be 14-20% higher than baseline. It's a dynamically shifting process.

If we assume the average 100 watt light bulb is on for about 12 hours, that would use 1.2 kilowatt-hours (kWh) of electricity.

In comparison - published studies have estimated the brain utilizes an extra 0.0055 - 0.011 kWh over a whole day of intensive learning compared to rest.

So taking the higher estimate of 0.011 kWh:

The brain's "extra" energy used during learning equates to having a 100 watt bulb turned on for:

0.011 kWh x (1000 watts / 100 watts-per-bulb) = 0.11 hours

Which equals about 7 minutes!

So the brain powering intense daily learning and memory formation requires about the same amount of "bonus" energy over rest as powering a 100 watt light bulb for 7 minutes. Pretty efficient when you consider all the encoding and recalling happening in neurons during that time!

Entering AI

In deep learning models, accuracy generally scales up logarithmically with increased computations/energy. So, going from 90% to 99% takes vastly more computation than going from 80% to 90%.

One analysis from OpenAI found that scaling a language model from 92% to 95% accuracy increased computing needs by 3-4x. Going to 99% may increase computing by 100x or more compared to the 92% model.

So, while the exact ratio depends on many factors, it's reasonable to estimate that going from 90% accuracy to 99% likely increases energy consumption by at least 10x for a deep neural network model, and potentially much more (100x+ in some cases).

The computations required for the last few accuracy gains tend to be very high. This leads to diminishing returns - a lot more energy for smaller accuracy improvements.

There are also model techniques like knowledge distillation that can help achieve higher accuracy without necessarily requiring hugely increased computations.

So in summary, while dependent on model specifics, 99% will generally require disproportionally more computations and energy than a 90% accurate model. The exact ratio could reasonably be estimated as anywhere from 10x to over 100x in typical deep learning scenarios

AI vs Humans Learning

The key training process that enables many AI systems today is called stochastic gradient descent (SGD). This requires feeding the AI model thousands to billions of examples to learn patterns.

• A benchmark often used to train common AI models like BERT requires over 3.3 billion words fed for analysis during SGD.

Let's assume an average human reads at 200-250 words per minute with full comprehension and memorization. At the higher end, that's 15,000 words per hour.

To read and memorize 3.3 billion words at 15,000 words per hour:

• 3,300,000,000 words / 15,000 words per hour = 220,000 hours
• Assuming an FTE of 40 hours per week:
• 220,000 hours / 40 hours per week = 5500 weeks = over 100 years!

So for a human to read and memorize a similar sized dataset to what AI models are trained on would take over 100 years of full-time effort.

But, if we imagine 3,000 people, the entire benchmark 3.3 billion word dataset could be read and memorized in just 1 day at a 15,000 words-per-hour pace per person.

• 3000 people x 7 minutes of 100W lightbulb per person of brain energy = 21,000 minutes
• 21,000 minutes / (60 minutes per hour) = 350 hours
• 350 hours x 100W = 35,000 Watt-hours

35,000 Wh = 35 kWh

So 3,000 people could handle the workload in 1 day, for a total energy expenditure equivalent to powering a 100W light bulb for 35,000 Watt-hours or 35 kilowatt-hours.

35 kWh is around the average daily electricity consumption for a single household in most European Union or USA locations. Now let’s considere the following :

• ChatGPT was trained on a dataset called the Pile which contains approximately 570 GB of text data, equalling roughly 1.95 trillion words.
• Anthropic has not publicly released the full training details of Claude, but it uses a similar self-supervised learning approach trained on diverse dialogue data over many months.
• OpenAI has stated ChatGPT required several thousand petaflop/s-days for training. That equates to millions of hours on extremely advanced supercomputers with thousands of GPUs.
• Anthropic also utilized considerable computing power of cloud TPUs for prolonged periods. Their fuller methodology will be detailed later.

So while a few thousand humans could manually process a couple billion words in a reasonable time span - state of the art AI today utilizes literally orders of magnitude larger datasets.

• In perspective, OpenAI’s GPT-3 model (predecessor to ChatGPT) required an estimated 300-400 megawatt-hours for training. That’s hundreds of homes worth of power used!

The human capability thought exercise makes for an eye-opening comparison to AI’s intense computational demands. We should be aware on the immense resources needed to create systems like Claude, Bard or ChatGPT with such breadth of knowledge and capability.

Conclusion

When it comes to scaling up advanced AI to be more flexible, broad, and powerful in the future, energy efficiency is arguably just as much a core challenge as further developing the machine learning theory and knowledge representation itself. A few reasons this is the case:

Unprecedented growth - The compute/energy needs for state-of-the-art AI have been doubling every few months. This exponential trajectory far outstrips efficiency gains in data centers. Without improved efficiency, energy needs could quickly become impractical.
Carbon footprint - The exponential growth also creates sustainability issues like larger carbon footprints. Further progress must happen responsibly and account for environmental impact through greener computing.
Economic viability - In addition to carbon footprints, the ballooning energy costs themselves threaten the economic viability of advancing AI systems. New hardware, software breakthroughs must constrain energy as much as focussing on accuracy gains.
Embedded/mobile applications - Beyond cloud-scale models, improved efficiency opens the door to more implementations of AI at the edge and applications like robotics. Energy pots major limitation on reach.

Yes the fundamental theory, knowledge representation, reasoning capabilities etc remain open frontiers for the AI techies.

But to enable AI’s responsibly applied potential at global scale, energy efficiency and overall compute constraints are equally crucial challenges needing even more crucial breakthroughs in the closest future.