The Alignment Problem: Machine Learning and Human Values

Cheery Friday Greetings to our Learning How to Learners!

The Alignment Problem: Machine Learning and Human Values, by Brian Christian, is a surprising contender for the most important book on learning of the decade. Christian tackles the complex challenge of ensuring that artificial intelligence remains aligned with human values as it grows more powerful.

The book masterfully weaves together insights from computer science, psychology, ethics, and more. Christian delves into key machine learning approaches and their implications for alignment, including:

• Reinforcement learning, where AI agents learn to take actions that maximize their rewards in an environment. Christian explains how difficult it is to specify reward functions that capture the full nuance of human values.
• Inverse reinforcement learning, which flips this problem by inferring the reward function an agent is optimizing from its observed behavior. This could allow AI to learn human values by watching what we do, rather than requiring us to specify them explicitly.
• Curiosity-driven learning techniques like empowerment and information gain, which motivate AI systems to explore and learn about the world without explicit rewards. This highlights the challenge of making sure AI pursues the right objectives.
• Cutting-edge approaches to better specify and pursue human preferences, even when they’re complex or uncertain:
  • Reward modeling, where AI tries to approximate the true reward function a human wants to optimize.
  • Inverse reward design, which accounts for the difficulty of translating values into concrete rewards.
  • Debate, where AI systems argue to surface flaws and refine their understanding of human values.
  • Cooperative inverse reinforcement learning and iterated amplification, where AI and humans collaborate to clarify and pursue the human’s goals.

Christian highlights both the promise and perils of these techniques, from the risk of encoding biases to the challenge of avoiding unintended consequences. 

The Alignment Problem is a critical framework for understanding the future of AI and its profound implications for humanity. Christian’s exploration of the frontiers of machine learning and the ethical quandaries they raise make the book essential reading. It’s a landmark work, not just for AI, but for understanding how we learn and create in an age where neural nets shape more and more of our lives. The Alignment Problem is an intellectual tour de force and a must-read for anyone seeking to grasp one of the defining challenges of our time.

Barb at the Said School of Business, Oxford University

In case you missed it, here is Barb’s talk on genAI and the brain—the room was packed, and over 1,700 were watching live via LinkedIn and YouTube.  Enjoy!

10 Times Scientists Admitted They Were Wrong, and What You Can Learn from Them

Here’s a worthwhile article by Hashem Elassad on what you can learn from admitting you were wrong—as modeled by great scientists (including our own beloved Santiago Ramón y Cajal).  

How Latin America started to beat inequality

This terrific article by Noah Smith helps us parse the data to see that attention to education can really matter in improving people’s lives. Key graf: “Better education boosts growth, especially in countries like Latin America where education had traditionally lagged. And it boosts growth in an equitable way, because when a larger chunk of the population is educated, the “skill premium” falls — you don’t just have a few elite workers taking all the cash, because they’re suddenly in direct competition with a much larger educated workforce.”

The Global Science of Learning Education Network presents their Scientist of the Month: Dr. Barbara Oakley

The Global Science of Learning community was originally born out of a need to assemble teams of experts prepared to address societal challenges surrounding learning at global scale and to devise a way to bring the widespread science of learning (SoL) community together. The science of learning movement was accelerated in the United States by significant investment by the National Science Foundation and propagated around the world. The outcome of the planning efforts is the Global Science of Learning for Education Network (GSoLEN), a network-of-networks aiming to develop a collaborative ecosystem for infusing science into widespread practice with the informed voices of researchers, educators, policy makers and philanthropists.  Professor Barb Oakley brings a unique perspective to GSoLEN, which is why she’s been honored as their Scientist of the Month. 

Podcast Conversation with MAYO Clinic host Stacy Craft

Stacy Craft is a Senior Instructional Designer and Assistant Professor of Medical Education at the Mayo Clinic. Check out her conversation with Barb “From Neurons to Networks: Understanding the Biology of Learning and Artificial Intelligence Parallels.”

That’s all for now. Have a happy week in Learning How to Learn!

Barb, Terry, and the entire Learning How to Learn team

• Uncommon Sense Teaching—the book and Coursera Specialization!

• Mindshift—the book and MOOC

• Learn Like a Pro—the book and MOOC

• The LHTL recommended text, A Mind for Numbers

• For kids and parents: Learning How to Learn—the book and MOOC. Pro tip—watch the videos and read the book together with your child. Learning how to learn at an early age will change their life!