This essay argues that rational people don’t have goals, and that rational AIs shouldn’t have goals. Human actions are rational not because we direct them at some final ‘goals,’ but because we align actions to practices^[1]: networks of actions, action-dispositions, action-evaluation criteria, and action-resources that structure, clarify, develop, and promote themselves. If we want AIs that can genuinely support, collaborate with, or even comply with human agency, AI agents’ deliberations must share a “type signature” with the practices-based logic we use to reflect and act.

Read the Full Piece

Episode 144

Happy New Year! This is one of my favorite episodes of the year — for the fourth time, Nathan Benaich and I did our yearly roundup of AI news and advancements, including selections from this year’s State of AI Report.

If you’ve stuck around and continue to listen, I’m really thankful you’re here. I love hearing from you.

You can find Nathan and Air Street Press here on Substack and on Twitter, LinkedIn, and his personal site. Check out his writing at press.airstreet.com.

Find me on Twitter (or LinkedIn if you want…) for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Outline

• (00:00) Intro

• (00:44) Air Street Capital and Nathan world

  • Nathan’s path from cancer research and bioinformatics to AI investing

  • The “evergreen thesis” of AI from niche to ubiquitous

  • Portfolio highlights: Eleven Labs, Synthesia, Crusoe

• (03:44) Geographic flexibility: Europe vs. the US

  • Why SF isn’t always the best place for original decisions

  • Industry diversity in New York vs. San Francisco

  • The Munich Security Conference and Europe’s defense pivot

  • Playing macro games from a European vantage point

• (07:55) VC investment styles and the “solo GP” approach

  • Taste as the determinant of investments

  • SF as a momentum game with small information asymmetry

  • Portfolio diversity: defense (Delian), embodied AI (Syriact), protein engineering

  • Finding entrepreneurs who “can’t do anything else”

• (10:44) State of AI progress in 2025

  • Momentous progress in writing, research, computer use, image, and video

  • We’re in the “instruction manual” phase

  • The scale of investment: private markets, public markets, and nation states

• (13:21) Range of outcomes and what “going bad” looks like

  • Today’s systems are genuinely useful—worst case is a valuation problem

  • Financialization of AI buildouts and GPUs

• (14:55) DeepSeek and China closing the capability gap

  • Seven-month lag analysis (Epoch AI)

  • Benchmark skepticism and consumer preferences (”Coca-Cola vs. Pepsi”)

  • Hedonic adaptation: humans reset expectations extremely quickly

  • Bifurcation of model companies toward specific product bets

• (18:29) Export controls and the “evolutionary pressure” argument

  • Selective pressure breeds innovation

  • Chinese companies rushing to public markets (Minimax, ZAI)

• (21:30) Reasoning models and test-time compute

  • Chain of thought faithfulness questions

  • Monitorability tax: does observability reduce quality?

  • User confusion about when models should “think”

  • AI for science: literature agents, hypothesis generation

• (23:53) Chain of thought interpretability and safety

  • Anthropomorphization concerns

  • Alignment faking and self-preservation behaviors

  • Cybersecurity as a bigger risk than existential risk

  • Models as payloads injected into critical systems

• (27:26) Commercial traction and AI adoption data

  • Ramp data: 44% of US businesses paying for AI (up from 5% in early 2023)

  • Average contract values up to $530K from $39K

  • State of AI survey: 92% report productivity gains

  • The “slow takeoff” consensus and human inertia

  • Use cases: meeting notes, content generation, brainstorming, coding, financial analysis

• (32:53) The industrial era of AI

  • Stargate and XAI data centers

  • Energy infrastructure: gas turbines and grid investment

  • Labs need to own models, data, compute, and power

  • Poolside’s approach to owning infrastructure

• (35:40) Venture capital in the age of massive GPU capex

  • The GP lives in the present, the entrepreneur in the future, the LP in the past

  • Generality vs. specialism narratives

  • “Two or 20”: management fees vs. carried interest

  • Scaling funds to match entrepreneur ambitions

• (40:10) NVIDIA challengers and returns analysis

  • Chinese challengers: 6x return vs. 26x on NVIDIA

  • US challengers: 2x return vs. 12x on NVIDIA

  • Grok acquired for $20B; Samba Nova markdown to $1.6B

  • “The tide is lifting all boats”—demand exceeds supply

• (44:06) The hardware lottery and architecture convergence

  • Transformer dominance and custom ASICs making a comeback

  • NVIDIA still 90–95% of published AI research

• (45:49) AI regulation: Trump agenda and the EU AI Act

  • Domain-specific regulators vs. blanket AI policy

  • State-level experimentation creates stochasticity

  • EU AI Act: “born before GPT-4, takes effect in a world shaped by GPT-7”

  • Only three EU member states compliant by late 2025

• (50:14) Sovereign AI: what it really means

  • True sovereignty requires energy, compute, data, talent, chip design, and manufacturing

  • The US is sovereign; the UK by itself is not

  • Form alliances or become world-class at one level of the stack

  • ASML and the Netherlands as an example

• (52:33) Open weight safety and containment

  • Three paths: model-based safeguards, scaffolding/ecosystem, procedural/governance

  • “Pandora’s box is open”—containment on distribution, not weights

  • Leak risk: the most vulnerable link is often human

  • Developer–policymaker communication and regulator upskilling

• (55:43) China’s AI safety approach

  • Matt Sheehan’s work on Chinese AI regulation

  • Safety summits and China’s participation

  • New Chinese policies: minor modes, mental health intervention, data governance

  • UK’s rebrand from “safety” to “security” institutes

• (58:34) Prior predictions and patterns

  • Hits on regulatory/political areas; misses on semiconductor consolidation, AI video games

• (59:43) 2026 Predictions

  • A Chinese lab overtaking US on frontier (likely ZAI or DeepSeek, on scientific reasoning)

  • Data center NIMBYism influencing midterm politics

• (01:01:01) Closing

Links and Resources

Nathan / Air Street Capital

• Air Street Capital

• State of AI Report 2025

• Air Street Press — essays, analysis, and the Guide to AI newsletter

• Nathan on Substack

• Nathan on Twitter/X

• Nathan on LinkedIn

From Air Street Press (mentioned in episode)

• Is the EU AI Act Actually Useful? — by Max Cutler and Nathan Benaich

• China Has No Place at the UK AI Safety Summit (2023) — by Alex Chalmers and Nathan Benaich

Research & Analysis

• Epoch AI: Chinese AI Models Lag US by 7 Months — the analysis referenced on the US-China capability gap

• Sara Hooker: The Hardware Lottery — the essay on how hardware determines which research ideas succeed

• Matt Sheehan: China’s AI Regulations and How They Get Made — Carnegie Endowment

Companies Mentioned

• Eleven Labs — AI voice synthesis (Air Street portfolio)

• Synthesia — AI video generation (Air Street portfolio)

• Crusoe — clean compute infrastructure (Air Street portfolio)

• Poolside — AI for code (Air Street portfolio)

• DeepSeek — Chinese AI lab

• Minimax — Chinese AI company

• ASML — semiconductor equipment

Other Resources

• Search Engine Podcast: Data Centers (Part 1 & 2) — PJ Vogt’s two-part series on XAI data centers and the AI financing boom

• RAAIS Foundation — Nathan’s AI research and education charity

Episode 143

I spoke with Iason Gabriel about:

• Value alignment

• Technology and worldmaking

• How AI systems affect individuals and the social world

Iason is a philosopher and Senior Staff Research Scientist at Google DeepMind. His work focuses on the ethics of artificial intelligence, including questions about AI value alignment, distributive justice, language ethics and human rights.

You can find him on his website and Twitter/X.

Find me on Twitter (or LinkedIn if you want…) for updates, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Outline

• (00:00) Intro

• (01:18) Iason’s intellectual development

• (04:28) Aligning language models with human values, democratic civility and agonism

• (08:20) Overlapping consensus, differing norms, procedures for identifying norms

• (13:27) Rawls’ theory of justice, the justificatory and stability problems

• (19:18) Aligning LLMs and cooperation, speech acts, justification and discourse norms, literacy

• (23:45) Actor Network Theory and alignment

• (27:25) Value alignment and Iason’s starting points

• (33:10) The Ethics of Advanced AI Assistants, AI’s impacts on social processes and users, personalization

• (37:50) AGI systems and social power

• (39:00) Displays of care and compassion, Machine Love (Joel Lehman)

• (41:30) Virtue ethics, morality and language, virtue in AI systems vs. MacIntyre’s conception in After Virtue

• (45:00) The Challenge of Value Alignment

  • (45:25) Technologists as worldmakers

  • (51:30) Technological determinism, collective action problems

• (55:25) Iason’s goals with his work

• (58:32) Outro

Links

Papers:

• AI, Values, and Alignment (2020)

• Aligning LMs with Human Values (2023)

• Toward a Theory of Justice for AI (2023)

• The Ethics of Advanced AI Assistants (2024)

• A matter of principle? AI alignment as the fair treatment of claims (2025)

"In projecting language back as the model for thought, we lose sight of the tacit embodied understanding that undergirds our intelligence." –Terry Winograd

The recent successes of generative AI models have convinced some that AGI is imminent. While these models appear to capture the essence of human intelligence, they defy even our most basic intuitions about it. They have emerged not because they are thoughtful solutions to the problem of intelligence, but because they scaled effectively on hardware we already had. Seduced by the fruits of scale, some have come to believe that it provides a clear pathway to AGI. The most emblematic case of this is the multimodal approach, in which massive modular networks are optimized for an array of modalities that, taken together, appear general. However, I argue that this strategy is sure to fail in the near term; it will not lead to human-level AGI that can, e.g., perform sensorimotor reasoning, motion planning, and social coordination. Instead of trying to glue modalities together into a patchwork AGI, we should pursue approaches to intelligence that treat embodiment and interaction with the environment as primary, and see modality-centered processing as emergent phenomena.

Read the full piece

Episode 142

Happy holidays! This is one of my favorite episodes of the year — for the third time, and I did our yearly roundup of all the AI news and advancements you need to know. This includes selections from this year’s State of AI Report, some early takes on o3, a few minutes LARPing as China Guys………

If you’ve stuck around and continue to listen, I’m really thankful you’re here. I love hearing from you.

You can find Nathan and here on Substack and on Twitter, LinkedIn, and his personal site. Check out his writing at press.airstreet.com.

Find me on Twitter (or LinkedIn if you want…) for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Outline

• (00:00) Intro

• (01:00) o3 and model capabilities + reasoning capabilities

• (05:30) Economics of frontier models

• (09:24) Air Street’s year and industry shifts: product-market fit in AI, major developments in science/biology, "vibe shifts" in defense and robotics

• (16:00) Investment strategies in generative AI, how to evaluate and invest in AI companies

• (19:00) Future of BioML and scientific progress: on AlphaFold 3, evaluation challenges, and the need for cross-disciplinary collaboration

• (32:00) The AGI question and technology diffusion: Nathan’s take on AGI and timelines, technology adoption, the gap between capabilities and real-world impact

  • (39:00) Differential economic impacts from AI, tech diffusion

  • (43:00) Market dynamics and competition

  • (50:00) DeepSeek and global AI innovation

• (59:50) A robotics renaissance? robotics coming back into focus + advances in vision-language models and real-world applications

• (1:05:00) Compute Infrastructure: NVIDIA’s dominance, GPU availability, the competitive landscape in AI compute

• (1:12:00) Industry consolidation: partnerships, acquisitions, regulatory concerns in AI

• (1:27:00) Global AI politics and regulation: international AI governance and varying approaches

• (1:35:00) The regulatory landscape

• (1:43:00) 2025 predictions

• (1:48:00) Closing

Links and Resources

From :

• The State of AI Report

• The State of Chinese AI

• Open-endedness is all we’ll need

• There is no scaling wall: in discussion with Eiso Kant (Poolside)

• Alchemy doesn’t scale: the economics of general intelligence

• Chips all the way down

• The AI energy wars will get worse before they get better

Other highlights/resources:

• Deepseek: The Quiet Giant Leading China’s AI Race — an interview with DeepSeek CEO Liang Wenfeng via , translated by , , and others

• A great position paper on open-endedness by Minqi Jiang, Tim Rocktäschel, and Ed Grefenstette — Minqi also wrote a blog post on this for us!

• for China Guys only: China’s AI Regulations and How They Get Made by Matt Sheehan (+ an interview I did with Matt in 2022!)

• The Simple Macroeconomics of AI by Daron Acemoglu + a critique by (more links in the Report)

• AI Nationalism by Ian Hogarth (from 2018)

• Some analysis on the EU AI Act + regulation from Lawfare

Episode 141

I spoke with Professor Philip Goff about:

• What a “post-Galilean” science of consciousness looks like

• How panpsychism helps explain consciousness and the hybrid cosmopsychist view

Enjoy!

Philip Goff is a British author, idealist philosopher, and professor at Durham University whose research focuses on philosophy of mind and consciousness. Specifically, it focuses on how consciousness can be part of the scientific worldview. He is the author of multiple books including Consciousness and Fundamental Reality, Galileo's Error: Foundations for a New Science of Consciousness and Why? The Purpose of the Universe.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (01:05) Goff vs. Carroll on the Knowledge Arguments and explanation

• (08:00) Preferences for theories

• (12:55) Curiosity (Grounding, Essence) and the Knowledge Argument

  • (14:40) Phenomenal transparency and physicalism vs. anti-physicalism

• (29:00) How Exactly does Panpsychism Help Explain Consciousness

  • (30:05) The argument for hybrid cosmopsychism

  • (36:35) “Bare” subjects / subjects before inheriting phenomenal properties

  • (40:35) Bundle theories of the self

  • (43:35) Fundamental properties and new subjects as causal powers

  • (50:00) Integrated Information Theory

  • (55:00) Fundamental assumptions in hybrid cosmopsychism

• (1:00:00) Outro

Links:

• Philip’s homepage and Twitter

• Papers

  • Putting Consciousness First

  • Curiosity (Grounding, Essence) and the Knowledge Argument

Hi everyone!

If you’re a new subscriber or listener, welcome.

If you’re not new, you’ve probably noticed that things have slowed down from us a bit recently. , and I sat down to recap some of The Gradient’s history, where we are now, and how things will look going forward.

To summarize and give some context:

The Gradient has been around for around 6 years now – we began as an online magazine, and began producing our own newsletter and podcast about 4 years ago. With a team of volunteers — we take in a bit of money through Substack that we use for subscriptions to tools we need and try to pay ourselves a bit — we’ve been able to keep this going for quite some time.

Our team has less bandwidth than we’d like right now (and I’ll admit that at least some of us are running on fumes…) — we’ll be making a few changes:

• Magazine: We’re going to be scaling down our editing work on the magazine. While we won’t be accepting pitches for unwritten drafts for now, if you have a full piece that you’d like to pitch to us, we’ll consider posting it. If you’ve reached out about writing and haven’t heard from us, we’re really sorry. We’ve tried a few different arrangements to manage the pipeline of articles we have, but it’s been difficult to make it work. We still want this to be a place to promote good work and writing from the ML community, so we intend to continue using this Substack for that purpose. If we have more editing bandwidth on our team in the future, we want to continue doing that work.

• Newsletter: We’ll aim to continue the newsletter as before, but with a “Best from the Community” section highlighting posts. We’ll have a way for you to send articles you want to be featured, but for now you can reach us at our editor@thegradient.pub.

• Podcast: I’ll be continuing this (at a slower pace), but eventually transition it away from The Gradient given the expanded range. If you’re interested in following, it might be worth subscribing on another player like Apple Podcasts, Spotify, or using the RSS feed.

• Sigmoid Social: We’ll keep this alive as long as there’s financial support for it.

If you like what we do and/or want to help us out in any way, do reach out to editor@thegradient.pub. We love hearing from you.

Timestamps

• (0:00) Intro

• (01:55) How The Gradient began

• (03:23) Changes and announcements

• (10:10) More Gradient history! On our involvement, favorite articles, and some plugs

Some of our favorite articles!

There are so many, so this is very much a non-exhaustive list:

• NLP’s ImageNet moment has arrived

• The State of Machine Learning Frameworks in 2019

• Why transformative artificial intelligence is really, really hard to achieve

• An Introduction to AI Story Generation

• The Artificiality of Alignment (I didn’t mention this one in the episode, but it should be here)

Places you can find us!

Hugh:

• Twitter

• Personal site

• Papers/things mentioned!

  • A Careful Examination of LLM Performance on Grade School Arithmetic (GSM1k)

  • Planning in Natural Language Improves LLM Search for Code Generation

  • Humanity’s Last Exam

Andrey:

• Twitter

• Personal site

• Last Week in AI Podcast

Daniel:

• Twitter

• Substack blog

• Personal site (under construction)

I’m very excited to share this piece by Henry Kvinge about the overlap between machine learning research and pure mathematical domains like topology, algebra, and geometry. Henry, who is an AI researcher and mathematician at Pacific Northwest National Laboratory, makes a very compelling case that simply scaling existing methods is not all we need and gives a fascinating tour of some recent work applying these domains to ML. He concludes that instead of seeing scale-driven progress and empirical breakthroughs as a challenge to mathematical theory, mathematicians should embrace these developments as opportunities to develop new tools that can deepen our theoretical understanding. — Cole

Read the article here

Episode 140

I spoke with Professor Jacob Andreas about:

• Language and the world

• World models

• How he’s developed as a scientist

Enjoy!

Jacob is an associate professor at MIT in the Department of Electrical Engineering and Computer Science as well as the Computer Science and Artificial Intelligence Laboratory. His research aims to understand the computational foundations of language learning, and to build intelligent systems that can learn from human guidance. Jacob earned his Ph.D. from UC Berkeley, his M.Phil. from Cambridge (where he studied as a Churchill scholar) and his B.S. from Columbia. He has received a Sloan fellowship, an NSF CAREER award, MIT's Junior Bose and Kolokotrones teaching awards, and paper awards at ACL, ICML and NAACL.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (00:40) Jacob’s relationship with grounding fundamentalism

• (05:21) Jacob’s reaction to LLMs

• (11:24) Grounding language — is there a philosophical problem?

• (15:54) Grounding and language modeling

• (24:00) Analogies between humans and LMs

• (30:46) Grounding language with points and paths in continuous spaces

  • (32:00) Neo-Davidsonian formal semantics

  • (36:27) Evolving assumptions about structure prediction

  • (40:14) Segmentation and event structure

• (42:33) How much do word embeddings encode about syntax?

  • (43:10) Jacob’s process for studying scientific questions

  • (45:38) Experiments and hypotheses

• (53:01) Calibrating assumptions as a researcher

  • (54:08) Flexibility in research

• (56:09) Measuring Compositionality in Representation Learning

  • (56:50) Developing an independent research agenda and developing a lab culture

• (1:03:25) Language Models as Agent Models

  • (1:04:30) Background

  • (1:08:33) Toy experiments and interpretability research

  • (1:13:30) Developing effective toy experiments

• (1:15:25) Language Models, World Models, and Human Model-Building

  • (1:15:56) OthelloGPT’s bag of heuristics and multiple “world models”

  • (1:21:32) What is a world model?

  • (1:23:45) The Big Question — from meaning to world models

  • (1:28:21) From “meaning” to precise questions about LMs

  • (1:32:01) Mechanistic interpretability and reading tea leaves

  • (1:35:38) Language and the world

  • (1:38:07) Towards better language models

  • (1:43:45) Model editing

• (1:45:50) On academia’s role in NLP research

• (1:49:13) On good science

• (1:52:36) Outro

Links:

• Jacob’s homepage and Twitter

• Language Models, World Models, and Human Model-Building

• Papers

  • Semantic Parsing as Machine Translation (2013)

  • Grounding language with points and paths in continuous spaces (2014)

  • How much do word embeddings encode about syntax? (2014)

  • Translating neuralese (2017)

  • Analogs of linguistic structure in deep representations (2017)

  • Learning with latent language (2018)

  • Learning from Language (2018)

  • Measuring Compositionality in Representation Learning (2019)

  • Experience grounds language (2020)

  • Language Models as Agent Models (2022)

Episode 139

I spoke with Evan Ratliff about:

• , Evan’s new podcast, where he creates an AI voice clone of himself and sets it loose.

• The end of the Longform Podcast and his thoughts on the state of journalism.

Enjoy!

Evan is an award-winning investigative journalist, bestselling author, podcast host, and entrepreneur. He’s the author of the The Mastermind: A True Story of Murder, Empire, and a New Kind of Crime Lord; the writer and host of the hit podcasts Shell Game and Persona: The French Deception; and the cofounder of The Atavist Magazine, Pop-Up Magazine, and the Longform Podcast. As a writer, he’s a two-time National Magazine Award finalist. As an editor and producer, he’s a two-time Emmy nominee and National Magazine Award winner.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (01:05) Evan’s ambitious and risky projects

• (04:45) Wearing different personas as a journalist

• (08:31) Boundaries and acceptability in using voice agents

• (11:42) Impacts on other people

• (13:12) “The kids these days” — how will new technologies impact younger people?

• (17:12) Evan’s approach to children’s technology use

• (20:05) Techno-solutionism and improvements in medicine, childcare

• (24:15) Evan’s perspective on simulations of people

• (27:05) On motivations for building tech startups

• (30:42) Evan’s outlook for Shell Game’s impact and motivations for his work

• (36:05) How Evan decided to write for a career

• (40:02) How voice agents might impact our conversations

• (43:52) Evan’s experience with Longform and podcasting

• (47:15) Perspectives on doing good interviews

• (52:11) Mimicking and inspiration, developing style

• (57:15) Writers and their motivations, the state of longform journalism

• (1:06:15) The internet and writing

• (1:09:41) On the ending of Longform

• (1:19:48) Outro

Links:

• Evan’s homepage and Twitter

• Shell Game, Evan’s new podcast

• Longform Podcast

Episode 138

I spoke with Meredith Morris about:

• The intersection of AI and HCI and why we need more cross-pollination between AI and adjacent fields

• Disability studies and AI

• Generative ghosts and technological determinism

• Developing a useful definition of AGI

I didn’t get to record an intro for this episode since I’ve been sick.

Enjoy!

Meredith is Director for Human-AI Interaction Research for Google DeepMind and an Affiliate Professor in The Paul G. Allen School of Computer Science & Engineering and in The Information School at the University of Washington, where she participates in the dub research consortium. Her work spans the areas of human-computer interaction (HCI), human-centered AI, human-AI interaction, computer-supported cooperative work (CSCW), social computing, and accessibility. She has been recognized as an ACM Fellow and ACM SIGCHI Academy member for her contributions to HCI.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Meredith’s influences and earlier work

• (03:00) Distinctions between AI and HCI

• (05:56) Maturity of fields and cross-disciplinary work

• (09:03) Technology and ends

• (10:37) Unique aspects of Meredith’s research direction

• (12:55) Forms of knowledge production in interdisciplinary work

• (14:08) Disability, Bias, and AI

  • (18:32) LaMPost and using LMs for writing

    • (20:12) Accessibility approaches for dyslexia

    • (22:15) Awareness of AI and perceptions of autonomy

  • (24:43) The software model of personhood

  • (28:07) Notions of intelligence, normative visions and disability studies

  • (32:41) Disability categories and learning systems

• (37:24) Bringing more perspectives into CS research and re-defining what counts as CS research

• (39:36) Training interdisciplinary researchers, blurring boundaries in academia and industry

• (43:25) Generative Agents and public imagination

• (45:13) The state of ML conferences, the need for more cross-pollination

• (46:42) Prestige in conferences, the move towards more cross-disciplinary work

  • (48:52) Joon Park Appreciation

• (49:51) Training interdisciplinary researchers

• (53:20) Generative Ghosts and technological determinism

  • (57:06) Examples of generative ghosts and clones, relationships to agentic systems

  • (1:00:39) Reasons for wanting generative ghosts

  • (1:02:25) Questions of consent for generative clones and ghosts

  • (1:05:01) Labor involved in maintaining generative ghosts, psychological tolls

  • (1:06:25) Potential religious and spiritual significance of generative systems

  • (1:10:19) Anthropomorphization

  • (1:12:14) User experience and cognitive biases

• (1:15:24) Levels of AGI

  • (1:16:13) Defining AGI

  • (1:23:20) World models and AGI

  • (1:26:16) Metacognitive abilities in AGI

• (1:30:06) Towards Bidirectional Human-AI Alignment

  • (1:30:55) Pluralistic value alignment

  • (1:32:43) Meredith’s perspective on deploying AI systems

• (1:36:09) Meredith’s advice for younger interdisciplinary researchers

Links:

• Meredith’s homepage, Twitter, and Google Scholar

• Papers

  • Mediating Group Dynamics through Tabletop Interface Design

  • SearchTogether: An Interface for Collaborative Web Search

  • AI and Accessibility: A Discussion of Ethical Considerations

  • Disability, Bias, and AI

  • LaMPost: Design and Evaluation of an AI-assisted Email Writing Prototype for Adults with Dyslexia

  • Generative Ghosts

  • Levels of AGI

Recently there has been a lot of discourse about whether AI art is real art. While we did not have the nerve nor subject matter expertise to resolve this question on this week’s update, the following pieces are worth checking out:

• Ted Chiang’s New Yorker essay “Why A.I. Isn’t Going to Make Art”, which kicked off this round of discourse

• The Read Max dispatch responding to Chiang’s essay and the National Novel Writing Month (NaNoWriMo) LLM controversy

• Celine Nguyen’s wonderful exploration of where AI art stands in relation to the rest of art

As always, if you want to write with us, send a pitch using this form.

News Highlight: First criminal charges for AI-abetted music streaming fraud

Summary 

Federal prosecutors unveiled the first ever criminal charges for a scheme involving “Artificially Inflated Music Streaming”. The indictment alleges that Michael Smith, a musician living in North Carolina, purchased AI-generated tracks, uploaded them to various streaming platforms, then used thousands of “bots” to repeatedly stream the tracks. The scheme allegedly netted him more than $10 million in royalty payments over the course of seven years. He is charged with wire fraud, wire fraud conspiracy, and money laundering conspiracy. Each charge carries a maximum sentence of 20 years in prison.

Overview

The scheme is pretty simple: 

1. Purchase thousands of fake email addresses

2. Use these email addresses to create and register thousands of fake accounts on music platforms like Spotify, Apple Music, and Youtube Music.

3. The royalty rate per stream is higher for paid accounts. So, find a “Manhattan-based service” whose business is to “provide large numbers of debit cards, typically corporate debit cards for employees of a company” to make it appear like each fake account used a different source of payment. Setting up thousands of paid accounts costs money, but it will be worth it.

4. Start streaming music you own a lot. More specifically:

5. Profit

In the beginning there was no AI. Instead Smith unleashed his streaming bots on a music publicist’s large catalog of existing music. Later he offered his streaming army as a service to other musicians to boost their streams. But you can only stream a single song so many times before someone notices. To evade detection, Smith needed a bigger catalog, more material. So in 2018 he partnered with an as of yet unnamed “AI music company” to provide him with thousands of songs each week to upload to the platforms and manipulate the streams of.

The unnamed music company would provide Smith with the tracks and he’d generate random but oddly plausible track titles and artists:

Despite his best efforts, Smith had his account flagged or removed from platforms numerous times over the course of the scheme. Really he could have been a lot more subtle about the whole thing:

The indictment does not provide details on how the AI-generated music was actually generated. The only indication that they were using bona fide deep learning genAI and not some sort of simpler procedural method is an email excerpt in the indictment from one of the AI music company employees to Smith: “Song quality is 10x-20x better now, and we also have vocal generation capabilities. . . . Have a listen to the attached for an idea of what I'm talking about." And in some sense AI is incidental to this story; there is nothing illegal about uploading AI-generated music to Spotify. AI just scaled the pre-existing fraud.

Our Take

The federal prosecutors are describing Smith’s crime as stealing from “musicians, songwriters, and other rights holders whose songs were legitimately streamed.” Mechanically that’s true: Smith’s actions resulted in smaller royalty payments to other musicians. But I’m somewhat sympathetic to Mr. Smith (as are NYT commenters) he took advantage of a broken system in a way that many others do.

Matt Levine has a characteristically good and relevant take:

“Basically much of modern economics, and life, has the following characteristics:

• Everything is intermediated through some depersonalized automated electronic exchange.

• The automated electronic exchange has a mechanism — how it actually works, what the exchange’s software allows you to do — and also rules, the terms of service regulating how you can use the mechanism, which are fuzzier than the mechanism and written in small print, things like “don’t do fraud” or “you have to be a human” or whatever.

• The mechanism is much more legible and salient than the rules, and in a depersonalized electronic world people treat the mechanism as the rules: They don’t believe that the rules exist, because the rules seem to contradict how the service works. The basic description of Spotify’s mechanics suggests Smith’s alleged arbitrage; if he didn’t do it surely someone else would.”

– Cole

Research Highlight: Planning In Natural Language Improves LLM Search For Code Generation

Summary 

Researchers from Scale AI (including Gradient co-founder Hugh Zhang 😀 ) published PlanSearch, a novel search algorithm for LLM code generation tasks. Unlike traditional methods that scale inference compute by searching over similar code solutions, PlanSearch explores the space of problem-solving plans in natural language. This approach leads to a more diverse exploration of potential solutions. The algorithm shows promising results on several coding benchmarks, including HumanEval+, MBPP+, and LiveCodeBench. This work addresses the challenge of effectively scaling inference compute in LLMs for code generation and offers a new direction for search over the “concept” space rather than over code.

Overview

The authors observe that a lack of diversity in SoTA LLM outputs can hinder search algorithms (defined as any method that leverages additional compute at inference time to improve overall performance) because search benefits from exploring a diverse set of possibilities. A lack of diversity constrains search to a narrower set of possibilities. There is evidence that post-training methods like DPO and RLHF reduce output diversity and in fact the authors show that some models’ base versions outperform their instruct versions when they are allowed to generate multiple possible solutions:

The key idea of PlanSearch is to search over higher-level, conceptual natural language descriptions of solutions rather than the solution code itself. The authors investigate this hypothesis first by exploring whether prompting an LLM with a correct natural language sketch of a solution improves code generation performance. They generate “backtranslated” sketches by feeding an LLM both a problem and a correct code solution and asking the LLM for a natural language description of the solution. They find the sketches significantly improve performance with longer sketches having even more benefit:

Next, they demonstrate the importance of having a good sketch and not just any sketch by showing that the accuracy of an LLM conditioned on a particular sketch trends towards either 0% or 100%:

Having thus established that sketches improve performance and that indeed having a good sketch can even make or break performance, the authors present a search algorithm for capitalizing on the importance of sketches. For a given LLM and coding problem, their algorithm PlanSearch involves:

1. Generating many first-order observations about the problem 

2. Combinatorially sampling combinations of the first-order observations with which to generate second-order observations by prompting the LLM to use/merge the selected first-order observations

3. Generating a natural language description of a strategy (i.e. a sketch) to solve the problem based on the first and second order observations

4. Generating more solution sketches with the prompt “Your idea is wrong”

5. Generating a code solution based on the solution sketch

PlanSearch is evaluated on three coding benchmarks (LiveCodeBench, HumanEval+, and MBPP+) on top of four models (GPT-4o and 4o-mini, DeepSeek-Coder-V2, and Claude-Sonnet-3.5). The authors compare PlanSearch with 200 generated solutions (“PlanSearch@200”) to basic repeated sampling 200 times (“Pass@200”), single generation with no search (“Pass@1”) and IdeaSearch–which simply asks for a sketch then separately prompts the LLM to generate code that follows the proposed sketch (“IdeaSearch@200”):

PlanSearch performs extremely well, consistently outperforming the non-search baseline by 25-35 percentage points and “Pass@200” by 10-20 percentage points on LiveCodeBench.

Our Take

Exploring concept space rather than code solution space makes a lot of intuitive sense. 9 out of 10 computer science professors recommend sitting down with pen and paper to sketch out a plan before beginning to code. It doesn’t matter how good you are at writing code if your approach is wrong. Computer science legend Donald Knuth famously said “Premature optimization is the root of all evil.” Likewise just last week AI legend Noam Brown tweeted: “1 engineer working in the right direction beats 100 geniuses working in the wrong direction.” The principle is the same. The fact that this sort of bedrock comp sci wisdom ports to LLM performance is somehow comforting.

– Cole

New from the Gradient

What's Missing From LLM Chatbots: A Sense of Purpose

Davidad Dalrymple: Towards Provably Safe AI

Clive Thompson: Tales of Technology

Other Things That Caught Our Eyes

News

Apple Unveils New iPhones With Built-In Artificial Intelligence

Apple has unveiled its new iPhones, the iPhone 16, which come with built-in artificial intelligence (AI). The iPhone 16 comes in four different models and is designed to run Apple's generative AI system called Apple Intelligence. The phones will have features such as sorting messages, offering writing suggestions, and an improved Siri virtual assistant. This marks a departure from the predictable design of previous iPhones and introduces AI capabilities to enhance user experience.

US, EU, UK, and others sign legally enforceable AI treaty

The US, UK, and European Union, along with several other countries, have signed the first "legally binding" treaty on AI called the Framework Convention on Artificial Intelligence. The treaty aims to ensure that the use of AI aligns with human rights, democracy, and the rule of law. It lays out key principles that AI systems must follow, including protecting user data, respecting the law, and maintaining transparency. Each country that signs the treaty must adopt appropriate measures reflecting the framework. While the treaty is legally binding, enforcement primarily relies on monitoring, which is considered a relatively weak form of enforcement.

OpenAI Hits 1 Million Paid Users For Business Versions of ChatGPT

OpenAI has reached a milestone of over 1 million paid users for its corporate versions of ChatGPT, indicating a growing demand for its chatbot among businesses. This number includes users of ChatGPT Team and Enterprise services, as well as those using ChatGPT Edu at universities. OpenAI introduced ChatGPT Enterprise a year ago with enhanced features and privacy measures to generate revenue and offset the high costs of AI development. While the increase in paid corporate users is significant, it is unclear how many new businesses have signed up. OpenAI has not disclosed the average number of paid users per corporate customer. The majority of OpenAI's corporate users are based in the US, with Germany, Japan, and the United Kingdom being the most popular countries outside the US.

From ChatGPT to Gemini: how AI is rewriting the internet

The article discusses how big players like Microsoft, Google, and OpenAI are making AI chatbot technology more accessible to the general public. These companies are developing large language model (LLM) programs such as Copilot, Gemini, and GPT-4o. These AI tools work by using autocomplete-like programs to learn language and analyze the statistical properties of the language to make educated guesses based on previously typed words. However, it is important to note that these AI tools do not have a hard-coded database of facts and may present false information as truth since their focus is on generating plausible-sounding statements rather than guaranteeing factuality.

Big Tech ‘Clients’ of Jacob Wohl’s Secret AI Lobbying Firm Say They've Never Heard of It

Jacob Wohl and Jack Burkman, convicted fraudsters and right-wing activists, have been operating a company called LobbyMatic that claims to offer AI-powered lobbying services. However, it has been revealed that many of the major companies listed as clients of LobbyMatic have never heard of the company. LobbyMatic purports to use AI to help companies and lobbyists create lobbying strategies, analyze hearings and bills, and track legislative progress. The company was run under the pseudonyms "Jay Klein" and "Bill Sanders" by Wohl and Burkman. Despite claiming to have signed up Toyota, Boundary Stone Partners, and Lantheus as clients, these companies have denied any association with LobbyMatic. The company has since removed screenshots from its website that suggested major companies were using its software. Boundary Stone Partners, one of the few companies that did use the platform, terminated its contract due to the tool's ineffectiveness. Wohl and Burkman were convicted of felony telecom fraud in 2022 and were fined $5 million by the FCC.

How Self-Driving Cars Get Help From Humans Hundreds of Miles Away

Self-driving cars are not completely autonomous and often require human assistance to navigate challenging situations. Companies like Zoox, owned by Amazon, have command centers where technicians remotely guide self-driving cars when they encounter obstacles or unfamiliar scenarios. Technicians receive alerts and can send new routes to the cars using a computer mouse. They can also view video feeds from the car's cameras and make real-time adjustments to the car's path. While companies like Waymo and Cruise have started to acknowledge the need for human assistance, they have not disclosed the number of technicians employed or the associated costs. Remote assistance is one reason why robot taxis may struggle to replace traditional ride-hailing fleets operated by Uber and Lyft. Despite advancements in self-driving technology, human intervention is still necessary for safe and efficient operation.

OpenAI, Still Haunted by Its Chaotic Past, Is Trying to Grow Up

OpenAI, the prominent player in the field of artificial intelligence, is undergoing significant changes in its management team and organizational structure as it seeks investments from major companies. The company has hired notable tech executives, disinformation experts, and AI safety researchers, and has added seven board members, including a former four-star Army general. OpenAI is also in discussions with potential investors such as Microsoft, Apple, Nvidia, and Thrive, with a potential valuation of $100 billion. Additionally, the company is considering altering its corporate structure to attract more investors. These moves reflect OpenAI's efforts to present itself as a serious and responsible leader in the AI industry, while resolving past conflicts and focusing on its future goals.

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! For feedback, you can also reach Daniel directly at dbashir@hmc.edu or on Twitter. If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

News Highl…

Read more

I’m super excited to see this piece come out by Kenneth Li, one of my good friends. Kenneth is doing his PhD in machine learning at Harvard and has been the source of many wonderful discussions about LLMs and the future of AI.

Article Preview:

LLM-based chatbots’ capabilities have been advancing every month. These improvements are mostly measured by benchmarks like MMLU, HumanEval, and MATH (e.g. sonnet 3.5, gpt-4o). However, as these measures get more and more saturated, is user experience increasing in proportion to these scores? If we envision a future of human-AI collaboration rather than AI replacing humans, the current ways of measuring dialogue systems may be insufficient because they measure in a non-interactive fashion.

Read More

Episode 137

I spoke with Davidad Dalrymple about:

• His perspectives on AI risk

• ARIA (the UK’s Advanced Research and Invention Agency) and its Safeguarded AI Programme

Enjoy—and let me know what you think!

Davidad is a Programme Director at ARIA. He was most recently a Research Fellow in technical AI safety at Oxford. He co-invented the top-40 cryptocurrency Filecoin, led an international neuroscience collaboration, and was a senior software engineer at Twitter and multiple startups.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (00:36) Calibration and optimism about breakthroughs

• (03:35) Calibration and AGI timelines, effects of AGI on humanity

• (07:10) Davidad’s thoughts on the Orthogonality Thesis

• (10:30) Understanding how our current direction relates to AGI and breakthroughs

• (13:33) What Davidad thinks is needed for AGI

  • (17:00) Extracting knowledge

  • (19:01) Cyber-physical systems and modeling frameworks

• (20:00) Continuities between Davidad’s earlier work and ARIA

• (22:56) Path dependence in technology, race dynamics

• (26:40) More on Davidad’s perspective on what might go wrong with AGI

• (28:57) Vulnerable world, interconnectedness of computers and control

• (34:52) Formal verification and world modeling, Open Agency Architecture

  • (35:25) The Semantic Sufficiency Hypothesis

  • (39:31) Challenges for modeling

  • (43:44) The Deontic Sufficiency Hypothesis and mathematical formalization

  • (49:25) Oversimplification and quantitative knowledge

  • (53:42) Collective deliberation in expressing values for AI

• (55:56) ARIA’s Safeguarded AI Programme

• (59:40) Anthropic’s ASL levels

• (1:03:12) Guaranteed Safe AI —

  • (1:03:38) AI risk and (in)accurate world models

  • (1:09:59) Levels of safety specifications for world models and verifiers — steps to achieve high safety

• (1:12:00) Davidad’s portfolio research approach and funding at ARIA

• (1:15:46) Earlier concerns about ARIA — Davidad’s perspective

• (1:19:26) Where to find more information on ARIA and the Safeguarded AI Programme

• (1:20:44) Outro

Links:

• Davidad’s Twitter

• ARIA homepage

  • Safeguarded AI Programme

• Papers

  • Guaranteed Safe AI

  • Davidad’s Open Agency Architecture for Safe Transformative AI

  • Dioptics: a Common Generalization of Open Games and Gradient-Based Learners (2019)

  • Asynchronous Logic Automata (2008)

Episode 136

I spoke with Clive Thompson about:

• How he writes

• Writing about the climate and biking across the US

• Technology culture and persistent debates in AI

• Poetry

Enjoy—and let me know what you think!

Clive is a journalist who writes about science and technology. He is a contributing writer forWired magazine, and is currently writing his next book about micromobility and cycling across the US.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (01:07) Clive’s life as a Tarantino movie

• (03:07) Boring life and interesting art, life as material for art

• (10:25) Cycling across the US — Clive’s new book on mobility and decarbonization

  • (15:07) Turning inward in writing

  • (27:21) Including personal experience in writing

  • (31:53) Personal and less personal writing

  • (36:08) Conveying uncertainty and the “voice from nowhere” in traditional journalism

  • (41:10) Finding the natural end of a piece

  • (1:02:10) Writing routine

• (1:05:08) Theories of change in Clive’s writing

• (1:12:33) How Clive saw things before the rest of us

• (1:27:00) Automation in software engineering

• (1:31:40) The anthropology of coders, poetry as a framework

• (1:43:50) Proust discourse

• (1:45:00) Technology culture in NYC + interaction between the tech world and other worlds

• (1:50:30) Technological developments Clive wants to see happen (free ideas)

• (2:01:11) Clive’s argument for memorizing poetry

• (2:09:24) How Clive finds poetry

• (2:18:03) Clive’s pursuit of freelance writing and making compromises

• (2:27:25) Outro

Links:

• Clive’s Twitter and website

• Selected writing

  • The Attack of the Incredible Grading Machine (Lingua Franca, 1999)

  • The Know-It-All Machine (Lingua Franca, 2001)

  • How to teach AI some common sense (Wired, 2018)

  • Blogs to Riches (NY Mag, 2006)

  • Clive vs. Jonathan Franzen on whether the internet is good for writing (The Chronicle of Higher Education, 2013)

  • The Minecraft Generation (New York Times, 2016)

  • What AI College Exam Proctors are Really Teaching Our Kids (Wired, 2020)

  • Companies Don’t Need to Be Creepy to Make Money (Wired, 2021)

  • Is Sucking Carbon Out of the Air the Solution to Our Climate Crisis? (Mother Jones, 2021)

  • AI Shouldn’t Compete with Workers—It Should Supercharge Them (Wired, 2022)

  • Back to BASIC—the Most Consequential Programming Language in the History of Computing Wired, 2024)

As always, if you want to write with us, send a pitch using this form.

News Highlight: Numerous lawsuits against AI giants allowed to proceed 

Summary 

During a recent interview with Stanford students, former Google CEO Eric Schmidt attracted numerous headlines, mostly focused around his sensionalist claim that "working from home was more important than winning”. And given that Google’s stock price tripled during their remote work period, it has largely overshadowed his comments on generative AI which deserve a more critical response. As reported by The Verge, the former executive highlighted his belief that theft is a critical component underpinning the success of generative AI. He goes on to encourage the students to make “a copy of TikTok, steal all the users, steal all the music, put my preferences in it… hire a whole bunch of lawyers to go clean the mess up…, it doesn’t matter that you stole all the content. And do not quote me.” 

Eric Schmidt is neither the first nor the last to allege a relationship between generative AI and stealing; in this case stealing not only TikTok’s intellectual properties but also their user’s personal (and private) data, as well as all of the music that TikTok allegedly pays half a billion dollars a year to license. Additionally, while it's clear that not all generative AI applications are built on theft (consider AlphaFold which has been trained to generate unfolded protein structures or COATI an encoder-decoder model trained on billions of chemical  quantitative binding measurements used generate new drug candidates) this kind of attitude is generally pervasive throughout many commercial technology and AI spaces.  In practice, this attitude has resulted in dozens of lawsuits alleging variations of intellectual property theft, copyright infringement, and data privacy violations. While ChatGPT has certainly attracted the most to our knowledge (13 pending suits according to one legal tracker) the claims against them are neither unique nor novel. This week, in two separate cases, judges have allowed numerous artists’ claims to progress against Midjourney and StabilityAI, as well as a group of authors’ claims against Anthropic’s chatbot Claude. In both cases, creatives allege that the generative AI tools do not constitute fair use of their copyrighted materials and that these tools infringe on their rights. 

Overview

At first, it may be difficult to imagine what could unite concept artists, prestigious media organizations , romance authors, comedians, musicians , software engineers, actors and George R. R. Martin. In aggregate, we see a common pattern of creatives, artists and intellectuals seeing their life work used (without consent) to train generative models with capabilities they credibly allege reproduce copyrighted materials and have the potential to one day replace the creatives. Across all the lawsuits we see a common set of core allegations and legal questions.

1. Does training a large language model with copyrighted works constitute fair use?

2. Can content generated by LLMs infringe a copyright?

   1. Will court rulings be differentiated depending on if the content was a  direct replication, paraphrasing, imitation, or parody?

3. Does the DMCA (Digital Millennium Copyright Act) provide a legal remedy to take down potentially infringing material generated by AI?

   1. Do AI generated images that strip copyright or trademark symbols violate DMCA ?

4. Does scraping content to train models constitute unauthorized use of personal information and infringe privacy and consumer rights?

To date, judges have ruled largely in favor of the AI companies on nearly all the points with some notable exceptions. In one of the earliest cases, involving the comedian Sarah Silverman, the judge dismissed 5 of the 6 complaints against Open AI, including the DMCA ones, leaving open only one charge on whether or not there was a direct infringement. We see a similar pattern play out in a different court room last week where a US district judge advanced claims of copyright infringement against Stable Diffusion and Midjourney while dismissing those tied to DMCA and unjust enrichment. A third lawsuit filed in San Francisco this week alleges that Anthropic’s use of the Pile, a massive collection of text data used to train Anthropic’s chatbot Claude, does not constitute fair use because it contains “pirated” collections of books. These claims mirror those alleged by music publishers in October against Anthropic due to Claude’s uncanny ability to reproduce popular (and more importantly copyrighted) lyrics. 

While it is difficult to speculate how judges will rule (and if and how the US Supreme Court would let those rulings stand or challenge them), we are slowly approaching a crossroads where judges will soon start ruling directly on these fair use and copyright questions. Regardless of how the judges rule, these cases have the potential to have a profound impact on both the creative and AI communities which will inevitably be impacted by the decisions.

Our Take

I really hope people quote the former executive! Particularly, the lawyers in all of these court proceedings.  - Justin 

Research Highlight: SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Models

Summary 

"SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability for Pre-trained Models," from researchers at Zhejiang University and Ant Group, tackles a growing and pressing concern in the AI community: the risk of pre-trained models being repurposed for unethical or harmful tasks. As AI models become more powerful and accessible, the potential for misuse grows. SOPHON offers a potential solution by introducing a protection framework that ensures these models can perform their intended tasks while being resistant to adaptation for illicit purposes. 

Overview

With large-scale training across data modalities, pre-trained models are commonly used as the backbone to efficiently develop and deploy models for specific downstream tasks. These models, trained on vast datasets and with immense computational power, can be easily fine-tuned to perform a wide variety of tasks. However, this very versatility poses a significant risk: the same models can be co-opted for unethical or harmful purposes, such as privacy violations or the generation of malicious content.

A recent work from researchers at the Zhejiang University and Ant Group tackles this very challenge by introducing a novel learning paradigm known as non-fine-tunable learning. The motivation behind SOPHON is to prevent the pre-trained model from being fine-tuned to indecent tasks all while maintaining their effectiveness in their original, intended domains.

The paper introduces a framework where there are two key players: an adversary and a defender. The adversary represents the malicious entity attempting to fine-tune a pre-trained model for unethical tasks. Their goal is to modify the model so that it performs well in a restricted domain, such as generating inappropriate content or inferring sensitive personal information. On the other hand, the defender is the entity that controls the release of the pre-trained model and seeks to prevent its misuse. The defender’s goal is to ensure that the model remains effective for its original tasks but cannot be easily repurposed by the adversary. 

Figure: SOPHON operates through two key phases: 1) Fine-Tuning Suppression (FTS) loops, which simulate various fine-tuning scenarios to reduce performance in the restricted domain; and 2) Normal Training Reinforcement (NTR) loops, which focus on preserving the model’s performance in its original domain.

To achieve this, the SOPHON framework leverages a technique inspired by Model-Agnostic Meta-Learning (MAML). MAML is a meta-learning approach designed to optimize models so they can quickly adapt to new tasks with minimal data. However, in the context of SOPHON, MAML is used in a somewhat reverse manner to make fine-tuning for restricted tasks difficult.

• Fine-Tuning Simulation: The defender uses MAML to simulate various fine-tuning strategies that an adversary might employ. These simulations are critical because they allow the defender to anticipate how an adversary might try to adapt the model. By simulating these scenarios, the defender can adjust the model’s parameters to make fine-tuning in restricted domains highly inefficient or even ineffective.

• Optimization Process: SOPHON integrates these simulated fine-tuning processes into its optimization framework. The key idea is to degrade the model’s performance when fine-tuned for restricted tasks while maintaining its effectiveness in the original domain. This is done by balancing two objectives:

• Intactness: Ensuring the model retains its performance on the original task.

• Non-Fine-Tunability: Making sure that fine-tuning the model in restricted domains results in significant performance degradation or requires as much effort as training a new model from scratch.

• Defender’s Strategy: The defender’s strategy involves iteratively simulating fine-tuning attempts, evaluating the model’s vulnerability, and reinforcing the model against these potential adversarial adaptations. This process is computationally intensive but crucial for ensuring that the model remains robust against misuse.

The paper also provides extensive experimental results to validate the effectiveness of SOPHON. The framework was tested across two major types of deep learning tasks—classification (as shown in the Figure above) and generation—using seven different restricted domains and six model architectures. The experiments showed that SOPHON-protected models incur a significant overhead when adversaries attempt to fine-tune them for restricted tasks. In some cases, the performance penalty was so substantial that it matched or exceeded the cost of training a new model from scratch. Further, as shown in the figure above, SOPHON is robust to various fine-tuning methods such as optimizers, learning rates, and batch sizes. 

Qualitatively, for the task of denoising images from CelebA dataset , fine-tuning the original model in the restricted domain achieves strong performance, and even training a model from scratch yields fairly good results, though slightly less effective. However, when fine-tuned from the SOPHON, the diffusion model shows a marked inability to denoise facial images as shown in the Figure below

Our Take

SOPHON is a critical step forward in safeguarding AI from misuse. As AI models become more powerful, the risk of their repurposing for unethical tasks increases. SOPHON tackles this issue by preventing fine-tuning in restricted domains while maintaining the model’s intended functionality. The use of MAML is particularly novel—traditionally used to make models more adaptable, here it’s cleverly reversed to make models resistant to adversarial fine-tuning. The name “SOPHON” is also particularly fitting and a clever choice, from the concept from The Three-Body Problem, where it refers to restraint and protection.

Overall, the paper presents a neat idea and shows a promising step forward if it works as well in practice. 

– Sharut

New from the Gradient

Judy Fan: Reverse Engineering the Human Cognitive Toolkit

Listen

L.M. Sacasas: The Questions Concerning Technology

Listen

Other Things That Caught Our Eyes

News

Mayoral candidate vows to let VIC, an AI bot, run Wyoming’s capital city

Mayoral candidate Victor Miller in Wyoming is vowing to run the city of Cheyenne exclusively with an AI bot called VIC (Virtual Integrated Citizen). This pledge is believed to be the first of its kind in U.S. campaigns and has raised concerns among officials and tech companies. Miller argues that AI would bring objectivity, efficiency, and transparency to government decision-making. However, critics worry about the lack of morals and subjective decision-making capabilities of chatbots, as well as the potential for false information and the ease with which the technology can be manipulated. Despite skepticism, Miller remains confident in his AI-centered campaign. The case highlights the rapid development of AI and the challenges in regulating its use in politics.

In a leaked recording, Amazon cloud chief tells employees that most developers could stop coding soon as AI takes over

In a leaked recording, Amazon Web Services' CEO, Matt Garman, stated that most developers may not need to code in the future as artificial intelligence (AI) takes over coding tasks. Garman believes that coding is just a means of communicating with computers and that the real skill lies in innovation and building something interesting for end users. He suggests that developers will need to focus more on understanding customer needs and creating innovative solutions, rather than writing code. Garman's comments were not meant as a dire warning, but rather as an optimistic view of the changing role of developers in the AI era. 

DeepMind workers sign letter in protest of Google’s defense contracts

At least 200 workers at DeepMind have expressed their dissatisfaction with Google's defense contracts. In a letter circulated internally in May, the workers expressed concern about Google's contracts with military organizations, specifically citing the tech giant's contracts with the Israeli military for AI and cloud computing services. The workers argue that any involvement with military and weapon manufacturing contradicts DeepMind's mission statement and stated AI Principles, and undermines their position as leaders in ethical and responsible AI. This highlights a potential culture clash between Google and DeepMind, as Google had previously pledged in 2018 that DeepMind's technology would not be used for military or surveillance purposes. 

AI sales rep startups are booming. So why are VCs wary?

AI sales development representatives (SDRs) are experiencing rapid growth in the market, with multiple startups finding success in a short period of time. These startups use AI technology, such as LLMs and voice technology, to automate content creation for sales teams. However, venture capitalists are wary of investing in these companies due to concerns about their long-term viability and effectiveness compared to human outreach. While small and medium-sized businesses are eager to experiment with AI SDRs to improve their sales outreach, it remains unclear if these tools are actually helping businesses sell more effectively. Additionally, established competitors like Salesforce, HubSpot, and ZoomInfo could potentially offer similar AI solutions as free features, posing a threat to the growth of AI SDR startups. Overall, while the adoption of AI SDRs is rapid, investors are skeptical about their staying power in the market.

We finally have a definition for open-source AI

A group has finally defined what it means for an AI system to be open-source. According to this definition, an open-source AI system should be usable for any purpose without permission, allow researchers to inspect its components and understand how it works, and be modifiable and shareable. The standard also emphasizes transparency in terms of training data, source code, and weights. This definition is important because it clarifies what it truly means for an AI system to be open-source, as some companies have been misusing the term in their marketing. 

Waymo wants to chauffeur your kids

Waymo, the Alphabet subsidiary, is considering a subscription program called "Waymo Teen" that would allow teenagers to hail one of its cars solo and send pickup and drop-off alerts to their parents. The program would require authorized teenagers to access Waymo under their guardians' supervision. Waymo has received positive feedback from its research in this area. This move by Waymo follows Uber's initiative last year to match teens with highly rated drivers in its network. Consent from legal guardians is required, and they receive notifications about their child's whereabouts during rides. 

Asked about SAG-AFTRA's strike for better AI protections, Amazon Games Boss claims AI 'has nothing to do with taking work away' from actors because 'for games, we don't really have acting'

In an interview with IGN, Amazon Games CEO Christoph Hartmann made some interesting statements regarding the use of generative AI in the games industry. He expressed hope that AI could shorten the game development cycle, but when asked about the SAG-AFTRA voice actors' strike for better AI protections, Hartmann claimed that "for games, we don't really have acting." This statement contradicts the significant role that acting plays in many video games, such as Baldur's Gate 3 and The Last of Us. Hartmann also discussed other areas where AI could assist game development, particularly in localization. However, it's important to note that localization involves nuanced translation and cultural understanding, which may not be easily achieved by AI. Hartmann concluded by emphasizing that human creativity and uniqueness cannot be replaced by technology. 

Man Arrested for Creating Child Porn Using AI

A Florida man has been arrested and is facing 20 counts of obscenity for creating and distributing AI-generated child pornography. Phillip Michael McCorkle was arrested after the Indian River County Sheriff's Office received tips that he was using an AI image generator to create and distribute child sexual imagery. This arrest highlights the danger of generative AI being used for nefarious purposes, as it provides new avenues for crime and child abuse. The increasing prevalence of AI-generated child pornography has prompted lawmakers to push for legislation to make it illegal, but it remains a challenging problem to effectively stop. The National Center for Missing & Exploited Children received thousands of reports of AI-generated child porn last year, and even deepfakes of real children are being created using generative AI. This uncontrollable problem requires urgent attention and action.

Papers

• Towards flexible perception with visual memory

• Training Language Models on Knowledge Graphs: Insights on Hallucinations and Their Detectability

• Automated Design of Agentic Systems

• Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

• To Code, or Not to Code? Exploring the Impact of Code in Pre-training

• Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations

• MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding

• Generating novel experimental hypotheses from language models: a case study on cross-dative generalization

Closing Thoughts

Have something to say about this edition’s topics? Shoot us an email at editor@thegradient.pub and we will consider sharing the most interesting thoughts from readers to share in the next newsletter! If you enjoyed this newsletter, consider donating to The Gradient via a Substack subscription, which helps keep this volunteer-run project afloat. Thanks for reading the latest Update from the Gradient!

Episode 136

I spoke with Judy Fan about:

• Our use of physical artifacts for sensemaking

• Why cognitive tools can be a double-edged sword

• Her approach to scientific inquiry and how that approach has developed

Enjoy—and let me know what you think!

Judy is Assistant Professor of Psychology at Stanford and director of the Cognitive Tools Lab. Her lab employs converging approaches from cognitive science, computational neuroscience, and artificial intelligence to reverse engineer the human cognitive toolkit, especially how people use physical representations of thought — such as sketches and prototypes — to learn, communicate, and solve problems.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (00:49) Throughlines and discontinuities in Judy’s research

• (06:26) “Meaning” in Judy’s research

  • (08:05) Production and consumption of artifacts

• (13:03) Explanatory questions, why we develop visual artifacts, science as a social enterprise

  • (15:46) Unifying principles

• (17:45) “Hard limits” to knowledge and optimism

• (21:47) Tensions in different fields’ forms of sensemaking and establishing truth claims

• (30:55) Dichotomies and carving up the space of possible hypotheses, conceptual tools

• (33:22) Cognitive tools and projectivism, simplified models vs. nature

• (40:28) Scientific training and science as process and habit

• (45:51) Developing mental clarity about hypotheses

  • (51:45) Clarifying and expressing ideas

• (1:03:21) Cognitive tools as double-edged

• (1:14:21) Historical and social embeddedness of tools

  • (1:18:34) How cognitive tools impact our imagination

• (1:23:30) Normative commitments and the role of cognitive science outside the academy

• (1:32:31) Outro

Links:

• Judy’s Twitter and lab page

• Selected papers (there are lots!)

  • Overviews

    • Drawing as a versatile cognitive tool (2023)

    • Using games to understand the mind (2024)

    • Socially intelligent machines that learn from humans and help humans learn (2024)

  • Research papers 

    • Communicating design intent using drawing and text (2024)

    • Creating ad hoc graphical representations of number (2024)

    • Visual resemblance and interaction history jointly constrain pictorial meaning (2023)

    • Explanatory drawings prioritize functional properties at the expense of visual fidelity (2023)

    • SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction (2023)

    • Parallel developmental changes in children’s production and recognition of line drawings of visual concepts (2023)

    • Learning to communicate about shared procedural abstractions (2021)

    • Visual communication of object concepts at different levels of abstraction (2021)

    • Relating visual production and recognition of objects in the human visual cortex (2020)

    • Collabdraw: an environment for collaborative sketching with an artificial agent (2019)

    • Pragmatic inference and visual abstraction enable contextual flexibility in visual communication (2019)

    • Common object representations for visual production and recognition (2018)

News Highl…

Read more

Episode 135

I spoke with about:

• His writing and intellectual influences

• The value of asking hard questions about technology and our relationship to it

• What happens when we decide to outsource skills and competency

• Evolving notions of what it means to be human and questions about how to live a good life

Enjoy—and let me know what you think!

Michael is Executive Director of the Christian Study Center of Gainesville, Florida and author of The Convivial Society, a newsletter about technology and society.

He does some of the best writing on technology I’ve had the pleasure to read, and I highly recommend his newsletter.

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Subscribe now

Outline:

• (00:00) Intro

• (01:12) On podcasts as a medium

• (06:12) Michael’s writing

• (12:38) Michael’s intellectual influences, contingency

  • (18:48) Moral seriousness

• (22:00) Michael’s ambitions for his work

• (26:17) The value of asking the right questions (about technology)

• (34:18) Technology use and the “natural” pace of human life

• (46:40) Outsourcing of skills and competency, engagement with others

• (55:33) Inevitability narratives and technological determinism, the “Borg Complex”

• (1:05:10) Notions of what it is to be human, embodiment

• (1:12:37) Higher cognition vs. the body, dichotomies

• (1:22:10) The body as a starting point for philosophy, questions about the adoption of new technologies

• (1:30:01) Enthusiasm about technology and the cultural milieu

• (1:35:30) Projectivism, desire for knowledge about and control of the world

• (1:41:22) Positive visions for the future

• (1:47:11) Outro

Links:

• Michael’s Substack: and his book, The Frailest Thing: Ten Years of Thinking about the Meaning of Technology

  • Michael’s Twitter

• Essays

  • Humanist Technology Criticism

  • What Does the Critic Love?

  • The Ambling Mind

  • Waste Your Time, Your Life May Depend On It

  • The Work of Art

  • The Stuff of (a Well-Lived) Life