Even networks long considered “untrainable” can learn effectively with a bit of a helping hand. Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have shown that a brief period of alignment between neural networks, a method they call guidance, can dramatically improve the performance of architectures previously thought unsuitable for modern tasks.

Their findings suggest that many so-called “ineffective” networks may simply start from less-than-ideal starting points, and that short-term guidance can place them in a spot that makes learning easier for the network. 

The team’s guidance method works by encouraging a target network to match the internal representations of a guide network during training. Unlike traditional methods like knowledge distillation, which focus on mimicking a teacher’s outputs, guidance transfers structural knowledge directly from one network to another. This means the target learns how the guide organizes information within each layer, rather than simply copying its behavior. Remarkably, even untrained networks contain architectural biases that can be transferred, while trained guides additionally convey learned patterns. 

“We found these results pretty surprising,” says Vighnesh Subramaniam ’23, MEng ’24, MIT Department of Electrical Engineering and Computer Science (EECS) PhD student and CSAIL researcher, who is a lead author on a paper presenting these findings. “It’s impressive that we could use representational similarity to make these traditionally ‘crappy’ networks actually work.”

Guide-ian angel 

A central question was whether guidance must continue throughout training, or if its primary effect is to provide a better initialization. To explore this, the researchers performed an experiment with deep fully connected networks (FCNs). Before training on the real problem, the network spent a few steps practicing with another network using random noise, like stretching before exercise. The results were striking: Networks that typically overfit immediately remained stable, achieved lower training loss, and avoided the classic performance degradation seen in something called standard FCNs. This alignment acted like a helpful warmup for the network, showing that even a short practice session can have lasting benefits without needing constant guidance.

The study also compared guidance to knowledge distillation, a popular approach in which a student network attempts to mimic a teacher’s outputs. When the teacher network was untrained, distillation failed completely, since the outputs contained no meaningful signal. Guidance, by contrast, still produced strong improvements because it leverages internal representations rather than final predictions. This result underscores a key insight: Untrained networks already encode valuable architectural biases that can steer other networks toward effective learning.

Beyond the experimental results, the findings have broad implications for understanding neural network architecture. The researchers suggest that success — or failure — often depends less on task-specific data, and more on the network’s position in parameter space. By aligning with a guide network, it’s possible to separate the contributions of architectural biases from those of learned knowledge. This allows scientists to identify which features of a network’s design support effective learning, and which challenges stem simply from poor initialization.

Guidance also opens new avenues for studying relationships between architectures. By measuring how easily one network can guide another, researchers can probe distances between functional designs and reexamine theories of neural network optimization. Since the method relies on representational similarity, it may reveal previously hidden structures in network design, helping to identify which components contribute most to learning and which do not.

Salvaging the hopeless

Ultimately, the work shows that so-called “untrainable” networks are not inherently doomed. With guidance, failure modes can be eliminated, overfitting avoided, and previously ineffective architectures brought into line with modern performance standards. The CSAIL team plans to explore which architectural elements are most responsible for these improvements and how these insights can influence future network design. By revealing the hidden potential of even the most stubborn networks, guidance provides a powerful new tool for understanding — and hopefully shaping — the foundations of machine learning.

“It’s generally assumed that different neural network architectures have particular strengths and weaknesses,” says Leyla Isik, Johns Hopkins University assistant professor of cognitive science, who wasn’t involved in the research. “This exciting research shows that one type of network can inherit the advantages of another architecture, without losing its original capabilities. Remarkably, the authors show this can be done using small, untrained ‘guide’ networks. This paper introduces a novel and concrete way to add different inductive biases into neural networks, which is critical for developing more efficient and human-aligned AI.”

Subramaniam wrote the paper with CSAIL colleagues: Research Scientist Brian Cheung; PhD student David Mayo ’18, MEng ’19; Research Associate Colin Conwell; principal investigators Boris Katz, a CSAIL principal research scientist, and Tomaso Poggio, an MIT professor in brain and cognitive sciences; and former CSAIL research scientist Andrei Barbu. Their work was supported, in part, by the Center for Brains, Minds, and Machines, the National Science Foundation, the MIT CSAIL Machine Learning Applications Initiative, the MIT-IBM Watson AI Lab, the U.S. Defense Advanced Research Projects Agency (DARPA), the U.S. Department of the Air Force Artificial Intelligence Accelerator, and the U.S. Air Force Office of Scientific Research.

Their work was recently presented at the Conference and Workshop on Neural Information Processing Systems (NeurIPS).

Fair use is not just an excuse to copy—it’s a pillar of online speech protection, and disregarding it in order to lash out at a critic should have serious consequences. That’s what we told a federal court in Channel 781 News v. Waltham Community Access Corporation, our case fighting copyright abuse on behalf of citizen journalists.

Waltham Community Access Corporation (WCAC), a public access cable station in Waltham, Massachusetts, records city council meetings on video. Channel 781 News (Channel 781), a group of volunteers who report on the city council, curates clips from those recordings for its YouTube channel, along with original programming, to spark debate on issues like housing and transportation. WCAC sent a series of takedown notices under the Digital Millennium Copyright Act (DMCA), accusing Channel 781 of copyright infringement. That led to YouTube deactivating Channel 781’s channel just days before a critical municipal election. Represented by EFF and the law firm Brown Rudnick LLP, Channel 781 sued WCAC for misrepresentations in its takedown notices under an important but underutilized provision of the DMCA.

The DMCA gives copyright holders a powerful tool to take down other people’s content from platforms like YouTube. The “notice and takedown” process requires only an email, or filling out a web form, in order to accuse another user of copyright infringement and have their content taken down. And multiple notices typically lead to the target’s account being suspended, because doing so helps the platform avoid liability. There’s no court or referee involved, so anyone can bring an accusation and get a nearly instantaneous takedown.

Of course, that power invites abuse. Because filing a DMCA infringement notice is so easy, there’s a temptation to use it at the drop of a hat to take down speech that someone doesn’t like. To prevent that, before sending a takedown notice, a copyright holder has to consider whether the use they’re complaining about is a fair use. Specifically, the copyright holder needs to form a “good faith belief” that the use is not “authorized by the law,” such as through fair use.

WCAC didn’t do that. They didn’t like Channel 781 posting short clips from city council meetings recorded by WCAC as a way of educating Waltham voters about their elected officials. So WCAC fired off DMCA takedown notices at many of Channel 781’s clips that were posted on YouTube.

WCAC claims they considered fair use, because a staff member watched a video about it and discussed it internally. But WCAC ignored three of the four fair use factors. WCAC ignored that their videos had no creativity, being nothing more than records of public meetings. They ignored that the clips were short, generally including one or two officials’ comments on a single issue. They ignored that the clips caused WCAC no monetary or other harm, beyond wounded pride. And they ignored facts they already knew, and that are central to the remaining fair use factor: by excerpting and posting the clips with new titles, Channel 781 was putting its own “spin” on the material - in other words, transforming it. All of these facts support fair use.

Instead, WCAC focused only on the fact that the clips they targeted were not altered further or put into a larger program. Looking at just that one aspect of fair use isn’t enough, and changing the fair use inquiry to reach the result they wanted is hardly the way to reach a “good faith belief.”

That’s why we’re asking the court to rule that WCAC’s conduct violated the law and that they should pay damages. Copyright holders need to use the powerful DMCA takedown process with care, and when they don’t, there needs to be consequences.

I’m sure there’s a story here:

Sources say the man had tailgated his way through to security screening and passed security, meaning he was not detected carrying any banned items.

The man deceived the BA check-in agent by posing as a family member who had their passports and boarding passes inspected in the usual way.

What a year it’s been. We’ve seen technology unfortunately misused to supercharge the threats facing democracy: dystopian surveillance, attacks on encryption, and government censorship. These aren’t abstract dangers. They’re happening now, to real people, in real time.

EFF’s lawyers, technologists, and activists are pushing back. But we need you in this fight.

JOIN EFF TODAY!

MAKE A YEAR END DONATION—HELP EFF UNLOCK CHALLENGE GRANTS!

If you donate to EFF before the end of 2025, you’ll help fuel the legal battles that defend encryption, the tools that protect privacy, and the advocacy that stops dangerous laws—and you’ll help unlock up to $26,200 in challenge grants. 

📣 Stand Together: That's How We Win 📣

The past year confirmed how urgently we need technologies that protect us, not surveil us. EFF has been in the fight every step of the way, thanks to support from people like you.

Get free gear when you join EFF!

This year alone EFF:

• Launched a resource hub to help users understand and fight back against age verification laws.
• Challenged San Jose's unconstitutional license plate reader database in court.
• Sued demanding answers when ICE spotting apps were mysteriously taken offline.
• Launched Rayhunter to detect cell site simulators.
• Pushed back hard against the EU's Chat Proposal that would break encryption for millions.

After 35 years of defending digital freedoms, we know what's at stake: we must protect your ability to speak freely, organize safely, and use technology without surveillance.

We have opportunities to win these fights, and you make each victory possible. Donate to EFF by December 31 and help us unlock additional grants this year!

Already an EFF Member? Help Us Spread the Word!

EFF Members have carried the movement for privacy and free expression for decades. You can help move the mission even further! Here’s some sample language that you can share with your networks:

We need to stand together and ensure technology works for us, not against us. Donate any amount to EFF by Dec 31, and you'll help unlock challenge grants! https://eff.org/yec
Bluesky | Facebook | LinkedIn | Mastodon
(more at eff.org/social)

_________________

EFF is a member-supported U.S. 501(c)(3) organization. We’re celebrating TWELVE YEARS of top ratings from the nonprofit watchdog Charity Navigator! Your donation is tax-deductible as allowed by law.

Conservatives have sought to block states from partnering with a Michael Bloomberg-backed fellowship program in environmental law.

Recipients of EPA’s largest climate grant program will get another shot at persuading the D.C. Circuit to restore $17 billion in awarded funds.

News about the threat to the National Center for Atmospheric Research hit "like a bomb" at the American Geophysical Union's fall meeting.

The General Assembly plowed ahead with plans to study data centers and climate impacts despite objections from Gov. Wes Moore (D).

The president's prime-time speech included false statements about the price of gas, the number of new power plants and the cost of electricity.

A growth engine for the economy is becoming a political albatross. Can messaging change that?

Interior officials are losing their patience with states as the West’s most important river teeters on the brink of crisis.

The International Energy Agency estimates that India and other nations could buoy the fuel through 2030.

The efforts reflect a push by companies and governments to close gaps in emissions monitoring as compliance demands rise and to supplement tools like satellites to deliver a greater degree of precision or to extend coverage to more sources of pollution.

Declining rainfall and rising temperatures have dried up pasture land at the same time agricultural use has expanded.

Carmakers and suppliers say the proposals still leave them exposed to factors they can’t control. Environmental groups, on the other hand, see loopholes that’ll weaken Europe’s climate strategy, slowing the uptake of EVs that has gained momentum over the last year.

Most languages use word position and sentence structure to extract meaning. For example, “The cat sat on the box,” is not the same as “The box was on the cat.” Over a long text, like a financial document or a novel, the syntax of these words likely evolves. 

Similarly, a person might be tracking variables in a piece of code or following instructions that have conditional actions. These are examples of state changes and sequential reasoning that we expect state-of-the-art artificial intelligence systems to excel at; however, the existing, cutting-edge attention mechanism within transformers — the primarily architecture used in large language models (LLMs) for determining the importance of words — has theoretical and empirical limitations when it comes to such capabilities.

An attention mechanism allows an LLM to look back at earlier parts of a query or document and, based on its training, determine which details and words matter most; however, this mechanism alone does not understand word order. It “sees” all of the input words, a.k.a. tokens, at the same time and handles them in the order that they’re presented, so researchers have developed techniques to encode position information. This is key for domains that are highly structured, like language. But the predominant position-encoding method, called rotary position encoding (RoPE), only takes into account the relative distance between tokens in a sequence and is independent of the input data. This means that, for example, words that are four positions apart, like “cat” and “box” in the example above, will all receive the same fixed mathematical rotation specific to that relative distance. 

Now research led by MIT and the MIT-IBM Watson AI Lab has produced an encoding technique known as “PaTH Attention” that makes positional information adaptive and context-aware rather than static, as with RoPE.

“Transformers enable accurate and scalable modeling of many domains, but they have these limitations vis-a-vis state tracking, a class of phenomena that is thought to underlie important capabilities that we want in our AI systems. So, the important question is: How can we maintain the scalability and efficiency of transformers, while enabling state tracking?” says the paper’s senior author Yoon Kim, an associate professor in the Department of Electrical Engineering and Computer Science (EECS), a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL), and a researcher with the MIT-IBM Watson AI Lab.

A new paper on this work was presented earlier this month at the Conference on Neural Information Processing Systems (NeurIPS). Kim’s co-authors include lead author Songlin Yang, an EECS graduate student and former MIT-IBM Watson AI Lab Summer Program intern; Kaiyue Wen of Stanford University; Liliang Ren of Microsoft; and Yikang Shen, Shawn Tan, Mayank Mishra, and Rameswar Panda of IBM Research and the MIT-IBM Watson AI Lab.

Path to understanding 

Instead of assigning every word a fixed rotation based on relative distance between tokens, as RoPE does, PaTH Attention is flexible, treating the in-between words as a path made up of small, data-dependent transformations. Each transformation, based on a mathematical operation called a Householder reflection, acts like a tiny mirror that adjusts depending on the content of each token it passes. Each step in a sequence can influence how the model interprets information later on. The cumulative effect lets the system model how the meaning changes along the path between words, not just how far apart they are. This approach allows transformers to keep track of how entities and relationships change over time, giving it a sense of “positional memory.” Think of this as walking a path while experiencing your environment and how it affects you. Further, the team also developed a hardware-efficient algorithm to more efficiently compute attention scores between every pair of tokens so that the cumulative mathematical transformation from PaTH Attention is compressed and broken down into smaller computations so that it’s compatible with fast processing on GPUs.

The MIT-IBM researchers then explored PaTH Attention’s performance on synthetic and real-world tasks, including reasoning, long-context benchmarks, and full LLM training to see whether it improved a model’s ability to track information over time. The team tested its ability to follow the most recent “write” command despite many distracting steps and multi-step recall tests, tasks that are difficult for standard positional encoding methods like RoPE. The researchers also trained mid-size LLMs and compared them against other methods. PaTH Attention improved perplexity and outcompeted other methods on reasoning benchmarks it wasn’t trained on. They also evaluated retrieval, reasoning, and stability with inputs of tens of thousands of tokens. PaTH Attention consistently proved capable of content-awareness.

“We found that both on diagnostic tasks that are designed to test the limitations of transformers and on real-world language modeling tasks, our new approach was able to outperform existing attention mechanisms, while maintaining their efficiency,” says Kim. Further, “I’d be excited to see whether these types of data-dependent position encodings, like PATH, improve the performance of transformers on structured domains like biology, in [analyzing] proteins or DNA.”

Thinking bigger and more efficiently 

The researchers then investigated how the PaTH Attention mechanism would perform if it more similarly mimicked human cognition, where we ignore old or less-relevant information when making decisions. To do this, they combined PaTH Attention with another position encoding scheme known as the Forgetting Transformer (FoX), which allows models to selectively “forget.” The resulting PaTH-FoX system adds a way to down-weight information in a data-dependent way, achieving strong results across reasoning, long-context understanding, and language modeling benchmarks. In this way, PaTH Attention extends the expressive power of transformer architectures. 

Kim says research like this is part of a broader effort to develop the “next big thing” in AI. He explains that a major driver of both the deep learning and generative AI revolutions has been the creation of “general-purpose building blocks that can be applied to wide domains,” such as “convolution layers, RNN [recurrent neural network] layers,” and, most recently, transformers. Looking ahead, Kim notes that considerations like accuracy, expressivity, flexibility, and hardware scalability have been and will be essential. As he puts it, “the core enterprise of modern architecture research is trying to come up with these new primitives that maintain or improve the expressivity, while also being scalable.”

This work was supported, in part, by the MIT-IBM Watson AI Lab and the AI2050 program at Schmidt Sciences.

Population decline often goes hand-in-hand with economic stagnation in rural areas — and the two reinforce each other in a cycle. Can digital technologies advance equitable innovation and, at the same time, preserve cultural heritage in shrinking regions?

A new open-access book, edited by MIT Vice Provost and Department of Urban Studies and Planning (DUSP) Professor Brent D. Ryan PhD ’02, Carmelo Ignaccolo PhD ’24 of Rutgers University, and Giovanna Fossa of the Politecnico di Milano, explores the transformative power of community-centered technologies in the rural areas of Italy.

“Small Town Renaissance: Bridging Technology, Heritage and Planning in Shrinking Italy” (Springer Nature, 2025) investigates the future of small towns through empirical analyses of cellphone data, bold urban design visions, collaborative digital platforms for small businesses, and territorial strategies for remote work. The work examines how technology may open up these regions to new economic opportunities. The book shares data-driven scholarly work on shrinking towns, economic development, and digital innovation from multiple planning scholars and practitioners, several of whom traveled to Italy in fall 2022 as part of a DUSP practicum taught by Ryan and Ignaccolo, and sponsored by MISTI Italy and Fondazione Rocca, in collaboration with Liminal.

“What began as a hands-on MIT practicum grew into a transatlantic book collaboration uniting scholars in design, planning, heritage, law, and telecommunications to explore how technology can sustain local economies and culture,” says Ignaccolo.

Now an assistant professor of city planning at Rutgers University’s E.J. Bloustein School of Planning and Public Policy, Ignaccolo says the book provides concrete and actionable strategies to support shrinking regions in leveraging cultural heritage and smart technologies to strengthen opportunities and local economies.

“Depopulation linked to demographic change is reshaping communities worldwide,” says Ryan. “Italy is among the hardest hit, and the United States is heading in the same direction. This project offered students a chance to harness technology and innovation to imagine bold responses to this growing challenge.”

The researchers note that similar struggles also exist in rural communities across Germany, Spain, Japan, and Korea. The book provides policymakers, urban planners, designers, tech innovators, and heritage advocates with fresh insights and actionable strategies to shape the future of rural development in the digital age. The book and chapters can be downloaded for free through most university libraries via open access.

The latest United Nations Climate Change Conference (COP30) concluded in November without a roadmap to phase out fossil fuels and without significant progress in strengthening national pledges to reduce climate-altering greenhouse gas emissions. In aggregate, today’s climate policies remain far too unambitious to meet the Paris Agreement’s goal of capping global warming at 1.5 degrees Celsius, setting the world on course to experience more frequent and intense storms, flooding, droughts, wildfires, and other climate impacts. A global policy regime aligned with the 1.5 C target would almost certainly reduce the severity of those impacts.

In the “2025 Global Change Outlook,” researchers at the MIT Center for Sustainability Science and Strategy (CS3) compare the consequences of these two approaches to climate policy through modeled projections of critical natural and societal systems under two scenarios. The Current Trends scenario represents the researchers’ assessment of current measures for reducing greenhouse gas (GHG) emissions; the Accelerated Actions scenario is a credible pathway to stabilizing the climate at a global mean surface temperature of 1.5 C above preindustrial levels, in which countries impose more aggressive GHG emissions-reduction targets.  

By quantifying the risks posed by today’s climate policies — and the extent to which accelerated climate action aligned with the 1.5 C goal could reduce them — the “Global Change Outlook” aims to clarify what’s at stake for environments and economies around the world. Here, we summarize the report’s key findings at the global level; regional details can also be accessed in several sections and through MIT CS3’s interactive global visualization tool.  

Emerging headwinds for global climate action 

Projections under Current Trends show higher GHG emissions than in our previous 2023 outlook, indicating reduced action on GHG emissions mitigation in the upcoming decade. The difference, roughly equivalent to the annual emissions from Brazil or Japan, is driven by current geopolitical events. 

Additional analysis in this report indicates that global GHG emissions in 2050 could be 10 percent higher than they would be under Current Trends if regional rivalries triggered by U.S. tariff policy prompt other regions to weaken their climate regulations. In that case, the world would see virtually no emissions reduction in the next 25 years.

Energy and electricity projections

Between 2025 and 2050, global energy consumption rises by 17 percent under Current Trends, with a nearly nine-fold increase in wind and solar. Under Accelerated Actions, global energy consumption declines by 16 percent, with a nearly 13-fold increase in wind and solar, driven by improvements in energy efficiency, wider use of electricity, and demand response. In both Current Trends and Accelerated Actions, global electricity consumption increases substantially (by 90 percent and 100 percent, respectively), with generation from low-carbon sources becoming a dominant source of power, though Accelerated Actions has a much larger share of renewables.   

“Achieving long-term climate stabilization goals will require more ambitious policy measures that reduce fossil-fuel dependence and accelerate the energy transition toward low-carbon sources in all regions of the world. Our Accelerated Actions scenario provides a pathway for scaling up global climate ambition,” says MIT CS3 Deputy Director Sergey Paltsev, co-lead author of the report.

Greenhouse gas emissions and climate projections

Under Current Trends, global anthropogenic (human-caused) GHG emissions decline by 10 percent between 2025 and 2050, but start to rise again later in the century; under Accelerated Actions, however, they fall by 60 percent by 2050. Of the two scenarios, only the latter could put the world on track to achieve long-term climate stabilization.  

Median projections for global warming by 2050, 2100, and 2150 are projected to reach 1.79, 2.74, and 3.72 degrees C (relative to the global mean surface temperature (GMST) average for the years 1850-1900) under Current Trends and 1.62, 1.56, and 1.50 C under Accelerated Actions. Median projections for global precipitation show increases from 2025 levels of 0.04, 0.11, and 0.18 millimeters per day in 2050, 2100, and 2150 under Current Trends and 0.03, 0.04, and 0.03 mm/day for those years under Accelerated Actions.

“Our projections demonstrate that aggressive cuts in GHG emissions can lead to substantial reductions in the upward trends of GMST, as well as global precipitation,” says CS3 deputy director C. Adam Schlosser, co-lead author of the outlook. “These reductions to both climate warming and acceleration of the global hydrologic cycle lower the risks of damaging impacts, particularly toward the latter half of this century.”

Implications for sustainability

The report’s modeled projections imply significantly different risk levels under the two scenarios for water availability, biodiversity, air quality, human health, economic well-being, and other sustainability indicators. 

Among the key findings: Policies that align with Accelerated Actions could yield substantial co-benefits for water availability, biodiversity, air quality, and health. For example, combining Accelerated Actions-aligned climate policies with biodiversity targets, or with air-quality targets, could achieve biodiversity and air quality/health goals more efficiently and cost-effectively than a more siloed approach. The outlook’s analysis of the global economy under Current Trends suggests that decision-makers need to account for climate impacts outside their home region and the resilience of global supply chains.  

Finally, CS3’s new data-visualization platform provides efficient, screening-level mapping of current and future climate, socioeconomic, and demographic-related conditions and changes — including global mapping for many of the model outputs featured in this report. 

“Our comparison of outcomes under Current Trends and Accelerated Actions scenarios highlights the risks of remaining on the world’s current emissions trajectory and the benefits of pursuing a much more aggressive strategy,” says CS3 Director Noelle Selin, a co-author of the report and a professor in the Institute for Data, Systems and Society and Department of Earth, Atmospheric and Planetary Sciences at MIT. “We hope that our risk-benefit analysis will help inform decision-makers in government, industry, academia, and civil society as they confront sustainability-relevant challenges.” 

This interview is part of a series of short interviews from the Department of Electrical Engineering and Computer Science (EECS). Each spotlight features a student answering their choice of questions about themselves and life at MIT. Today’s interviewee, senior Diego Temkin, is double majoring in courses 6-3 (Computer Science and Engineering) and 11 (Urban Planning). The McAllen, Texas, native is involved with MIT’s Dormitory Council (DormCon), helps to maintain Hydrant (formerly Firehose)/CourseRoad, and is both a member of the Student Information Processing Board (MIT’s oldest computing club) and an Advanced Undergraduate Research Opportunities Program (SuperUROP) scholar.

Q: What’s your favorite key on a standard computer keyboard, and why?

A: The “1” key! During Covid, I ended up starting a typewriter collection and trying to fix them up, and I always thought it was interesting how they didn’t have a 1 key. People were just expected to use the lowercase “l,” which presumably makes anyone who cares about ASCII very upset.

Q: Tell us about a teacher from your past who had an influence on the person you’ve become.

A: Back in middle school, everyone had to take a technology class that taught things like typing skills, Microsoft Word and Excel, and some other things. I was a bit of a nerd and didn’t have too many friends interested in the sort of things I was, but the teacher of that technology class, Mrs. Camarena, would let me stay for a bit after school and encouraged me to explore more of my interests. She helped me become more confident in wanting to go into computer science, and now here I am. 

Q: What’s your favorite trivia factoid?

A: Every floor in Building 13 is painted as a different MBTA line. I don’t know why and can’t really find anything about it online, but once you notice it you can’t unsee it!

Q: Do you have any pets? 

A: I do! His name is Skateboard, and he is the most quintessentially orange cat. I got him off reuse@mit.edu during my first year here at MIT (shout out to Patty K), and he’s been with me ever since. He’s currently five years old, and he’s a big fan of goldfish and stepping on my face at 7 a.m. Best decision I’ve ever made. 

Q: Are you a re-reader or a re-watcher? If so, what are your comfort books, shows, or movies?

A: Definitely a re-watcher, and definitely “Doctor Who.” I’ve watched far too much of that show and there are episodes I can recite from memory (looking at you, “The Eleventh Hour”). Anyone I know will tell you that I can go on about that show for hours, and before anyone asks, my favorite doctor is Matt Smith (sorry to the David Tennant fans; I like him too, though!)

Q: Do you have a bucket list? If so, share one or two of the items on it.

A: I’ve been wanting to take a cross-country Amtrak trip for a while … I think I might try going to the West Coast and some national parks during IAP [Independent Activities Period], if I have the time. Now that it’s on here, I definitely have to do it!

Why did humans evolve the eyes we have today?

While scientists can’t go back in time to study the environmental pressures that shaped the evolution of the diverse vision systems that exist in nature, a new computational framework developed by MIT researchers allows them to explore this evolution in artificial intelligence agents.

The framework they developed, in which embodied AI agents evolve eyes and learn to see over many generations, is like a “scientific sandbox” that allows researchers to recreate different evolutionary trees. The user does this by changing the structure of the world and the tasks AI agents complete, such as finding food or telling objects apart.

This allows them to study why one animal may have evolved simple, light-sensitive patches as eyes, while another has complex, camera-type eyes.

The researchers’ experiments with this framework showcase how tasks drove eye evolution in the agents. For instance, they found that navigation tasks often led to the evolution of compound eyes with many individual units, like the eyes of insects and crustaceans.

On the other hand, if agents focused on object discrimination, they were more likely to evolve camera-type eyes with irises and retinas.

This framework could enable scientists to probe “what-if” questions about vision systems that are difficult to study experimentally. It could also guide the design of novel sensors and cameras for robots, drones, and wearable devices that balance performance with real-world constraints like energy efficiency and manufacturability.

“While we can never go back and figure out every detail of how evolution took place, in this work we’ve created an environment where we can, in a sense, recreate evolution and probe the environment in all these different ways. This method of doing science opens to the door to a lot of possibilities,” says Kushagra Tiwary, a graduate student at the MIT Media Lab and co-lead author of a paper on this research.

He is joined on the paper by co-lead author and fellow graduate student Aaron Young; graduate student Tzofi Klinghoffer; former postdoc Akshat Dave, who is now an assistant professor at Stony Brook University; Tomaso Poggio, the Eugene McDermott Professor in the Department of Brain and Cognitive Sciences, an investigator in the McGovern Institute, and co-director of the Center for Brains, Minds, and Machines; co-senior authors Brian Cheung, a postdoc in the  Center for Brains, Minds, and Machines and an incoming assistant professor at the University of California San Francisco; and Ramesh Raskar, associate professor of media arts and sciences and leader of the Camera Culture Group at MIT; as well as others at Rice University and Lund University. The research appears today in Science Advances.

Building a scientific sandbox

The paper began as a conversation among the researchers about discovering new vision systems that could be useful in different fields, like robotics. To test their “what-if” questions, the researchers decided to use AI to explore the many evolutionary possibilities.

“What-if questions inspired me when I was growing up to study science. With AI, we have a unique opportunity to create these embodied agents that allow us to ask the kinds of questions that would usually be impossible to answer,” Tiwary says.

To build this evolutionary sandbox, the researchers took all the elements of a camera, like the sensors, lenses, apertures, and processors, and converted them into parameters that an embodied AI agent could learn.

They used those building blocks as the starting point for an algorithmic learning mechanism an agent would use as it evolved eyes over time.

“We couldn’t simulate the entire universe atom-by-atom. It was challenging to determine which ingredients we needed, which ingredients we didn’t need, and how to allocate resources over those different elements,” Cheung says.

In their framework, this evolutionary algorithm can choose which elements to evolve based on the constraints of the environment and the task of the agent.

Each environment has a single task, such as navigation, food identification, or prey tracking, designed to mimic real visual tasks animals must overcome to survive. The agents start with a single photoreceptor that looks out at the world and an associated neural network model that processes visual information.

Then, over each agent’s lifetime, it is trained using reinforcement learning, a trial-and-error technique where the agent is rewarded for accomplishing the goal of its task. The environment also incorporates constraints, like a certain number of pixels for an agent’s visual sensors.

“These constraints drive the design process, the same way we have physical constraints in our world, like the physics of light, that have driven the design of our own eyes,” Tiwary says.

Over many generations, agents evolve different elements of vision systems that maximize rewards.

Their framework uses a genetic encoding mechanism to computationally mimic evolution, where individual genes mutate to control an agent’s development.

For instance, morphological genes capture how the agent views the environment and control eye placement; optical genes determine how the eye interacts with light and dictate the number of photoreceptors; and neural genes control the learning capacity of the agents.

Testing hypotheses

When the researchers set up experiments in this framework, they found that tasks had a major influence on the vision systems the agents evolved.

For instance, agents that were focused on navigation tasks developed eyes designed to maximize spatial awareness through low-resolution sensing, while agents tasked with detecting objects developed eyes focused more on frontal acuity, rather than peripheral vision.

Another experiment indicated that a bigger brain isn’t always better when it comes to processing visual information. Only so much visual information can go into the system at a time, based on physical constraints like the number of photoreceptors in the eyes.

“At some point a bigger brain doesn’t help the agents at all, and in nature that would be a waste of resources,” Cheung says.

In the future, the researchers want to use this simulator to explore the best vision systems for specific applications, which could help scientists develop task-specific sensors and cameras. They also want to integrate LLMs into their framework to make it easier for users to ask “what-if” questions and study additional possibilities.

“There’s a real benefit that comes from asking questions in a more imaginative way. I hope this inspires others to create larger frameworks, where instead of focusing on narrow questions that cover a specific area, they are looking to answer questions with a much wider scope,” Cheung says.

This work was supported, in part, by the Center for Brains, Minds, and Machines and the Defense Advanced Research Projects Agency (DARPA) Mathematics for the Discovery of Algorithms and Architectures (DIAL) program.