In the world of artificial intelligence, the debate around how reinforcement learning (RL) contributes to reasoning capabilities is heating up. Does RL help create new skills, or does it just amplify what’s already there? A team of researchers from various institutions has been digging into this question, and their findings are quite intriguing.
The researchers focused on a complex task called Complementary Reasoning, which involves blending internal knowledge with external information. They used a synthetic dataset of human biographies to break this task down into two simpler skills: Parametric Reasoning (using internal knowledge) and Contextual Reasoning (using external information). This breakdown allowed them to evaluate how well different AI models could generalize their learning across various difficulty levels: I.I.D. (Independent and Identically Distributed), Composition, and Zero-shot settings.
They found that Supervised Fine-Tuning (SFT) works well for in-distribution performance but struggles with out-of-distribution (O.O.D.) generalization, especially in Zero-shot settings where the relationships are new. This led them to discover the SFT Generalization Paradox: models that are supervised only on the composite task can achieve near-perfect accuracy within the training distribution but fail miserably when faced with new, unseen scenarios. This suggests that these models are essentially memorizing shortcuts rather than truly understanding the task.
In contrast, RL seems to act more like a reasoning synthesizer rather than just a probability amplifier. However, there’s a catch: RL can only synthesize these complex strategies if the base model has already mastered the individual atomic skills through SFT. This means that RL builds on a foundation of well-learned basics to create more complex reasoning strategies without needing explicit supervision for those strategies.
The implications of this research are significant. It challenges the notion that RL is merely an amplifier of existing behaviors and suggests that, with the right foundational skills, RL can actively synthesize complex reasoning strategies. This could pave the way for more scalable and generalized AI models, capable of handling a wide range of complex tasks. As we continue to push the boundaries of AI, understanding these mechanisms will be crucial in developing more intelligent and adaptable systems.
