The Problematic Approach

Classical reinforcement learning operates on a simple principle: the agent maintains one policy (a mapping from states to actions), interacts with the environment, receives rewards, and updates that single policy. This creates a bottleneck: learning happens sequentially, one experience at a time.

Understanding Quantum Superposition

In quantum mechanics, superposition is the ability of a particle to exist in multiple states simultaneously. The classic example is Schrödinger's cat: until you open the box, the cat is in a superposition of being both alive and dead. When you observe it (measure it), the superposition "collapses" to one definite state.

Mathematically, a quantum state is represented as:

where |0⟩ and |1⟩ are basis states (like "alive" and "dead"), and |α|² + |β|² = 1. The probability of measuring |0⟩ is |α|², and the probability of measuring |1⟩ is |β|².

Applying Superposition to Reinforcement Learning

In classical RL, an agent maintains one policy π(s) that maps states to actions. In QiRL, we maintain a superposition of policies:

Each |πᵢ⟩ is a base policy (a different strategy), and |αᵢ|² is the probability of selecting that policy. When the agent needs to act, it "measures" the superposition, collapsing to one policy with probability |αᵢ|².

Example: Frozen Lake - In the classic Frozen Lake environment, an agent must navigate from start to goal on a frozen lake, avoiding holes. A classical agent might try one strategy (e.g., "always go right"), learn from it, then try another. A QiRL agent maintains multiple strategies simultaneously: one that prefers going right, one that prefers going down, one that's more cautious, etc. All strategies learn from the same experience in parallel.

Implementing Superposition: A Concrete Example

Let's see how superposition works in practice using the Frozen Lake environment. We'll create multiple base policies (different navigation strategies) and maintain them in superposition.

Base Policies: We might have π₁ (prefer right), π₂ (prefer down), π₃ (cautious, avoids holes), π₄ (exploratory). Each is a Q-function or policy network. The superposition maintains weights α₁, α₂, α₃, α₄ such that |α₁|² + |α₂|² + |α₃|² + |α₄|² = 1.

When acting, we "measure" the superposition: sample a policy i with probability |αᵢ|², then use that policy to select an action. All policies update from the same experience, but the weights evolve to favor better-performing policies.

import numpy as np
import gymnasium as gym

class SuperposedPolicy:
    """
    Quantum-inspired superposition of multiple RL policies.
    Example: Frozen Lake with multiple navigation strategies.
    """
    def __init__(self, base_policies, initial_amplitudes=None):
        # Base policies: different strategies (e.g., prefer right, prefer down, cautious)
        self.base_policies = base_policies  # [π₁, π₂, ..., πₖ]
        
        # Quantum amplitudes: |αᵢ|² is probability of selecting policy i
        if initial_amplitudes is None:
            # Start with uniform superposition: equal probability for all
            k = len(base_policies)
            self.amplitudes = np.ones(k, dtype=complex) / np.sqrt(k)
        else:
            self.amplitudes = np.array(initial_amplitudes, dtype=complex)
    
    def act(self, state):
        """
        Quantum measurement: collapse superposition to one policy.
        Probability of selecting policy i is |αᵢ|²
        """
        # Compute probabilities from amplitudes
        probabilities = np.abs(self.amplitudes) ** 2
        probabilities = probabilities / probabilities.sum()  # Normalize
        
        # Measure: sample policy according to probabilities
        selected_policy_idx = np.random.choice(len(self.base_policies), p=probabilities)
        selected_policy = self.base_policies[selected_policy_idx]
        
        # Use selected policy to choose action
        return selected_policy.act(state)
    
    def update_from_experience(self, state, action, reward, next_state, done):
        """
        Update all base policies in parallel (quantum parallelism).
        Then update amplitudes using amplitude amplification.
        """
        # Parallel update: all policies learn from same experience
        for π in self.base_policies:
            π.update(state, action, reward, next_state, done)
        
        # Amplitude amplification: increase amplitude for better policies
        self._amplify_successful_policies(reward)
    
    def _amplify_successful_policies(self, reward):
        """
        Grover-inspired amplitude amplification.
        Rotate amplitudes toward policies with higher expected returns.
        """
        # Estimate performance of each policy (simplified)
        policy_returns = [π.estimate_return() for π in self.base_policies]
        max_return = max(policy_returns) if policy_returns else 1.0
        
        # Quantum rotation: rotate amplitude based on performance
        for i, (amp, perf) in enumerate(zip(self.amplitudes, policy_returns)):
            if max_return > 0:
                angle = np.arctan(perf / max_return)
                # Rotate amplitude in complex plane
                self.amplitudes[i] = amp * np.exp(1j * angle)
        
        # Normalize to maintain probability distribution
        norm = np.sqrt(np.sum(np.abs(self.amplitudes) ** 2))
        self.amplitudes = self.amplitudes / norm

def grover_amplitude_amplification(amplitudes, oracle_scores, num_iterations=1):
    """
    Grover's algorithm for amplitude amplification.
    Increases probability of "good" policies exponentially.
    
    Args:
        amplitudes: complex array of quantum amplitudes
        oracle_scores: performance scores for each policy (higher = better)
        num_iterations: number of Grover iterations
    
    Returns:
        Updated amplitudes with amplified good policies
    """
    n = len(amplitudes)
    
    for iteration in range(num_iterations):
        # Phase inversion: flip sign of amplitudes for good policies
        for i, score in enumerate(oracle_scores):
            if score > np.mean(oracle_scores):  # "Good" policy
                amplitudes[i] = -amplitudes[i]
        
        # Inversion about mean: amplify all amplitudes
        mean_amp = np.mean(amplitudes)
        amplitudes = 2 * mean_amp - amplitudes
    
    # Normalize
    norm = np.sqrt(np.sum(np.abs(amplitudes) ** 2))
    return amplitudes / norm

def parallel_td_update(superposed_q_values, states, actions, rewards, next_states):
    """
    Quantum parallelism: update Q-values for all policies simultaneously.
    In true quantum computing, this would use unitary operators.
    Here we simulate parallel updates classically.
    """
    # Unitary operator U for temporal difference update
    # U|Q⟩ = |Q + α(r + γ max Q(s') - Q(s,a))⟩
    
    updated_q_values = []
    for q_values in superposed_q_values:  # One Q-function per base policy
        # Standard TD update, but applied to all policies in parallel
        td_error = rewards + 0.99 * np.max(q_values[next_states]) - q_values[states, actions]
        updated_q = q_values.copy()
        updated_q[states, actions] += 0.1 * td_error  # Learning rate 0.1
        updated_q_values.append(updated_q)
    
    return updated_q_values

# Example usage with Frozen Lake
env = gym.make('FrozenLake-v1')
base_policies = [
    QLearningPolicy(prefer_direction='right'),
    QLearningPolicy(prefer_direction='down'),
    QLearningPolicy(exploration_rate=0.3),  # More cautious
    QLearningPolicy(exploration_rate=0.7)   # More exploratory
]

superposed_policy = SuperposedPolicy(base_policies)

for episode in range(1000):
    state, _ = env.reset()
    done = False
    while not done:
        action = superposed_policy.act(state)
        next_state, reward, done, _, _ = env.step(action)
        superposed_policy.update_from_experience(state, action, reward, next_state, done)
        state = next_state

Key Insights

Quantum superposition in reinforcement learning demonstrates that we don't need to commit to a single policy. By maintaining a superposition of multiple strategies, we can explore the solution space in parallel, achieving significant improvements in sample efficiency and learning speed.