The Problematic Approach

Classical RL algorithms like Q-learning assume we know the environment dynamics P(s'|s,a) and reward function R(s,a). Even when these are learned, the agent typically commits to a single estimate and acts as if it's certain.

Example: Stochastic Frozen Lake with Unknown Map - In stochastic Frozen Lake, we know the ice is slippery (stochastic transitions), but we might not know the exact map. Classical Q-learning would estimate transition probabilities from data, pick the most likely map, and act as if that map is definitely correct. This ignores uncertainty about which map we're actually in.

The Quantum-Inspired Solution

In quantum mechanics, a mixed state is a density matrix ρ = Σᵢ pᵢ |ψᵢ⟩⟨ψᵢ| representing a probabilistic mixture of pure states. We apply this to RL by maintaining a belief distribution over possible environments.

Instead of committing to a single environment model, we maintain multiple hypotheses about the environment (different maps, different transition probabilities, etc.) and keep a probability distribution over them. The policy then acts based on this mixed state, accounting for uncertainty.

Example: Stochastic Frozen Lake with Unknown Map - We maintain beliefs about multiple possible maps: Map A (holes in positions X), Map B (holes in positions Y), etc. Each map has probability pᵢ. The agent's policy considers all maps weighted by their probabilities, making decisions that are robust to this uncertainty.

The policy takes the belief distribution as input and chooses tests or treatments that are robust across likely worlds, while still being efficient.

Mixed-State Implementation

Below we sketch a mixed-state agent with multiple world models and a risk-aware policy that conditions on the belief distribution.

class MixedStateAgent:
    def __init__(self, world_models, policy):
        self.world_models = world_models  # list of models M₁, M₂, …
        self.belief = np.ones(len(world_models)) / len(world_models)
        self.policy = policy

    def update_belief(self, observation):
        # Bayes-like update (simplified)
        likelihoods = np.array([
            M.likelihood(observation) for M in self.world_models
        ])
        self.belief *= likelihoods
        self.belief /= self.belief.sum()

    def act(self, state):
        # Pass both state and belief to the policy
        return self.policy.act(state, belief=self.belief)

def cvar_loss(returns, alpha=0.1):
    """
    Conditional Value at Risk (CVaR) at level alpha.
    Focus on the worst alpha fraction of outcomes.
    """
    sorted_returns = np.sort(returns)
    cutoff = int(len(sorted_returns) * alpha)
    tail = sorted_returns[:cutoff] if cutoff > 0 else sorted_returns
    return -tail.mean()

Key Insights

Quantum mixed states provide a natural framework for handling epistemic uncertainty in reinforcement learning. By maintaining a probability distribution over possible environments rather than committing to a single model, agents can make robust decisions that account for what they don't know.

Stochastic Frozen Lake Example: When the map is uncertain, a mixed-state agent maintains beliefs about multiple possible maps. It avoids paths that are safe in some maps but dangerous in others, naturally exhibiting risk-averse behavior without explicit risk penalties.