MONTE CARLO VALUE CONTOUR
An ace currently counts as 11Usable ace
player sum
dealer showing
01
THE PROBLEM
Interactive lesson · Blackjack
Learn a value function from complete games.
In Blackjack, the player can hit (draw another card) or stick (stop drawing). A total above 21 loses. After the player sticks, the dealer hits until reaching 17 or more. The higher non-bust total wins; equal totals draw.
Choose a fixed policy, then evaluate it from experience. The current rule is hit below 20; stick on 20–21. Unlike the Jack’s Car Rental dynamic-programming labs, Monte Carlo prediction is not given a transition table. It learns only from the states, actions, and final reward observed in sampled games.
States & rewards
STATE(player sum, dealer card, usable ace?)
POLICY πhit below 20 · stick on 20–21
TERMINAL REWARDwin +1 · draw 0 · loss −1
DISCOUNTγ = 1 for this episodic task
What is unknown? The simulator knows how cards are dealt, but the learner does not use those probabilities. It simply plays many games and averages the returns observed after visiting each state. States with player sums below 12 are omitted because the player always hits there and cannot bust.
Two different states, not just two labels
A usable ace is already in the hand and currently counts as 11. It automatically counts as 1 if a draw would otherwise cause a bust—this is not an extra player action. No usable ace means no ace, or aces already counted as 1.
18 + a drawn 5: A + 7 becomes 13 with no usable ace; 10 + 8 busts at 23. A single game may visit both value maps.
The policy
CHOOSE THE POLICY TO EVALUATE
Stick when the player sum reaches 20
Hit below 20; stick on 20–21. Moving the slider starts a fresh Monte Carlo estimate for the new policy.
Sample games
Cards are dealt randomly, including any initial draws needed to reach 12. An ace may be dealt initially or drawn later. Only states actually visited update their matching map.
EPISODES OBSERVED0complete games
STATES VISITED0of 200
WINS / DRAWS / LOSSES0 / 0 / 0sample outcomes
SELECTED STATE VALUE—0 first visits
Value estimates
MONTE CARLO VALUE CONTOUR
No ace, or aces count as 1No usable ace
player sum
dealer showing
−1 likely loss0 roughly even+1 likely win· unvisited
LATEST SAMPLED EPISODE
reward GNo game sampled yet
Sample one game to reveal the trajectory the learner actually observed.
One outcome becomes a return for every visited state.
G ← Rt+1 + γGThe terminal reward will appear here, starting from the last decision and moving back to the first.
Incremental sample averageV(s) ← V(s) + [G − V(s)] / N(s)Each first visit contributes one complete observed return. As N(s) grows, any single lucky or unlucky game has less influence.
What “toward convergence” means here
1 game
A few visited cells jump directly to −1, 0, or +1. That is one outcome, not yet a dependable value.
Hundreds
Common states receive repeated returns. The maps begin to show where the fixed policy tends to win or lose.
Thousands
Sample averages move less with each episode and approach the true Vπ for states that keep being visited.
Monte Carlo control
Learn both actions. Improve the policy.
Prediction above asks “How good is this fixed policy?” Control asks “Which action should I choose?” We now estimate Q(s, hit) and Q(s, stick) from completed games, then choose the larger estimate in each state. The prediction slider does not control this separate experiment.
Start with a random state and action
Each game starts in one of the 200 states, chosen uniformly, with hit or stick chosen with equal probability. This is exploring starts: even an action the policy currently avoids gets tried. We construct an actual hand matching the state’s sum and ace status.
Average returns for each (state, action)
Q(s,a) ← Q(s,a) + [G − Q(s,a)] / N(s,a)The first visit to each state-action pair in a game contributes its final result: +1, 0, or −1. All earlier rewards are zero and γ = 1.
Keep the better action
π(s) ← argmaxa Q(s,a)After the game, compare hit and stick at updated states. The next game follows this improved policy after its random first action. Ties keep the existing action.
The learner is not given transition or reward probabilities. The game simulator deals cards and reports outcomes; the learner only uses those episodes. Exploring starts is a teaching assumption—we can choose a starting hand here, unlike a normal casino deal.
TRAINING GAMES0sampled episodes
ACTION-PAIR COVERAGE0/400sampled at least once
BOTH ACTIONS SAMPLED0/200states with a comparison
POLICY CELLS CHANGED0net change this batch
CURRENT GREEDY POLICY
No ace, or all aces count as 1No usable ace
player sum
dealer showing
CURRENT GREEDY POLICY
Ace counts as 11; can become 1Usable ace
player sum
dealer showing
H = hit S = stick. White ring = policy changed in this batch. Gray = one or both actions not yet sampled. Click a cell to compare both actions.
Inspecting state
(18, 10, no usable ace)
| Action | Q estimate | N(s,a) |
|---|---|---|
| hitcurrent choice | 0 (initial) | 0 |
| stick | 0 (initial) | 0 |
π(s) = hit. Not enough coverage to compare both actions yet. Zero is an initial estimate, not evidence that an action is worth zero.
What does Q mean? Take that action now, then follow the policy for the rest of the game. As the policy improves, the continuation changes too; these running averages include returns collected under earlier policies.
Latest training game · 0
Try one game to see the update
Q starts at zero for all 400 state-action pairs. The initial policy hits below 20 and sticks on 20–21. Training replaces it with state-specific choices.
Is this the optimal policy yet? The maps show the current learned greedy policy, moving toward an optimal policy for these simplified rules—not a certified optimum. One sample per action is not enough. Keep training, inspect sample counts, and expect close decisions to fluctuate. This is not casino Blackjack: no doubling, splitting, finite shoe, or special payout for a natural 21; the dealer sticks at 17, including soft 17.
What to carry forward
Learn values. Improve decisions.
01
Prediction vs control
Prediction estimates Vπ for a fixed policy. Control estimates Q(s,a) and uses it to improve the policy.
02
Finish the episode
The final win, draw, or loss determines the return for every visited state in this undiscounted game.
03
Average first visits
Prediction averages returns for states; control averages them for state-action pairs. Each first visit contributes once per game.
04
Coverage matters
Rare states need more games. A noisy estimate or an unvisited cell is not evidence of its true value.
Want to compare with model-based backups? Explore dynamic programming with Jack’s Car Rental →