*This is Part 6 of our GPT from scratch series. Having upgraded to modern tokenization in Part 5, we now tackle the data limitation by training on a much larger corpus and exploring pretraining + finetuning.*

## Training on a Larger Corpus

Since we have a larger model, we now want to train on a larger dataset. I'm still interested in generating poems, so let's use this [poetry dataset from Kaggle](https://www.kaggle.com/datasets/tgdivy/poetry-foundation-poems) which gives us a CSV file containing a list of poems from [Poetry Foundation](https://www.poetryfoundation.org/). 

We first have to do some data cleaning with pandas to get the newline Unicode characters consistent, and concatenate all the poems into a single text file.

```
Line terminators found in the poems:
----------------------------------------
Mac Classic (\r only): 417494 occurrences
Double CR + LF (\r\r\n): 379146 occurrences
Windows (\r\n): 379146 occurrences
Reversed (\n\r): 19556 occurrences
Double CR (\r\r): 10527 occurrences
Line Separator (U+2028): 419 occurrences
Paragraph Separator (U+2029): 1 occurrences
----------------------------------------

Successfully concatenated 13854 poems into 'all_poems_cleaned.txt'

Sample cleaning (poem at index 101):
Original: '\r\r\nThe sound of a guitar drifts through the air.\r\r\nCupped in my hand, a snowflake quivers lightly.\r\r'...
Cleaned: 'The sound of a guitar drifts through the air.\nCupped in my hand, a snowflake quivers lightly.\nThick '...
```

This file is around 20MB and 400,000 lines long, which is 20 times the size and 10 times the number of lines of the Shakespeare dataset.

## Training Results on Poetry Dataset

```
step 0: train loss 10.8247, val loss 10.8255
step 100: train loss 6.2405, val loss 6.5146
step 200: train loss 5.7141, val loss 6.2657
step 300: train loss 5.5298, val loss 6.1561
step 400: train loss 5.4678, val loss 5.8279
step 500: train loss 5.5526, val loss 5.7174
step 600: train loss 5.2497, val loss 5.6578
step 700: train loss 5.1549, val loss 5.6028
step 800: train loss 5.0009, val loss 5.7194
step 900: train loss 5.2021, val loss 5.5778
step 1000: train loss 4.8988, val loss 5.5454
step 1100: train loss 4.9935, val loss 5.4814
step 1200: train loss 4.9520, val loss 5.3551
step 1300: train loss 5.0129, val loss 5.3685
step 1400: train loss 4.9534, val loss 5.3807
step 1500: train loss 4.7947, val loss 5.3912
step 1600: train loss 5.0557, val loss 5.2993
step 1700: train loss 4.8419, val loss 5.2611
step 1800: train loss 4.7657, val loss 5.0324
step 1900: train loss 4.4551, val loss 5.0815
step 2000: train loss 4.6674, val loss 5.1306
step 2100: train loss 4.6318, val loss 5.0484
step 2200: train loss 4.6357, val loss 5.2188
step 2300: train loss 4.5006, val loss 4.9910
step 2400: train loss 4.3960, val loss 5.0353
step 2500: train loss 4.4227, val loss 5.0051
step 2600: train loss 4.4193, val loss 4.9974
step 2700: train loss 4.4097, val loss 4.9683
step 2800: train loss 4.2323, val loss 4.9965
step 2900: train loss 4.3671, val loss 4.8900
step 3000: train loss 4.3548, val loss 4.8972
step 3100: train loss 4.3923, val loss 4.7957
step 3200: train loss 4.4397, val loss 4.7694
step 3300: train loss 4.3573, val loss 4.9981
step 3400: train loss 4.3772, val loss 4.8391
step 3500: train loss 4.3908, val loss 4.8069
step 3600: train loss 4.2172, val loss 4.7400
step 3700: train loss 4.1613, val loss 4.8643
step 3800: train loss 4.2559, val loss 4.9014
step 3900: train loss 4.0760, val loss 4.7784
step 4000: train loss 3.9681, val loss 4.7814
step 4100: train loss 4.0662, val loss 4.8351
step 4200: train loss 4.1156, val loss 4.9005
step 4300: train loss 4.1036, val loss 4.7409
step 4400: train loss 4.0790, val loss 4.6941
step 4500: train loss 4.1168, val loss 4.8207
step 4600: train loss 3.8932, val loss 4.7466
step 4700: train loss 4.0971, val loss 4.9108
step 4800: train loss 4.0825, val loss 4.8164
step 4900: train loss 4.0249, val loss 4.7728
step 4999: train loss 4.0157, val loss 4.7919
```

The model generates much more diverse and poetic text:

```
! but ...  
beautise on a head, everything's
served, seemed—
Lewash on the back of the garbled pllesh,
are not to be vigilant
by food, tears, eyes burnt,
dust and screaming. Just say —I've brought
Grandter aesthetics—
maybe God like John Fanny or that.7.
Everything's that alonghed in.
Butoried to me
when he yelled.

Just urboilherer appeared. Then he were falling, sits crying and prayed to his brother, being missing. They started on time they went up, though
God said his father had to stop a good reporters
stand, locked, hooves, feet up tighter, "Is smooth?"
The Morning
Season explores the world of the walls
Gazels through the light. Later each day,
A reasonably thing is beauties
At evening a while Frains unself from a walk.Zusted in light old
The way used to the land. On the snow?
There said, "Not offensive." "but that's the trietry, "Nor does
say, but it" with footage,
 Maoaching the crowd."You like the power actual
or on in their apartmentrelices, and the vinyl themselves lie together by that the Jewish sound." So, in one hand,
you could folding it backdroaches and wall you made
had keep what'﻿s done." and I stared, picking a grump offpour, and stared on the smoke, tapped to a wheel and the twos winkboat.
The house was a cigar on my stopes of dilelict and
those threatened hunger from how to be at moving steadily, could never believe
as white but a man with pens would not take  to. I could remember that night.
The moon was thick, the wind rivening the waves of the sea, the swift dark whiteerly recedes
from are it and its silkency of    paths comprising sleep, a rail of o'clock. Outspread it somewhere,
and if it ever repeated; soon, just angularling forward, the magager and schimology.The rifle wasn't wobbling paths and blinked,
```

## Further Training Attempts

To try and further improve, we try training longer and with a learning rate that is ten times smaller. However, both training and validation seem to have saturated. We also try a cosine learning rate with multiple restarts, but the results are not very good:

```python
step_lr = config.learning_rate*np.cos(0.5*np.pi*iter/(config.max_iters-1))
for group in optimizer.param_groups:
  group['lr'] = step_lr
```

```
step 0: train loss 10.8202, val loss 10.8287
...
step 3000: train loss 4.5538, val loss 5.0448
...
step 1000: train loss 4.5272, val loss 4.9344
...
step 999: train loss 4.2557, val loss 4.8072
...
step 999: train loss 4.0713, val loss 4.7289
...
step 999: train loss 4.1581, val loss 4.8483
```

Next, we try reshuffling the training and validation split of the input data by using the first 10% for validation instead. We find that training loss is now higher, another sign we were overfitting. After further training, we end up with training loss and validation loss being about 4. The final result is decent:

```
Summer dazed
with its theatrical designs, tubes, guts.
I clump the bripscrewed black.
I share the precise whatever
not actually needed.
I say when it grazed the time,
things myself between them
and there's little wings forth
English stains, the slick cold force.
Running the basement that moved
never. I wonder whether it meant
that this sad adventure within and
unlemishes the dead.
I was writing that Old Friend was
always working on I have
seen many plans seeing unsatisfied,
she was always ashamed
of this region.
I think of the Second Coming
asleep clear as I thought
waswitch recognized yet because of himself
were not afraid of me
but this sequence after the war
that had been doing to me
don't hurt me I meant so many years ago.
I wasn't my brother.
```

## Finetuning on Shakespeare

Lastly, we try finetuning on the tiny Shakespeare dataset we started with. We have essentially pretrained our larger model on the larger poetry dataset and now use a much lower learning rate (3e-6) and 10,000 training iterations to refine on the smaller tiny Shakespeare dataset.

```
step 10000: train loss 3.1057, val loss 4.1152
```

This is much better than when we just tried to train our larger model with the GPT-2 tokenizer on the Shakespeare data alone. This is the advantage of pretraining! Here are the results:

```
Is he contented himself with his exercise,
Even he that fall'd upon his tent with him
The ground o'er his bed, his dole from the north,
And cries 'He well.'
And Tranio, which consummate honour, in turn
Proclaims him on his way; and he cried,
Lay him that comments mortals play out.

LADY CAPULET:
My lords, I will not hear my traitor.

DUKE VINCENTIO:
Wrut, away! thou shalt never weep;
And yet I have received thy royal presence,
Having done the corse. But thou'll hear be
My father's son, blessing in the cruel gown,
though mine own buried heir
Our father's poor son, my poor daughter, my child,
Where I am resign'd me; and I will be king,
And pain'd the house of this churchyard call.
Thy father, and I seek thee here;
So, newly wench'd with that great aspect,
And thy servile' mortal presence made me,
And thou wouldst board thy mortal bones
In readiness, he may hear it bear.
Ah, art thou law to be made my nurse.

LADY CAPULET:
Ay, I'll win thee would not woe awhile.

DUKE OF YORK:
O gentle Plantagenet queen,

ISABELLA:
Ah, my lord, thou art safe and younger!

DUKE OF YORK:
Farewell with us, bleated those that have said,
That we should plague thee for many days.

KING RICHARD II:
Give me my father to your brother's house,
Or else send him by your fortunes to the street.
What art thou slain?

GLOUCESTER:
Harry of York, I tell thee that kill thy life:
And thou, contracted hither to my farm,
Thou art so happy to be repelling'd at thy mind!
As on a day that pageant thou sufficiency,
Now let thy father speak before thy function.

KING RICHARD II:
O coward to this day; there comes Hereford with thee!
```

Wow! While it's still not perfectly coherent, it seems much better than our output from the character-level tokenizer. Some characters even have multiple lines!

## Key Insights

This part demonstrates several important concepts in modern language model training:

1. **Dataset Scale Matters**: The larger poetry dataset (20x bigger) significantly improved text quality and diversity.

2. **Data Cleaning is Critical**: Proper handling of line terminators and text formatting is essential for good training data.

3. **Pretraining + Finetuning Works**: The two-stage approach of pretraining on a large diverse corpus, then finetuning on a specific domain, produces better results than training on the target domain alone.

4. **Overfitting Remains a Challenge**: Even with more data, the model still shows signs of overfitting, indicating we need even more data or better regularization.

5. **Modern Tokenization Benefits**: Subword tokens produce more natural-looking text even when the model isn't perfectly trained.

The progression from character-level Shakespeare generation to subword-level poetry and then Shakespeare finetuning shows the power of scaling both model size and training data in the right sequence.

Building GPT from Scratch - Part 6: Pretraining and Finetuning

*This is Part 5 of our GPT from scratch series. Having built and optimized our transformer in Parts 1-4, we now upgrade from character-level tokenization to modern subword tokenization using tiktoken.*

## What is Tokenization?

**Tokenization** is the process of breaking text into smaller units called *tokens*. A *tokenizer* handles this task and assigns each token a unique integer ID from its vocabulary. Andrej Karpathy gives a great overview of tokenization in [this lecture](https://www.youtube.com/watch?v=zduSFxRajkE).

## Tokenizer Vocabulary

The tokenizer's vocabulary is the full set of tokens it knows how to process. Tokenizers are *trained* on large text corpora—though this training differs from training a neural network. You can even train your own tokenizer and control its vocabulary size and tokenization rules.

## How Strings Are Tokenized

In English, tokens usually range from single characters to whole words—like `"t"` or `" great"`. Different languages and models may result in longer or shorter tokens. Most modern LLMs use *subword* tokenization methods, which balance vocabulary size with coverage of possible inputs. Some common tokenization algorithms include **Byte Pair Encoding (BPE)**, **WordPiece** and **Unigram**.

## Why Tokenization Matters

If your input text includes tokens *not* in the tokenizer's vocabulary, things can go wrong—often in frustrating or subtle ways. Many quirks and failures in LLM behavior can be traced back to poor or unexpected tokenization.

> Tokenization issues are a major source of bugs in LLM applications.

## Using Tiktoken

We'll use [`tiktoken`](https://github.com/openai/tiktoken/blob/main/README.md), OpenAI's fast, open-source tokenizer. Given a text (e.g., `"tiktoken is great!"`) and an encoding (e.g., `"cl100k_base"`), it produces a tokenized output like:

```python
["t", "ik", "token", " is", " great", "!"]
```

You can test tokenizations interactively using:

- [**OpenAI Tokenizer Tool**](https://beta.openai.com/tokenizer)
- [**Tiktokenizer Web App**](https://tiktokenizer.vercel.app/)

This is as simple as letting

```python
enc = tiktoken.encoding_for_model("gpt-2")
```

and then calling `enc.encode` and `enc.decode`.

## Scaling Challenges with Larger Vocabularies

Let's try `o200k_base`, which is used in `gpt-4o`. The vocab size is now 200,000 and our number of parameters in the model jumps to 164M! This is because the embedding and unembedding layers scale with the vocab size. Since `o200k_base` is too big, let's try the GPT-2 model which has a vocab size of around 50,000. Our model is now reduced to 50M parameters, still much bigger than what we had previously, with most of the weights in the embedding and unembedding layers. 

Currently, our dataset that we train on is small, so this is overkill. Most tokens probably don't even appear! We can see this disparity when we try to train the model:

```
step 0: train loss 10.8323, val loss 10.8315
step 500: train loss 4.8271, val loss 5.3269
step 1000: train loss 3.9008, val loss 4.8286
step 1500: train loss 3.3493, val loss 4.7968
step 2000: train loss 2.7602, val loss 4.9501
step 2500: train loss 2.1606, val loss 5.2748
step 3000: train loss 1.4815, val loss 5.7240
```

The validation loss is terrible, i.e. next token prediction is bad. Note that our tokens no longer correspond to individual characters. The training loss keeps going down since we now have an even more extreme example of overfitting to the data with increased parameters. 

## Improved Text Generation

Interestingly, the output is still quite readable since tokens are sub-word and not character level, showing the advantage of an increased vocabulary:

```
what stay to-morrow! I must desire her:
Hard shrowa thousand pound a book
Our friend. The one cockues!

FLORDS:
Ay, good testimony, come.

Servant:
My lord!

MIONwell, my lord, the heavens to cry't: look down,
And occupations speak again, I do come on,
Is't speak to see you. All my lord--
Alack, your lords! my lords, who, I have
Upon my heart:--your convey him, my lord,
Mowius; from my blood against your hand,
Out on your stables there! O injurious sin!

ISABELLHORIZELO:
Stay, good heaven;
My soul is some good's liver for her true self,
Whose settled good edencing trade here.
```

Despite the overfitting issues, the subword tokenization produces more natural-sounding text compared to our character-level approach. Words are more complete and the overall flow is more coherent.

## Key Takeaways

We have improved the tokenization part of our model, but we need to train on a larger dataset to properly utilize the expanded vocabulary. The main insights from this upgrade:

1. **Modern tokenization** uses subword algorithms that balance vocabulary efficiency with coverage
2. **Vocabulary size directly impacts model parameters** - larger vocabularies mean more embedding parameters
3. **Dataset size must match vocabulary size** - small datasets can't effectively train large vocabularies
4. **Subword tokens produce more natural text** even when the model is overfitting

In the next section, we'll address the dataset limitation by training on a much larger corpus to properly leverage our improved tokenization.

Building GPT from Scratch - Part 5: Tokenization

*This is Part 4 of our GPT from scratch series. Having built a complete transformer in Parts 1-3, we now look at further optimizations for more efficient training.*

## Restructuring and Configuration

Now that we have successfully implemented a transformer network, we can make a number of adjustments to optimize for more efficient training.

First, we restructure our script and put all hyperparameters into a config class:

```python
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
print(f'using device: {DEVICE}')

config = Config(
  batch_size = 64,
  block_size = 64,
  max_iters = 5000,
  eval_interval = 500,
  learning_rate = 3e-4,
  eval_iters = 10,
  n_embd = 128,
  n_head = 4,
  n_layer = 6,
  dropout = 0.2,
  vocab_size = vocab_size,
)

model = GPTLanguageModel(config, device=DEVICE)
model.to(DEVICE)
model = torch.compile(model) # requires PyTorch 2.0
```

As you can see, we also compile the model before training to improve training speed.

## Efficient Multi-Head Attention

The next big change we make is to compute the keys, queries, and values for the multiple heads all in a single matrix multiplication. This means we combine the 'Head' and 'Multi-Head' classes into one single class, which processes the heads as a new batch dimension.

```python
class MultiHeadAttention(nn.Module):
  """ multiple heads of self-attention in parallel """

  def __init__(self, config):
    super().__init__()
    self.n_head = config.n_head
    self.n_embd = config.n_embd
    self.dropout = config.dropout

    # key, query, value projections for all heads, but in a batch
    self.attn = nn.Linear(config.n_embd, 3 * config.n_embd, bias=False)
    self.proj = nn.Linear(config.n_embd, config.n_embd, bias=False)
    self.attn_dropout = nn.Dropout(config.dropout)
    self.resid_dropout = nn.Dropout(config.dropout)
    self.register_buffer("bias",
      torch.tril(torch.ones(1, 1, config.block_size, config.block_size)))

  def forward(self, x):
    B, T, C = x.size()
    q, k, v  = self.attn(x).split(self.n_embd, dim=2)
    # reshape and move head dimension forward using einops
    k = rearrange(k, 'b t (nh hs) -> b nh t hs', nh=self.n_head)
    q = rearrange(q, 'b t (nh hs) -> b nh t hs', nh=self.n_head)
    v = rearrange(v, 'b t (nh hs) -> b nh t hs', nh=self.n_head)

    att = (q @ k.transpose(-2, -1)) * k.size(-1)**-0.5
    att = att.masked_fill(self.bias[:,:,:T,:T] == 0, float('-inf'))
    att = F.softmax(att, dim=-1)
    att = self.attn_dropout(att)
    y = att @ v # (B, nh, T, T) x (B, nh, T, hs) -> (B, nh, T, hs)

    # re-assemble all head outputs side by side
    y = rearrange(y, 'b nh t hs -> b t (nh hs)')
    y = self.resid_dropout(self.proj(y))
    return y
```

## Flash Attention

Finally, we implement flash attention, which reorders the attention computation such that computations can be tiled which greatly speeds up the attention step.

```python
# In MultiHeadAttention class
self.flash = hasattr(torch.nn.functional, 'scaled_dot_product_attention')
# ... in forward pass ...
y = torch.nn.functional.scaled_dot_product_attention(q, k, v, 
                                                     attn_mask=None, 
                                                     dropout_p=dropout_p,
                                                     is_causal=True)
```

## Training Performance and Scaling

Training on a T4 GPU, a 1.2M parameters model now trains in 2 minutes. We are ready to scale up to the size shown in the tutorial video:

```python
config = Config(
  batch_size = 64,
  block_size = 256,
  max_iters = 5000,
  eval_interval = 500,
  learning_rate = 3e-4,
  eval_iters = 10,
  n_embd = 384,
  n_head = 6,
  n_layer = 6,
  dropout = 0.2,
  vocab_size = vocab_size,
)
```

This has 10 million parameters and trains in 30 minutes! Let's take a look at the output:

```
BRAKENBURY:
Despite of this seven sit in the noble
Of Lord Hastings, and my grave is my charged mine;
For George shall not speak not pass'd it:
The valour upon it. Is deliver'd it with me?

BRUTUS:
Yea, beggar, by your voices and hearts,
But since she changed in your packs and bloody,
Your joy your might in him writ
Be punk'd between pains. I am struckcomment
To son I writ you that yet you did love:
If it were example to your knees to express
By your fleships of state? Exeter, me
And wring, my son, come on, my sorrow ladys,
Whose profession joy wings and me down,
```

Pretty good! Most words are recognizable and the sentences have more structure. The overall form is also consistent with the training data with readable names (Brutus is a familiar one!)

## Bias-Variance Trade-off
If we plot the losses against training steps for our small 0.3M model and our final 10M model:
![Validation loss comparison](/images/training_validation_comparison.png)
The larger model generalizes better (has a lower validation loss) but overfits more (training loss is much lower than validation loss). This represents a classic **bias-variance trade-off** in neural scaling. The larger model's increased parameter capacity enables better feature representation learning and pattern recognition, resulting in lower validation error. However, this same capacity creates **memorization potential** for training data.

## Conclusion

These optimizations demonstrate how production transformer implementations differ from educational versions. The key improvements - configuration management, batched attention computation, model compilation, and flash attention - provide significant training speedups while maintaining the same underlying architecture.

The scaling results show that our simple transformer can achieve reasonable quality when given sufficient parameters and training time, following the same principles that power modern large language models.

Building GPT from Scratch - Part 4: Further Optimizations

*This is Part 3 of a 3-part series on building a Generative Pretrained Transformer from scratch. In Parts 1 and 2, we built the data pipeline and attention mechanism. Now we'll complete our transformer with the remaining key components.*

## Feed-Forward Networks

After our attention mechanism learns to communicate between tokens, we need to give the model computation time to process what it has learned. This is where feed-forward networks come in - they're simply MLPs with ReLU activation that allow the model to "think" about the information it just gathered.

```python
class FeedFoward(nn.Module):
    """ a simple linear layer followed by a non-linearity """

    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, n_embd),
            nn.ReLU(),
        )

    def forward(self, x):
        return self.net(x)
```

We add this to our model after the attention:

```python
# In our model:
self.sa_head = MultiHeadAttention(4, n_embd//4)
self.ffwd = FeedFoward(n_embd) # MLP with RELU

# In forward pass:
x = self.sa_head(x)
x = self.ffwd(x)
logits = self.lm_head(x)
```

## Transformer Blocks: Communication + Computation

The pattern of attention (communication) followed by feed-forward (computation) is so fundamental that we package it into reusable **Transformer Blocks**. These blocks can be stacked to create deeper networks that alternate between communication and computation phases.

```python
n_head = 4
n_layer = 2

class Block(nn.Module):
    """ Transformer block: communication followed by computation """

    def __init__(self, n_embd, n_head):
        # n_embd: embedding dimension, n_head: the number of heads we'd like
        super().__init__()
        head_size = n_embd // n_head
        self.sa = MultiHeadAttention(n_head, head_size)
        self.ffwd = FeedFoward(n_embd)

    def forward(self, x):
        x = self.sa(x)
        x = self.ffwd(x)
        return x
```

We can then stack multiple blocks:

```python
# In our model:
self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])

# In forward pass:
x = tok_emb + pos_emb # (B,T,C)
x = self.blocks(x)
logits = self.lm_head(x)
```

At this point, our model starts generating recognizable English words!

```
RILELE
MAssell use you.
Nt I RYan lake off.

Ficink'd
hote carewtledeche quie to whanl Gatt Mejesery ely:
The if, bet leveses it theave be ry skit you file.

Kay wred tome dake stance, suks,
Adech JORo!

ALOUSBET:
Wis brake grourst and ald creapsss,
Andite
noat,
Amothery are doreast is
```

## Training Optimizations

As our network gets deeper with multiple transformer blocks, we need optimizations to improve training stability and performance.

### Residual Connections

The first optimization is **residual/skip connections**, introduced in the [Deep Residual Learning for Image Recognition](https://arxiv.org/abs/1512.03385) paper. These connections allow gradients to flow directly through the network, avoiding training bottlenecks in deep networks.

Instead of 'x = self.sa(x)', we do 'x = x + self.sa(x)'. The computation now returns a residual that gets added to the original input:

```python
class Block(nn.Module):
    def forward(self, x):
        x = x + self.sa(x)  # residual connection
        x = x + self.ffwd(x)  # residual connection
        return x
```

### Projection Layers

We also add projection layers to map back to the embedding space after our multi-head attention and expand our feed-forward networks as in the original paper:

```python
class MultiHeadAttention(nn.Module):
    """ multiple heads of self-attention in parallel """

    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
        self.proj = nn.Linear(head_size * num_heads, n_embd)

    def forward(self, x):
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.proj(out)
        return out

class FeedFoward(nn.Module):
    """ a simple linear layer followed by a non-linearity """

    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),  # expand by 4x as in the paper
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),  # project back down
        )

    def forward(self, x):
        return self.net(x)
```

### Layer Normalization

Layer normalization helps stabilize training by normalizing inputs to have zero mean and unit variance. We apply it before both the self-attention and feed-forward computations in each block, and once more after all blocks:

```python
class Block(nn.Module):
    """ Transformer block: communication followed by computation """

    def __init__(self, n_embd, n_head):
        super().__init__()
        head_size = n_embd // n_head
        self.sa = MultiHeadAttention(n_head, head_size)
        self.ffwd = FeedFoward(n_embd)
        self.ln1 = nn.LayerNorm(n_embd)
        self.ln2 = nn.LayerNorm(n_embd)

    def forward(self, x):
        x = x + self.sa(self.ln1(x))
        x = x + self.ffwd(self.ln2(x))
        return x
```

### Dropout

Finally, we add **dropout** for regularization, which randomly sets some activations to zero during training to prevent overfitting:

```python
class FeedFoward(nn.Module):
    def __init__(self, n_embd):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_embd, 4 * n_embd),
            nn.ReLU(),
            nn.Linear(4 * n_embd, n_embd),
            nn.Dropout(dropout),  # add dropout
        )

class MultiHeadAttention(nn.Module):
    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])
        self.proj = nn.Linear(head_size * num_heads, n_embd)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        out = torch.cat([h(x) for h in self.heads], dim=-1)
        out = self.dropout(self.proj(out))
        return out
```

## Final Model and Hyperparameters

We now have a complete decoder-only transformer ready to scale and train! Here are the hyperparameters for our final model:

```python
# hyperparameters
batch_size = 32
block_size = 32
max_iters = 5000
eval_interval = 500
learning_rate = 3e-4
eval_iters = 200
n_embd = 64
n_head = 6
n_layer = 6
dropout = 0.2
```

## Final Results

After training our complete transformer, we get much more structured output:

```
DOF GlLN:
They good, then then, bladgy bone not thindnes
I way Jeain, fainly!

ISABELLA:
I way, thou fourd to havary flown dews'm-sine.

NowZALLET:
Which here old thy's warring hiod
On dearys hory be wive to more; greseli!

But nighd Wart, prance:
Barch,
And prayem not welld she you, coldinger:
O I the oldst God somed:
Sirry is let never To be whith, I new'n be thy limpiny
Word: where deblitiss for give upon the conqueennifult,
And so pobeterl. by they she thy truge,
If you his let a brotess.
```

While still not perfect, the improvement is dramatic! The model now:
- Uses proper character names (ISABELLA, etc.)
- Maintains somewhat consistent dramatic structure
- Shows more coherent word formation
- Demonstrates longer-range dependencies

## What We've Accomplished

In this three-part series, we've built a complete transformer from scratch:

1. **Part 1**: Data loading, tokenization, and a simple bigram baseline
2. **Part 2**: The attention mechanism - the core innovation of transformers
3. **Part 3**: Complete transformer architecture with all the essential components

### Key Components We've Implemented:

- **Self-attention mechanisms** for learning relationships between tokens
- **Multi-head attention** for capturing different types of relationships
- **Feed-forward networks** for computation after communication
- **Transformer blocks** that stack attention and computation
- **Residual connections** for stable training of deep networks
- **Layer normalization** for training stability
- **Positional embeddings** so the model understands sequence order
- **Dropout** for regularization and preventing overfitting

## The Path Forward

Our small transformer demonstrates the core principles, but to achieve GPT-level performance, you'd need to scale up significantly:

- **More parameters**: Modern language models have billions of parameters
- **More data**: Training on much larger text corpora
- **More compute**: Training for weeks or months on powerful hardware
- **Better tokenization**: Using subword tokenizers like BPE or SentencePiece
- **Longer context windows**: Supporting much longer sequences

## Key Takeaways

1. **Attention is the core innovation**: The self-attention mechanism allows tokens to communicate and share information based on content rather than just position.

2. **Transformers are surprisingly simple**: The architecture is just attention + feed-forward blocks stacked together with some normalization and residual connections.

3. **Scale matters**: The same architecture that gives us semi-coherent Shakespeare can generate human-level text when scaled up with more parameters, data, and compute.

4. **Training stability is crucial**: Residual connections, layer normalization, and proper weight initialization are essential for training deep networks.

## Conclusion

This tutorial has shown how to build a transformer model from scratch using PyTorch. We've implemented all the key components of the decoder transformer architecture and seen how they work together to create a language model capable of generating structured text.

The progression from random gibberish (bigram model) to semi-coherent Shakespeare-like text (full transformer) demonstrates the power of the attention mechanism and proper architectural choices. While our small model doesn't generate fully coherent text yet, scaling up the parameters, training data, and computation would lead to increasingly capable language models.

The complete nanoGPT implementation, with additional optimizations and the ability to train larger models, can be found in [Andrej Karpathy's repository](https://github.com/karpathy/nanoGPT). This serves as an excellent foundation for understanding and experimenting with transformer architectures.

Building GPT from Scratch - Part 3: Building the Complete Transformer

*This is Part 2 of a 3-part series on building a Generative Pretrained Transformer from scratch. In Part 1, we built a simple bigram model. Now we'll introduce the core innovation that makes transformers work: Attention.*

## The Mathematical Foundation: Weighted Aggregation

Before diving into attention, let's understand a neat mathematical trick that forms its foundation. By multiplying a row-normalized T×T lower triangular matrix with a T×C matrix, we can get a T×C matrix which calculates a running average of the preceding rows.

When we combine this with softmax (setting -inf instead of 0), we get a more general weighted aggregation of past elements:

```python
# Weighted aggregation with softmax
torch.manual_seed(1337)
B,T,C = 4,8,2
x = torch.randn(B,T,C)

tril = torch.tril(torch.ones(T,T)) # lower triangular
wei = torch.zeros((T,T))
wei = wei.masked_fill(tril==0, float('-inf'))
wei = F.softmax(wei, dim=-1)
res = wei @ x
res.shape
```

This is the mathematical foundation of attention - we're learning to compute weighted averages of past tokens, where the weights are determined by the content of the tokens themselves.

## Single Head Self-Attention

Now for the meat of transformers: the attention mechanism. We create an attention space of dimension 'head_size'. We project each token into this space using three matrices: **Key (K)**, **Query (Q)**, and **Value (V)**.

Here's the intuition:
- **Query (Q)**: What the token is looking for
- **Key (K)**: What the token contains/offers
- **Value (V)**: What the token communicates if paid attention to

When K and Q are similar (measured using dot product), they have high affinity and receive higher attention weights. We don't aggregate tokens directly - we aggregate V(X), which represents what each token communicates when attended to.

The normalization by 1/√(head_size) is crucial. When Q and K have unit variance, this ensures the attention weights also have unit variance. Without this normalization, softmax would saturate towards one-hot vectors, meaning tokens would only aggregate information from a single previous token.

```python
class Head(nn.Module):
    """ one head of self-attention """

    def __init__(self, head_size):
        super().__init__()
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))

    def forward(self, x):
        # input of size (batch, time-step, channels)
        # output of size (batch, time-step, head size)
        B,T,C = x.shape
        k = self.key(x)   # (B,T,hs)
        q = self.query(x) # (B,T,hs)
        # compute attention scores ("affinities")
        wei = q @ rearrange(k, 'b t h -> b h t') * C**-0.5
        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf')) # (B, T, T)
        wei = F.softmax(wei, dim=-1) # (B, T, T)
        # perform the weighted aggregation of the values
        v = self.value(x) # (B,T,hs)
        out = wei @ v # (B, T, T) @ (B, T, hs) -> (B, T, hs)
        return out
```

Note that we register 'tril' as a buffer since it's not a learnable parameter of the model.

## Important Notes About Attention

Here are some crucial points about how attention works:

- **Note 1**: Attention is a communication mechanism. You have a directed graph and you want every node to take weighted sums of every node pointing to it.

- **Note 2**: Positional encoding must be embedded in the nodes, since the structure of the attention mechanism doesn't inherently understand position.

- **Note 3**: Batches are independent - there's no cross-communication between different sequences in a batch.

- **Note 4**: Here we ignore future tokens using triangular masking. If you were doing sentiment analysis, you might want to allow this communication. Encoder blocks would allow bidirectional attention, decoder blocks would not. The difference is just the triangular masking.

- **Note 5**: This is called "self-attention" because K, Q, V all come from the same input X. If the keys and values came from somewhere else, you'd have "cross-attention", which is used when there's a separate source of information to pull from.

## Building the Complete Single-Head Model

Let's put everything together into a complete model with single-head attention. We'll make several improvements to our training setup:

```python
# hyperparameters
batch_size = 32
block_size = 8 
max_iters = 5000
eval_interval = 500
learning_rate = 1e-3
device = 'cuda' if torch.cuda.is_available() else 'cpu'
eval_iters = 200
n_embd = 32
torch.manual_seed(1337)

# data loading
def get_batch(split):
    # generate a small batch of data of inputs x and targets y
    data = train_data if split == 'train' else val_data
    ix = torch.randint(len(data) - block_size, (batch_size,))
    x = torch.stack([data[i:i+block_size] for i in ix])
    y = torch.stack([data[i+1:i+block_size+1] for i in ix])
    x, y = x.to(device), y.to(device)
    return x, y

@torch.no_grad()
def estimate_loss():
    out = {}
    model.eval()
    for split in ['train', 'val']:
        losses = torch.zeros(eval_iters)
        for k in range(eval_iters):
            X, Y = get_batch(split)
            logits, loss = model(X, Y)
            losses[k] = loss.item()
        out[split] = losses.mean()
    model.train()
    return out
```

Key improvements in this version:

1. **GPU Support**: We move data and model to GPU using 'device = 'cuda''
2. **Better Loss Estimation**: 'estimate_loss()' averages losses over multiple batches for more accurate estimates
3. **No Gradient Context**: '@torch.no_grad()' tells PyTorch this function won't be backpropagated through
4. **Embedding Dimension**: Our logits are now embedded in an embedding dimension, so we need an "unembedding" linear layer
5. **Positional Embeddings**: We add position information so tokens know where they are in the sequence

```python
class GPTLanguageModel(nn.Module):

    def __init__(self, vocab_size):
        super().__init__()
        # each token directly reads off the logits for the next token from a lookup table
        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)
        self.position_embedding_table = nn.Embedding(block_size, n_embd)
        self.sa_head = Head(n_embd)
        self.lm_head = nn.Linear(n_embd, vocab_size)

        # better init, not covered in the original GPT video, but important
        self.apply(self._init_weights)
    
    def _init_weights(self, module):
        if isinstance(module, nn.Linear):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)
            if module.bias is not None:
                torch.nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)

    def forward(self, idx, targets=None):
        B, T = idx.shape

        # idx and targets are both (B,T) tensor of integers
        tok_emb = self.token_embedding_table(idx) # (B,T,C)
        pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)
        x = tok_emb + pos_emb # (B,T,C)
        x = self.sa_head(x)
        logits = self.lm_head(x)

        if targets is None:
            loss = None
        else:
            B, T, C = logits.shape
            logits = logits.view(B*T, C)
            targets = targets.view(B*T)
            loss = F.cross_entropy(logits, targets)

        return logits, loss

    def generate(self, idx, max_new_tokens):
        # idx is (B, T) array of indices in the current context
        for _ in range(max_new_tokens):
            # crop idx to the last block_size tokens
            idx_cond = idx[:, -block_size:]
            # get the predictions
            logits, loss = self(idx_cond)
            # focus only on the last time step
            logits = logits[:, -1, :] # becomes (B, C)
            # apply softmax to get probabilities
            probs = F.softmax(logits, dim=-1) # (B, C)
            # sample from the distribution
            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)
            # append sampled index to the running sequence
            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)
        return idx
```

## Results with Single-Head Attention

After training, we get improved results that show more structure, though still not quite Shakespeare:

```
Arrg hor; ho, ho aak.

LUCKUCARE Fqhelree ndel te:
Thece waplt ko.
FOCI
MHABRIEKY tur waverer orid betievis dyof b.

Bo,
ABIUCE:
N-, evesune athid nt cobasr!, Go hern, alsemin rsin varit ther I;
ANher en:ouingaroua lis py Bh mithe ast prird band bad youun theis pioat hed man ile ere sty hanonoue avillars d ty hon I:
So Ilar chy w'guma my tono yat d,
I more hathive, ha utithe baag tee
Iile athadog at y hoke hay hate whinsintoourtarenol usdon co whe sou; mer wif yoh hlele coud illesen?o I choteban
```

The attention mechanism has given our model the ability to look back at previous characters and make more informed predictions. We can see more English-like structure emerging.

## Multi-Head Attention

Instead of using a single attention head, we can use multiple smaller heads in parallel. This allows the model to attend to different types of relationships simultaneously - some heads might focus on syntax, others on semantics, etc.

```python
class MultiHeadAttention(nn.Module):
    """ multiple heads of self-attention in parallel """

    def __init__(self, num_heads, head_size):
        super().__init__()
        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)])

    def forward(self, x):
        return torch.cat([h(x) for h in self.heads], dim=-1)
```

In our model, we replace the single head with:
```python
self.sa_head = MultiHeadAttention(4, n_embd//4)
```

This gives us slightly better loss and more diverse attention patterns.

## What's Next?

We now have a working attention mechanism! However, our model is still missing several key components that make modern transformers work well:

- **Feed-forward networks** for computation after attention
- **Transformer blocks** that repeat the attention + computation pattern
- **Residual connections** and **layer normalization** for training stability
- **Dropout** for regularization

In Part 3, we'll add these final components and see our model start generating much more coherent text. The attention mechanism we've built here is the foundation - everything else builds on top of it to create the full transformer architecture.

Building GPT from Scratch - Part 2: The Attention Mechanism


*This is Part 1 of a 3-part series on building a Generative Pretrained Transformer from scratch, based on Andrej Karpathy's excellent tutorial.*

## Introduction

The goal of this series is to build a 'Generative Pretrained Transformer' from scratch, following the attention architecture from the original 2017 [**Attention is all you need**](https://arxiv.org/abs/1706.03762) paper and OpenAI's GPT-2. We'll be building a character-level generative model - essentially a next character predictor that can generate Shakespeare-like text.

This tutorial is based on Andrej Karpathy's video [**Let's build GPT: from scratch, in code, spelled out**](https://www.youtube.com/watch?v=kCc8FmEb1nY&t=10s), which has gained 6 million views in 2 years! Transformers get their name from their original use case of machine translation.

## Data Loading

The first step in building any language model is preparing our data. We'll use the tiny Shakespeare dataset for this tutorial.

```python
# Download tiny shakespeare dataset
!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt

# read and inspect the file
with open('input.txt', 'r') as f:
  text = f.read()
print("Length of dataset in characters: ", len(text))
print(text[:1000])
```

### Tokenization

The first interesting concept is **tokenization**. We want to convert our input text into integers that our model can work with. The model will predict the next integer, which we can then decode back into a character.

Our simple approach creates a dictionary of all available characters (including newline) and assigns each an integer. For more sophisticated treatments, you'd want to look at SentencePiece by Google or tiktoken by OpenAI. There's always a tradeoff between dictionary size and encoding lengths.

```python
chars = sorted(list(set(text)))
vocab_size = len(chars)
print(f"{vocab_size} characters in dictionary : {''.join(chars)}")
# First one is a new line!

# Create a mapping from characters to integers (A Tokenizer)
# This is called tokenizing. Google uses SentencePiece. OpenAI and GPT uses tiktoken
itos = { i:ch for i, ch in enumerate(chars)}
stoi = { ch:i for i, ch in enumerate(chars)}
encode = lambda s: [stoi[c] for c in s] # encoder: convert string to list of integers
decode = lambda l: ''.join([itos[i] for i in l]) # decoder: convert list of integers to string

# Now encode the whole text dataset and store it as a pytorch Tensor
import torch
data = torch.tensor(encode(text), dtype=torch.long)
print(data.shape, data.dtype)
print(data[:100])
print(decode([int(v) for v in (data[:100].data)]))

# Split the data into training and validation sets
n = int(0.9*len(data))
train_data = data[:n]
val_data = data[n:]
```

### Block Size and Context Windows

The second important concept is **block size**, which sets the context window length. This is the size of the 'memory' of the model. A larger block size would give more long-range associations, but is probably more expensive to train and would require more data.

With block size defined, we can input batch_size rows of block_size integers for each data batch. Training in batches is faster since multiple independent blocks can be processed simultaneously.

```python
block_size = 8 # or alternatively context lengths
train_data[:block_size+1]
# in a sample of 9 characters, we have 8 examples with different context lengths.

torch.manual_seed(1337)
batch_size = 4
block_size = 8

def get_batch(split):
  # generate a small batch of data of inputs x and targets y
  data = train_data if split == 'train' else val_data
  ix = torch.randint(len(data) - block_size , (batch_size,)) # should have a -1
  x = torch.stack([data[i:i+block_size] for i in ix])
  y = torch.stack([data[i+1:i+1+block_size] for i in ix])
  return x, y
```

## The Bigram Model

Now let's implement our first model - a simple bigram model. This model predicts the next character based only on the current character (context length of 1) and stores probabilities for the next character.

We use a log-likelihood/cross-entropy loss. Initially, we expect the loss to be around ln(1/65) ≈ 4.17, since we have 65 characters and initially each character should be equally likely.

```python
import torch.nn as nn
from torch.nn import functional as F
torch.manual_seed(1337)
from einops import rearrange

class BigramLanguageModel(nn.Module):

  def __init__(self, vocab_size):
    super().__init__() # initialize the parent class Module
    # each token directly reads off the logits for the next token from a lookup table
    self.token_embedding_table = nn.Embedding(vocab_size, vocab_size)

  def forward(self, idx, targets=None):
    # idx and targets and both (B,T) tensors of integers
    logits = self.token_embedding_table(idx) # (B,T,C) batch time channel

    if targets is None:
      loss = None
    else:
      logits = rearrange(logits,'b t c -> (b t) c')
      targets = rearrange(targets, 'b t -> (b t)')
      loss = F.cross_entropy(logits, targets)

    return logits, loss

  def generate(self, idx, max_new_tokens):
    # idx is (B, T) array of indicies in the current context
    for _ in range(max_new_tokens):
      # get the predictions
      logits, loss = self(idx)
      # focus only on last time step
      logits = logits[:, -1 , :] # becomes B x C
      # apply softmax to get probabilties
      probs = F.softmax(logits, dim=1) # becomes B x C
      # sample from distribution
      idx_next = torch.multinomial(probs, num_samples=1) # B x 1
      # append sampled index to running sequence
      idx = torch.cat((idx, idx_next), dim=1) # B x T+1
    return idx

m = BigramLanguageModel(vocab_size)
logits, loss = m(xb, yb)
print(logits.shape)
print(loss)

idx = torch.zeros((1,1), dtype=torch.long)
print(decode(m.generate(idx, max_new_tokens=100)[0].tolist()))
```

Note that for PyTorch, when we subclass 'nn.Module', calling the model (including 'self') actually calls the 'forward' method!

## Training the Bigram Model

We can train our model by choosing an **optimizer** and a **learning rate**. A good learning rate is typically 1e-3, but for smaller networks you can get away with higher rates.

```python
# training
batch_size = 32

optimizer = torch.optim.AdamW(m.parameters(), lr=1e-3)

for steps in range(10000):
  # sample a batch of data
  xb, yb = get_batch('train')

  logits, loss = m(xb, yb)
  optimizer.zero_grad(set_to_none=True)
  loss.backward()
  optimizer.step()

print(loss.item())
```

## Results

Unfortunately, the results using the bigram model aren't great:

```
Iyoteng h hasbe pave pirance
Rie hicomyonthar's
Plinseard ith henoure wounonthioneir thondy, y heltieiengerofo'dsssit ey
KIN d pe wither vouprrouthercc.
hathe; d!
My hind tt hinig t ouchos tes; st yo
# Not quite Shakespeare
```

The bigram model is too simple - it can only look at one character back to predict the next. We need something more sophisticated that can consider longer context windows and learn more complex patterns.

## What's Next?

In the next part of this series, we'll introduce the key mathematical trick that makes transformers possible: the attention mechanism. We'll start with a mathematical foundation for weighted aggregation and build up to single-head self-attention.

The bigram model gives us a solid foundation to build upon, and more importantly, it establishes our training pipeline and data handling. Now we're ready to dive into the real magic of transformers!

Building GPT from Scratch - Part 1: Data Loading and the Bigram Model


This is a pretty long paper, and is divided into 12 sections (although the last 3 are discussion, related work and comments).

## Section 1: Background & Motivation

The first section provides an introduction and summary of the main results in this paper. They begin by mentioning that it would be nice if each neuron had a one to one correspondence with a specific feature such as dog’s snout. This has been shown empirically to sometimes happen, but this is not always the case especially in LLMs. So the motivating question is when and why this happens (or doesn’t happen). By using toy models which are small ReLU networks trained to learn five features of varying importance and varying sparsity, they show that models can store more features than they have dimensions (**superposition**), and they can even perform computations in this superposed state. They hypothesize that the small networks simulate large sparse networks.

This seems like an interesting way to view superposition —- as a way of compressing information through continuous representation, but also as a computation mechanism (cue the name superposition and some possible relation to QM). When they say `no interference', this is what I think of as a circuit, with each neuron activating on, and only on, specific features.

They go on to say the main contribution is that they directly demonstrate superposition occurs relatively natural and offer a theory (phase diagram!) of when this happens. They make lots of parallels to physics phenomena, which I am excited to read given my physics background. There is an interesting note for example, that superposition organizes features into geometric structures, which immediately signals to be some kind of energy (loss function) minimizing geometries. They have not defined **sparsity** yet, although they discuss it, but I’m sure that is coming next. 

## Section 1b: Definitions and Motivation: Features, Directions, and Superposition

The next part of this section kicks off by discussing the Linear Representation Hypothesis ([see here](https://arxiv.org/abs/2311.03658)), which is the idea that certain directions in a representation space correspond to specific high level concepts, in other words that they carry semantic meaning. The claim that these are linear also seems important since that makes them easy to compose and decompose, and even to do vector arithmetic. This is more an empirical observation, e,g, see Mikolov et al for a famous example in word embeddings.

In their words, ‘features of the input’ are captured as ‘directions in activation space’. If we can figure out these vector representations, we can decompose activations into something we can more easily understand — as a basis of concepts each activated with a certain strength. 

One natural basis in a neural network by virtue of its structure is having each neuron correspond to one feature. Naturally this would be limited to the number of neurons you have (This is a privileged basis, and there are incentives to used this basis such as activation functions, but this doesn’t mean that the basis will be used)! However, if you have more features than neutrons, a different strategy is required. This is where the idea of superposition comes in, where the features are captured by a certain direction in the high dimensional input space. I assume this is akin to creating a non-orthogonal basis. 

There is some interesting discussion of what actually constitutes a feature, since `concept' is a pretty vague word. They settle on defining it as a property which will be an individual neuron in a sufficiently large network for the same problem. Does this mean that any neural network that exhibits superposition has an equivalent much larger neural network that works in the basis where each neuron is one feature? What implications does that have for discrete valued/low bit neural networks? What are the trade offs for being able to do this network compression?

Next, a strong case is made for linear representations, of which I like the argument that linear representations make features linearly accessible, meaning that since neural network layers are linear functions followed by a non-linearity, a feature must be linear in order for a neuron to consistently excite or inhibit on that feature in a single step. 

Lastly, there is more discussion on **the superposition hypothesis**, which I think is the core of this paper.  Basically, if a neural network needs to represent more features than there are neurons, features can’t align with the basis and hence polysemanticity (single neuron represents to multiple geatures) is unavoidable. While you can only have n orthogonal vectors in n dimensions, you can have an exponential exp(n) almost orthogonal vectors. So for large-n, this would be very efficient at cramming more features in than neurons! The cost is that each feature activating creates some small amount of noise by making it looks as some other features slightly activating. 

The other big idea here is that sparsity is important. From compressed sensing, it is generally difficult to reconstruct a high dimensional vector projected into a low dimensional space since information is lost, but it is possible if you can assume the original vector is sparse. So sparsity here means that activations don’t happen together, and that means this `noise' is minimized. If the activations are not sparse, multiple present features will interfere.

Again the hypothesis is that small networks with polysemanticity are low dimensional projections of larger networks. The requirements for this to be effective seem pretty strict, so it would nice to be able to better quantify them and to understand how important this is in most networks.

Mech Interp: Toy Models Part I


With so much to look at, it is difficult to decide where to start, but it is probably best to just start somewhere. To start off, I think I will read [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html) (Elhage et al 2022). I remember really liking the findings and approach of the paper, since trying to understand simple toy models seems like a very intuitive approach to me. I have found this very useful even in my own research, where instead of directly trying to understand the results of large simulations with lots of moving pieces, we try to understand much simple toy model simulations to build a scaffold for understanding the complex behavior we observe. Since the work deals with toy models, I should also be able to play around and run some of these experiments fairly easily. This should also be a natural start to understanding sparse autoencoders (SAEs) since they seem to be the current main approach towards untangling this problem of superposition.

Before that I would like to read some resource that gives a higher level view of the history of the field and where it is going, to get a better sense of the bird’s eye view landscape. At the same time I am keeping a list of concepts I want to learn more about as I come across them, and hopefully this list does not get too long.

I decided to begin by reading [this blog post](https://www.alignmentforum.org/posts/beREnXhBnzxbJtr8k/mech-interp-is-not-pre-paradigmatic) on the alignment forum by Lee Sharkey titled “Mech interp is not pre-paradigmatic”. In particular, since I am looking for some initial paradigm in which to frame mech interp research, the title of this post indicated it would present some form of such a framework. So as the title suggests, this post basically begins by arguing that the field of mech interp is **not** in what might be called a pre-paradigmatic phase, and instead is actually at the mature stage of a second mini-paradigmic wave. Pre-paradigmatic here refers to a stage in the development of a scientific field before it has established a dominant theoretical framework or "paradigm." This concept comes from philosopher Thomas Kuhn's influential work "The Structure of Scientific Revolutions”, where he outlines how a field progresses through these stages. While some argue mech interp, being a relatively new field, is in this pre-paradigmatic phase, this post pushes the idea that mech interp actually begun as an offshoot of computational neuroscience and hence inherits many of the ideas and concepts established there. In fact, the argument is made that it has been rediscovering many of the same ideas explored in neuroscience! I certainly lack sufficient knowledge in either area to judge this statement, but I don’t find this claim surprising given the parallels. Further, given the greater tractability of working with neural networks, perhaps progress might actually eventually flow in the other direction! I think the worry lies in the fact that neuroscience seems to be struggling to make recent progress and that mech interp might reach a similar wall. The article suggests the following **Three Waves of Mech Interp**:

1. **First Wave** (2010s): Focused on demonstrating that interpretable structure exists in deep neural networks. 'Features are the fundamental unit of neural networks' and 'features are connected by weights, forming circuits' . This wave ended when researchers discovered polysemantic neurons (neurons that respond to multiple unrelated concepts).
2. **Second Wave** (2022-present): Emerged after the "Toy Models of Superposition" paper, introducing sparse dictionary learning (SDLs) to address polysemanticity. However, this wave now faces its own anomalies.
3. **Potential Third Wave?** The post suggests "Parameter Decomposition" as a promising approach that could resolve Second-Wave anomalies by decomposing neural network parameters into interpretable components representing computational mechanisms. Worth nothing that this is what the author is working on at present.

At the same time, I think the transition from first to second wave lines up with the rise of LLMs, since much early work was done with CNNs. Much larger LLMs clearly showed much richer representation and required understanding polysemanticity and superposition. As for parameter decomposition, I will have to come back to this again since I don’t understand enough to appreciate its arguments. This outline also makes me feel like the Toy Models paper is indeed the best place to start if it marks this big transition in the thinking in the field. I think it would be good for me to try and define mech interp, superposition and polysemancity, perhaps after going through the paper, and for example think about this particular quote from the post:

> The idea that 'networks represent more features than they have neurons'. It is a natural corollary of the superposition hypothesis that neurons would exhibit polysemanticity, since there cannot be a one-to-one relationship between neurons and 'features'.

The article also presents a list of anomalies in this Second Wave that I would like to return to once I have built more understanding, since these should be the open problems the field is looking at now. While this post is just one perspective, I think it is a sufficiently good mental framework to start with, with lots of references and ideas to think about. Alright, onwards to Toy Models!

Mech Interp: Paradigms


## Starting my Mech Interp Journey

Change is never easy, and stepping away from astronomy after many years feels like giving up part of my identity. I don't regret my decision to pursue graduate school, those years have been nothing but fulfilling. It's a great privilege to dedicate time and effort to a pursuit of such intellectual purity. However, at the end of graduate school, one must once again cast thoughts toward the future.
Remaining on the academic path brings uncertainty in many areas of life and often demands personal sacrifices: constant relocation, distance from loved ones, evolving responsibilities as teaching loads and funding pressures mount. Balancing these demands with other personal priorities is far from easy, and I found myself unable to envision navigating this path happily. I've also always been acutely aware that while astronomy is fascinating, there are infinite equally interesting areas of study and work—many of more immediate relevance to humanity.
While changing trajectory has been difficult, deciding what to do next has been even more challenging. One area I'm strongly interested in pursuing is mechanistic interpretability.

## The Rise of Neural Networks

Over the years, I've observed the explosion of machine learning, or more accurately, the rise of neural networks, deep learning, and large language models as they evolved from esoteric topics to universal adoption, taking root in all aspects of society. The social implications are tremendous, and it really does appear to be a watershed moment in how humans interact with technology.
Yet so much remains mysterious about how these networks actually work. We train these systems on vast amounts of data, but their resultant capabilities have repeatedly exceeded expectations while theoretical understanding struggles to catch up. This reminds me of complex systems that display emergent behavior even with simple, local rule-based evolution. We know the full state of a trained neural network (every weight, bias, and computation that flows through it), but its overall capabilities still baffle us.
Information is being processed in ways that seem opaque to us, so a field has crystallized around making sense of this opacity in ways we can understand and interpret. This is mechanistic interpretability (or "mech interp"), the science of understanding how machine learning networks learn to process information, much like neuroscience tries to understand how the biological brain does something similar.
While young, this field is moving at lightning speed. I strongly believe that progress in mech interp is among the most important research being done today, given the reach and rate of growth of this technology. Preventing these systems from pursuing unintended goals (known as AI alignment) surely requires developing an understanding of how these networks do what they do.

## A New Experience

While I've maintained interest in machine learning and neural networks over the years, my direct experience has been limited. The applications to my research were never convincing or promising enough (a lack of interpretability makes neural networks problematic for theoretical applications).
Recently, however, I experimented with using Neural ODEs as a natural way of extending our usual process of modeling physical systems with differential equations through deep learning, while exploring symbolic regression to improve interpretability and generalizability. I also attended NeurIPS 2024 and got a feel for what the field was excited about, including in the context of scientific applications.
It's been over half a year since then, and many of the big ideas such as MCP, multimodal inputs, and agentic AI have dominated advances in that time. Along the way, I've repeatedly encountered work being done on mechanistic interpretability, including papers from the Anthropic team. Beyond skimming these papers, I haven't devoted time to thinking more deeply about these ideas.
Given that this is a field I'm interested in pursuing, I've decided to invest time in diving deeper into the ideas and research in the literature, and exploring where I might be able to contribute.

## Why This Blog

Starting this research blog serves three purposes.
1. Mainly, I hope it will document my thoughts and ideas as they change and evolve while I learn and explore this new field. As I progress, it will be useful to return to earlier thoughts.
2. Second, I want to improve the clarity of my writing, since I often find it difficult to express thoughts without extensive refinement.
3. Finally, I'm hoping this imposes some level of self-accountability to keep at it regularly, since this will be on the side and doesn't overlap with my current work.

Mech Interp Day 0: Motivations


The previous iteration of this website featured an interactive 3D pulsating sphere built using Three.js. For the sake of a cleaner feel, I decided to remove it for this iteration, but found the experience rewarding. The animation I originally had can be found at the botto of this post! The capability for 3D graphics that Three.js provide is rich in potential, especially since modern web development has evolved far beyond static pages. Today's users expect rich, interactive experiences that feel more like native applications than traditional websites.

## The Technology Stack

Combining several powerful technologies can create truly engaging web experiences. The combination I initially went with included:

- **Three.js** for 3D graphics and WebGL rendering
- **Framer Motion** for smooth animations and transitions
- **React** for component-based architecture
- **Next.js** for performance optimization

## Why 3D on the Web?

Three.js has revolutionized how we think about web interfaces. Here's a simple example of creating a rotating cube:

```javascript
import * as THREE from 'three';

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer();

const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);

scene.add(cube);
camera.position.z = 5;

function animate() {
    requestAnimationFrame(animate);
    cube.rotation.x += 0.01;
    cube.rotation.y += 0.01;
    renderer.render(scene, camera);
}

animate();
```

## Animation with Framer Motion

Framer Motion makes it incredibly easy to add smooth animations to React components:

```jsx
import { motion } from 'framer-motion';

const AnimatedCard = () => {
  return (
    
      Interactive Card
      This card animates on load and hover!
    
  );
};
```

## My Experience

The combination of Three.js and Framer Motion in a React/Next.js environment provides an incredibly powerful toolkit for creating memorable web experiences. The key is finding the right balance between visual impact and performance. The rest of this post will go through how I set up my initial animation, with the final product at the end!

## Uniform Sampling of Points on A Sphere

The key to creating a convincing sphere lies in proper point distribution. Unlike naive approaches that create clustering at poles, we use **spherical coordinates** with uniform random sampling:

```javascript
const points = [];
for (var i = 0; i < 1500; i++) {
  var vertex = new THREE.Vector3();

  // Uniform random sampling on sphere surface
  var u = THREE.MathUtils.randFloat(0, 1);
  var v = THREE.MathUtils.randFloat(0, 1);
  var theta = 2 * Math.PI * u;           // Azimuthal angle
  var phi = Math.acos(2 * v - 1);        // Polar angle

  // Convert to Cartesian coordinates
  vertex.x = 3.5 * Math.sin(phi) * Math.cos(theta);
  vertex.y = 3.5 * Math.sin(phi) * Math.sin(theta);
  vertex.z = 3.5 * Math.cos(phi);

  points.push(vertex);
}
```

This approach ensures **uniform distribution** across the sphere surface, avoiding the visual artifacts of simpler random placement methods.

## GLSL Shader

The pulsating light effect is achieved through custom **vertex and fragment shaders** that run directly on the GPU:

### Shader Uniforms Setup

```javascript
const shaderPoint = THREE.ShaderLib.points;
const uniforms = THREE.UniformsUtils.clone(shaderPoint.uniforms);
uniforms.time = { value: 0 };
uniforms.color = { type: "v3", value: colorValue };

const pMaterial = new THREE.ShaderMaterial({
  uniforms,
  transparent: true,
  depthWrite: false,
  blending: THREE.AdditiveBlending,  // Creates light emission effect
  vertexShader,
  fragmentShader,
});
```

### The Shader Pipeline

This particle system consists of two interconnected shaders:

- **Vertex Shader** ("vert.glsl") - Handles particle positioning, sizing, and movement
- **Fragment Shader** ("frag.glsl") - Controls particle appearance, color, and glow effects

### 3D Simplex Noise Implementation

The vertex shader begins with a complete **Simplex noise** implementation - a sophisticated algorithm for generating natural-looking randomness:

```glsl
float snoise(vec3 v) {
    // 85 lines of math
    // Creates smooth, continuous 3D noise
    return 42.0 * dot(m*m, vec4(dot(p0,x0), dot(p1,x1), dot(p2,x2), dot(p3,x3)));
}
```

**Why Simplex Noise?** Unlike basic random functions, Simplex noise provides:
- **Continuous gradients** - no sudden jumps or artifacts
- **3D coherence** - neighboring points have similar values
- **Performance optimization** - faster than Perlin noise
- **Natural patterns** - mimics organic movement and growth

### Organic Particle Movement

The main vertex shader transforms each particle's position using time-based trigonometric functions:

```glsl
vec3 newPos = position;

newPos.x += sin(time + position.x * position.y) * 0.08;
newPos.y += cos(time + position.x * position.y * 1.1) * 0.08;
newPos.z += cos(time + position.x * position.y * 1.3) * 0.08;
```

**Mathematical Breakdown:**
- **Base oscillation**: "sin(time)" and "cos(time)" create rhythmic movement
- **Position coupling**: "position.x * position.y" makes each particle's movement unique
- **Frequency variation**: Multipliers prevent synchronized motion
- **Amplitude control**: "0.08" keeps movement subtle and elegant

This creates a **Lissajous-like pattern** where each particle follows its own complex orbital path, determined by its starting position.

### Dynamic Particle Sizing

The most sophisticated aspect is the noise-driven size variation:

```glsl
gl_PointSize = 50. + snoise(position * 0.05 + vec3(0, 0, vtime * 0.1)) * 50.;
gl_PointSize *= 0.5;
```

**Size Calculation Explained:**
- **Base size**: "50." pixels provides consistent minimum visibility
- **Noise sampling**: "snoise(position * 0.05 + ...)" creates spatial variation
- **Temporal drift**: "vec3(0, 0, vtime * 0.1)" makes noise evolve over time
- **Amplitude**: "* 50." allows sizes to range from 0 to 100 pixels
- **Final scaling**: "* 0.5" reduces overall scale to 0-50 pixels

The result is particles that **breathe** - growing and shrinking organically as the noise field evolves through time.

### Time-Based Animation

The pulsating effect is driven by a time uniform that updates every frame:

```javascript
const animate = (time) => {
  // Update shader time for pulsating effect
  pMaterial.uniforms.time.value = time * 0.004;
  
  // Continue animation loop
  requestAnimationFrame(animate);
  renderer.render(scene, camera);
}
```

The **AdditiveBlending** mode creates the characteristic light emission, making particles appear to glow and blend naturally when they overlap.

## Responsive Color System

The particle system adapts to the user's color mode preference through dynamic uniform updates:

```javascript
// Real-time color mode detection and adaptation
if (localStorage.getItem("chakra-ui-color-mode") === "dark") {
  pMaterial.uniforms.color.value = new THREE.Color(0xffffff);  // White particles
} else {
  pMaterial.uniforms.color.value = new THREE.Color(0x000000);  // Black particles
}
```

This creates a seamless experience where the 3D scene automatically adapts to the user's interface preferences without requiring page refreshes.

## Smooth Camera Animations

The initial camera movement uses an **easing function** to create natural motion:

```javascript
function easeOutCirc(x) {
  return Math.sqrt(1 - Math.pow(x - 1, 4));
}

// Camera animation during first 100 frames
if (frame <= 100) {
  const rotSpeed = -easeOutCirc(frame / 120) * Math.PI * 20;
  
  camera.position.x = p.x * Math.cos(rotSpeed) + p.z * Math.sin(rotSpeed);
  camera.position.z = p.z * Math.cos(rotSpeed) - p.x * Math.sin(rotSpeed);
  camera.lookAt(target);
} else {
  // Switch to user-controlled orbit after animation
  controls.update();
}
```

After the initial animation completes, control transitions to **OrbitControls** for user interaction, with automatic rotation enabled.

## Performance Optimizations

### GPU-Accelerated Rendering
- **ShaderMaterial** for GPU-based calculations

### Efficient Animation Loop
```javascript
let req = null;
const animate = (time) => {
  req = requestAnimationFrame(animate);
  
  // Minimal CPU calculations
  // GPU handles particle transformations
  
  renderer.render(scene, camera);
}

// Proper cleanup
return () => {
  cancelAnimationFrame(req);
  renderer.domElement.remove();
  renderer.dispose();
}
```

## Final Product
Here is the final animation, best viewed in dark mode!
<hr />
<voxel-art />

Building Interactive Web Experiences


This document summarizes the complete implementation of a markdown-based blog system using Next.js, Chakra UI, react-markdown, and KaTeX for mathematical equations.

## Tech Stack

- **Next.js** - React framework with static site generation
- **Chakra UI** - Component library for styling
- **react-markdown** - Markdown to React component converter
- **KaTeX** - LaTeX equation rendering
- **gray-matter** - Frontmatter parsing
- **Framer Motion** - Animations

## Required Dependencies

```bash
npm install react-markdown remark-gfm rehype-highlight rehype-raw gray-matter katex rehype-katex remark-math
```

## File Structure

```
project/
├── components/
│   └── BlogPost.jsx
├── lib/
│   └── posts.js
├── pages/
│   ├── writing.js (blog index)
│   └── blog/
│       └── [slug].js (individual posts)
├── posts/
│   ├── my-first-post.md
│   └── second-blog-post.md
└── pages/_app.js
```

## Core Components

### lib/posts.js - File System Functions
```javascript
import fs from 'fs';
import path from 'path';
import matter from 'gray-matter';

const postsDirectory = path.join(process.cwd(), 'posts');

export function getAllPosts() {
  const fileNames = fs.readdirSync(postsDirectory);
  const allPostsData = fileNames.map((fileName) => {
    const slug = fileName.replace(/\.md$/, '');
    const fullPath = path.join(postsDirectory, fileName);
    const fileContents = fs.readFileSync(fullPath, 'utf8');
    const { data, content } = matter(fileContents);

    return {
      slug,
      content,
      ...data,
    };
  });

  return allPostsData.sort((a, b) => (a.date < b.date ? 1 : -1));
}

export function getPostBySlug(slug) {
  const fullPath = path.join(postsDirectory, `${slug}.md`);
  const fileContents = fs.readFileSync(fullPath, 'utf8');
  const { data, content } = matter(fileContents);

  return {
    slug,
    content,
    ...data,
  };
}
```

### components/BlogPost.jsx - Markdown Renderer
```javascript
import ReactMarkdown from 'react-markdown';
import remarkGfm from 'remark-gfm';
import remarkMath from 'remark-math';
import rehypeHighlight from 'rehype-highlight';
import rehypeKatex from 'rehype-katex';
import { 
  Box, 
  Heading, 
  Text, 
  Code, 
  Divider, 
  Link,
  UnorderedList,
  OrderedList,
  ListItem,
  useColorModeValue
} from '@chakra-ui/react';

const BlogPost = ({ content }) => {
  const blockquoteBg = useColorModeValue('gray.50', 'gray.700');
  const inlineCodeBg = useColorModeValue('gray.100', 'gray.600');
  const inlineCodeColor = useColorModeValue('gray.800', 'gray.100');

  return (
    <Box maxW="800px" mx="auto" p={6}>
      <ReactMarkdown
        remarkPlugins={[remarkGfm, remarkMath]}
        rehypePlugins={[rehypeHighlight, rehypeKatex]}
        components={{
          h1: ({ children }) => (
            <Heading as="h1" size="2xl" mb={6} mt={8}>
              {children}
            </Heading>
          ),
          h2: ({ children }) => (
            <Heading as="h2" size="xl" mb={4} mt={6}>
              {children}
            </Heading>
          ),
          h3: ({ children }) => (
            <Heading as="h3" size="lg" mb={3} mt={5}>
              {children}
            </Heading>
          ),
          p: ({ children }) => (
            <Text mb={4} lineHeight="1.7">
              {children}
            </Text>
          ),
          ul: ({ children }) => (
            <UnorderedList mb={4} spacing={2}>
              {children}
            </UnorderedList>
          ),
          ol: ({ children }) => (
            <OrderedList mb={4} spacing={2}>
              {children}
            </OrderedList>
          ),
          li: ({ children }) => (
            <ListItem>{children}</ListItem>
          ),
          code: ({ inline, children, className }) => {
            return inline ? (
              <Code 
                px={2} 
                py={1} 
                bg={inlineCodeBg}
                color={inlineCodeColor}
                borderRadius="md"
                suppressHydrationWarning={true}
              >
                {children}
              </Code>
            ) : (
              <Box mb={4}>
                <Code
                  as="pre"
                  display="block"
                  p={4}
                  bg="gray.900"
                  color="white"
                  _dark={{ bg: "gray.800" }}
                  borderRadius="md"
                  overflow="auto"
                  suppressHydrationWarning={true}
                >
                  {children}
                </Code>
              </Box>
            );
          },
          a: ({ href, children }) => (
            <Link href={href} color="blue.500" isExternal>
              {children}
            </Link>
          ),
          blockquote: ({ children }) => (
            <Box
              as="blockquote"
              borderLeft="4px solid"
              borderColor="blue.500"
              pl={4}
              py={2}
              mb={4}
              fontStyle="italic"
              bg={blockquoteBg}
              _dark={{ bg: "gray.700", borderColor: "blue.300" }}
              borderRadius="md"
              suppressHydrationWarning={true}
            >
              {children}
            </Box>
          ),
          hr: () => <Divider my={6} />,
        }}
      >
        {content}
      </ReactMarkdown>
    </Box>
  );
};

export default BlogPost;
```

### pages/writing.js - Blog Index
```javascript
import NextLink from 'next/link'
import { Box, Container, Heading, Link, Text } from '@chakra-ui/react'
import Layout from '../components/layouts/article'
import Section from '../components/section'
import { getAllPosts } from '../lib/posts'

const Writing = ({ posts = [] }) => {
  return (
    <Layout title="Posts">
      <Container>
        <Heading as="h3" fontSize={20} mb={4} mt={5}>
          Posts
        </Heading>

        {posts && posts.map((post, index) => (
          <Section key={post.slug} delay={0.1 + (index + 1) * 0.1}>
            <Box my={4}>
              <Text fontSize="sm" color="gray.500" display="inline" mr={3}>
                {new Date(post.date + 'T00:00:00').toLocaleDateString('en-US', {
                  year: 'numeric',
                  month: 'short',
                  day: 'numeric',
                  timeZone: 'UTC'
                })}
              </Text>
              <Link as={NextLink} href={`/blog/${post.slug}`}>
                {post.title}
              </Link>
            </Box>
          </Section>
        ))}

      </Container>
    </Layout>
  )
}

export async function getStaticProps() {
  try {
    const posts = getAllPosts();
    return {
      props: {
        posts,
      },
    };
  } catch (error) {
    console.error('Error in getStaticProps:', error);
    return {
      props: {
        posts: [],
      },
    };
  }
}

export default Writing
```

### pages/blog/[slug].js - Individual Post Pages
```javascript
import { getAllPosts, getPostBySlug } from '../../lib/posts';
import BlogPost from '../../components/BlogPost';
import { Box, Heading, Text } from '@chakra-ui/react';

export default function Post({ post }) {
  return (
    <Box>
      <Box textAlign="center" mb={8}>
        <Heading as="h1" size="3xl" mb={4}>
          {post.title}
        </Heading>
        <Text color="gray.600">{post.date}</Text>
      </Box>
      <BlogPost content={post.content} />
    </Box>
  );
}

export async function getStaticProps({ params }) {
  const post = getPostBySlug(params.slug);
  return {
    props: {
      post,
    },
  };
}

export async function getStaticPaths() {
  const posts = getAllPosts();
  const paths = posts.map((post) => ({
    params: { slug: post.slug },
  }));

  return {
    paths,
    fallback: false,
  };
}
```

## Markdown Features Supported

### Basic Formatting
- **Bold text** and *italic text*
- Headers (H1, H2, H3)
- Paragraphs with proper spacing
- Horizontal rules
- Links (internal and external)

### Lists
- Unordered lists with bullet points
- Ordered lists with numbers
- Proper spacing between items

### Code
- Multi-line code blocks with language-specific highlighting
- Dark/light mode responsive styling

### Blockquotes
> Styled blockquotes with left border. Dark/light mode responsive

### Mathematical Equations
- Inline math: $E = mc^2$
- Block equations: $$\int_{-\infty}^{\infty} e^{-x^2} dx = \sqrt{\pi}$$
- Complex expressions with proper LaTeX rendering

## Frontmatter Structure
```yaml
---
title: "Post Title"
date: "2025-01-15"
excerpt: "Brief description of the post"
tags: ["tag1", "tag2", "tag3"]
---
```

## Development Workflow

1. Create markdown files in posts/ directory
2. Posts automatically appear on /writing page

## Troubleshooting Notes

- **Date Issues**: Use `new Date(post.date + 'T00:00:00')` to prevent timezone shifts
- Inline equations not working

Blog Implementation Summary


## Welcome to My Blog

This is my first blog post! I will just test functionality in this post, and write up how I implemented post functionality in a future post.

## Code Example

```python
def hello_world():
    print("Hello, World!")
    return True
``` 

## More Examples

> This is a blockquote with some important information.

Check out [this link](https://example.com) for more info.

Here's some **bold text** and *italic text*.

### Lists

Unordered list:
- Item 1
- Item 2
- Item 3

Ordered list:
1. First item
2. Second item
3. Third item 

### Equations
This is an inline equation: $E = mc^2$.

In web animations, we often use trigonometric functions. The general form of a sine wave is:

$$
y(x,t) = A \sin(kx - \omega t + \phi)
$$

Where:
- $A$ is the amplitude
- $k$ is the wave number  
- $\omega$ is the angular frequency
- $\phi$ is the phase shift

This equation is perfect for creating smooth, natural animations in Three.js!

That's all for now!
