SIPL is a programming language. We write programs in SIPL.
A language model predicts the next token in a sequence.
We build a tokenizer, train a Transformer, and generate text.
Attention connects each token to the tokens before it.
Training improves predictions by reducing the loss.
The model learns from its own local text corpus.
Save a checkpoint to continue training another day.
Question: What is SIPL? Answer: SIPL is a programming language.
Question: What does attention do? Answer: Attention connects tokens.
Starling is a project that uses the siplAI library.
A project owns its interface. A library provides functions.
We can include the library in another SIPL program.
The compiler includes the library in a standalone executable.
We train a small model from text that we choose.
Each training step updates the model to improve its prediction.
A tokenizer turns text into tokens. The model predicts tokens.
SIPL runs our code. The GPU can compute matrix products.
We save our model and continue training from a checkpoint.
