Create a Gemma compatible transformer implementation #30
Labels
enhancement
New feature or request
help wanted
Extra attention is needed
project:TTP
For the Tiny Transformer Playground
Different transformer implementations have variations (e.g. in positional encoding, where skip connections are, use of MQA, etc). Lets provide a Gemma standard implementation of transformers. This could be verified by being able to load and evaluate with a Gemma weights file.
The text was updated successfully, but these errors were encountered: