Code

Rwkv x eagle notebooks#75

Code

Merged

PicoCreator merged 31 commits intomainRWKV/RWKV-infctx-trainer:mainfrom rwkv-x-eagle-notebooksRWKV/RWKV-infctx-trainer:rwkv-x-eagle-notebooksCopy head branch name to clipboardFeb 6, 2024

+8,710-766

Commits

Commits on Feb 3, 2024

• change logging format, to reduce confusion

  pic-o

  committed
  4ddbf83

  Copy full SHA for 4ddbf83

• move to experiment folder

  pic-o

  committed
  b3a32c0

  Copy full SHA for b3a32c0

• preparing experiment notebooks

  pic-o

  committed
  695545d

  Copy full SHA for 695545d

• Added notes on runnign the runner

  pic-o

  committed
  d4e2d3e

  Copy full SHA for d4e2d3e

• Merge remote-tracking branch 'origin/main' into rwkv-x-eagle-notebooks

  pic-o

  committed
  fea8459

  Copy full SHA for fea8459

• include any option

  pic-o

  committed
  d9a6211

  Copy full SHA for d9a6211

Commits on Feb 4, 2024

• wip benchmarks

  PicoCreator

  committed
  3d5fc0f

  Copy full SHA for 3d5fc0f

• tweak

  PicoCreator

  committed
  9c7744c

  Copy full SHA for 9c7744c

• Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV-infctx-trainer into rwkv-x-eagle-notebooks

  PicoCreator

  committed
  59dc19e

  Copy full SHA for 59dc19e

• Fixing the LR for batched

  pic-o

  committed
  c42c7c0

  Copy full SHA for c42c7c0

• Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV-infctx-trainer into rwkv-x-eagle-notebooks

  pic-o

  committed
  fe9bd2c

  Copy full SHA for fe9bd2c

• fixing mask sum calc

  pic-o

  committed
  475558d

  Copy full SHA for 475558d

• wip calibration

  PicoCreator

  committed
  0fd1387

  Copy full SHA for 0fd1387

• enwiki 16k test

  PicoCreator

  committed
  a2443d7

  Copy full SHA for a2443d7

• Update notebook title in enwiki-16k-3e-5.ipynb

  PicoCreator

  committed
  85ce0c1

  Copy full SHA for 85ce0c1

• Fixing multipack

  PicoCreator

  committed
  18f52d0

  Copy full SHA for 18f52d0

• loss validation run

  PicoCreator

  committed
  ade9b35

  Copy full SHA for ade9b35

• WIP datapack fixing code

  PicoCreator

  committed
  efced7f

  Copy full SHA for efced7f

• Update learning rate initialization and finalization values

  PicoCreator

  committed
  e5deb69

  Copy full SHA for e5deb69

• WIP benchmarks

  PicoCreator

  committed
  5e1b715

  Copy full SHA for 5e1b715

• drop dataset_name and dataset_index

  PicoCreator

  committed
  6d7ca1c

  Copy full SHA for 6d7ca1c

• Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV-infctx-trainer into rwkv-x-eagle-notebooks

  PicoCreator

  committed
  dd9c79d

  Copy full SHA for dd9c79d

• the datapack code

  PicoCreator

  committed
  0ee162d

  Copy full SHA for 0ee162d

• support custom dataset split

  PicoCreator

  committed
  44fb0be

  Copy full SHA for 44fb0be

• config update

  PicoCreator

  committed
  6f1e02e

  Copy full SHA for 6f1e02e

• WIP tweaks

  PicoCreator

  committed
  601986c

  Copy full SHA for 601986c

• wip MultiPack train

  PicoCreator

  committed
  dfccbc7

  Copy full SHA for dfccbc7

Commits on Feb 6, 2024

• tweaks

  PicoCreator

  committed
  74deedb

  Copy full SHA for 74deedb

• prototype train/test split swap - because sometimes you need that

  pic-o

  committed
  429c8c1

  Copy full SHA for 429c8c1

• Merge branch 'rwkv-x-eagle-notebooks' of https://github.com/RWKV/RWKV-infctx-trainer into rwkv-x-eagle-notebooks

  pic-o

  committed
  0654172

  Copy full SHA for 0654172

• fixing multi-gpu sync

  pic-o

  committed
  e02907a

  Copy full SHA for e02907a