add jupyter-structured scripts

bigcode-project · May 11, 2023 · 588df43 · 588df43
1 parent 966a101
commit 588df43
Show file tree

Hide file tree

Showing 4 changed files with 848 additions and 0 deletions.
diff --git a/preprocessing/jupyter-structured/README.md b/preprocessing/jupyter-structured/README.md
@@ -0,0 +1,17 @@
+## Creating Jupyter-structured dataset
+
+### Step 1
+Parse Jupyter notebooks from `the Stack`.  
+```
+python jupyter-segment-notebooks.py
+```
+
+### Step 2
+Generate markdown-code-output triplets.
+```
+python jupyter-generate-triplets.py
+```
+
+### Step 3
+Create notebook-level structured dataset using `jupyter-structured.ipynb`.
+