update correct scripts and readme.md

bigcode-project · May 12, 2023 · af99753 · af99753
1 parent a39c4c0
commit af99753
Show file tree

Hide file tree

Showing 4 changed files with 88 additions and 546 deletions.
diff --git a/README.md b/README.md
@@ -21,4 +21,5 @@ necessary used for model training.
   - Filters for GitHub Issues
   - Filters for Git Commits
   - Script to convert Jupyter notebooks to scripts
+  - Scripts to convert Jupyter notebooks to structured markdown-code-output triplets
 - `decontamination`: script to remove files that match test-samples from code generation benchmarks.
diff --git a/preprocessing/README.md b/preprocessing/README.md
@@ -40,3 +40,22 @@ We release the Jupyter scripts dataset as part of [StarCoderData](https://huggin
 python jupyter_script_conversion.py
 ```
 
+# Creating Jupyter-structured dataset
+
+## Step 1
+Parse Jupyter notebooks from `the Stack`.  
+```
+python jupyter-structured/jupyter-segment-notebooks.py
+```
+
+## Step 2
+Generate markdown-code-output triplets.
+```
+python jupyter-structured/jupyter-generate-triplets.py
+```
+
+## Step 3
+Create notebook-level structured dataset using `jupyter-structured/jupyter-structured.ipynb`.
+
+
+
diff --git a/preprocessing/jupyter-structured/README.md b/preprocessing/jupyter-structured/README.md