Jupyter Notebook READMEs on GitHub

GitHub.com has a feature whereby a README.md file containing text and Markdown markup present in any directory is rendered to HTML below the list of files in that directory. See https://github.com/SalishSeaCast/analysis-ben/tree/master/notebooks as an example.

We can use that feature in directories that contain Jupyter Notebook files to provide links to our notebooks rendered to HTML by the Jupyter nbviewer service. Doing so makes the notebooks easily visible to anyone without having to run Jupyter Notebook. It is also an easy way to generate notebook viewer links to paste into the Google Drive “whiteboard” documents for weekly group meetings.

You could hand edit the README.md file, but that’s tedious and error prone, so it is an obvious candidate for code automation. Here is a prototype make_readme.py module that provides that automation:

  1"""Jupyter Notebook collection README generator
  2
  3Copyright by the UBC EOAS MOAD Group and The University of British Columbia.
  4
  5Licensed under the Apache License, Version 2.0 (the "License");
  6you may not use this file except in compliance with the License.
  7You may obtain a copy of the License at
  8
  9   https://www.apache.org/licenses/LICENSE-2.0
 10
 11Unless required by applicable law or agreed to in writing, software
 12distributed under the License is distributed on an "AS IS" BASIS,
 13WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 14See the License for the specific language governing permissions and
 15limitations under the License.
 16
 17When you add a new notebook to this directory,
 18rename a notebook,
 19or change the description of a notebook in its first Markdown cell,
 20please generate a updated `README.md` file with:
 21
 22    python3 -m make_readme
 23
 24and commit and push the updated `README.md` to GitHub.
 25"""
 26import json
 27from pathlib import Path
 28import re
 29
 30
 31NBVIEWER = "https://nbviewer.org/github"
 32GITHUB_ORG = "SalishSeaCast"
 33REPO_NAME = "analysis-casey"
 34DEFAULT_BRANCH_NAME = "main"
 35CREATOR_NAME = "Casey Lawrence"
 36TITLE_PATTERN = re.compile("#{1,6} ?")
 37
 38
 39def main():
 40    cwd_parts = Path.cwd().parts
 41    repo_path = Path(*cwd_parts[cwd_parts.index(REPO_NAME)+1:])
 42    url = f"{NBVIEWER}/{GITHUB_ORG}/{REPO_NAME}/blob/{DEFAULT_BRANCH_NAME}/{repo_path}"
 43
 44    readme = f"""\
 45The Jupyter Notebooks in this directory are made by
 46{CREATOR_NAME} for sharing of Python code techniques
 47and notes.
 48
 49The links below are to static renderings of the notebooks via
 50[nbviewer.org](https://nbviewer.org/).
 51Descriptions below the links are from the first cell of the notebooks
 52(if that cell contains Markdown or raw text).
 53
 54"""
 55    for fn in Path(".").glob("*.ipynb"):
 56        readme += f"* ## [{fn}]({url}/{fn})  \n    \n"
 57        readme += notebook_description(fn)
 58
 59    license = f"""
 60## License
 61
 62These notebooks and files are copyright by the
 63[UBC EOAS MOAD Group](https://github.com/UBC-MOAD/docs/blob/main/CONTRIBUTORS.rst)
 64and The University of British Columbia.
 65
 66They are licensed under the Apache License, Version 2.0.
 67https://www.apache.org/licenses/LICENSE-2.0
 68Please see the LICENSE file in this repository for details of the license.
 69"""
 70
 71    with open("README.md", "wt") as f:
 72        f.writelines(readme)
 73        f.writelines(license)
 74
 75
 76def notebook_description(fn):
 77    description = ""
 78    with open(fn, "rt") as notebook:
 79        contents = json.load(notebook)
 80    try:
 81        first_cell = contents["worksheets"][0]["cells"][0]
 82    except KeyError:
 83        first_cell = contents["cells"][0]
 84    first_cell_type = first_cell["cell_type"]
 85    if first_cell_type not in "markdown raw".split():
 86        return description
 87    desc_lines = first_cell["source"]
 88    for line in desc_lines:
 89        suffix = ""
 90        if TITLE_PATTERN.match(line):
 91            line = TITLE_PATTERN.sub("**", line)
 92            suffix = "**"
 93        if line.endswith("\n"):
 94            description += f"    {line[:-1]}{suffix}\n"
 95        else:
 96            description += f"    {line}{suffix}"
 97    description += "\n" * 2
 98    return description
 99
100
101if __name__ == "__main__":
102    main()

Here’s how to set up and use this script:

  1. Put the code above into a file called make_readme.py in a directory that contains Jupyter Notebook files.

  2. Edit line 34 to the GitHub organization that your repository is in. If you are setting this up for a repository in the UBC-MOAD organization on GitHub, you should change line 34 from:

    GITHUB_ORG = "SalishSeaCast"

    to:

    GITHUB_ORG = "UBC-MOAD"

  3. Edit line 35 to the name of your repository. If the local clone of the repository you are working is called ch3-paper/, you should change line 35 from:

    REPO_NAME = "analysis-casey"

    to:

    REPO_NAME = "ch3-paper"

  4. Edit line 36 to the name of your repository’s default branch. (You can check the name of your default branch with git symbolic-ref --short HEAD) If the name of your default branch is master, you should change line 36 from:

    DEFAULT_BRANCH_NAME = "main"

    to:

    DEFAULT_BRANCH_NAME = "master"

  5. Edit line 37 to your name for the “notebooks made by …” message; i.e. change line 37 from:

    CREATOR_NAME = "Casey Lawrence"

    to:

    CREATOR_NAME = "Your Name"

  6. Edit lines 45-47 to describe what your notebooks are about. You can put as much text as you want there. It is the beginning of the text that will appear between the list of files on the GitHub page and the list of links to the nbviewer renderings of your notebooks. Don’t forget to change line 44 to your name!

  7. Save the make_readme.py file. You won’t need to edit it again unless you want to change the preamble text starting at line 43.

  8. Run the make_readme.py script to create your README.md file:

    $ python3 -m make_readme

  9. Use Git to add, commit, and push to GitHub your new notebook(s), the make_readme.py script, and the README.md file:

    $ git add make_readme.py README.md MyNotebook.ipynb
$ git commit -m"Add new notebook, make_readme script and README file."
$ git push

  10. Use your browser to navigate to the repository and directory on GitHub and you should see the rendered README.md showing your notebook name(s) as a link to the nbviewer rendering(s) for your notebook(s).

  11. Each time you create a new notebook in the directory, run python3 -m make_readme to update the README.md file and commit it along with your new notebook.

The make_readme.py script reads the first cell of each notebook in the directory and, if that cell contains text, adds it to the README.md file. That lets you include a title and brief description of your notebooks along with the links on the GitHub page. If you change the contents of that 1st cell in an existing notebook you need to run python3 -m make_readme, commit the README.md changes, and push them to GitHub in order to update the page there.