Merge pull request #164 from slacgismo/dev

Revamped Docs

.gitignore

@@ -17,6 +17,7 @@
 *.png
 notebooks/*.pkl
 notebooks/data/*
+docs/source/getting_started/notebooks/inputs
 
 ### Database ###
 *.accdb

.readthedocs.yaml

@@ -0,0 +1,25 @@
+# Read the Docs configuration file for Sphinx projects
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the OS, Python version and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.10"
+
+# Build documentation in the "docs/" directory with Sphinx
+sphinx:
+  configuration: docs/source/conf.py
+  fail_on_warning: true
+
+# Install regular dependencies.
+# Then, install special pinning for RTD.
+python:
+  install:
+    - requirements: requirements.txt
+    - method: pip
+      path: .
+

CODE_OF_CONDUCT.md

@@ -1,4 +1,3 @@
-
 # Contributor Covenant Code of Conduct
 
 ## Our Pledge

CONTRIBUTING.md

@@ -1,3 +1,72 @@
-[To be added]
+# Contribution Guidelines
 
-For now, please contact [Bennet Meyers](mailto:[email protected]) - bennetm at stanford.edu
+Contributions are welcome, and are greatly appreciated! Even small contributions are helpful, and credit will always be given!
+
+## Types of Contributions
+
+### Report Bugs
+
+To report a bug, first make sure there isn't already an open [GitHub Issue](https://github.com/slacgismo/solar-data-tools/issues)
+about it. If not, you can submit a new issue, making sure to include:
+
+* Your OS name and version.
+* Any details about your local setup (_e.g._, your Python environment setup) that might be helpful in troubleshooting.
+* The Solar Data Tools version you are using.
+* Detailed steps to reproduce the bug.
+* A tag, if appropriate (_e.g._, "bug" or "enhancement").
+
+### Fix Open Issues
+
+Look through the [GitHub Issues](https://github.com/slacgismo/solar-data-tools/issues) for any 
+reported bugs. Anything tagged with "bug" and "help wanted" is open to contributions.
+
+### Implement Features
+
+Look through the [GitHub Issues](https://github.com/slacgismo/solar-data-tools/issues) for any 
+feature requests. Anything tagged with "enhancement" and "help  wanted" is open to contributions.
+
+### Write Documentation
+
+You can never have enough documentation! Please feel free to contribute to any
+part of the documentation, such as the getting started guides or the docstrings/API reference.
+
+## Get Started!
+
+Ready to contribute? Here's how to set up Solar Data Tools for local development.
+
+1. You will need to have `git` installed on your local machine, and you will need some way to create and manage
+Python virtual environments. We recommend using `conda`.
+2. Create your own fork of the Solar Data Tools [repository](https://github.com/slacgismo/solar-data-tools).
+3. Clone your repository, for example by typing in your terminal: 
+`git clone https://github.com/<your_username>/solar-data-tools.git`.
+4. Install `solardatatools` as editable in your new environment:
+
+    ```console
+    $ pip install -e .
+    ```
+   If you would like to install optional dependencies, such as MOSEK or docs-specific dependencies, you can do so by:
+    ```console
+    $ pip install -e ".[docs]"
+    ```
+5. Create a branch for local development and make your changes:
+
+    ```console
+    $ git checkout -b name-of-your-bugfix-or-feature
+    ```
+
+6. When you're done making changes, check that your changes conform to any code formatting requirements and pass any tests.
+
+7. Commit and push your changes to your fork, and open a pull request.
+
+## Pull Request Guidelines
+
+Before you submit a pull request, check that it meets these guidelines:
+
+1. The pull request should include additional tests if appropriate.
+2. If the pull request adds functionality, the docs should be updated.
+3. The pull request should work for all currently supported operating systems and versions of Python.
+
+## Code of Conduct
+
+Please note that the Solar Data Tools project is released with a
+[Code of Conduct](getting_started/code_of_conduct.md). By contributing to this project you agree to abide by its terms.

README.md

@@ -1,6 +1,21 @@
-![SDT_v1_primary_blue_text](https://github.com/user-attachments/assets/99dd3103-12ea-413b-9871-1b45f8eaf95d)
-
-<table>
+<!-- HEADER -->
+<br />
+<p align="center">
+  <a href="#">
+    <img src="docs/source/_static/SDT_v1_secondary_blue_text.png" width="600">
+  </a>
+
+  <p align="center">
+    <br />
+    <a href="https://solar-data-tools.readthedocs.io/"><strong>Explore our documentation </strong></a>
+    ·
+    <a href="https://github.com/slacgismo/solar-data-tools/issues"><strong>Report Issue </strong></a>
+    <br />
+    <br />
+</p>
+</p>    
+
+<table  align="center" >
 <tr>
   <td>Latest Release</td>
   <td>
@@ -29,12 +44,11 @@
         <img src="https://readthedocs.org/projects/solar-data-tools/badge/?version=stable" alt="documentation build status" />
     </a>
         <a href="https://github.com/slacgismo/solar-data-tools/actions/workflows/test.yml">
-        <img src="https://github.com/slacgismo/solar-data-tools/actions/workflows/test.yml/badge.svg?branch=master" alt="Actions build status" />
-    </a>
-    <!-- switch below from tadatoshi to gismo -->
-    <a href="https://travis-ci.com/tadatoshi/solar-data-tools.svg?branch=development">
-        <img src="https://travis-ci.com/tadatoshi/solar-data-tools.svg?branch=development">
+        <img src="https://github.com/slacgismo/solar-data-tools/actions/workflows/test.yml/badge.svg?branch=main" alt="Actions build status" />
     </a>
+    <a href="https://github.com/slacgismo/solar-data-tools/actions/workflows/build.yml">
+        <img src="https://github.com/slacgismo/solar-data-tools/actions/workflows/build.yml/badge.svg">
+    </a> 
   </td>
 </tr>
 <tr>
@@ -68,18 +82,26 @@
     </td>
 </tr>
 </table>
+
 
-Solar Data Tools is an open-source Python library for analyzing PV power (and irradiance) time-series data. It provides
-methods for data I/O, cleaning, filtering, plotting, and analysis. These methods are largely automated and require little
-to no input from the user regardless of system type—from utility tracking systems to multi-pitch rooftop systems. Solar Data Tools
+Solar Data Tools is an open-source Python library for analyzing PV power (and irradiance) time-series data. It
 was developed to enable analysis of _unlabeled_ PV data, i.e. with no model, no meteorological data, and no performance index required,
 by taking a statistical signal processing approach in the algorithms used in the package’s main data processing pipeline.
-Head over to our Getting Started pages in our [docs](https://solar-data-tools.readthedocs.io/) for a demo of Solar Data Tools!
+Solar Data Tools empowers PV system fleet owners or operators to analyze system performance a hundred times faster even when 
+they only have access to the most basic data stream—power output of the system.
 
-For an in-depth tutorial on Solar Data Tools, we recommend taking a look at the recent webinar 
+Solar Data Tools provides methods for data I/O, cleaning, filtering, plotting, and analysis. These methods are largely automated and require little
+to no input from the user regardless of system type—from utility tracking systems to multi-pitch rooftop systems. 
+Head over to our Getting Started pages in our [documentation](https://solar-data-tools.readthedocs.io/) for a demo! For an in-depth tutorial on Solar Data Tools, we recommend taking a look at the recent webinar 
 we did with the DOE's Solar Energy Technologies Office (SETO) with our colleagues at NREL, linked below:
 
-[![IMAGE ALT TEXT HERE](https://img.youtube.com/vi/XKbqIlAEwOQ/hq1.jpg)](https://www.youtube.com/watch?v=XKbqIlAEwOQ)
+<p align="center">
+  <a href="https://www.youtube.com/watch?v=XKbqIlAEwOQ">
+ <img src="https://img.youtube.com/vi/XKbqIlAEwOQ/hq1.jpg" />
+  </a>
+  <br/>
+  </img>
+</p>
 
 You can also check the [notebooks](https://github.com/slacgismo/solar-data-tools/blob/main/notebooks/examples) folder in this repo for more examples.
 
@@ -132,13 +154,18 @@ $ conda install solar-data-tools -c conda-forge -c slacgismo -c stanfordcvxgrp
 
 ### Solvers
 
-#### QSS & CLARABEL
+#### CLARABEL
 
-By default, [QSS](https://github.com/cvxgrp/qss) and CLARABEL solvers are used for non-convex and convex problems, respectively. Both are supported by [OSD](https://github.com/cvxgrp/signal-decomposition/tree/main), the modeling language used to solve signal decomposition problems in Solar Data Tools, and both are open source. 
+By default, the [CLARABEL](https://clarabel.org/stable/) solver is used to solve the signal decomposition problems. CLARABEL (as well as other solvers) is compatible with [OSD](https://github.com/cvxgrp/signal-decomposition/tree/main), the modeling language used to solve signal decomposition problems in Solar Data Tools. Both are open source and are dependencies of Solar Data Tools. 
 
 #### MOSEK
 
-MOSEK is a commercial software package. Since it is more stable and offers faster solve times, we provide continuing support for it, however you will still need to obtain a license. If installing with pip, you can install the optional MOSEK dependency by running `pip install "solar-data-tools[mosek]"`. If installing from conda, you will have to manually install MOSEK if you desire to use it as conda does not support optional dependencies like pip. 
+MOSEK is a commercial software package. Since it is more stable and offers faster solve times,
+we provide continuing support for it (with signal decomposition problem formulations using CVXPY). However,
+you will still need to obtain a license. If installing with pip, you can install the optional MOSEK dependency by running 
+`pip install "solar-data-tools[mosek]"`. 
+If installing from conda, you will have to manually install MOSEK if you desire to use it as 
+conda does not support optional dependencies like pip. 
 
 More information about MOSEK and how to obtain a license is available here:
 
@@ -147,8 +174,9 @@ More information about MOSEK and how to obtain a license is available here:
 * [Personal academic license](https://www.mosek.com/products/academic-licenses/)
 
 ## Usage
-
-Users will primarily interact with this software through the `DataHandler` class. If you would like to specify a solver, just pass the keyword argument `solver` to `dh.pipeline` with the solver of choice. Passing QSS will keep the convex problems solver as OSQP, unless `solver_convex=QSS` is passed as well. Setting `solver=MOSEK` will set the solver to MOSEK for convex and non-convex problems by default.
+Users will primarily interact with this software through the `DataHandler` class. By default, Solar Data 
+Tools uses [CLARABEL](https://clarabel.org/stable/) as the solver all signal decomposition problems. If you would like 
+to specify another solver (such as MOSEK), just pass the keyword argument `solver` to `DataHandler.pipeline` with the solver of choice.
 
 ```python
 from solardatatools import DataHandler
@@ -159,42 +187,55 @@ pv_system_data = get_pvdaq_data(sysid=35, api_key='DEMO_KEY', year=[2011, 2012,
 dh = DataHandler(pv_system_data)
 dh.run_pipeline(power_col='dc_power')
 ```
-If everything is working correctly, you should see something like the following
+If everything is working correctly, you should see a run summary like the following
 
 ```
-total time: 24.27 seconds
+total time: 25.99 seconds
 --------------------------------
 Breakdown
 --------------------------------
-Preprocessing              11.14s
-Cleaning                   0.94s
-Filtering/Summarizing      12.19s
-    Data quality           0.25s
-    Clear day detect       1.75s
-    Clipping detect        7.77s
-    Capacity change detect 2.42s
+Preprocessing              6.76s
+Cleaning                   0.41s
+Filtering/Summarizing      18.83s
+    Data quality           0.21s
+    Clear day detect       0.44s
+    Clipping detect        15.51s
+    Capacity change detect 2.67s
 ```
 
-## Contributors
+You can also find more in-depth tutorials and guides in [our documentation](https://solar-data-tools.readthedocs.io/).
 
-Must enable pre-commit hook before pushing any contributions
-```
-pip install pre-commit
-pre-commit install
-```
 
-Run pre-commit hook on all files
-```
-pre-commit run --all-files
-```
+## Contributing
 
-## Test Coverage
+We welcome contributions of any form! Please see our [Contribution Guidelines](./CONTRIBUTING.md) for more information.
+
+## Citing Solar Data Tools
+
+If you use Solar Data Tools in your research, please cite:
+
+**Recommended citations**
+
+  Bennet E. Meyers, Elpiniki Apostolaki-Iosifidou and Laura Schelhas, "Solar Data Tools: Automatic Solar 
+  Data Processing Pipeline," 2020 47th IEEE Photovoltaic Specialists Conference (PVSC), Calgary, AB, Canada, 2020,
+  pp. 0655-0656, doi: [10.1109/PVSC45281.2020.9300847](https://doi.org/10.1109/PVSC45281.2020.9300847).
+
+  Bennet E. Meyers, Sara A. Miskovich, Duncan Ragsdale, Mitchell Victoriano, Aramis Dufour, 
+  Nimish Telang, Nimish Yadav, Elpiniki Apostolaki-Iosifidou, Claire Berschauer, Chengcheng Ding, 
+  Jonathan Goncalves, Victor-Haoyang Lian, Tristan Lin, Alejandro Londono-Hurtado, Junlin Luo, Xiao Ming, 
+  David Jose Florez Rodriguez, Derin Serbetcioglu, Shixian Sheng, Jose St Louis, Tadatoshi Takahashi, and Haoxi Zhang. (2024). 
+  slacgismo/solar-data-tools. Zenodo. doi: [10.5281/zenodo.5056959](https://zenodo.org/doi/10.5281/zenodo.5056959)
+
+**Citing technical details (_e.g._, SDT algorithms)**
+
+  Bennet E. Meyers, PVInsight (Final Technical Report). United States. [https://doi.org/10.2172/1897181](https://doi.org/10.2172/1897181)
+
+**Citing a specific version**
+
+You can also cite the DOI corresponding to the specific version of
+Solar Data Tools that you used. Solar Data Tools DOIs are listed at
+[here](https://zenodo.org/search?q=parent.id%3A5056959&f=allversions%3Atrue&l=list&p=1&s=10&sort=version).
 
-In order to view the current test coverage metrics, run:
-```
-coverage run --source solardatatools -m unittest discover && coverage html
-open htmlcov/index.html
-```
 
 ## Versioning
 

docs/README.md

@@ -1,6 +1,7 @@
 ## Instructions to getting started locally:
 
-Navigate to ./docs and run (make sure pvi-user conda env is active)
+Install the package as editable with the docs optional dependencies: `pip install -e ".[docs]"`.
+Then, navigate to `./docs` and run 
 
 ```sh
 $ make html
@@ -9,13 +10,18 @@ $ open _build/html/index.html
 
 ## Updating the documentation:
 
-All the documentation files live in docs/source. Index.rst is the landing page. Under contents is a list of all the sections to this documentation which are linked to their own rst files.
+All the documentation files live in docs/source. Index.rst is the landing page. Under contents is a 
+list of all the sections to this documentation which are linked to their own rst or md files. 
+We use `MyST` to integrate markdown into our docs pages.
 
-In order to add a new section, first update the index.rst to have the name of the section under contents. Then create a "section_name".rst file making sure that the "section_name" matches the one in index.rst. If it makes it easier, https://cloudconvert.com/md-to-rst, allows to convert mark down files to rst format.
+In order to add a new section, first update the index.rst to have the name of the section under contents. 
+Then create a "section_name".rst file making sure that the "section_name" matches the one in index.rst.
+Alternatively you can write Markdown files following the MyST guidelines for rst integration.
 
-## Auto generate API Documentation Based on Python Files in solardatatools directory
+## Auto generate API Documentation 
 
-force deletes only the existing rst files that were generated by this command to update with the new documentation in the solardatatools directory
+Force deletes only the existing rst files that were generated by this command to update with
+the new documentation in the solardatatools directory:
 
 ```sh
 $ sphinx-apidoc -o ./source ../solardatatools -f
@@ -29,9 +35,9 @@ $ make html
 $ open _build/html/index.html
 ```
 
-## Changes merged to master auto update on [solar-data-tools docs](https://solar-data-tool.readthedocs.io/en/latest/) via a web hook
+## Changes merged to main auto update on [solar-data-tools docs](https://solar-data-tool.readthedocs.io/en/latest/) via a web hook
 
 
 ## Link for [getting started with sphinx](https://docs.readthedocs.io/en/stable/intro/getting-started-with-sphinx.html)
 
-Credential for read the docs can be found in gismo team drive under PV-Insight.
+Credentials for the Read The Docs account can be found in GISMo Dashlane account.