Skip to content

Commit

Permalink
commit
Browse files Browse the repository at this point in the history
  • Loading branch information
Jacob Knightley authored and Jacob Knightley committed Jan 7, 2025
1 parent 6a09db7 commit 9c394f4
Show file tree
Hide file tree
Showing 22 changed files with 284 additions and 240 deletions.
7 changes: 5 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ instance/
.scrapy

# Sphinx documentation
docs/_build/
.readthedocs/_build/

# PyBuilder
.pybuilder/
Expand Down Expand Up @@ -143,6 +143,9 @@ venv.bak/
.dmypy.json
dmypy.json

# ruff
.ruff_cache/

# Pyre type checker
.pyre/

Expand All @@ -157,4 +160,4 @@ cython_debug/
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
#.idea/
26 changes: 26 additions & 0 deletions .readthedocs/config/build_local.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
$ErrorActionPreference = "Stop"

# Define the paths relative to the script location
$rootDir = (Resolve-Path "$PSScriptRoot\..\..\").Path
$buildDir = "$rootDir\.readthedocs\_build"
$docsDir = "$rootDir\.readthedocs\docs"
$confDir = "$rootDir\.readthedocs\config"
$requirements = "$confDir\requirements.txt"
$packageRequirements = "$rootDir\requirements.txt"

# Remove the contents of the build directory if it exists
if (Test-Path -Path $buildDir) {
Remove-Item -Recurse -Force $buildDir/*
}

# Install the requirements
pip install -r $requirements

# Install the package requirements
pip install -r $packageRequirements

# Run the Sphinx build
sphinx-build -b html $docsDir $buildDir -c $confDir

# Serve the built documentation
Start-Process -NoNewWindow -FilePath "python" -ArgumentList "-m http.server --directory `"$buildDir`""
33 changes: 33 additions & 0 deletions .readthedocs/config/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
import os
import sys

# Set location of importable modules
root_path = str(os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")))
sys.path.insert(0, root_path + "/src")
sys.path.insert(0, root_path + "/sample")

# Configuration file for the Sphinx documentation builder.

project = "fabric-cicd"
copyright = "Microsoft Corporation"
author = "Microsoft Corporation"

# Load the version
with open(root_path + "/VERSION") as version_file:
release = version_file.read().strip()

extensions = [
"sphinx.ext.autodoc",
"sphinx_autodoc_typehints",
"myst_parser",
]

html_theme = "sphinx_rtd_theme"

autodoc_default_options = {
"members": True,
"show-inheritance": True,
"special-members": "__init__",
}

suppress_warnings = ["myst.header"]
16 changes: 16 additions & 0 deletions .readthedocs/config/readthedocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Read the Docs configuration file for Sphinx projects

version: 2

build:
os: ubuntu-22.04
tools:
python: "3.10"

sphinx:
configuration: conf.py

python:
install:
- requirements: requirements.txt
- requirements: ../../requirements.txt
4 changes: 4 additions & 0 deletions .readthedocs/config/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
sphinx
sphinx_rtd_theme
sphinx_autodoc_typehints
myst_parser
11 changes: 11 additions & 0 deletions .readthedocs/docs/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
fabric_cicd
=========================

.. toctree::
:maxdepth: 4
:caption: Contents:

overview
library_reference/index
installation
usage
8 changes: 8 additions & 0 deletions .readthedocs/docs/installation.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Installation
============

To install fabric-cicd, run:

.. code-block:: bash
pip install fabric-cicd
7 changes: 7 additions & 0 deletions .readthedocs/docs/library_reference/fabric_workspace.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
fabric_workspace Module
=========================

.. automodule:: fabric_cicd.fabric_workspace
:members:
:undoc-members:
:show-inheritance:
8 changes: 8 additions & 0 deletions .readthedocs/docs/library_reference/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Library Reference
=================

.. toctree::
:maxdepth: 2

fabric_workspace
publish
7 changes: 7 additions & 0 deletions .readthedocs/docs/library_reference/publish.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
publish Module
=================

.. automodule:: fabric_cicd.publish
:members:
:undoc-members:
:show-inheritance:
6 changes: 6 additions & 0 deletions .readthedocs/docs/overview.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Overview
=========================

.. include:: ../../README.md
:parser: myst_parser.sphinx_
:start-line: 12
32 changes: 32 additions & 0 deletions .readthedocs/docs/usage.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
Usage
=====

Authenticate with Default Credentials
--------------------------------------

.. literalinclude:: ../../sample/auth_default_credential.py
:language: python

Authenticate with SPN Credentials
--------------------------------------

.. literalinclude:: ../../sample/auth_spn_secret.py
:language: python

Setting Variables within Azure DevOps Release
----------------------------------------------

.. literalinclude:: ../../sample/set_vars_from_ado_build.py
:language: python

Setting Variables within Local GIT Branch
----------------------------------------------

.. literalinclude:: ../../sample/set_vars_from_git_branch.py
:language: python

Using Optional Parameters
----------------------------------------------

.. literalinclude:: ../../sample/optional_parameters.py
:language: python
174 changes: 112 additions & 62 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,81 +1,131 @@
# Fabric CICD
[![PyPI version](https://badge.fury.io/py/fabric-cicd.svg)](https://badge.fury.io/py/fabric-cicd)

[![Language](https://img.shields.io/badge/language-Python-blue.svg)](https://www.python.org/)
[![TestPyPI version](https://img.shields.io/badge/TestPyPI-fabric--cicd-blue)](https://test.pypi.org/project/fabric-cicd/)
[![Read The Docs](https://readthedocs.org/projects/fabric-cicd/badge/?version=latest&style=flat)](https://readthedocs.org/projects/fabric-cicd/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Downloads](https://static.pepy.tech/badge/fabric-cicd)](https://pepy.tech/project/fabric-cicd)
[![Code style: ruff](https://img.shields.io/badge/code%20style-ruff-000000.svg)](https://github.com/charliermarsh/ruff)

<!-- [![PyPI version](https://badge.fury.io/py/fabric-cicd.svg)](https://badge.fury.io/py/fabric-cicd) -->
<!-- [![Downloads](https://static.pepy.tech/badge/fabric-cicd)](https://pepy.tech/project/fabric-cicd) -->

[Read full documentation on ReadTheDocs](https://fabric-cicd.readthedocs.io/en/latest/)

---
[Read the documentation on ReadTheDocs!](https://fabric-cicd.readthedocs.io/en/stable/)
---

Fabric CICD is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library is intended to support code first CICD integrations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal of this library is to support the CICD developers that don't want to dig into the weeds of interacting directly with the Microsoft Fabric APIs.
## Project Overview

Fabric CICD is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library is intended to support code first CICD integrations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal of this library is to support the CICD developers that don't want to dig into the weeds of interacting directly with the Microsoft Fabric APIs.
Fabric CICD is a Python library designed for use with [Microsoft Fabric](https://learn.microsoft.com/en-us/fabric/) workspaces. This library is intended to support code first CICD integrations to seamlessly integrate Source Controlled workspaces into a deployment framework. The goal of this library is to support the CICD developers that don't want to dig into the weeds of interacting directly with the Microsoft Fabric APIs.

If you encounter any issues, please [raise a bug](https://github.com/microsoft/fabric-cicd/issues/new?assignees=&labels=&projects=&template=bug_report.md&title=).

If you have ideas for new features/functions, please [request a feature](https://github.com/microsoft/fabric-cicd/issues/new?assignees=&labels=&projects=&template=feature_request.md&title=).

## Featured Scenarios
### Featured Scenarios

- Deploying Items hosted in a repository
- 100% deployment, does not consider diffs
- Deploys into the tenant of the executing identity

### In Scope Item Types
- Notebooks
- Data Pipelines
- Environments


- Notebooks
- Data Pipelines
- Environments


- Notebooks
- Data Pipelines
- Environments

### Limitations
- Notebooks Limitations
- Attached lakehouses are not changed during the deployment
- Attached resources are not included in the deployment
- Data Pipelines
- Connections are not changed during the deployment
- Environments
- Custom and Public Libraries are not included in the deployment
- Folders
- Sub folders are not included in the deployment

## How to Use
```shell
pip install fabric-cicd
```

```python
from fabric_cicd import FabricWorkspace, publish_all_items, unpublish_all_orphan_items

# Initialize the FabricWorkspace object with the required parameters
target_workspace = FabricWorkspace(
workspace_id='',
environment='',
repository_directory=r'',
item_type_in_scope=[]
)

# Publish all items defined in item_type_in_scope
publish_all_items(target_workspace)

# Unpublish all items defined in item_type_in_scope not found in repository
unpublish_all_orphan_items(target_workspace)
```

## Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide
a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions
provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
contact [[email protected]](mailto:[email protected]) with any additional questions or comments.

- **Notebooks**:
- Attached lakehouses are not changed during deployment
- Attached resources are not included in deployment
- **Data Pipelines**:
- Connections are not changed during deployment
- **Environments**:
- Custom and public libraries are not included in deployment
- **Folders**:
- Sub folders are not included in deployment

### Authentication

- Optionally provide your own credential object aligned with the [TokenCredential class](https://learn.microsoft.com/en-us/dotnet/api/azure.core.tokencredential) in the `token_credential` parameter.
- If the `token_credential` parameter is omitted, the library uses the Azure SDK `DefaultAzureCredential` for authentication.
- Refer to the [Azure SDK documentation](https://learn.microsoft.com/en-us/azure/developer/python/sdk/authentication/credential-chains?tabs=dac#defaultazurecredential-overview) for the order in which credential types are attempted.
- For local development with a UPN, install either the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows) or [Az.Accounts](https://www.powershellgallery.com/packages/Az.Accounts/2.2.3).
- Note: When no credential is provided, the `DefaultAzureCredential` may choose an unexpected identity. For instance, if you log in to the Azure CLI with an SPN but log in to Az.Account with a UPN, the `DefaultAzureCredential` will choose the CLI authentication first.

## Contribute

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit [https://cla.opensource.microsoft.com](https://cla.opensource.microsoft.com).

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.

### Requirements

Before you begin, ensure you have the following installed:

- [Python](https://www.python.org/downloads/) (version 3.10 or higher)
- [PowerShell](https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell)
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-windows) or [Az.Accounts PowerShell module](https://www.powershellgallery.com/packages/Az.Accounts/2.2.3)
- [Visual Studio Code (VS Code)](https://code.visualstudio.com/)

### Initial Configuration

1. Clone the repository:

```sh
git clone https://github.com/microsoft/fabric-cicd.git /your/target/directory
cd /your/target/directory
```

1. Create a virtual environment:

```sh
python -m venv venv
```

1. Activate the virtual environment:

- On Windows:

```sh
.\venv\Scripts\activate
```

- On macOS and Linux:

```sh
source venv/bin/activate
```

1. Install the dependencies:

```sh
pip install -r requirements.txt
pip install -r requirements-dev.txt
```

1. Open the project in VS Code and ensure the virtual environment is selected:

- Open the Command Palette (Ctrl+Shift+P) and select `Python: Select Interpreter`.
- Choose the interpreter from the venv directory.
## Support
### How to file issues and get help
This project uses GitHub Issues to track bugs, feature requests, and questions. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug, feature request, or question as a new Issue.
### Microsoft Support Policy
Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft
trademarks or logos is subject to and must follow
[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
Any use of third-party trademarks or logos are subject to those third-party's policies.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.
Loading

0 comments on commit 9c394f4

Please sign in to comment.