Python Workflow (PyWf) for Internal Proprietary Projects

Overview

PyWf provides a comprehensive automation framework for managing the entire lifecycle of internal proprietary Python projects. By consolidating various development tasks into a consistent interface, it eliminates the cognitive overhead of switching between different tools and commands.

Getting Started

how_to_use.py
 1# -*- coding: utf-8 -*-
 2
 3from pathlib import Path
 4from pywf_internal_proprietary.api import PyWf
 5
 6# Initialize the PyWf object.
 7pywf = PyWf.from_pyproject_toml(Path("/path/to/pyproject.toml"))
 8
 9# Perform common development operations.
10pywf.create_virtualenv()
11pywf.remove_virtualenv()
12pywf.poetry_authorization()
13pywf.poetry_lock()
14pywf.poetry_install_only_root()
15pywf.poetry_install()
16pywf.poetry_install_dev()
17pywf.poetry_install_test()
18pywf.poetry_install_doc()
19pywf.poetry_install_all()
20pywf.run_unit_test()
21pywf.run_cov_test()
22pywf.view_cov()
23pywf.build_doc()
24pywf.view_doc()
25pywf.create_cloudflare_pages_project()
26pywf.deploy_cloudflare_pages()
27pywf.poetry_build()
28pywf.publish_to_codeartifact()
29pywf.publish_to_github_release()
30pywf.setup_codecov_io_upload_token_on_github()
31pywf.setup_cloudflare_pages_upload_token_on_github()

Core Functionality

Virtual Environment Management

  • create_virtualenv(): Creates a virtual environment using Poetry with the specified Python version

  • remove_virtualenv(): Removes the virtual environment directory

Dependency Management

  • poetry_lock(): Resolves and locks dependencies in the poetry.lock file

  • poetry_install_only_root(): Installs only the package in development mode without dependencies

  • poetry_install(): Installs the package and its core dependencies

  • poetry_install_dev(): Installs development dependencies

  • poetry_install_test(): Installs testing dependencies

  • poetry_install_doc(): Installs documentation dependencies

  • poetry_install_all(): Installs all dependency groups

Testing

  • run_unit_test(): Executes unit tests with pytest

  • run_cov_test(): Runs code coverage tests and generates reports

  • view_cov(): Opens the coverage report in your browser

Documentation

  • build_doc(): Builds documentation using Sphinx

  • view_doc(): Opens the documentation in your browser

  • deploy_versioned_doc(): Deploy documentation website to AWS S3 with version

  • deploy_latest_doc(): Deploy documentation website to AWS S3 as latest version

  • view_latest_doc(): View latest version of documentation website on AWS S3

  • create_cloudflare_pages_project(): Create Cloudflare pages project

  • deploy_cloudflare_pages(): Deploy Cloudflare pages project from docs/build/html folder

Building and Publishing

  • poetry_build(): Creates distribution packages using poetry

  • publish_to_codeartifact(): Publishes the package to AWS CodeArtifact using twine

CI/CD Integration

  • setup_codecov_io_upload_token_on_github(): Configures Codecov.io integration with GitHub Actions

  • publish_to_github_release(): Creates a GitHub Release for version tracking

Configuration

PyWf reads configuration from the [tool.pywf] section in your pyproject.toml:

TODO need to update this

[tool.pywf]
dev_python = "3.11.8"
github_account = "YourUsername"
# Create GitHub token in https://github.com/settings/tokens and put the token at
# ``${HOME}/.github/${github_account}/pac/${github_token_name}.txt``
github_token_name = "your-token-file-name"
codecov_account = "YourUsername"
# Create Codecov token in https://app.codecov.io/account/gh/${codecov_account}/access and put the token at
# ``${HOME}/.codecov/github/${codecov_account}/${codecov_token_name}.txt``
codecov_token_name = "your-token-file-name"
readthedocs_username = "yourusername"
readthedocs_project_name = "your-project-file-name"
# Create Readthedocs token in https://app.readthedocs.org/accounts/tokens/ and put the token at
# ``${HOME}/.readthedocs/${readthedocs_username}/${readthedocs_token_name}.txt``
readthedocs_token_name = "your-token"

Unified Command System

To simplify your workflow and avoid memorizing complex commands, PyWf includes a lightweight command pattern that can be integrated with Makefile support:

Command Wrappers

The bin/ directory contains thin Python wrappers for all PyWf functionality:

bin/s01_1_venv_create.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.create_virtualenv(real_run=True, verbose=True)
bin/s01_2_venv_remove.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.remove_virtualenv(real_run=True, verbose=True)
bin/s02_0_poetry_lock.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_lock(real_run=True, verbose=True)
bin/s02_1_install_only_root.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_only_root(real_run=True, verbose=True)
bin/s02_2_install.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install(real_run=True, verbose=True)
bin/s02_3_install_dev.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_dev(real_run=True, verbose=True)
bin/s02_4_install_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_test(real_run=True, verbose=True)
bin/s02_5_install_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_doc(real_run=True, verbose=True)
bin/s02_6_install_automation.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_auto(real_run=True, verbose=True)
bin/s02_7_install_all.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_all(real_run=True, verbose=True)
bin/s02_8_poetry_export.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_export(with_hash=False, real_run=True, verbose=True)
bin/s03_1_run_unit_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_unit_test(real_run=True, verbose=True)
bin/s03_2_run_cov_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_cov_test(real_run=True, verbose=True)
bin/s03_3_view_cov_result.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.view_cov(real_run=True, verbose=True)
bin/s03_4_run_int_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_int_test(real_run=True, verbose=True)
bin/s03_5_run_load_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_load_test(real_run=True, verbose=True)
bin/s04_1_build_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.build_doc(real_run=True, verbose=True)
bin/s04_2_view_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.view_doc(real_run=True, verbose=True)
bin/s04_3_deploy_versioned_doc.py
bin/s04_4_deploy_latest_doc.py
bin/s04_5_view_latest_doc.py
bin/s04_6_create_cloudflare_pages_project.py
bin/s04_7_deploy_cloudflare_pages.py
bin/s05_1_build_package.py
 1#!/usr/bin/env python
 2# -*- coding: utf-8 -*-
 3
 4"""
 5We primarily use poetry to build the package.
 6"""
 7
 8from pywf import pywf
 9
10# pywf.python_build(real_run=True, verbose=True)
11pywf.poetry_build(real_run=True, verbose=True)
bin/s05_2_publish_package.py
 1#!/usr/bin/env python
 2# -*- coding: utf-8 -*-
 3
 4"""
 5We primarily use twine to publish the package for open source project.
 6"""
 7
 8from pywf import pywf
 9
10pywf.twine_upload()
bin/s05_3_remove_package_version.py
bin/s05_4_create_release.py
bin/s06_1_setup_codecov.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.setup_codecov_io_upload_token_on_github(real_run=True, verbose=True)

These wrappers initialize PyWf using your project configuration and execute specific functions with sensible defaults.

Makefile Integration

The included Makefile provides a unified command interface:

# -*- coding: utf-8 -*-

help: ## ** Show this help message
    @perl -nle'print $& if m{^[a-zA-Z_-]+:.*?## .*$$}' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-40s\033[0m %s\n", $$1, $$2}'


venv-create: ## ** Create Virtual Environment
    ~/.pyenv/shims/python ./bin/s01_1_venv_create.py


venv-remove: ## ** Remove Virtual Environment
    ~/.pyenv/shims/python ./bin/s01_2_venv_remove.py


poetry-source-add: ## Add AWS CodeArtifact as secondary source in poetry
    python ./bin/s02_0_poetry_source_add.py


poetry-lock: poetry-source-add ## Resolve dependencies using poetry, update poetry.lock file
    ~/.pyenv/shims/python ./bin/s02_0_poetry_lock.py


install-root: ## Install Package itself without any dependencies
    ~/.pyenv/shims/python ./bin/s02_1_install_only_root.py


install: ## ** Install main dependencies and Package itself
    ~/.pyenv/shims/python ./bin/s02_2_install.py


install-dev: ## Install Development Dependencies
    ~/.pyenv/shims/python ./bin/s02_3_install_dev.py


install-test: ## Install Test Dependencies
    ~/.pyenv/shims/python ./bin/s02_4_install_test.py


install-doc: ## Install Document Dependencies
    ~/.pyenv/shims/python ./bin/s02_5_install_doc.py


install-automation: ## Install Dependencies for Automation Script
    ~/.pyenv/shims/python ./bin/s02_6_install_automation.py


install-all: ## Install All Dependencies
    ~/.pyenv/shims/python ./bin/s02_7_install_all.py


poetry-export: ## Export dependencies to requirements.txt
    ~/.pyenv/shims/python ./bin/s02_8_poetry_export.py


test: install install-test test-only ## ** Run test


test-only: ## Run test without checking test dependencies
    ~/.pyenv/shims/python ./bin/s03_1_run_unit_test.py


cov: install install-test cov-only ## ** Run code coverage test


cov-only: ## Run code coverage test without checking test dependencies
    ~/.pyenv/shims/python ./bin/s03_2_run_cov_test.py


int: install install-test int-only ## ** Run integration test


int-only: ## Run integration test without checking test dependencies
    ~/.pyenv/shims/python ./bin/s03_3_run_int_test.py


view-cov: ## View code coverage test report
    ~/.pyenv/shims/python ./bin/s03_3_view_cov_result.py


build-doc: install install-doc ## Build documentation website locally
    ~/.pyenv/shims/python ./bin/s04_1_build_doc.py


view-doc: ## View documentation website locally
    ~/.pyenv/shims/python ./bin/s04_2_view_doc.py


deploy-versioned-doc: install install-doc ## Deploy documentation website to AWS S3 with version
    ./.venv/bin/python ./bin/s04_3_deploy_versioned_doc.py


deploy-latest-doc: install install-doc ## Deploy documentation website to AWS S3 as latest version
    ./.venv/bin/python ./bin/s04_4_deploy_latest_doc.py


view-latest-doc: install install-doc ## View latest version of documentation website on AWS S3
    ./.venv/bin/python ./bin/s04_5_view_latest_doc.py


create-pages-project: ## Create Cloudflare pages project
    ./.venv/bin/python ./bin/s04_6_create_cloudflare_pages_project.py


deploy-pages: ## Deploy Cloudflare pages project from docs/build/html folder
    ./.venv/bin/python ./bin/s04_7_deploy_cloudflare_pages.py


build: ## Build Python library distribution package
    ~/.pyenv/shims/python ./bin/s05_1_build_package.py


publish: build ## Publish Python library to AWS CodeArtifact
    ~/.pyenv/shims/python ./bin/s05_2_publish_package.py


remove: ## Remove Python package version from AWS CodeArtifact
    ~/.pyenv/shims/python ./bin/s05_3_remove_package_version.py


release: ## Create Github Release using current version
    ~/.pyenv/shims/python ./bin/s05_4_create_release.py


setup-codecov: ## Setup Codecov Upload token in GitHub Action Secrets
    ~/.pyenv/shims/python ./bin/s06_1_setup_codecov.py

When you type make, you will see:

$ make
help                                     ** Show this help message
venv-create                              ** Create Virtual Environment
venv-remove                              ** Remove Virtual Environment
poetry-source-add                        Add AWS CodeArtifact as secondary source in poetry
poetry-lock                              Resolve dependencies using poetry, update poetry.lock file
install-root                             Install Package itself without any dependencies
install                                  ** Install main dependencies and Package itself
install-dev                              Install Development Dependencies
install-test                             Install Test Dependencies
install-doc                              Install Document Dependencies
install-automation                       Install Dependencies for Automation Script
install-all                              Install All Dependencies
poetry-export                            Export dependencies to requirements.txt
test                                     ** Run test
test-only                                Run test without checking test dependencies
cov                                      ** Run code coverage test
cov-only                                 Run code coverage test without checking test dependencies
int                                      ** Run integration test
int-only                                 Run integration test without checking test dependencies
view-cov                                 View code coverage test report
build-doc                                Build documentation website locally
view-doc                                 View documentation website locally
deploy-versioned-doc                     Deploy documentation website to AWS S3 with version
deploy-latest-doc                        Deploy documentation website to AWS S3 as latest version
view-latest-doc                          View latest version of documentation website on AWS S3
create-pages-project                     Create Cloudflare pages project
deploy-pages                             Deploy Cloudflare pages project from docs/build/html folder
build                                    Build Python library distribution package
publish                                  Publish Python library to AWS CodeArtifact
remove                                   Remove Python package version from AWS CodeArtifact
release                                  Create Github Release using current version
setup-codecov                            Setup Codecov Upload token in GitHub Action Secrets

When you type make cov, it actually runs python bin/s03_2_run_cov_test.py

You may also edit the Makefile yourself to use different global Python instead of ~/.pyenv/shims/python.

This approach offers several advantages:

  • Consistent command syntax across projects

  • Self-documenting commands with make help

  • No need to remember underlying tools or syntax

Getting Started with Cookiecutter

For new projects, use our Cookiecutter Template cookiecutter-pywf_internal_proprietary automatically set up the entire PyWf structure:

The template provides:

  • Pre-configured directory structure

  • Default Makefile with all common commands

  • Command wrappers in the bin/ directory

  • Properly configured pyproject.toml

  • GitHub Actions integration files

  • Documentation templates

Simply run:

pip install "cookiecutter>=2.6.0,<3.0.0" && cookiecutter https://github.com/MacHu-GWU/cookiecutter-pywf_internal_proprietary

Then follow the prompts to create a new project with the entire PyWf infrastructure ready to use.