Python Workflow (PyWf) for Internal Proprietary Projects¶
Overview¶
PyWf provides a comprehensive automation framework for managing the entire lifecycle of internal proprietary Python projects. By consolidating various development tasks into a consistent interface, it eliminates the cognitive overhead of switching between different tools and commands.
Getting Started¶
how_to_use.py
1# -*- coding: utf-8 -*-
2
3from pathlib import Path
4from pywf_internal_proprietary.api import PyWf
5
6# Initialize the PyWf object.
7pywf = PyWf.from_pyproject_toml(Path("/path/to/pyproject.toml"))
8
9# Perform common development operations.
10pywf.create_virtualenv()
11pywf.remove_virtualenv()
12pywf.poetry_authorization()
13pywf.poetry_lock()
14pywf.poetry_install_only_root()
15pywf.poetry_install()
16pywf.poetry_install_dev()
17pywf.poetry_install_test()
18pywf.poetry_install_doc()
19pywf.poetry_install_all()
20pywf.run_unit_test()
21pywf.run_cov_test()
22pywf.view_cov()
23pywf.build_doc()
24pywf.view_doc()
25pywf.create_cloudflare_pages_project()
26pywf.deploy_cloudflare_pages()
27pywf.poetry_build()
28pywf.publish_to_codeartifact()
29pywf.publish_to_github_release()
30pywf.setup_codecov_io_upload_token_on_github()
31pywf.setup_cloudflare_pages_upload_token_on_github()
Core Functionality¶
Virtual Environment Management
create_virtualenv(): Creates a virtual environment using Poetry with the specified Python versionremove_virtualenv(): Removes the virtual environment directory
Dependency Management
poetry_source_add_codeartifact(): Add CodeArtifact as a secondary sourcepoetry_authorization(): Set environment variables to allow Poetry to authenticate with CodeArtifact.poetry_lock(): Resolves and locks dependencies in the poetry.lock filepoetry_install_only_root(): Installs only the package in development mode without dependenciespoetry_install(): Installs the package and its core dependenciespoetry_install_dev(): Installs development dependenciespoetry_install_test(): Installs testing dependenciespoetry_install_doc(): Installs documentation dependenciespoetry_install_all(): Installs all dependency groups
Testing
run_unit_test(): Executes unit tests with pytestrun_cov_test(): Runs code coverage tests and generates reportsview_cov(): Opens the coverage report in your browser
Documentation
build_doc(): Builds documentation using Sphinxview_doc(): Opens the documentation in your browserdeploy_versioned_doc(): Deploy documentation website to AWS S3 with versiondeploy_latest_doc(): Deploy documentation website to AWS S3 as latest versionview_latest_doc(): View latest version of documentation website on AWS S3create_cloudflare_pages_project(): Create Cloudflare pages projectdeploy_cloudflare_pages(): Deploy Cloudflare pages project from docs/build/html folder
Building and Publishing
poetry_build(): Creates distribution packages using poetrypublish_to_codeartifact(): Publishes the package to AWS CodeArtifact using twine
CI/CD Integration
setup_codecov_io_upload_token_on_github(): Configures Codecov.io integration with GitHub Actionsedit_github_repo_metadata(): Edit GitHub repo metadata such as description and homepage URLpublish_to_github_release(): Creates a GitHub Release for version tracking
Configuration
PyWf reads configuration from the [tool.pywf] section in your pyproject.toml:
# python workflow tool config
[tool.pywf]
# The specific python version you use for local development
dev_python = "3.11.8"
# --- github.com
github_account = "MacHu-GWU"
# Create GitHub token in https://github.com/settings/tokens and put the token in
# ``${HOME}/home_secret.json``
github_token_field = "providers.github.accounts.sh.users.sh.secrets.dev.value"
# --- codecov.io (for code coverage test report)
codecov_account = "MacHu-GWU"
# Create Codecov token in https://app.codecov.io/account/gh/${codecov_account}/access
# and put the token in ``${HOME}/home_secret.json``
codecov_token_field = "providers.codecov_io.accounts.sh.users.sh.secrets.dev.value"
# --- readthedocs.org (for documentation hosting)
# Create Readthedocs token in https://app.readthedocs.org/accounts/tokens/
# and put the token at ``${HOME}/home_secret.json``
readthedocs_token_field = "providers.readthedocs.accounts.sh.users.sh.secrets.dev.value"
# Readthedocs project name, usually it is the same as your project name
readthedocs_project_name = "pywf_internal_proprietary"
Unified Command System¶
To simplify your workflow and avoid memorizing complex commands, PyWf includes a lightweight command pattern that can be integrated with Makefile support:
Command Wrappers
The bin/ directory contains thin Python wrappers for all PyWf functionality:
bin/g1_t1_s1_bootstrap.py
bin/g1_t2_s1_venv_create.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.create_virtualenv(real_run=True, verbose=True)
bin/g1_t2_s2_venv_remove.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.remove_virtualenv(real_run=True, verbose=True)
bin/g2_t1_s1_poetry_source_add.py
bin/g2_t1_s2_poetry_login.py
bin/g2_t1_s3_pip_login.py
bin/g2_t1_s4_twine_login.py
bin/g2_t1_s5_poetry_lock.py
bin/g2_t1_s6_poetry_export.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_export(with_hash=False, real_run=True, verbose=True)
bin/g2_t1_s7_uv_login.py
bin/g2_t1_s8_uv_lock.py
bin/g2_t2_s1_install_only_root.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_only_root(real_run=True, verbose=True)
bin/g2_t2_s2_install.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install(real_run=True, verbose=True)
bin/g2_t2_s3_install_dev.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_dev(real_run=True, verbose=True)
bin/g2_t2_s4_install_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_test(real_run=True, verbose=True)
bin/g2_t2_s5_install_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_doc(real_run=True, verbose=True)
bin/g2_t2_s6_install_automation.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_auto(real_run=True, verbose=True)
bin/g2_t2_s7_install_all.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.poetry_install_all(real_run=True, verbose=True)
bin/g3_t1_s1_run_unit_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_unit_test(real_run=True, verbose=True)
bin/g3_t2_s1_run_cov_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_cov_test(real_run=True, verbose=True)
bin/g3_t2_s2_view_cov_result.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.view_cov(real_run=True, verbose=True)
bin/g3_t3_s1_run_int_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_int_test(real_run=True, verbose=True)
bin/g3_t4_s1_run_load_test.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.run_load_test(real_run=True, verbose=True)
bin/g4_t1_s1_nb_to_md.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.notebook_to_markdown(real_run=True, verbose=True)
bin/g4_t2_s1_build_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.build_doc(real_run=True, verbose=True)
bin/g4_t2_s2_view_doc.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.view_doc(real_run=True, verbose=True)
bin/g4_t3_s1_deploy_versioned_doc.py
bin/g4_t3_s2_deploy_latest_doc.py
bin/g4_t3_s3_view_latest_doc.py
bin/g4_t4_s1_create_cloudflare_pages_project.py
bin/g4_t4_s2_deploy_cloudflare_pages.py
bin/g5_t1_s1_build_package.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4"""
5We primarily use poetry to build the package.
6"""
7
8from pywf import pywf
9
10# pywf.python_build(real_run=True, verbose=True)
11pywf.poetry_build(real_run=True, verbose=True)
bin/g5_t2_s1_publish_package.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4"""
5We primarily use twine to publish the package for open source project.
6"""
7
8from pywf import pywf
9
10pywf.twine_upload()
bin/g5_t2_s2_remove_package_version.py
bin/g5_t2_s3_create_release.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.publish_to_github_release(real_run=True, verbose=True)
bin/g6_t1_s1_setup_codecov.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.setup_codecov_io_upload_token_on_github(real_run=True, verbose=True)
bin/g6_t1_s2_setup_cloudflare_pages_upload_token_on_github.py
bin/g6_t1_s3_edit_github_repo.py
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4from pywf import pywf
5
6pywf.edit_github_repo_metadata(real_run=True, verbose=True)
These wrappers initialize PyWf using your project configuration and execute specific functions with sensible defaults.
Makefile Integration
The included Makefile provides a unified command interface:
1# -*- coding: utf-8 -*-
2
3help: ## ⭐ Show this help message
4 @perl -nle'print $& if m{^[a-zA-Z_-]+:.*?## .*$$}' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-40s\033[0m %s\n", $$1, $$2}'
5
6
7venv-create: ## ⭐ Create Virtual Environment
8 ~/.pyenv/shims/python ./bin/g1_t2_s1_venv_create.py
9
10
11venv-remove: ## Remove Virtual Environment
12 ~/.pyenv/shims/python ./bin/g1_t2_s2_venv_remove.py
13
14
15poetry-lock: ## ⭐ Resolve dependencies using poetry, update poetry.lock file
16 ~/.pyenv/shims/python ./bin/g2_t1_s1_poetry_lock.py
17
18
19poetry-export: ## Export dependencies to requirements.txt
20 ~/.pyenv/shims/python ./bin/g2_t1_s6_poetry_export.py
21
22
23install-root: ## Install Package itself without any dependencies
24 ~/.pyenv/shims/python ./bin/g2_t2_s1_install_only_root.py
25
26
27install: ## ⭐ Install main dependencies and Package itself
28 ~/.pyenv/shims/python ./bin/g2_t2_s2_install.py
29
30
31install-dev: ## Install Development Dependencies
32 ~/.pyenv/shims/python ./bin/g2_t2_s3_install_dev.py
33
34
35install-test: ## Install Test Dependencies
36 ~/.pyenv/shims/python ./bin/g2_t2_s4_install_test.py
37
38
39install-doc: ## Install Document Dependencies
40 ~/.pyenv/shims/python ./bin/g2_t2_s5_install_doc.py
41
42
43install-automation: ## Install Dependencies for Automation Script
44 ~/.pyenv/shims/python ./bin/g2_t2_s6_install_automation.py
45
46
47install-all: ## Install All Dependencies
48 ~/.pyenv/shims/python ./bin/g2_t2_s7_install_all.py
49
50
51test-only: ## Run test without checking test dependencies
52 ~/.pyenv/shims/python ./bin/g3_t1_s1_run_unit_test.py
53
54
55test: install install-test test-only ## ⭐ Run test
56
57
58cov-only: ## Run code coverage test without checking test dependencies
59 ~/.pyenv/shims/python ./bin/g3_t2_s1_run_cov_test.py
60
61
62cov: install install-test cov-only ## ⭐ Run code coverage test
63
64
65view-cov: ## ⭐ View code coverage test report
66 ~/.pyenv/shims/python ./bin/g3_t2_s2_view_cov_result.py
67
68
69int-only: ## Run integration test without checking test dependencies
70 ~/.pyenv/shims/python ./bin/g3_t3_s1_run_int_test.py
71
72
73int: install install-test int-only ## ⭐ Run integration test
74
75
76nb-to-md: ## Convert Notebook to Markdown
77 ~/.pyenv/shims/python ./bin/g4_t1_s1_nb_to_md.py
78
79
80build-doc: install install-doc ## ⭐ Build documentation website locally
81 ~/.pyenv/shims/python ./bin/g4_t2_s1_build_doc.py
82
83
84view-doc: ## ⭐ View documentation website locally
85 ~/.pyenv/shims/python ./bin/g4_t2_s2_view_doc.py
86
87
88build: ## Build Python library distribution package
89 ~/.pyenv/shims/python ./bin/g5_t1_s1_build_package.py
90
91
92publish: build ## ⭐ Publish Python library to Public PyPI
93 ~/.pyenv/shims/python ./bin/g5_t2_s1_publish_package.py
94
95
96release: ## ⭐ Create Github Release using current version
97 ~/.pyenv/shims/python ./bin/g5_t2_s3_create_release.py
98
99
100setup-codecov: ## ⭐ Setup Codecov Upload token in GitHub Action Secrets
101 ~/.pyenv/shims/python ./bin/g6_t1_s1_setup_codecov.py
102
103
104setup-rtd: ## ⭐ Create ReadTheDocs Project
105 ~/.pyenv/shims/python ./bin/g6_t1_s2_setup_readthedocs.py
106
107
108edit-github: ## ⭐ Edit GitHub Repository Metadata
109 ~/.pyenv/shims/python ./bin/g6_t1_s3_edit_github_repo.py
When you type make, you will see:
$ make
help ⭐ Show this help message
bootstrap ⭐ Install dependencies for Python development workflow automation
venv-create ⭐ Create Virtual Environment
venv-remove Remove Virtual Environment
poetry-source-add Add AWS CodeArtifact as secondary source in poetry
poetry-login Configure poetry with AWS CodeArtifact
pip-login Configure pip with AWS CodeArtifact
twine-login Configure twine with AWS CodeArtifact
poetry-lock ⭐ Resolve dependencies using poetry, update poetry.lock file
poetry-export ⭐ Export dependencies to requirements.txt
uv-login Configure uv with AWS CodeArtifact
uv-lock ⭐ Resolve dependencies using uv, update uv.lock file
install-root Install Package itself without any dependencies
install ⭐ Install main dependencies and Package itself
install-dev Install Development Dependencies
install-test Install Test Dependencies
install-doc Install Document Dependencies
install-automation Install Dependencies for Automation Script
install-all Install All Dependencies
test-only Run test without checking test dependencies
test ⭐ Run test
cov-only Run code coverage test without checking test dependencies
cov ⭐ Run code coverage test
view-cov ⭐ View code coverage test report
int-only Run integration test without checking test dependencies
int ⭐ Run integration test
nb-to-md Convert Notebook to Markdown
build-doc ⭐ Build documentation website locally
view-doc ⭐ View documentation website locally
deploy-versioned-doc Deploy documentation website to AWS S3 with version
deploy-latest-doc Deploy documentation website to AWS S3 as latest version
view-latest-doc View latest version of documentation website on AWS S3
create-pages-project ⭐ Create Cloudflare pages project
deploy-pages ⭐ Deploy Cloudflare pages project from docs/build/html folder
build Build Python library distribution package
publish ⭐ Publish Python library to AWS CodeArtifact
remove Remove Python package version from AWS CodeArtifact
release ⭐ Create Github Release using current version
setup-codecov ⭐ Setup Codecov Upload Token in GitHub Action Secrets
setup-cloudflare-token ⭐ Setup Cloudflare Pages Upload Token in GitHub Action Secrets
edit-github ⭐ Edit GitHub Repository Metadata
setup-github-env-var ⭐ Setup GitHub Secret Environment Variables
When you type make cov, it actually runs python bin/g3_t2_s1_run_cov_test.py
You may also edit the Makefile yourself to use different global Python instead of ~/.pyenv/shims/python.
This approach offers several advantages:
Consistent command syntax across projects
Self-documenting commands with
make helpNo need to remember underlying tools or syntax