You ship the job. It passes CI. Then, a colleague runs it in a notebook environment, and suddenly your shared utility library throws an ImportError because it pulled in a different version of pandas than the one used in your orchestration layer.
Many data teams still manage shared code by appending directory paths to sys.path or bundling messy setup.py scripts that haven't been updated since the mid-2010s. This is a fragile architecture that invites inconsistency. If you want a robust data platform, you should stop treating your internal tooling like an afterthought and start packaging it as a proper Python distribution using pyproject.toml.
Why the common approach falls short
The "copy-paste-folder" or "sys.path.append" strategy is an engineering debt bomb. When you rely on environment-level path hacks, you lose the ability to manage versioning, track dependencies, and enforce consistent execution environments across local development, CI/CD runners, and production compute clusters.
Legacy setup.py files are equally problematic. They rely on arbitrary code execution during the installation process, which is opaque, slow, and increasingly discouraged by the Python Packaging Authority (PyPA). If your build process requires a Python script to resolve its own environment, you’ve already lost control of your supply chain.
The transition to PEP 517 and PEP 621 standards, centered on the pyproject.toml file, moves configuration away from executable code into static metadata. This allows build tools to understand your project structure without running arbitrary code, leading to safer, more predictable builds.
Standardizing with pyproject.toml
To build a modern shared library, you need to define your metadata and build backend in pyproject.toml. I recommend using hatchling or setuptools (>= 61.0.0). Here is a standard configuration that ensures your library is installable and its dependencies are tracked:
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "data-utility-lib"
version = "0.1.0"
authors = [
{ name = "Data Platform Team", email = "[email protected]" },
]
description = "Shared utilities for data pipelines"
requires-python = ">=3.9"
dependencies = [
"pandas>=2.0.0",
"pydantic>=2.0.0",
"pyarrow>=12.0.0",
]
[project.optional-dependencies]
dev = [
"pytest",
"black",
"ruff",
]
By defining these requirements in a static file, you enable tools like pip or uv to resolve the entire dependency graph before installation begins. This prevents the "dependency drift" that occurs when an environment installs one version of a library at the start of a pipeline and a conflicting version halfway through a dependency resolution chain.
Enforcing consistent environments
Once your library is packaged, you should treat it as a primary dependency in your project-specific environments. In your downstream data pipelines, you no longer rely on shared folders; you reference the library in your pyproject.toml or requirements.txt via a private package registry or a direct git URL.
When you install your library, you are pulling a defined artifact. If you need to upgrade a core dependency like pyarrow, you update the pyproject.toml in your library, bump the version, and propagate that change to your pipelines. This creates an auditable trail of who changed which dependency and when—a critical requirement for any regulated environment where data lineage and software provenance are non-negotiable.
Photo by Testalize.me on Unsplash
The objections (and my answers)
"But we have a small team, and packaging feels like overkill."
It feels like overkill until you spend four hours debugging why a UDF in a Spark job works locally but crashes on the cluster. Packaging your code forces you to define exactly what your code needs to run. If you can't state your dependencies explicitly in a pyproject.toml, you don't actually know what your code is doing.
"Won't this slow down our development velocity?"
Initially, yes. You have to learn how to version your internal library. However, the alternative is "dependency hell," where every engineer has a slightly different version of numpy or pandas in their local environment, leading to "works on my machine" syndromes that cost far more developer time than a formal packaging process ever will.
"What about internal package management?"
You don't need a complex Artifactory setup to start. You can install directly from a private Git repository using pip install git+https://github.com/org/[email protected]. Once your team matures, you can move to a simple private PyPI mirror, but the pyproject.toml format remains the constant that ensures your setup is compatible with modern ecosystem tooling.
Conclusion
Packaging your shared code is not just about "cleaner project structure"; it is about platform stability. By moving away from imperative setup.py scripts and loose file structures toward declarative pyproject.toml configurations, you transform your shared library from a liability into a reliable piece of infrastructure.
Stop appending to sys.path. Start building artifacts. Your pipelines—and your future self—will thank you when you can finally deploy with confidence.
Cover photo by Juanjo Jaramillo on Unsplash.
This article was drafted with AI assistance and is intended for technical review and fact-checking before publication.
Top comments (0)