Python Packaging

Overview

Create python packages that can be distributed with pip from GitHub (and not PyPi).

Configuration

Setup.py

A setup config file, setup.py, in the repo root directory should contain a single call to setuptools.setup(), like so:

from setuptools import setup

setup(name='namespace-my-package',
      version='0.1',
      description='My package description',
      url='http://github.com/dbarchowsky/namespace-my-package',
      author='Damien Barchowsky',
      author_email='dbarchowsky@gmail.com',
      license='MIT',
      packages=['namespace.my_package'],
      install_requires=['list', 
            'of', 
            'dependencies',
            ],    
      zip_safe=False)

After which the package can be installed locally with

$ pip install .

And once installed the package can be used like this:

>>> from namespace.my_package import my_module
>>> my_module.some_routine()

Namespaces

It's recommended to use namespaces when creating packages. Even if the package is not being publicly distributed, the names of dependencies from other packages can clash.

The recommended file structure is this:

mynamespace-subpackage-a/
    setup.py
    mynamespace/
        subpackage_a/
            __init__.py
            module_aa.py

Note that there is no __init__.py in the mynamespace directory. Only packages should have __init__.py files.

Including sub-packages in distribution

Each module in the package must be explicitly added to the distribution under the packages setting in setup.py.

For a project with packages under the top-level package, e.g.:

mynamespace-subpackage-a/
    setup.py
    mynamespace/
        my_package/
            __init__.py
            subpackage_a/
                __init__.py
            subpackage_b/
                __init__.py
            module_aa.py

To include the non-top-level packages, add them to packages in setup.py:

from setuptools import setup

setup(name='mynamespace-my-package',
    ''' [...] '''
    packages=['mynamespace.my_package',
              'mynamespace.my_package.subpackage_a',
              'mynamespace.my_package.subpackage_b',
    ],
    ''' [...] '''

Installation from GitHub

Prerequisites

Create and store a local SSH key. (One each for all computers where development is happening.) See Generating a new SSH key and adding it to the ssh-agent (GitHub Help)

Register the key on the GitHub account. See Adding a new SSH key to your GitHub account (GitHub Help)

One key pair will suffice for all ssh requests, i.e. for a computer with the key installed, the key can be used for all repositories in the GitHub account.

Installing a package

To install a package from a private GitHub repository:

$ pip install git+ssh://git@github.com/username/my-repo.git

The packages dependencies are listed under install_requires=[] in the package's setup.py.

Updating a package

After changes have been committed to a repository, they can be retrieved with:

$ pip install git+ssh://git@github.com/username/my-repo.git --upgrade

(or) 

$ pip install git+ssh://git@github.com/username/my-repo.git -U