Title

class: center, middle, inverse

# Introduction to Python packaging

## Uwe Schmitt

## uwe.schmitt@id.ethz.ch

## 26. July 2016

---
class: middle

### Preparation

````bash
$ pip install -U pip
$ pip install -U wheel
$ pip install -U virtualenv  # skip this line if you use anaconda or miniconda
````
---
class: middle

### Topics

- `pip` and Python cheese shop (Python Package index)

- `virtualenv` for isolated Python development environments

- Python packages

- Code layout for testing

- Distributing Python packages

---
class: middle

### pip

* `pip` is the recommended tool for installation and deinstallation of Python packages.

* It is able to resolve dependencies (you install package A which needs package B which
is not yet installed on your machine)

* https://pip.pypa.io/en/latest/index.html

### PyPI aka "cheese shop"

Python Package Index (http://pypi.python.org) is the common site to share public
Python packages and is used by tools like `pip` as the default source for fetching
and installing packages.

---
class: middle

### virtualenv

* `virtualenv` creates isolated Python installations on your machine and so helps to avoid
version conflicts and keeps your machine clean on the long term.

* I use a separate virtual environment for every script, library I implement
and every test installation of external packages (disk space is cheap these
days).

* https://virtualenv.pypa.io/en/latest/index.html

* if you use Andaconda lookup the `conda create` command instead, skip the following demos for `virtualenv`.

---
class: middle

### Create a virtual environment

````bash
$ cd
$ mkdir -p tmp
$ cd tmp
$ mkdir demo
$ cd demo
$ virtualenv env
...
````

This created a folder named `env` which holds all the files related to this virtual environment:
````bash
$ ls env
bin/  lib/  includes/

$ ls env/bin
activate  python  python2.7
````

On windows it will more look like this:
```makefile
C:\...> dir env
Scripts  lib  includes

C:\...> dir env\Scripts
activate  activate.bat  python.exe  python2.7.exe
```

---
class:middle

### Use a virtual environment

Activate the virtual environment (on Mac and linux):
````bash
$ source env/bin/activate
$ which python
env/bin/python

$ python -c "import os; print(os.__file__)"
env/lib/python2.7/os.pyc
````

Activate the virtual environment (on Windows):
````makefile
C:\...> env\Scripts\activate.bat
C:\...> where python
env/bin/python
C:\...> python -c "import os; print(os.__file__)"
env\lib\python2.7\os.pyc
````

---
class: middle

### Install packages into the virtual environment
````bash
$ python -c "import requests"
...
ImportError: No module named requests

$ pip install requests
$ python -c "import requests; print(requests.__file__)"
env/lib/python2.7/site-packages/requests/__init__.pyc

$ pip uninstall requests
$ python -c "import requests"
...
ImportError: No module named requests
````

---
class: middle

### To "switch back" to the global installation
````bash
$ deactivate
$ which python
/usr/local/bin/python
````

Comment: `deactivate` is not needed if you activate another virtual environment.

### If you remove the `venv` folder the full environment disappears, no traces left.
---
class: middle
### Minimal Python package: setup folders and virtualenv

Please execute the following statements to setup a folder structure for an experimental
Python project, this is a prerequisite for the
following examples.

You can skip the two lines related to `git` if you do not use git (yet !!!)

We use `vim` in the examples below, you can use any text editor you like.

````bash
$ cd ~/tmp
$ mkdir package_demo_project
$ cd package_demo_project
$ virtualenv env
$ source env/bin/activate

$ mkdir src
$ cd src
$ git init .    # only if you use git

$ vim .gitignore    # only if you use git
````

Insert the following line in `.gitignore` if you use `git`:
````git
*.pyc
````

Now git will ignore the `.pyc` files.

---
class: middle
### Minimal Python package

* single Python files are named "modules"
* a collection of Python modules can be organized as an "package".
* this can be nested, so a package may have sub packages.

**When reproducing the examples below replace `_XXX` in the package and folder names
by a unique name** (for example your NETHZ name). Later we will upload the package to the Python package index which
requires a unique package name !

Setup package folders and files
````bash
$ pwd
~/tmp/package_demo_project

$ mkdir tutorial_package_XXX
$ touch tutorial_package_XXX/__init__.py
````

The marker file `__init__.py` we created above is necessary, else importing modules inside
`tutorial_package_XXX` will not work !

---
class: middle
### Minimal Python package continued

This is the current layout of our project:
````bash
* package_demo_project
├── env
│   └── ...
└── tutorial_package_XXX
    └── __init__.py
````

The `*` above marks the current working folder !

````bash
$ vim tutorial_package_XXX/greet.py
````

edit `greet.py`:
````python
def say_hello(who):
    print(greeting(who))

def greeting(who):
    return "hi %s" % who
````

````bash
$ ls tutorial_package_XXX
__init__.py  greet.py
````

---
class: middle

### Check your package

After the previous steps we can import `tutorial_package_XXX` now:

````bash
* package_demo_project
├── env
│   └── ...
└── tutorial_package_XXX
    ├── __init__.py
    └── greet.py
````

````bash
$ python -c "import tutorial_package_XXX.greet as m; m.say_hello('urs')"
hi urs
````

This only works because the folder `tutorial_package_XXX` is located in our
current working directory.  We will see how to fix this later (see "Install
package locally in development mode").

---
class: middle

### Improvement 1

edit `greet.py`:
````python
def say_hello(who):
    print(greeting(who))

def greeting(who):
    return "hi %s" % who

def main():
    import sys
    say_hello(sys.argv[1])

if __name__ == "__main__":
    main()
````
```` bash
$ python -c "import tutorial_package_XXX.greet"
$ cd tutorial_package_XXX
$ python greet.py uwe
hi uwe
$ cd ..
````

* `main()` is not executed when importing the module
* ... but if you run the module like a Python script.
* can be used to write preliminar test code for a single module

---
class: middle

### Improvement 2

If you just import the package the modules are not loaded automatically, so you get
the following error:

````bash
$ python -c "import tutorial_package_XXX.greet"
$ python -c "import tutorial_package_XXX as m; m.greet.say_hello('uwe')"
...
ImportError: No module named tutorial_package_XXX.greet
````

To fix this:
```` bash
$ vim tutorial_package_XXX/__init__.py
````

add one line to `__init__.py`:

````python
import greet
````

`__init__.py` is loaded and executed if you import the package. This is the place
for package initialization code. Now the `import` shown above works:

```` bash
$ python -c "import tutorial_package_XXX as m; m.greet.say_hello('uwe')"
hi uwe
````
---
class: middle

### For `git` users

````bash
$ pwd
~/tmp/package_demo_project
$ git add tutorial_package_XXX
$ git commit -m "initial version"
````
---
class: middle
### setup.py

* `setup.py` provides package meta data needed for distributing the package

First check if your working directory is one level above your package:

````bash
$ pwd
~/tmp/package_demo_project
$ vim setup.py
````

edit `setup.py`:
````python
from setuptools import setup

````

`setup()` accepts many other keyworad arguments to provide information about your package:
for example a longer textual description, dependencies on other packages or
search terms for PyPI.
---
class: middle

### Install package locally in development mode

````bash
* package_demo_project
├── env
│   └── ...
├─ setup.py
└── tutorial_package_XXX
    ├── __init__.py
    └── greet.py
````

````bash
$ pip install -e .
````

This installs your current package into the current virtual environment in *edit mode*:

- all changes of the source code of your package are available immediately **without** running `pip install -e .` again.
- You only must run `pip install -e .` if you change `setup.py`.

---
class:middle
### For `git` users

````bash
$ pwd
~/tmp/package_demo_project
$ git setup.py
$ git commit -m "added setup.py"
````
---
class: middle

### Check installation

The package can now be imported from any folder. The `cd` changes to your home directory:
````bash
$ cd
$ pwd
/home/schmittu
$ python -c "import tutorial_package_XXX as m; m.greet.say_hello('uwe')"
hi uwe
````
Now we go back to our project folder, `cd -` jumps back to the location before your last `cd`:
````bash
$ cd -
$ pwd
~/tmp/package_demo_project
````

---
class: middle

### Create a "wheel" file for manual distribution

http://pythonwheels.com:
"Wheels are the new standard of python distribution and are intended to replace eggs."

Wheels can handle platform dependend installations if you package contains some C code.

````bash
$ python setup.py bdist_wheel
$ ls dist/
tutorial_package_XXX-0.0.1-py2-none-any.whl
````
You can send this file to colleagues who can install your package with `pip`:

````bash
(other host)$ pip install tutorial_package_XXX-0.0.1-py2-none-any.whl
(other host)$ pip uninstall tutorial_package_XXX
````

### Alternative: source distributions
````bash
$ python setup.py sdist
$ ls dist/
tutorial_package_XXX-0.0.1-py2-none-any.whl  tutorial_package_XXX-0.0.1.tar.gz
````

---
class: middle
### Minimal setup for unit testing

I recommend `py.test` (http://pytest.org/latest/) for running and writing tests in Python.

````bash
$ pip install pytest
$ mkdir tests
$ vim tests/test_main.py
````
````python
def test_import():
    import tutorial_package_XXX

def test_greeting():
    import tutorial_package_XXX
    assert tutorial_package_XXX.greet.greeting("monty") == "hi monty"
````

As we installed our package in edit mode, the import statements in the tests
will work whatever your current working directory when running the tests is.
---
class: middle

### Run the tests

````python
* package_demo_project
├── env
│   └── ...
├─ setup.py
├── tutorial_package_XXX
│   ├── __init__.py
│   └── greet.py
└── tests
    └── test_main.py
````

And now we run the tests in the `test` folder:

````bash
$ py.test tests/
============================= test session starts ==============================
platform darwin -- Python 2.7.8 -- py-1.4.26 -- pytest-2.7.0
rootdir: /Users/uweschmitt/Projects/hands_on_python_packaging, inifile: 
collected 2 items

tests/test_say_hello.py ..

=========================== 2 passed in 0.01 seconds ===========================
````

---
class:middle
### For `git` users

````bash
$ pwd
~/tmp/package_demo_project
$ git add tests
$ git commit -m "added first tests"
````
---
class: middle
### Minimal setup for unit testing continued

We modify now the `test_greeting` function so that the test will fail:

````python
def test_greeting():
    import tutorial_package_XXX
    assert tutorial_package_XXX.greet.greeting("monty") == "hello monty"
````

Running `py.test` detects the broken test and provides helpful information:

````bash
$ py.test tests/
============================= test session starts ==============================
platform darwin -- Python 2.7.8 -- py-1.4.27 -- pytest-2.7.0
rootdir: /Users/uweschmitt/tmp, inifile: 
collected 2 items

tests/test_main.py .F

=================================== FAILURES ===================================
________________________________ test_greeting _________________________________

def test_greeting():
        import tutorial_package_XXX
>       assert tutorial_package_XXX.greet.greeting("monty") == "hello monty"
E       assert 'hi monty' == 'hello monty'
E         - hi monty
E         + hello monty

tests/test_main.py:8: AssertionError
====================== 1 failed, 1 passed in 0.01 seconds ======================
````

---
class:middle
### For `git` users: undo latest changes

````bash
$ pwd
~/tmp/package_demo_project
$ git checkout tests
$ py.test tests/

============================= test session starts ==============================
platform darwin -- Python 2.7.8 -- py-1.4.26 -- pytest-2.7.0
rootdir: /Users/uweschmitt/Projects/hands_on_python_packaging, inifile: 
collected 2 items

tests/test_say_hello.py ..

=========================== 2 passed in 0.01 seconds ===========================
````
---
class: middle
### How to upload your package to PyPI

* First create an account at http://pypi.python.org if you do not have one !
* Do this once to register your package:
````bash
$ python setup.py register
````
* Do this the first time and for every update:
````bash
$ python setup.py bdist_wheel upload
````
* Now look at http://pypi.python.org and search for the package

---
class: middle

### Now install from PyPI

Ask your course neighbour about his package name and install it with `pip` in a 
fresh virtual environment !

````bash
$ cd ~/tmp
$ virtualenv test_env
$ source test_env/bin/activate # this automatically deactivates the previous env
$ pip install tutorial_package_XXX
...
Successfully installed tutorial-package-uwe-0.0.1
$ python -c "import tutorial_package_XXX as m; m.greet.say_hello('uwe')"
hi uwe
````

And we clean up:
````bash
$ deactivate
$ rm -rf test_env
````
---
class: middle

### Installation dependencies

On the last slides we show how to specify installation requirements. Here we want
to extend our package so that it greets in big letters:

````bash
$ pwd
~/tmp/package_demo_project
````

edit `setup.py`:
````python
from setuptools import setup

setup(
    name="tutorial_package_XXX",
    version="0.0.1",
    author="Uwe Schmitt",
    description="An demonstration of how to create and publish python packages",
    license="BSD",
    packages=['tutorial_package_XXX'],
    install_requires=['big_letters'],
)
````

As we modified `setup.py` we have to run the local installation again, and you should see 
in the output that
`big_letters` package is installed:

````bash
$ pip install -e .
````

---
class: middle
### Now we extend our package

edit `greet.py`:

````python
from big_letters import print_big

def say_hello(who):
    print(greeting(who))

def greeting(who):
    return "hi %s" % who

def greet_big(who):
    print()
    print_big("hi %s" % who)

def main():
    import sys
    say_hello(sys.argv[1])

if __name__ == "__main__":
    main()
````

---
class: middle

### Check our changes

As we installed the package in edit mode those changes are immediately available:

````bash
$ python -c "import tutorial_package_XXX as m; m.greet.greet_big('uwe')"

*   *  ***         *   *  *     *  *****
*   *   *          *   *  *     *  *
*****   *          *   *  *  *  *  ***
*   *   *          *   *  *  *  *  *
*   *  ***          ***    ** **   *****
````
---
class:middle
### For `git` users

````bash
$ pwd
~/tmp/package_demo_project
$ git add setup.py
$ git add tutorial_package_XXX/greet.py
$ git commit -m "added greeting in big letters"
````

---
class:middle
### Upload the newest version to PyPI

In order to upload the modifications of your package to PyPI **you must increase the version
numer** in `setup.py`:

````bash
$ vim setup.py
....
````
For `git` users:
````bash
$ git add setup.py
$ git commit -m "release 0.0.2"
````

After uploading the updates with

````bash
$ python setup.py bdist_wheel upload
````

every user of your package only needs to run

````bash
$ pip install -U demo_package_XXX
````

which will install your modifications as well as the needed package `big_letters`.

---
class: middle
### Run a specific Python function from the command line

````bash
$ vim setup.py
````
add `entry_points` setting:
````python
from setuptools import setup

Impact of this modification:

* After installation of `tutorial_package_XXX` you can run `greeter`  on the command line
* this will run the Python function `main` which is located in the module `greet.py` inside the `tutorial_package_XXX` package !

---
class: middle
### Update your packages installation

Because we modified `setup.py`:
````bash
$ pip install -e .
````

### Run your script

````bash
$ which greeter
env/bin/greeter

$ greeter urs
hi urs
````

* Everybody who installs your package with `pip install tutorial_package_XXX` now has the command line tool `greeter` available.
* This is the recommended way to implement larger applications too !

Impacts:

* Distribution over PyPI
* Installation, upgrade and deinstallation with `pip`
* Clean installation in virtual environments.
* Clean installation of command line tool

---
class:middle
### For `git` users

````bash
$ pwd
~/tmp/package_demo_project
$ git add setup.py
$ git commit -m "implemented command line tool 'greeter'"
````
---
class: middle

### Some links:

Packaging introductions and references:

* http://www.scotttorborg.com/python-packaging/minimal.html
* https://packaging.python.org/en/latest/

Easy and elegant command line interfaces:

* Use http://click.pocoo.org/5/ !!!

General site about Python best practices:

* Hitchhikers Guide to Python: http://docs.python-guide.org/en/latest/

---

class: center, middle, inverse

This slide show was created with 
http://remarkjs.com/