Faster python deployments with wheels

One of the most annoying issues I have with python packaging system is time it takes to deploy any non-trivial app. Recent projects I was working on have large list of several packages they depend on,
which again have their own dependencies. This is typically specified as requirement.txt file that can be processed by pip (pip install -r requirements.txt), which may look like this:

django==1.6
djangorestframework==2.3.10
psycopg2==2.5.2
south==0.8.4
...

(small tip: if you want quickly discover latest version of package, use yolk).

Such list tends to grow with your project, and its hard to ever remove anything from it.
The main problem with the way pip handles it is that:

1. Pip processes it sequentially, so your 16 cores and your network pipe are underutilised, and all download times just add up.
2. Compilation of  complex extensions take forever. (and amazon micro instances requires setting up swap to even be able to do that).

There are few ways to deal with this problem. You can start with setting up download cache for pip,

which obviously will help with download times. You can create and reuse single environment, which will store all packages and pip will only install or update packages that were changed or added. This approach generally works, but once in a while update goes wrong and you may spend long time trying to figure how to fix it, so I prefer to build fresh environment every time. Or you can invent your own way of doing it. Either way until very recently setting up deployment properly required certain amount of tinkering with the way python packages are build and deployed. (Well, it still does, but the amount has been greatly reduced.)

So if you hate wasted time and complexity introduced by compiling c extensions on every host, you will (almost) love wheel. Wheel is new format for storing and deploying python packages, and main advantage is that it allows to include compiled code in it. So finally its possible to compile all packages on build machine,

and deploy binary form to all target hosts easily. This is still a bit of a bleeding edge, as only recently released pip 1.5.4 fixed a bug related to downloading dependencies that was making wheels practically useless.

It is however working properly now, so lets enter brave new world:

mkdir test && cd test

virtualenv .

. ./bin/activate

pip install wheel

pip install --upgrade pip>=1.5.4

mkdir wheels

and finally the most important bit:

pip wheel --wheel-dir wheels -r requirements.txt

(one more tip: with recent version of pip and certain packages you run into problems with pip not willing to download externally hosted files. In that case you may want to add them as exceptions with --allow-external and --allow-unverified flags).

This will create wheels containing all required packages (and their dependencies) which can be distributed with your app (at least to machines with the same architecture/os/lib versions, which is all I care for).

The only issue I have is that for reasons I completely don't understand, pip wheel command

does not use wheel directory as a cache, building everything from scratch every time. Sequentially of course.

So just putting it into deployment script still will result in great amount of wasted time.

Luckily this simple script will solve the problem:

$ cat build_new_wheels.py

#!/usr/bin/env python
"""
Obtain packages listed in requirement file
and download/build wheels for them as needed

USAGE:

wheels.py WHEEL_DIR REQUIREMENTS_FILE

"""
import os
import sys
import subprocess

def check_wheel(pkg, ver, wheels):
"""
Check if there is wheel for given pkg/version. Note that python version and arch is ignored here, so it will break if you mix them.

"""
_pkg = pkg.lower().replace('-', '_')
s = _pkg
if ver:
s = '{0}-{1}-'.format(_pkg, ver)
for wheel in wheels:
if wheel.lower().startswith(s):
return True
return False


WHEEL_DIR = sys.argv[-2]
WHEELS = os.listdir(WHEEL_DIR)
REQ_FILE = sys.argv[-1]
PACKAGES = []
lines = []
with open(REQ_FILE) as f:
lines = f.readlines()
for line in lines:
line = line.strip()
if line and not line.startswith('#'):
if '==' in line:
pkg, ver = line.split('==')
else:
pkg, ver = line, None
PACKAGES.append((pkg,ver))

for pkg, ver in PACKAGES:
build = True
if not check_wheel(pkg, ver, WHEELS):
print 'building', pkg, ver
pkg_spec = pkg
if ver:
pkg_spec = '{0}=={1}'.format(pkg, ver)
exit_code = subprocess.call(['pip', 'wheel', '--wheel-dir', WHEEL_DIR, pkg_spec])
if exit_code != 0:
sys.stderr.write('Error building wheel for {0}\n'.format(pkg_spec))
os.exit(1)

exit_code = subprocess.call(['pip', 'install', '--no-index', '--find-links', WHEEL_DIR, '-r', REQ_FILE])
if exit_code != 0:
print 'pip exited with non-zero exit code'
os.exit(1)

You can use it simply by specifying wheel directory and requirement file:

$ ./build_new_wheels.py wheels requirements.txt

and it will only build wheels that don't exists in wheels directory. Note that while this script is rather proof of concept and does not support all features that can be used in requirements file or all wheel options (>= operator, separation of various python versions, git or file repositories, mixing python versions),

it allows me to only build wheels for packages that were introduced or changed since last build.

Parallel processing could also be easily added here  thanks to multiprocessing module.

I really would like to see this (or similar) behaviour added to pip, as that would finally make it fully usable without custom work.

If you want to know more about wheel format, go rigtht there: http://wheel.readthedocs.org/en/latest/

UPDATE: Work on caching wheels is happening here: https://github.com/pypa/pip/pull/1572

Choosing Elasticsearch client for python

Recently one of my co-workers asked me about my choice of Elasticsearch python client. This the is longer version of my answer.
(tl;dr - I've chosen pyes because it has batteries included).

First: Why do I need a client and what do I need it for?

Elasticsearch is a webservice. All you need is to make http call.
In a simplest case, with one server and fairly straightforward queries,
anything that can make GET and POST request (like requests - this really should in python standard library)
will work just fine. What I need however is far from simple case.

First of all, when I'm accessing ES cluster with several nodes,
I need to deal with occasional failures. At the very list client should be able
to specify connection timeout and amount of retries.

Some client implement connection pooling, loadbalancing and failover, but since dedicated
loadbalancer is much better at handling all of those, I don't care about client support for that.
(this also the reason for using http instead of thrift).

Second: while simple ES queries are easy to write by hand, this is what I'm frequently dealing with:

{
  "sort": [
    {
      "follows.date_added": {
        "order": "desc",
        "nested_filter": {
          "terms": {
            "follows.owner_id": [
              1
            ]
          }
        }
      }
    },
    {
      "entries.usd_price": {
        "order": "asc",
        "nested_filter": {
          "bool": {
            "must": [
              {
                "bool": {
                  "must_not": [
                    {
                      "term": {
                        "entries.disallow_countries": "US"
                      }
                    }
                  ],
                  "must": [
                    {
                      "terms": {
                        "entries.allow_countries": [
                          "*",
                          "US"
                        ]
                      }
                    }
                  ]
                }
              },
              {
                "terms": {
                  "stock_status": [
                    3
                  ]
                }
              }
            ]
          }
        }
      }
    }
  ],
  "from": 0,
  "facets": {
    "color_not_analyzed": {
      "facet_filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "gender_not_analyzed": [
                  "Men"
                ]
              }
            }
          ]
        }
      },
      "terms": {
        "field": "color_not_analyzed",
        "size": 50
      }
    },
    "subcategory_not_analyzed": {
      "facet_filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "gender_not_analyzed": [
                  "Men"
                ]
              }
            }
          ]
        }
      },
      "terms": {
        "field": "subcategory_not_analyzed",
        "size": 50
      }
    },
    "category_not_analyzed": {
      "facet_filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "gender_not_analyzed": [
                  "Men"
                ]
              }
            }
          ]
        }
      },
      "terms": {
        "field": "category_not_analyzed",
        "size": 50
      }
    },
    "retailer_slug": {
      "facet_filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "gender_not_analyzed": [
                  "Men"
                ]
              }
            }
          ]
        }
      },
      "terms": {
        "field": "retailer_slug",
        "size": 50
      }
    },
    "gender_not_analyzed": {
      "terms": {
        "field": "gender_not_analyzed",
        "size": 50
      }
    },
    "product_type_not_analyzed": {
      "facet_filter": {
        "bool": {
          "must": [
            {
              "terms": {
                "gender_not_analyzed": [
                  "Men"
                ]
              }
            }
          ]
        }
      },
      "terms": {
        "field": "product_type_not_analyzed",
        "size": 50
      }
    }
  },
  "filter": {
    "bool": {
      "must": [
        {
          "terms": {
            "gender_not_analyzed": [
              "Men"
            ]
          }
        }
      ]
    }
  },
  "query": {
    "filtered": {
      "filter": {
        "bool": {
          "must": [
            {
              "nested": {
                "filter": {
                  "bool": {
                    "must": [
                      {
                        "bool": {
                          "must_not": [
                            {
                              "term": {
                                "entries.disallow_countries": "US"
                              }
                            }
                          ],
                          "must": [
                            {
                              "terms": {
                                "entries.allow_countries": [
                                  "*",
                                  "US"
                                ]
                              }
                            }
                          ]
                        }
                      },
                      {
                        "terms": {
                          "stock_status": [
                            3
                          ]
                        }
                      }
                    ]
                  }
                },
                "path": "entries"
              }
            },
            {
              "nested": {
                "filter": {
                  "terms": {
                    "follows.owner_id": [
                      1
                    ]
                  }
                },
                "path": "follows"
              }
            }
          ]
        }
      },
      "query": {
        "match_all": {
          
        }
      }
    }
  },
  "size": 10
}

(and this is not most complicated query I'm doing, far from it). There are few problems with such complex queries, which require support from the client:

- you have to keep up with quickly evolving ES syntax. If you are using deprecated or obsolete feature, client should warn you.

- you don't want to spend hours chasing typo, starring at ES "parsing error near..." response. Queries should be generated.

- you need to be able to easily modify queries to use ES efficiently. Client should provide high-level interface to do it.

- but you need to get everything out of ES - client should support every available feature and syntax option.

Beside that, I have standard expectations for every library:

- keep up with ES development
- fix bugs and release often
- provide good documentation

What I don't need:

- as mentioned: any advanced connection management
- integration with any framework. While useful at the beginning, it gets in the way later,
and can become a limitation. In my case ES index is highly independent from my database models.

Considering those requirements, what were my options then?


First lets have a brief overview of ES libraries that I'm not even considering as usable:

pyelasticsearch
ESClient
rawes

all of them (and many others you can find on pypi) provide not much more then thin wrapper over http request. While they are useful, and for most people are simply good enough, they really are not an option for me.

Here is the list of clients that I was looking at:

elasticutils

This one was really promising, as it allows you to write this:

In [1]: elasticutils.S().filter(foo__gte=4, baz__startswith='bar').order_by('-baz').facet('foo')
Out[1]: <s {'filter': {'and': [{'range': {'foo': {'gte': 4}}}, {'prefix': {'baz': 'bar'}}]}, 'sort': [{'baz': 'desc'}], 'facets': {'foo': {'terms': {'field': 'foo'}}}}>

which is absolutely amazing, comparing with raw ES syntax. If you are choosing ES library now, you definitely should consider it.
Unfortunately when I was looking at it, it was relying on pyelasticsearch that wasn't compatible with recent ES version, making it completely useless.
I hope this has been fixed, but I moved one since then, so I don't know for sure. The only objection I would have would be lack of support for nested documents.
Other then that, it really makes using ES a pleasure.

elasticfun
haystack

Both provide similar queryset-ish syntax, although support much smaller subset of ES features. Likely good enough for many people, but not me.
Haystack supports many search engines, so you can't expect integration with ES as good as dedicated client.

And the winner is ... pyes:

Pyes provides:

- support for nearly every ES feature, via object-oriented interface. If there is anything missing (happened few times),
its really easy to add.

- queryset, for convenience:

In [1]: queryset.QuerySet(index='index', type='type').filter(foo=3, bar__startswith='joe').order_by('bar').facet('baz')._build_search().serialize()
Out[1]: 
{'facets': {'baz': {'terms': {'field': 'baz', 'size': 10}}},
 'from': 0,
 'query': {'filtered': {'filter': {'and': [{'term': {'bar.startswith': 'joe'}},
     {'term': {'foo': 3}}]},
   'query': {'match_all': {}}}},
 'sort': [{'bar': 'asc'}]}

unfortunatelly queryset itself does not support nested documents, but all other pyes classes do.

- simple way of dealing with complex queries. Basically pyes provides python class
for every part of ES query, like filters, facets or queries. This gives you query generation (each class has serialize method that generates relevant part of ES syntax),
and yet allows to go as low-level as needed, to tweak anything you want. This oo-based approach makes pyes (and anything that uses it)
very easy to inspect and debug, which is something I frequently do. You have to deal with whole complexity of ES of course, but that is exactly what I often need to do.

- good (but not perfect) support for recent ES versions. While there were few details I had to fix or enhance, at least it was never completely broken (pointing finger at elasticutils here).

- it does support specifying connection timeouts and retries. Actually it does much more - I don't need it, but its good to have a choice.

- straightforward translation to ES syntax makes it easy to understand if you know ES syntax (otherwise it makes it very, very hard to understand anything)

The cons also exist:

- while actively maintained, official releases are rare. Use master. This is the biggest drawback.

- if you know nothing about full-text search engines, this may not be the best choice for you. It will allow you to dive as deep into ES as needed, but there is little automation. In that case, haystack might be the best choice.

- following standards set by ES itself, documentation sucks. You can easily do hello-world query, but then there is a lot of undocumented methods that accept **kwargs. Source is easy to read though.

but they don't outweigh the pros and for my needs there really was no other choice.

(if any of pyes authors is reading it, here is my wishlist: provide official releases, gather and publish list of unsupported ES features and keep up the good work you are doing)