Asynchronous programming is hard. In the past, the Nova project used Tornado, then Twisted and it is now using eventlet which also became the defacto standard in OpenStack. Eventlet is not perfect, we will explain why we consider that eventlet has major flaws. We will then introduce the asyncio module of Python 3.4, how we plan to use it in OpenStack, to finish with the current status of this integration.
What’s wrong with eventlet?
Eventlet is “a concurrent networking library for Python that allows you to change how you run your code, not how you write it” according to its documentation.
The eventlet module has two major issues: debugging is hard because of the implicit task switching and it does not support Python 3 (see the Support Python 3.3 issue). The eventlet dependency blocks at least the porting to Python 3 of 9 OpenStack servers (ceilometer, cinder, glance, heat, horizon, keystone, neutron, nova, swift).
In OpenStack, eventlet is used with monkey-patching enabled. Monkey-patching the Python standard libraries replaces blocking functions with asynchronous functions using greenthread. For example, the Python time.sleep() function which “suspends execution for the given number of seconds” is replaced with greenthread.sleep which “yields control to another eligible coroutine until at least seconds have elapsed”.
The problem is that it becomes hard to guess if a function may or may not switch to another task (“green thread”). Or worse, the function may switch in a new version of a module, you won’t notice the change. The developer must be aware of that and write the code with this fact in mind. This has led to real bugs, remember this OpenStack reaction: when adding sleep(0) fixes an eventlet test. 
Just one example of eventlet.sleep(0): Sleep to simulate concurrency and allow other threads to work in Ceilometer tests (most concurrency tests of Nova use eventlet.sleep).
Eventlet also monkey-patches the threading module so it is not possible to use operating system threads, green threads are used everywhere.
Eventlet relies on the greenlet module to manage green threads. The greenlet module is not portable: it uses assembler code with one version per architecture and per platform to switch between green threads. Greenlet contains code specific to CPython and so cannot be used with Jython or IronPython. PyPy supports greenlet since PyPy 2.0. Even if “it works”, eventlet will not be integrated in CPython, it is difficult to maintain and so is not a sustainable solution.
The new asyncio module (PEP 3156) and the Trollius project
For a nice introduction to asyncio, read the report of Guido Van Rossum’s keynote at Pycon US 2013: PyCon: Asynchronous I/O by Jake Edge (March, 2013). The development started outside Python in a project called Tulip for Python 3.3, the name of the Python module is “asyncio” (use import asyncio in Python).
The asyncio module has been designed as a superset of existing libraries: Twisted, Tornado, gevent, eventlet, etc. The design is the PEP 3156 “Asynchronous IO Support Rebooted: the asyncio Module” written by Guido van Rossum, author of the Python language. The PEP has been written with the help of developers of the existing libraries and the design reuses good ideas of existing libraries, like transports and protocols of Twisted. Asyncio offers a nice syntax for coroutines using the new yield from syntax of Python 3.3: PEP 380 “Syntax for Delegating to a Subgenerator”.
The asyncio module is written in pure Python, and therefore portable. It is built on top of existing modules like concurrent.futures, subprocess, threading and the new selectors module (which is itself built on top on the select module). asyncio is a glue between modules of the Python standard library and existing asynchronous frameworks, but it can be used alone. It also adds new features like tasks using coroutines. It supports blocking functions as well with its executor API which uses a pool of threads by default. It uses operating system threads which can run in parallel (especially when executing C functions releasing the GIL). The default executor can be replaced to use a pool of processes, or a custom executor.
PEP 3156 has been accepted and asyncio is part of the Python 3.4 standard library. Asyncio is now very well tested by a farm of more than 33 buildbots on various architectures (x86, x86_64, PPC64, ARM7, SPARC) and operating systems (Linux, Windows 2003/XP/7, Mac OS X 10.4/10.6, FreeBSD 6/7/9/10, OpenBSD, Solaris 10, OpenIndiana). Enovance ported asyncio to Python 2.6 and 2.7 as the new Trollius project: Trollius works on Python 2.6-3.4. We also wrote the documentation for the asyncio module: asyncio – Asynchronous I/O, event loop, coroutines and tasks.
Even before the official release of Python 3.4 (scheduled for March 16th), there are already many projects supporting asyncio. There are asyncio event loops for greenlet, gevent, libuv, GLib, Tornado and 0MQ. There are database drivers for PostgreSQL, Redis, MongoDB and memcached. There are HTTP clients and servers, web sockets, and a gunicorn worker. See asyncio third party for the whole list, some of them are still experimental.
For more information on asyncio, see my notes about asyncio: talks about asyncio, Tulip backends and event loops, Tulip projects, etc. For a more technical comparison of the different asynchronous I/O models in Python, read the nice Async I/O and Python article by Mark McLoughlin (June, 2013).
Differences between Trollius and Tulip
The Trollius project is based on Tulip: in the Mercurial repository, it is a branch based on the Tulip branch, so Trollius can easily merge new changes from Tulip. We are working directly on asyncio and update the Trollius project regularly to keep it up to date.
The major difference between Tulip and Trollius is the syntax of coroutines: Trollius uses yield From(...) instead of yield from ..., and raise Return(x) instead of return x. Except for the syntax of coroutines, the API is the same: classes, methods, functions, etc. It is possible to use the same code base for Trollius and Tulip using callbacks. For example, this solution is used by the AutobahnPython project which also supports Twisted.
Also read the Trollius documentation.
Plan to use asyncio in OpenStack
Straightforward dropping of Python 2 support and replacing eventlet with asyncio is not possible: there are a lot of existing OpenStack setup running on Python 2. The Tulip project cannot be used in OpenStack, because the porting of OpenStack to Python 3 is still in progress. Trollius is a more realistic option because it works on Python 2 and 3.
There are different options to use asyncio in OpenStack:
Replace eventlet completely with Trollius
Plug Trollius into eventlet: eventlet would execute Trollius tasks; eventlet would be the Trollius event loop
Plug eventlet into Trollius: Trollius would execute eventlet tasks; Trollius would be an eventlet hub
Replacing eventlet with Trollius at once is not possible, it’s not how OpenStack is developed. All changes in OpenStack are carefully reviewed, well tested, and usually very small. The development is incremental.
Plugging Trollius into eventlet can be implemented using the greenio project which is an asyncio event loop using greenlet. It doesn’t look to be the preferred option of the OpenStack developers.
Plugging eventlet into Trollius can be implemented using the eventlet hub API. We are working on a proof-of-concept. This way would be a smooth transition to asyncio. Existing code would continue to use eventlet, and new code can directly use the asyncio API.
Also read recent threads on the openstack-dev mailing list about asynchronous programming, asyncio and Python 3:
[openstack-dev] Asynchrounous programming: replace eventlet with asyncio
[openstack-dev] [solum] async / threading for python 2 and 3
[openstack-dev] Python 3 compatibility
Status of asyncio in OpenStack
The first step was to add Trollius to the global-requirements.txt: the patch has been merged.
The Oslo Messaging project was selected to experiment asyncio because it was designed to support different asynchronous libraries. Oslo Messaging has an “executor API” with two available implementations: blocking and eventlet. The blueprint Oslo/blueprints/asyncio proposes to add a new Trollius executor, implementation: Add a new asynchronous executor based on Trollius.
The next step is to write an experimental eventlet hub on Trollius.