A neat Python debugger command

pdb is a console-mode debugger built into Python. Out of the box, it has basic features like variable inspection, breakpoints, and stack frame walking, but it lacks more advanced capabilities.

Fortunately, it can be customized with a .pdbrc file in the user's home directory. Ned Batchelder has several helpful commands in his .pdbrc file:

  • pl: print local variables
  • pi obj: print the instance variables of obj
  • ps: print the instance variables of self

Printing instance variables is great for quickly inspecting objects, but it shows only one half of the picture. What about the class-side of objects? Properties and methods are crucial for understanding what can actually be done with an object, in contrast to what data it encapsulates.

Since I couldn't find a readily available pdb command for listing class contents, I wrote my own:

# Print contents of an object's class (including bases).
alias pc for k,v in sorted({k:v for cls in reversed(%1.__class__.__mro__) for k,v in cls.__dict__.items() if cls is not object}.items()): print("%s%-20s= %-80.80s" % ("%1.",k,repr(v)))

pc lists the contents of an object's class and its base classes. Typically, these are the properties and methods supported by the object. It is used like this:

# 'proc' is a multiprocessing.Process() instance.
(Pdb) pc proc
...
proc.daemon              = <property object at 0x036B9A20>
proc.exitcode            = <property object at 0x036B99C0>
proc.ident               = <property object at 0x036B9A50>
proc.is_alive            = <function BaseProcess.is_alive at 0x033E4618>
proc.join                = <function BaseProcess.join at 0x033E45D0>
proc.name                = <property object at 0x036B99F0>
proc.pid                 = <property object at 0x036B9A50>
proc.run                 = <function BaseProcess.run at 0x033E4A98>
proc.start               = <function BaseProcess.start at 0x033E4DB0>
proc.terminate           = <function BaseProcess.terminate at 0x033E4DF8>

Note the difference to pi, which lists the contents of the proc instance:

(Pdb) pi proc       # In contrast, here is the image dictionary.
proc._args          = ()
proc._config        = {'authkey': b'\xd0\xc8\xbd\xd6\xcf\x7fo\xab\x19_A6\xf8M\xd4\xef\x88\xa9;\x99c\x9
proc._identity      = (2,)
proc._kwargs        = {}
proc._name          = 'Process-2'
proc._parent_pid    = 1308
proc._popen         = None
proc._target        = None

In general, pc focuses on the interface while pi examines the state of the object. The two complement each other nicely. Especially when working with an unfamiliar codebase, pc is helpful for quickly figuring out how to use a specific class.

pc works with both Python 2 and Python 3 (on Python 2 it only shows new-style classes). Add it to your .pdbrc and give it a try. Let me know what you think!

Python's GIL and atomic reference counting

I love Python because it's an incredibly fun, expressive and productive language. However, it's often criticised for being slow. I think the correct answer to that is two-fold:

  • Use an alternative Python implementation for impressive speed-ups. For example, PyPy is on average 7 times faster than the standard CPython.
  • Not all parts of a program have to be blazingly fast. Use Python for all non-performance-critical areas, such as the UI and database access, and drop to C or C++ only when it's required for for CPU intensive tasks. This is easy to achieve with language binding generators such as CFFI, SWIG or SIP.

A further performance-related issue is Python's Global Interpreter Lock (GIL), which ensures that only one Python thread can run at a single time. This is a bit problematic, because it affects PyPy as well, unless you want to use its experimental software transactional memory support.

Why is this such a big deal? With the rise of multi-core processors, multithreading is becoming more important as well. This not only affects performance on large servers, it impacts desktop programs and is crucial for battery life on mobile phones (race to idle). Further, other programming languages make multi-threaded programming easier and easier. C, C++, and Java have all moved to a common memory model for multithreading. C++ has gained std::atomic, futures, and first-class thread support. C# has async and await, which is proposed for inclusion in C++ as well. This trend will only accelerate in the future.

With this in mind, I decided to investigate the CPython GIL. Previous proposals for its removal have failed, but I thought it's worth a look — especially since I couldn't find any recent attempts.

The results were not encouraging. Changing the reference count used by Python objects from a normal int to an atomic type resulted in a ~23% slowdown on my machine. This is without actually changing any of the locking. This penalty could be moderated for single-threaded programs by only using atomic instructions once a second thread is started. This requires a function call and an if statement to check whether to use atomics or not in the refcount hot path. Doing this still results in an 11% slowdown in the single-threaded case. If hot-patching was used instead of the if, a 6% slowdown remained.

refcount slowdown
Atomic refcounts slow down the overall Python execution speed by >20%.

The last result looks promising, but is deceiving. Hot-patching would rely on having a single function to patch. Alas, the compiler decided to mostly inline the Py_INCREF/Py_DECREF function calls. Disabling inlining of these functions gives a 16% slowdown, which is worse than the "call + if" method. Furthermore, hot-patching is probably not something that could be merged in CPython anyway.

So what's the conclusion? Maybe turning Py_INCREF and Py_DECREF into functions and living with the 11% slowdown of the single-threaded case would be sell-able, if compelling speed-ups of multithreaded workloads could be shown. It should be possible to convert one module at a time from the GIL to fine-grained locking, but performance increases would only be expected once at least a couple of core modules are converted. That would take a substantial amount of work, especially given the high risk that the resulting patches wouldn't be accepted upstream.

Where does this leave Python as a language in the multi-threaded world? I'm not sure. Since PyPy is already the solution to Python's performance issue, perhaps it can solve the concurrency problem as well with its software transactional memory mode.

PS: My profiling showed that the reference counting in Python (Py_INCREF() and Py_DECREF()) takes up to about 5–10% of the execution time of benchmarks (not including actual object destruction), crazy!

Subclassing C++ in Python with SIP

I use SIP in MapsEvolved to generate bindings for interfacing Python with C++. I really like SIP due to its straight-forward syntax that mostly allows just copying class definitions over from C++. Further, it's really well thought out and contains support for a number of advanced use cases.

One such feature is implementing a C++ interface in Python. The resulting class can then even be passed back to C++, and any methods called on it will be forwarded to the Python implementation. Sweet!

Here is an example I originally wrote for this Stack Overflow question. It illustrates how ridiculously easy it is to get this working:

visitor.h:

class EXPORT Node {
public:
    int getN() const;
    ...
};
struct EXPORT NodeVisitor {
    virtual void OnNode(Node *n) = 0;
};
void visit_graph_nodes(NodeVisitor *nv);

visitor.sip:

%Module pyvisit

%ModuleHeaderCode
#include "visitor.h"
%End

class Node {
public:
    int getN() const;
    ...
};

struct NodeVisitor {
    virtual void OnNode(Node* n) = 0;
};

void visit_graph_nodes(NodeVisitor *nv);

Using it from Python:

>>> import pyvisit
>>> class PyNodeVisitor(pyvisit.NodeVisitor):
>>>     def OnNode(self, node):
>>>         print(node.getN())
>>> pnv = PyNodeVisitor()
>>> visit_graph_nodes(pnv)
1
2
3
...

Here, the C++ function visit_graph_nodes() calls the Python method pnv.OnNode() for every node in its (internal) graph. A zip file with the full working source code of this example can be downloaded here.

The subclassing capabilities of SIP don't stop at interfaces, either. It's possible to derive from any C++ class, abstract or not, inheriting (or overriding) existing method implementations as needed. This gives a lot of flexibility and makes it easy to have classes with some parts implemented in C++, and others being in Python.

Configuring SSL on Apache 2.4

Configuring a modern web server to employ strong encryption and forward secrecy doesn't have to be hard. There is excellent documentation from Mozilla and from the OWASP.

Obtaining an SSL certificate

One major stumbling block is where to obtain an SSL certificate. In the future, this should hopefully be easy with Let's Encrypt. Until that is actually functional, StartSSL offers free SSL certificates. The process takes a bit of patience, but it's not difficult. There's also a StartSSL HOWTO from h-online.com.

While I've used StartSSL in the past, I had some trouble with them because 10 years after I registered greek0.net, someone grabbed greekO.net and StartSSL was alleging I was trying to mislead users?! So that was the end of my business with them...

I've now switched to Comodo's Positive SSL Certificate, which I like for a couple of reasons:

  • it lasts for 3 years,
  • it's really uncomplicated, and
  • it's crazy cheap: 7.45$ per year.

The process of getting the cert from them was super easy, simpler than StartSSL. About 3 hours from going to their website to having the certificate installed on my server, with most of it waiting email verifications. Credit card payment was quick and easy. 10/10, would buy again :-)

Apache configuration

With the certificate acquisition out of the way, here are the juicy bits from my Apache config.

mod_ssl config:

# Enable only cyphers that support forward secrecy.
# See these two links for reference:
# https://stackoverflow.com/questions/17308690
# https://wiki.mozilla.org/Security/Server_Side_TLS#Non-Backward_Compatible_Ciphersuite
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK

# Use server priorities for cipher algorithm choice.
SSLHonorCipherOrder on

# With Apache 2.4, SSLv2 is gone and only SSLv3 and TLSv* are supported.
# Disable SSLv3, all TLS protocols are OK.
SSLProtocol all -SSLv3

# Enable OCSP stapling
# With this, the client can verify that our certificate isn't revoked
# without having to query an external OCSP service.
SSLUseStapling On
SSLStaplingCache shmcb:${APACHE_RUN_DIR}/ssl_stapling(32768)

The per-site configuration:

SSLEngine On
SSLCertificateKeyFile   /path/to/serverkey.key    # The private server key.
SSLCertificateFile      /path/to/certificate.crt  # The certificate provided by CA.
SSLCertificateChainFile /path/to/cert-bundle      # A separate download from your CA.


# Use a customized prime group for DH key exchange (vs Logjam attack).
# Generate a DH group file with:
#    openssl dhparam -out dhparams.pem 2048
#
# Newer Apache versions support the following command to set the dhparams:
SSLOpenSSLConfCmd DHParameters "/path/to/dhparams.pem"

# If Apache reports an error for the above line, remove it and include
# the dhparams in the certificate:
#   cat <CERT>.crt dhparams.pem > cert-with-dhparams.crt
#   SSLCertificateFile cert-with-dhparams.crt


# HSTS: Force browsers to require SSL for this domain for the next year.
# Down-grade to HTTP will cause browsers to abort with a security error.
Header always set Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"


# HPKP: Pin the current key for the next two months.
# Generate hash using:
#   openssl rsa -in <serverkey>.key -outform der -pubout | \
#   openssl dgst -sha256 -binary | openssl enc -base64
#
# You ideally want to generate a backup key and include that here as well,
# in case the primary key is lost or compromised.
# Also note the implications for key rollover.
# See: https://developer.mozilla.org/en-US/docs/Web/Security/Public_Key_Pinning
Header always set Public-Key-Pins "pin-sha256=\"<HASH>\"; max-age=5184000; includeSubDomains"


# Disable compression to avoid BREACH HTTPS/SSL attack.
<Location />
    SetEnv no-gzip
</Location>

That should cover the basics.

Testing

As for SSL connection testing, I found the Qualys SSL Labs Test helpful. It shows what browsers (browser versions) will get which encryption quality (forward secrecy or not) and highlights common problems such as certificate chain issues.

Hope this helps someone out there!

Cleaning up unused Docker images and containers

Docker doesn't delete old/unused images or containers by itself, even if they weren't used for a long time or were only intermediary steps on the way to another image. This leads to an image sprawl that eats up a lot of disk space if not kept in check.

The right way to solve this would be to parse the output of docker inspect and remove containers and images based on certain policies. Unfortunately, a quick internet search did not turn up a script that does this.

Since I didn't want to spend the time to write such a thing myself, I resorted to what – sadly – seems to be state-of-the-art docker image management: a cronjob running those two lines:

docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rm
docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi >/dev/null 2>&1

The first line removes containers that are older than two weeks and are not currently running (docker rm simply will not remove running containers). The second line removes images that are not used by any container and are not tagged (i.e. don't have proper repository name).

These two invocations are based on this Stack Overflow question and on this blog post by Jim Hoskins.

This solution works well enough, you probably shouldn't use it on production servers, though. :-)