Month: July 2014

SSL for Python (M2Crypto) on Windows

M2Crypto is the most versatile and popular SSL library for Python. Naturally, it takes a predictable amount of burden getting it to work under Windows.

If you’re lucky, you can find a precompiled binary online, and circumvent the heartache. Though many pages have come and gone, here is one that works, courtesy of the grr project: M2Crypto.

Not only do they provide a [non-trivial] set of instructions on how to build the binaries yourself, but they present binaries, as well. Though the binaries are hosted on Google Code (and unlikely to go away), I’ve hosted them, too, for brevity:

M2CryptoWindows

Note that these binaries, as given, are not installable Python packages. I have produced and published two such packages to PyPI, for your convenience:

M2CryptoWin32
M2CryptoWin64

SQLAlchemy and MySQL Encoding

I recently ran into an issue with the encoding of data coming back from MySQL through sqlalchemy. This is the first time that I’ve encountered such issues since this project first came online, months ago.

I am using utf8 encoding on my database, tables, and columns. I just added a new column, and suddenly my pages and/or AJAX calls started failing with one of the following two messages, respectively:

  • UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0x96 in position 5: ordinal not in range(128)
  • UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x96 in position 5: invalid start byte

When I tell the stored procedure to return an empty string for the new column instead of its data, it works. The other text columns have an identical encoding.

It turns out that SQLAlchemy defaults to the latin1 encoding. If you need something different, than you’re in for a surprise. The official solution is to pass the “encoding” parameter to create_engine. This is the example from the documentation:

engine = create_engine("mysql://scott:tiger@hostname/dbname", encoding='latin1', echo=True)

In my case, I tried utf8. However, it still didn’t work. I don’t know if that ever works. It wasn’t until I uncovered a StackOverflow entry that I found the answer. I had to append “?charset=utf8” to the DSN string:

mysql+mysqldb://username:password@hostname:port/database_name?charset=utf8

The following are the potential explanations:

  • Since I copy and pasted values that were set into these columns, I accidentally introduced a character that was out of range.
  • The two encodings have an overlapping set of codes, and I finally introduced a character that was supported by one but not the other.

Whatever the case, it’s fixed and I’m a few hours older.

Using ctypes to Read Binary Data from a Double-Pointer

This is a sticky and exotic use-case of ctypes. In the example below, we make a call to some library function that treats ptr like a double-pointer, and sets ptr to point to a buffer and sets count with the number of bytes that are available there. The data at the pointer may have one or more NULL bytes that should not be interpreted as terminators.

from ctypes import *

ptr = ctypes.c_char_p()
count = ctypes.c_size_t()

r = library.some_call(
        ctypes.cast(ctypes.byref(ptr), 
                    ctypes.POINTER(ctypes.c_void_p)), 
        ctypes.byref(count))

if r != 0:
    raise ValueError("Library call failed.")

data = ctypes.string_at(ptr, count.value)

Snappy for Very Easy Compression

Snappy is a fast compression algorithm by Google. When I’ve used it, it’s been for socket compression, though it can be used for file compression, too.

For socket compression in Python, the examples are embarrassingly simple. First, import the module:

import snappy

When you establish the socket (which we’ll refer to as s), create the compressor and decompressor:

c = snappy.StreamCompressor()
d = snappy.StreamDecompressor()

From that point on, just pass all of your outgoing data through c.add_chunk(), and all of your read data through d.decompress(). add_chunk() and decompress() may return an empty string from time-to-time (obviously).

Beautiful Flowcharts and UML Without Installation

Unfortunately, there are no Google Searches that render good suggestions for cheap sites that create beautiful flowcharts. The recommended products seem to either be locally installable or exorbitantly priced.

I now present two. They can both import from Visio, can both integrate with Confluence, and both cost between $5 and $10 a month.

Gliffy

  • Slighty more “comical” and flowing, but the difference may be negligible
  • Google Documents integration available with all paid accounts
  • Has an academic discount

Gliffly Flowchart Example

Lucidchart

  • Less effort and less clicks to control alignment and establish links than most services
  • You can leave comments on components
  • Revision history
  • Integrates with Google Documents, but only with team accounts
  • You can do Hangouts and chat from the design screen.

Lucidchart Flowchart Example

Edit: I also received a last-minute mentioned for draw.io from @DiegoTerzano. Though the designs are thinner/leaner, it’s not a shoddy interface.

Use ca_kit to Rapidly Establish a CA

I began using ca_kit so often that it became inconvenient not having it formally packaged and uploaded to PyPI. So, I’ve built it into a formal package. The following scripts are published into the path, upon install:

  • ck_create_ca: Create CA certificates
  • ck_create: Create regular certificates
  • ck_sign: Sign a regular certificate against the CA certificate
  • ck_verify_ca: Verify that a signed certificate matches the CA certificate

I hope this is as indisposable to you as it is to me.

DEFLATE Socket Compression in Python

Properly-working DEFLATE compression is elusive in Python. Thanks to wtolson, I’ve found such a solution.

This is an example framework for establishing the compression and decompression objects:

def activate_zlib():
    import zlib

    wbits = -zlib.MAX_WBITS

    compress = zlib.compressobj(level, zlib.DEFLATED, wbits)

    compressor = lambda x: compress.compress(x) + \
                            compress.flush(zlib.Z_SYNC_FLUSH)

    decompress = zlib.decompressobj(wbits)
    
    decompressor = lambda x: decompress.decompress(x) + \
                                decompress.flush()
    
    return (compressor, decompressor)

With this example, you’ll pass all of your outgoing data through the compressor, and all of your incoming data to the decompressor. As always, you’ll do a read-loop until you’ve decompressed the expected number of bytes.

Method Overloads in Python 3.4

Python 3.4 added a “singledispatch” decorator to functools, which provides method overloads. This enables you to perform different operations based on the type of the first argument.

By default, it prefers to work with static methods. This mostly comes from the link above:

import functools


class TestClass(object):
    @functools.singledispatch
    def test_method(arg):
        print("Let me just say,", end=" ")
        print(arg)

    @test_method.register(int)
    def _(arg):
        print("Strength in numbers, eh?", end=" ")
        print(arg)

    @test_method.register(list)
    def _(arg):
        print("Enumerate this:")

        for i, elem in enumerate(arg):
            print(i, elem)

if __name__ == '__main__':
    TestClass.test_method(55555)
    TestClass.test_method([33, 22, 11])

However, there is a low-impact way to get overloading on instance-methods, too. We’ll just place our own wrapper around the standard singledispatch wrapper, and hijack the bulk of the functionality:

import functools

def instancemethod_dispatch(func):
    dispatcher = functools.singledispatch(func)
    def wrapper(*args, **kw):
        return dispatcher.dispatch(args[1].__class__)(*args, **kw)
    wrapper.register = dispatcher.register
    functools.update_wrapper(wrapper, func)
    return wrapper


class TestClass2(object):
    @instancemethod_dispatch
    def test_method(self, arg):
        print("2: Let me just say,", end=" ")
        print(arg)

    @test_method.register(int)
    def _(self, arg):
        print("2: Strength in numbers, eh?", end=" ")
        print(arg)

    @test_method.register(list)
    def _(self, arg):
        print("2: Enumerate this:")

        for i, elem in enumerate(arg):
            print(i, elem)

if __name__ == '__main__':
    t = TestClass2()
    t.test_method(55555)
    t.test_method([33, 22, 11])

Aside from superficial changes to the original example, we just added the instancemethod_dispatch function and updated the methods to take a “self” argument.

A special thanks to Zero Piraeus for penning the instancemethod_dispatch method (under the original name of “methdispatch”).