Colored Logging in Python Under Jenkins

Setting-up the coloredlogs package is easy. Just install the coloredlogs package from PIP, import coloredlogs, and initialize like the following:

coloredlogs.install(isatty=True)

Note the isatty parameter. By default, coloredlogs detects whether or not you’re in a terminal and enabled ANSI output accordingly. This means that you won’t see ANSI output in Jenkins unless you explicitly pass True.

Make sure to check “Color ANSI Console Output” in your job config. Using “xterm” for “ANSI color map” worked fine for me.

As this will add another handler to the logging, you may see duplicate logging if you have any other handlers.

For more information on configuration, see the documentation.

Advertisements

The Numb-Nuts Tutorial to the Celery Distributed Task Queue (using Python)

Celery is a distributed queue that is very easy to pick-up. I’ll do two quick examples: one that sends a job and returns and another that sends a job and then retrieves a result. I’m going to use SQLite for this example (which is interfaced via SQLAlchemy). Since Celery seems to have some issues importing SQLite under Python 3, we’ll use Python 2.7 . Make sure that you install the “celery” and “sqlalchemy” Python packages.

Without Results

Define the tasks module and save it as sqlite_queue_without_results.py:

import celery

_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'
_APP = celery.Celery('sqlite_queue_without_results', broker=_BACKEND_URI)

@_APP.task
def some_task(incoming_message):
    return "ECHO: {0}".format(incoming_message)

Start the server:

$ celery -A sqlite_queue_without_results worker --loglevel=info

Execute the following Python to submit a job:

import sqlite_queue_without_results

_ARG1 = "Test message"
sqlite_queue_without_results.some_task.delay(_ARG1)

That’s it. You’ll see something similar to the following in the server window:

Celery (Without Result)

With Results

This time, when we define the tasks module, we’ll provide Celery a results backend. Call this module sqlite_queue_with_results.py:

import celery

_RESULT_URI = 'db+sqlite:///results.sqlite'
_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'

_APP = celery.Celery('sqlite_queue_with_results', broker=_BACKEND_URI, backend=_RESULT_URI)

@_APP.task
def some_task(incoming_message):
    return "ECHO: {0}".format(incoming_message)

Start the server:

$ celery -A sqlite_queue_with_results worker --loglevel=info

Execute the following Python to submit a job:

import sqlite_queue_with_results

_ARG1 = "Test message"
r = sqlite_queue_with_results.some_task.delay(_ARG1)
value = r.get(timeout=2)

print("Result: [{0}]".format(value))

Since we’re using a traditional DBMS (albeit a fast, local one) to store our results, we’ll be internally polling for a state change and then fetching the result. Therefore, it is a more costly operation and I’ve used a two-second timeout to accommodate this.

The server output will be similar to the following:

Celery (With Result)

The client output will look like:

Result: [ECHO: Test message]

Celery has many more features not explored by this tutorial, including:

  • exception propagation
  • custom task states (including providing metadata that can be read by the client)
  • task ignore/reject/retry responses
  • HTTP-based tasks (for calling your tasks in another place or language)
  • task routing
  • periodic/scheduled tasks
  • workflows
  • drawing visible graphs in order to inspect behavior

For more information, see the user guide.

Installing Xcode Command Line Tools for Mavericks (Problems)

I had a perfectly running development environment under Mavericks 10.9.1 . I’m not, by nature, someone who would prefer to use a Mac, but sometimes we have to take what we’re given, and, at least, it’s Unix-based.

I was surviving without having to install Xcode, until recently when I had to investigate Apple’s illegally-modified “pngcrush” utility. I required Xcode in order to get the iPhone optimizations. Otherwise, I just got the standard version of the open-source utility. So, I installed it.

Yesterday, I had to install/build the Python “cryptography” module, which requires a C build-environment. Now, I had some cc/gcc discrepancies, and one unsupported command-line argument. Obviously, it’s Xcode. So, I uninstalled it. I innocently also installed the 10.9.1->10.9.2 Mavericks update at the same time.

Catastrophe. Now, the same stuff is broken, and I get warnings every single time I invoke Brew:

Warning: No developer tools installed.
You should install the Command Line Tools.
Run `xcode-select --install` to install them.

When I run xcode-select, I get a dialog that says the command-line tools are required, one button for installing them, and another for the full Xcode install. When I click to install the tools, I got the EULA and then a progress-bar that said “Finding Software”, only to give me a message:

Can't install the software because it is not currently available from the Software Update server.

I had to physically go and download the dmg package: http://developer.apple.com/downloads

However, when I dragged the pkg file into Applications and ran it, I inevitably ran into the following message, every time:

The installation failed.

The Installer can't locate the data it needs to install the software. Check your install media or Internet connection and try again, or contact the software manufacturer for assistance.

It turns out that it expected to be run directly from the dmg container. It looks like everything is working now (with only the command-line tools, and not requiring the whole Xcode install).

Though I’m still investigating the build errors I now having, the emphasis of this post is how to remedy the Xcode/tools errors that I was seeing.