Add Timestamps to Your Bash History

Put this in your profile script (.bashrc):

export HISTTIMEFORMAT="%d/%m/%y %T "

I also find it helpful to increase the size of my command-line history:

export HISTFILESIZE=10000
export HISTSIZE=10000

The first controls how many are stored. The second controls how many are kept in memory. You can also tweak it to hold an actual unlimited number if you so wish.

Advertisements

Colored Logging in Python Under Jenkins

Setting-up the coloredlogs package is easy. Just install the coloredlogs package from PIP, import coloredlogs, and initialize like the following:

coloredlogs.install(isatty=True)

Note the isatty parameter. By default, coloredlogs detects whether or not you’re in a terminal and enabled ANSI output accordingly. This means that you won’t see ANSI output in Jenkins unless you explicitly pass True.

Make sure to check “Color ANSI Console Output” in your job config. Using “xterm” for “ANSI color map” worked fine for me.

As this will add another handler to the logging, you may see duplicate logging if you have any other handlers.

For more information on configuration, see the documentation.

The Numb-Nuts Tutorial to the Celery Distributed Task Queue (using Python)

Celery is a distributed queue that is very easy to pick-up. I’ll do two quick examples: one that sends a job and returns and another that sends a job and then retrieves a result. I’m going to use SQLite for this example (which is interfaced via SQLAlchemy). Since Celery seems to have some issues importing SQLite under Python 3, we’ll use Python 2.7 . Make sure that you install the “celery” and “sqlalchemy” Python packages.

Without Results

Define the tasks module and save it as sqlite_queue_without_results.py:

import celery

_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'
_APP = celery.Celery('sqlite_queue_without_results', broker=_BACKEND_URI)

@_APP.task
def some_task(incoming_message):
    return "ECHO: {0}".format(incoming_message)

Start the server:

$ celery -A sqlite_queue_without_results worker --loglevel=info

Execute the following Python to submit a job:

import sqlite_queue_without_results

_ARG1 = "Test message"
sqlite_queue_without_results.some_task.delay(_ARG1)

That’s it. You’ll see something similar to the following in the server window:

Celery (Without Result)

With Results

This time, when we define the tasks module, we’ll provide Celery a results backend. Call this module sqlite_queue_with_results.py:

import celery

_RESULT_URI = 'db+sqlite:///results.sqlite'
_BACKEND_URI = 'sqla+sqlite:///tasks.sqlite'

_APP = celery.Celery('sqlite_queue_with_results', broker=_BACKEND_URI, backend=_RESULT_URI)

@_APP.task
def some_task(incoming_message):
    return "ECHO: {0}".format(incoming_message)

Start the server:

$ celery -A sqlite_queue_with_results worker --loglevel=info

Execute the following Python to submit a job:

import sqlite_queue_with_results

_ARG1 = "Test message"
r = sqlite_queue_with_results.some_task.delay(_ARG1)
value = r.get(timeout=2)

print("Result: [{0}]".format(value))

Since we’re using a traditional DBMS (albeit a fast, local one) to store our results, we’ll be internally polling for a state change and then fetching the result. Therefore, it is a more costly operation and I’ve used a two-second timeout to accommodate this.

The server output will be similar to the following:

Celery (With Result)

The client output will look like:

Result: [ECHO: Test message]

Celery has many more features not explored by this tutorial, including:

  • exception propagation
  • custom task states (including providing metadata that can be read by the client)
  • task ignore/reject/retry responses
  • HTTP-based tasks (for calling your tasks in another place or language)
  • task routing
  • periodic/scheduled tasks
  • workflows
  • drawing visible graphs in order to inspect behavior

For more information, see the user guide.