Verifying Gerrit CRs to Your Jenkins’ Pipeline’s Shared Libraries

Jenkins’ pipelines represent a totally different direction from traditional, script-based jobs. Instead of specifying your SCM configuration and other build semantics in your job, you mostly script them out via a pipeline (“Jenkinsfile”) file, which is a heterogeneous script/declarative mess. Although you can be purely declarative, this is sometimes too strict to be useful, e.g. not being able to have traditional variable assignments in order to pass information between steps. Even though there are drawbacks, your whole workflow is largely version-controlled.

One of the drawbacks is the complexity of managing shared-library dependencies that you might have in order to make some of your Java/Groovy logic reusable. You can define these in your project (or, the case of multibranch pipelines, the folder) or at the admin level. You can also define these on the fly in the code.

Gerrit change-requests are applied essentially by fetching on a pseudo-refspec location (refs/changes/), and then cherry-picking it in. Therefore, in order to use one, you need to 1) clone, 2) fetch, and 3) either cherry-pick or checkout (or a couple of other methods). Although you can do this with a little effort with your actual Jenkinsfile (which is configured in the job; you can take the refspec from the environment during a verification and then use “FETCH_HEAD” as your branch), these are not intuitively available for the shared-libraries that you might be importing into your pipeline.

It turns out that you can massage the on-the-fly library loader to do this for you.

if (env.GERRIT_PATCHSET_REVISION) {
  echo("Using shared-library for verification.")

  library([
    identifier: 'myLibrary@' + env.GERRIT_PATCHSET_REVISION,
    retriever: modernSCM([
      $class: 'GitSCMSource',
      remote: 'https://repo.host/pipeline/library',
      traits: [
        [$class: 'jenkins.plugins.git.traits.BranchDiscoveryTrait'],
        [
          $class: 'RefSpecsSCMSourceTrait',
          templates: [
            [value: '+refs/heads/*:refs/remotes/@{remote}/*'], 
            [value: "+refs/changes/*:refs/remotes/@{remote}/*"]
          ]
        ]
      ]
    ])
  ])
} else {
  echo("Using shared-library from branch (not a verification).")

  library("myLibrary@" + env.BRANCH_NAME)
}

The principal things to notice are:

  1. We are telling it to bring all of the change-requests into scope (“+refs/changes/:refs/remotes/@{remote}/“).
  2. We are telling Jenkins to import exactly the library version tied to the change (“‘myLibrary@’ + env.GERRIT_PATCHSET_REVISION”). This wouldn’t be accessible without (1).

It works great.

I generated the original version of the code by using the Snippet Generator with the “library” step and then modifying according to the above.

Note that this pipeline can be used both in a multibranch pipeline job context as well as in the normal [single-branch] pipeline job used for verification (because we would only want to kick-off verification jobs just for the branch of the change). env.BRANCH_NAME will automatically be defined in the multibranch context.

Advertisements

Python: Retrieve User Info from LDAP

Supports returning the full DN for the user as well as a list of groups that they are a member of:

import logging
import ldap
import collections

# Install:
#
# apt: libsasl2-dev
# pip: python-ldap
#

_USER_QUERY = '(&(objectClass=USER)(sAMAccountName={username}))'
_GROUP_QUERY = '(&(objectClass=GROUP)(cn={group_name}))'

_LOGGER = logging.getLogger(__name__)

_USER = \
    collections.namedtuple(
        '_USER', [
            'dn',
            'attributes',
            'groups',
        ])


class NotFoundException(Exception):
    pass


class LdapAdapter(object):
    def __init__(self, host_and_port, username, password, base_dn):
        self.__host_and_port = host_and_port
        self.__username = username
        self.__password = password
        self.__base_dn = base_dn

        self.__raw_resource = None

    @property
    def __resource(self):
        if self.__raw_resource is None:
            self.__raw_resource = self.__auth()

        return self.__raw_resource

    def __auth(self):
        conn = ldap.initialize('ldap://' + self.__host_and_port)
        conn.protocol_version = 3
        conn.set_option(ldap.OPT_REFERRALS, 0)

        try:
            conn.simple_bind_s(self.__username, self.__password)
        except ldap.INVALID_CREDENTIALS:
            # Pinned, for the future.

            raise
        except ldap.SERVER_DOWN:
            # Pinned, for the future.

            raise
        except ldap.LDAPError:
            _LOGGER.error("LDAP error content:\n{}".format(e.message))

            if issubclass(e.message.__class__, dict) is True and \
               'desc' in e.message:
                raise Exception("LDAP: {}".format(e.message['desc']))

            raise
        else:
            return conn

    def get_dn_by_username(self, username):
        """Return user information. See _USER."""

        uf = _USER_QUERY.format(username=username)
        results_raw = \
            self.__resource.search_s(
                self.__base_dn,
                ldap.SCOPE_SUBTREE,
                uf)

        if not results_raw:
            raise NotFoundException(username)

        results = []
        for dn, attributes in results_raw:
            if dn is None:
                continue

            u = _USER(
                    dn=dn,
                    attributes=attributes,
                    groups=attributes['memberOf'])

            results.append(u)

        assert \
            len(results) == 1, \
            "More than one result was found for user [{}], which doesn't " \
            "make sense: {}".format(username, results)

        return results[0]

    def get_group_members(self, group_name):
        """Return a list of DNs."""

        gf = _GROUP_QUERY.format(group_name=group_name)
        results_raw = \
            self.__resource.search_s(
                self.__base_dn,
                ldap.SCOPE_SUBTREE,
                gf)

        if not results_raw:
            raise NotFoundException(group_name)

        collections = []
        for dn, attributes in results_raw:
            if dn is None:
                continue

            collections.append(attributes['member'])

        if not collections:
            raise NotFoundException(group_name)

        assert \
            len(collections) == 1, \
            "Too many sets of results returned: {}".format(collections)

        return collections[0]

go-exif-knife: One Exif Command-Line Tool to [Nearly] Rule Them All

go-exif-knife is a tool that will allow you to parse Exif from JPEG and PNG images and to do a brute-force parse of Exif embedded in any other format. You can cherry-pick specific IFDs or tags to print, and print them both as normal and JSON-formatted text. You can can also print parsed GPS data and timestamps and even produce a Google S2 geohash from the GPS data, and dump the thumbnail. If using JPEG or PNG, you can also update or add new Exif data.

This project is built on top of go-jpeg-image-structure, go-png-image-structure, and go-exif. PNG added support for Exif only in the last year, and this project was in service of providing useful Exif support for PNG.

Binary downloads are available here.

 

 

Go: Exif Reader/Writer Library

The go-exif project is now available. It allows you to parse and enumerate/visit/search/dump the existing IFDs/tags in an Exif blob, instantiate a builder to create and construct a new Exif blob, and create a builder from existing IFDs/tags (so you can add/remove starting from what you have). There are also utility functions to make the GPS data manageable.

There are currently 140 unit-tests in the CI process and tested examples covering enumeration, building, thumbnails, GPS, etc…

I have also published go-jpeg-image-structure and go-png-image-structure to actually implement reading/writing Exif in those corresponding formats. PNG adopted Exif support in 2017 and this project was primarily meant to provide PNG with fully-featured Exif-writer support both via library and via command-line tool.

go-exif includes a command-line utility to generally find and parse Exif data in any blob of data. This works for TIFF right off the bat (TIFF is the underlying format of Exif), which I did not specifically write a wrapper implementation for.

 

Git: Annotate Recent Changes in Blame

Pretty awesome. Pass a duration of time and the blame output will mark the lines from older commits with a “^” prefix.

$ git blame --since=3.weeks -- work_deserving_a_promotion.py

Output:

^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1285)                     remote_fil
^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1286)                     attributes
3386b3595 (Dustin Oprea 2018-05-25 19:27:55 -0400 1287) 
^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1288)             elif fnmatch.fnmat
aac11271e (Dustin Oprea 2018-05-27 02:52:29 -0400 1289)                 # If we're bui
aac11271e (Dustin Oprea 2018-05-27 02:52:29 -0400 1290)                 # and test-key

Thanks to this SO.

Go: Implementing Subcommands With go-flags

github.com/jessevdk/go-flags is the go-to tool for argument processing. It supports subcommands but understanding how to do it is a feat of reverse-engineering. So, here is an example.

Code:

package main

import (
    "os"

    "github.com/jessevdk/go-flags"
)

type readParameters struct {
}

type writeParameters struct {
}

type parameters struct {
    Verbose bool `short:"v" long:"verbose" description:"Display logging"`
    Read readParameters `command:"read" alias:"r" description:"Read functions"`
    Write readParameters `command:"write" alias:"w" description:"Write functions"`
}

var (
    arguments = new(parameters)
)

func main() {
    p := flags.NewParser(arguments, flags.Default)

    _, err := p.Parse()
    if err != nil {
        os.Exit(-1)
    }

    switch p.Active.Name {
    case "read":
        //...
    case "write":
        //...
    }
}

If you were to save it as "args.go", this is what the help and the usage would look like:

$ go run args.go -h
Usage:
  args [OPTIONS] 

Application Options:
  -v, --verbose  Display logging

Help Options:
  -h, --help     Show this help message

Available commands:
  read   Read functions (aliases: r)
  write  Write functions (aliases: w)

exit status 255

$ go run args.go read 

Git: Putting All Submodules on Their Branches

By default, submodules are initialized in a detached-head state and not made to track specific branches, even when you specify a branch when initially adding the submodule. This means that any commits you produce will not be on a particular branch and the head commit will not be updated to point to new commits (you would not be able to push any new commits, at least not in the way you expect). This is fine where there is no active development, but, otherwise, you would likely need to intervene and individually checkout each project to the branches.

Assuming you specified a branch when you added the submodule, you can use the “git submodule foreach” subcommand to automate this:

git submodule foreach --recursive 'git checkout $(git config -f .gitmodules --get submodule.$name.branch)'

You can run this from your supermodule project or qualify the “.gitmodule” filename with its path.

If you need something more complicated, you can obviously write a script and call it from this context.