Git: Annotate Recent Changes in Blame

Pretty awesome. Pass a duration of time and the blame output will mark the lines from older commits with a “^” prefix.

$ git blame --since=3.weeks -- work_deserving_a_promotion.py

Output:

^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1285)                     remote_fil
^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1286)                     attributes
3386b3595 (Dustin Oprea 2018-05-25 19:27:55 -0400 1287) 
^4412d8c5 (Dustin Oprea 2018-05-17 18:56:11 -0400 1288)             elif fnmatch.fnmat
aac11271e (Dustin Oprea 2018-05-27 02:52:29 -0400 1289)                 # If we're bui
aac11271e (Dustin Oprea 2018-05-27 02:52:29 -0400 1290)                 # and test-key

Thanks to this SO.

Advertisements

Git: Putting All Submodules on Their Branches

By default, submodules are initialized in a detached-head state and not made to track specific branches, even when you specify a branch when initially adding the submodule. This means that any commits you produce will not be on a particular branch and the head commit will not be updated to point to new commits (you would not be able to push any new commits, at least not in the way you expect). This is fine where there is no active development, but, otherwise, you would likely need to intervene and individually checkout each project to the branches.

Assuming you specified a branch when you added the submodule, you can use the “git submodule foreach” subcommand to automate this:

git submodule foreach --recursive 'git checkout $(git config -f .gitmodules --get submodule.$name.branch)'

You can run this from your supermodule project or qualify the “.gitmodule” filename with its path.

If you need something more complicated, you can obviously write a script and call it from this context.

Git: Automatically Squashing at the Prompt

I do a huge amount of squashing, every day of the week. Ever the kind of engineer who wishes to optimize every single redundant operation, I wrote a simple script and then aliased it in my shell. When I do a commit that I know I will be squashing into the previous commit, I simply do a “git commit -m SQUASH -a” and then run “SQUASH_LAST” (my alias, which is autocompleted) to run the squash. The script verifies that the last commit message starts with “SQUASH” (for verification/sanity), executes the squash, and then prints the current commit, previous commit, and final commit revisions.

It is extremely convenient and saves a ton of time and annoyingly-repetitive steps.

The script (which I put in my home):

#!/bin/bash -e

HEAD_COMMIT_MESSAGE=$(git log --format=%B -1 HEAD)

# For safety. Our use-case is usually to always just squash into a commit
# that's associated with an active change. We really don't want lose our head
# and accidentally squash something that wasn't intended to be squashed.
if [[ "${HEAD_COMMIT_MESSAGE}" != SQUASH* ]]; then
    echo "SQUASH: Commit to be squashed should have 'SQUASH' as its commit-message."
    exit 1
fi

_FILEPATH=$(mktemp)
git log --format=%B -1 HEAD~1 >"${_FILEPATH}"

echo "Initial head: $(git rev-parse HEAD)"

git reset --soft HEAD~2 >/dev/null

echo "Head after reset: $(git rev-parse HEAD)"

git commit -F $_FILEPATH >/dev/null
rm $_FILEPATH

echo "Head after commit: $(git rev-parse HEAD)"

echo

The alias (for completeness):

alias SQUASH_LAST='<filepath>'

It really is about the little things.

I have also put the script into a gist.

Go: Testing Against Application Binaries

Unit-testing in Go is simple and a pleasure. The minimum structure required to do unit-tests is scarcely more than that required to write any kind of code. In fact, most of the time it is so easy that you are almost, arguably, guaranteed to waste time doing any debugging at all before you have written unit-tests.

However, it may take a little more thought to test your executables. Even though you can still have a unit-testing source-file (“*_test.go”) and you can call your main() to do something, it’s non-trivial to capture your output and/or pass arguments:

  • You might end-up using os.Pipe() to hook stdout/stderr and launching a goroutine to read from the other end, but you might have issues.
  • Your test might call back into the execute in os.Args[0] (the tests run from a test-specific binary generated by the testing process), but this won’t accept the arbitrary command-line arguments required by your application.
  • You might wrap a call to “go test” and try to pass “-args ” (“-args” is like “–” for tests, where all following arguments are passed verbatim), but I have had issues with this.

Naturally, you want to avoid having to kick-off a build of your application at the top of the tests in order to have something to test against.

You can use “go run” with exec.Command (in os/exec) to easily accomplish all of this while still avoiding a manual build. You can even provide it alternative io.Writer instances in order to capture stdout/stderr output.

Example:

package main

import (
    "testing"
    "os"
    "path"
    "bytes"
    "fmt"

    "os/exec"
)

var (
    assetsPath = ""
    appFilepath = ""
)

func TestMain(t *testing.T) {
    imageFilepath := path.Join(assetsPath, "NDM_8901.jpg")

    cmd := exec.Command(
            "go", "run", appFilepath,
            "-filepath", imageFilepath)

    b := new(bytes.Buffer)
    cmd.Stdout = b
    cmd.Stderr = b

    err := cmd.Run()
    actual := b.String()

    if err != nil {
        fmt.Printf(actual)
        panic(err)
    }

    expected := `IFD=[IfdIdentity] ID=(0x010f) NAME=[Make] COUNT=(6) TYPE=[ASCII] VALUE=[Canon]
IFD=[IfdIdentity] ID=(0x0110) NAME=[Model] COUNT=(22) TYPE=[ASCII] VALUE=[Canon EOS 5D Mark III]
IFD=[IfdIdentity] ID=(0x0112) NAME=[Orientation] COUNT=(1) TYPE=[SHORT] VALUE=[1]
IFD=[IfdIdentity] ID=(0x011a) NAME=[XResolution] COUNT=(1) TYPE=[RATIONAL] VALUE=[72/1]
IFD=[IfdIdentity] ID=(0x011b) NAME=[YResolution] COUNT=(1) TYPE=[RATIONAL] VALUE=[72/1]
IFD=[IfdIdentity] ID=(0x0128) NAME=[ResolutionUnit] COUNT=(1) TYPE=[SHORT] VALUE=[2]
...
IFD=[IfdIdentity] ID=(0x0128) NAME=[ResolutionUnit] COUNT=(1) TYPE=[SHORT] VALUE=[2]
IFD=[IfdIdentity] ID=(0x0201) NAME=[JPEGInterchangeFormat] COUNT=(1) TYPE=[LONG] VALUE=[11444]
IFD=[IfdIdentity] ID=(0x0202) NAME=[JPEGInterchangeFormatLength] COUNT=(1) TYPE=[LONG] VALUE=[21491]
`

    if actual != expected {
        t.Fatalf("Output not as expected:\n%s", actual)
    }
}

func init() {
    goPath := os.Getenv("GOPATH")

    assetsPath = path.Join(goPath, "src", "github.com", "dsoprea", "go-exif", "assets")
    appFilepath = path.Join(goPath, "src", "github.com", "dsoprea", "go-exif", "exif-read-tool", "main.go")
}

Python: Writing Hex Values into YAML

YAML has the ability to express hex-values, which are then decoded as numbers. However, when you want to dump a YAML document, strings will be quoted and numbers will be decimals. In order to write actual hex-values, you need to wrap your value in another type and then tell the YAML encoder how to handle it.

This is specifically possible with the ruamel YAML encoder (pypi).

An example of how to do this:

import sys

import ruamel.yaml


class HexInt(int):
    pass

def representer(dumper, data):
    return \
        ruamel.yaml.ScalarNode(
            'tag:yaml.org,2002:int',
            '0x{:04x}'.format(data))

ruamel.yaml.add_representer(HexInt, representer)

data = {
    'item1': {
        'string_value': 'some_string',
        'hex_value': HexInt(641),
    }
}

ruamel.yaml.dump(data, sys.stdout, default_flow_style=False)

Output:

item1:
  hex_value: 0x0281
  string_value: some_string

Please note that I require that my hex-values are two bytes and padded with zeroes, so the example above will print four characters (plus the prefix): 0x{:04x} . If this doesn’t work for you, change it to whatever you require.

Thanks to this post.

Git: Producing a Revert Commit for a Previous Change

Create an inverse commit to flip a previous change. Child’s play:

$ git revert <REFSPEC>

The new commit looks like:

commit 09cc98e3fa121774750728f5fa337befeb02d914
Author: Dustin Oprea <dustin@randomingenuity.com>
Date:   Tue Mar 27 16:02:25 2018 -0400

    Revert "What's the worst that could happen?"

    This reverts commit cf4fc9a50a20a633b82ee28ef9efa46df86db18d.

A lot more nicer and a lot more professional than copying-and-pasting into a new commit or dropping an old commit with a rebase.

It is nearly identical to similar, existing features provided by many version-control review systems.

GoogleAutoAuth: Automate the Google API Authentication Flow in Your Project

I write a ton of system software and tools. I’ve written a few independent tools against the Google APIs. They use OAuth, as most reputable APIs do.

Unfortunately, manually integrating the authentication flow in your system (read: headless, non-interactive) tools is painful after doing it a couple of times and is at least as painful for your users to deal with, especially when they have to log-in to a system that they do not usually have to touch just to periodically debug authentication.

The normal flow:

  1. Developer: Request a URL from the Google client-tools.
  2. Developer: Present the URL to the user and have them open it in a browser.
  3. User: Logs-in.
  4. User: Acknowledge that the tool will be able to access user’s data.
  5. Google: Authorization portal provides a code/token.
  6. User: Provides the code to the tool at the command-line.
  7. Developer: Does a final authorization with Google using the token.

With some mild wizardry in our Python tools, we can reduce this down to two basic steps:

  1. Developer: Initialize the auto-authentication framework with Google application-identity information at program-load.
  2. User: Call the tool and authenticate and authorize when prompted.

The tool will automatically launch a webserver on an open port, open the default browser with the Google login and authorization prompt, and then write the authorization to disk.

In the event that you want or need to do things manually, a generic tool is provided that can produce URLs and accept authorization codes.

For more information, see python-googleautoauth.