Go: Image-Processing Microservice

Imaginary is a fun little package that can be Dockerized and deployed next to the rest of your services to offload image-processing:

Fast HTTP microservice written in Go for high-level image processing backed by bimg and libvips. imaginary can be used as private or public HTTP service for massive image processing with first-class support for Docker & Heroku. It’s almost dependency-free and only uses net/http native package without additional abstractions for better performance.

Advertisements

Go: Functions that satisfy interfaces

I ran across this while sifting through the http package:

// The HandlerFunc type is an adapter to allow the use of
// ordinary functions as HTTP handlers. If f is a function
// with the appropriate signature, HandlerFunc(f) is a
// Handler that calls f.
type HandlerFunc func(ResponseWriter, *Request)

// ServeHTTP calls f(w, r).
func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
    f(w, r)
}

A function that satisfies http.HandlerFunc will also automatically get a function-on-a-function (who knew) that makes it look like a struct that satisfies the http.Handler interface as well.

I had looked for a footnote about this in the section on function declarations in the Go language specification but was not successful. The receiver is little more than just a repositioned first-argument. There’s nothing that says “MUST be a struct or a func”.

Resolving Go Import URLs

The package path that you import may not directly correlate to a repository URL. If Go can not find the package in your GOPATH, it will load the URL, redirecting as necessary, and will either look for a “meta” tag with a “name” attribute having value “go-import” or take whatever URL we are at when no more HTTP redirects are necessary (if any).

To speed things up, I wrote a short Python tool to do this and stashed it in a gist.

Make sure to install the dependencies:

  • requests
  • beautifulsoup4

Source:

import sys
import argparse
import urllib.parse

import requests
import bs4

def _get_cycle(url):
    while 1:
        print("Reading: {}".format(url))

        r = requests.get(url, allow_redirects=False)
        r.raise_for_status()

        bs = bs4.BeautifulSoup(r.text, 'html.parser')

        def meta_filter(tag): 
            # We're looking for meta-tags like this:
            #
            # <meta name="go-import" content="googlemaps.github.io/maps git https://github.com/googlemaps/google-maps-services-go">
            
            return \
                tag.name == 'meta' and \
                tag.attrs.get('name') == 'go-import'

        for m in bs.find_all(meta_filter):
            phrase = m.attrs['content']
            _, vcs, repo_url_root = phrase.split(' ')
            if vcs != 'git':
                continue

            return repo_url_root

        next_url = r.headers.get('Location')
        if next_url is None:
            break

        p = urllib.parse.urlparse(next_url)
        if p.netloc == '':
            # Take the schema, hostname, and port from the last URL.
            p2 = urllib.parse.urlparse(url)
            updated_url = '{}://{}{}'.format(p2.scheme, p2.netloc, next_url)
            print("  [{}] => [{}]".format(next_url, updated_url))

            url = updated_url
        else:
            url = next_url

    return url

def _main():
    description = "Determine the import URL for the given Go import path"
    parser = argparse.ArgumentParser(description=description)
    
    parser.add_argument(
        'import_path',
        help='Go import path')

    args = parser.parse_args()

    initial_url = "https://{}".format(args.import_path)
    final_url = _get_cycle(initial_url)

    print("Final URL: [{}]".format(final_url))

if __name__ == '__main__':
    _main()

Creating TAR Archives in Go

A short program to show how to write TAR-GZ and TAR-XZ (LZMA) archives. Note that I have not included an example for TAR-BZ2 because there is no easily-findable public library for doing so.

package main

import (
    "archive/tar"
    "compress/gzip"

    "fmt"
    "os"
    "io"
    "time"

    "github.com/ulikunitz/xz"
)

func addFile(tw *tar.Writer, filepath string) {
    data := fmt.Sprintf("I am data: %s\n", filepath)

    h := new(tar.Header)
    h.Name = filepath
    h.Size = int64(len(data))
    h.Mode =  int64(0666)
    h.ModTime = time.Now()

    // write the header to the tarball archive
    if err := tw.WriteHeader(h); err != nil {
        panic(err)
    }

    // copy the file data to the tarball 
    if _, err := io.WriteString(tw, data); err != nil {
        panic(err)
    }
}

func createTarGz() {
    f, err := os.Create("output.tar.gz")
    if err != nil {
        panic(err)
    }

    defer f.Close()

    gw := gzip.NewWriter(f)
    defer gw.Close()

    tw := tar.NewWriter(gw)
    defer tw.Close()

    addFile(tw, "aa")
    addFile(tw, "bb/cc")
}

func createTarXz() {
    f, err := os.Create("output.tar.xz")
    if err != nil {
        panic(err)
    }

    defer f.Close()

    xw, err := xz.NewWriter(f)
    if err != nil {
        panic(err)
    }

    defer xw.Close()

    tw := tar.NewWriter(xw)
    defer tw.Close()

    addFile(tw, "dd")
    addFile(tw, "ee/ff")
}

func main() {
    createTarGz()
    createTarXz()
}

Examine the outputs:

$ tar tzf output.tar.gz 
aa
bb/cc
$ tar xz -O - -f output.tar.gz aa
I am data: aa
$ tar xz -O - -f output.tar.gz bb/cc
I am data: bb/cc

$ tar tJf output.tar.xz
dd
ee/ff
I am data: bb/cc
$ tar xJ -O - -f output.tar.xz dd
I am data: dd
$ tar xJ -O - -f output.tar.xz ee/ff
I am data: ee/ff

Go: Encoding Maps with Non-String Keys to JSON

I ran into some issues with 1.6.2 encoding a map to a JSON structure because it had int/int64 keys. Sure, JSON prescribes string-only keys, but I incorrectly made the reasonable assumption that Go would coerce these to strings. It turns out that this change was made in the very recent past and that, as of the last release (1.7.1) (and probably earlier ones) this should no longer be a problem.

Unfortunately, AppEngine still runs on 1.6.2, for now. So, if you had the same problem, you will have to go the conventional routeĀ and translate these keys to strings, yourself, prior to marshaling.

Using the Google Maps Client Library for Go in AppEngine

The default HTTP transport implementation for Go isn’t supported when running in AppEngine. Trying to use it will result in the following error:

http.DefaultTransport and http.DefaultClient are not available in App Engine. See https://cloud.google.com/appengine/docs/go/urlfetch/

To fix this, you need to use the http.Client implementation from AppEngine’s urlfetch package (imported from google.golang.org/appengine/urlfetch).

uc := urlfetch.Client(ctx)

options := []maps.ClientOption {
    maps.WithHTTPClient(uc),
    maps.WithAPIKey(GoogleApiKey),
}

c, err := maps.NewClient(options...)
if err != nil {
    panic(err)
}

nsr := &maps.NearbySearchRequest{
    Location: &maps.LatLng {
        Lat: latitude,
        Lng: longitude,
    },
    Radius: radius,
    OpenNow: true,
    RankBy: maps.RankByProminence,
    Type: maps.PlaceTypeRestaurant,
}

psr, err := c.NearbySearch(ctx, nsr)
if err != nil {
    panic(err)
}