golang

Go: Build-Time Variables

Go allows you to override global variables at build time during the linker stage:

$ go build -o /tmp/prog-custom -ldflags "-X main.overrideableValuePhrase=123456" ./cmd/prog

Requirements:

  • It must be a global variable and not a constant.
  • It might be in the package that you are telling it to build. This definitely works for executables and, with the removal of support for binary-only packages (BOPs), it probably doesn’t apply whatsoever to intermediate packages.
  • It must be a string (so, process it from the init() function).

It doesn’t matter whether it is an exported or unexported symbol.

 

Go: Write RPC-Connected Plugins

Plugins are Go’s system for developing shared-libraries. However, this is backed by a general system that can also have alternative implementations. In this case, you can write a plugin in Go that runs from one system and load that plugin in Go, on the fly, from a million other systems. Courtesy of Hashicorp.

https://github.com/hashicorp/go-plugin

 

 

Go: Parsing Time Expressions

go-time-parse will parse time expressions into time.Duration quantities. From the example:

actualDuration, phraseType, err := ParseDuration("24 days from now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour/24, phraseType)

actualDuration, phraseType, err = ParseDuration("now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration, phraseType)

actualDuration, phraseType, err = ParseDuration("12m")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Minute, phraseType)

actualDuration, phraseType, err = ParseDuration("every 6 hours")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour, phraseType)

Output:

24 [time]
0 [time]
12 [interval]
6 [interval]

Identifying Nearest Major Cities

go-geographic-attractor is a new project that indexes world city and population data and can match a given coordinate to either the nearest major city or the nearest city (if no major city is near) in near-instantaneous time.

From the example:

// Load countries.

countryDataFilepath := path.Join(appPath, "test", "asset", "countryInfo.txt")

f, err := os.Open(countryDataFilepath)
log.PanicIf(err)

defer f.Close()

countries, err := geoattractorparse.BuildGeonamesCountryMapping(f)
log.PanicIf(err)

// Load cities.

gp := geoattractorparse.NewGeonamesParser(countries)

cityDataFilepath := path.Join(appPath, "index", "test", "asset", "allCountries.txt.detroit_area_handpicked")
g, err := os.Open(cityDataFilepath)
log.PanicIf(err)

defer g.Close()

ci := NewCityIndex()

err = ci.Load(gp, g)
log.PanicIf(err)

// Do the query.

clawsonCoordinates := []float64{42.53667, -83.15041}

sourceName, visits, cr, err := ci.Nearest(clawsonCoordinates[0], clawsonCoordinates[1])
log.PanicIf(err)

// Print the results.

for _, vhi := range visits {
    fmt.Printf("%s: %s\n", vhi.Token, vhi.City)
}

fmt.Printf("\n")

fmt.Printf("Source: %s\n", sourceName)
fmt.Printf("ID: %s\n", cr.Id)
fmt.Printf("Country: %s\n", cr.Country)
fmt.Printf("City: %s\n", cr.City)
fmt.Printf("Population: %d\n", cr.Population)
fmt.Printf("Latitude: %.10f\n", cr.Latitude)
fmt.Printf("Longitude: %.10f\n", cr.Longitude)

Go: Read and Browse Ext4 Filesystems in User-Space

go-ext4 is a pure Go implementation of an Ext4 reader with support for reading the journal. An example of how to walk the file-structure:

inodeNumber := InodeRootDirectory

filepath := path.Join(assetsPath, "hierarchy_32.ext4")

f, err := os.Open(filepath)
log.PanicIf(err)

defer f.Close()

_, err = f.Seek(Superblock0Offset, io.SeekStart)
log.PanicIf(err)

sb, err := NewSuperblockWithReader(f)
log.PanicIf(err)

bgdl, err := NewBlockGroupDescriptorListWithReadSeeker(f, sb)
log.PanicIf(err)

bgd, err := bgdl.GetWithAbsoluteInode(inodeNumber)
log.PanicIf(err)

dw, err := NewDirectoryWalk(f, bgd, inodeNumber)
log.PanicIf(err)

allEntries := make([]string, 0)

for {
    fullPath, de, err := dw.Next()
    if err == io.EOF {
        break
    } else if err != nil {
        log.Panic(err)
    }

    description := fmt.Sprintf("%s: %s", fullPath, de.String())
    allEntries = append(allEntries, description)
}

sort.Strings(allEntries)

for _, entryDescription := range allEntries {
    fmt.Println(entryDescription)
}

// Output:
//
// directory1/fortune1: DirectoryEntry
// directory1/fortune2: DirectoryEntry
// directory1/fortune5: DirectoryEntry
// directory1/fortune6: DirectoryEntry
// directory1/subdirectory1/fortune3: DirectoryEntry
// directory1/subdirectory1/fortune4: DirectoryEntry
// directory1/subdirectory1: DirectoryEntry
// directory1/subdirectory2/fortune7: DirectoryEntry
// directory1/subdirectory2/fortune8: DirectoryEntry
// directory1/subdirectory2: DirectoryEntry
// directory1: DirectoryEntry
// directory2/fortune10: DirectoryEntry
// directory2/fortune9: DirectoryEntry
// directory2: DirectoryEntry
// lost+found: DirectoryEntry
// thejungle.txt: DirectoryEntry

This project is used to directly read the filesystem, file, and journal data without the support of kernel or the FUSE interface. Therefore, no elevated privileges are required.

Go: Image-Processing Microservice

Imaginary is a fun little package that can be Dockerized and deployed next to the rest of your services to offload image-processing:

Fast HTTP microservice written in Go for high-level image processing backed by bimg and libvips. imaginary can be used as private or public HTTP service for massive image processing with first-class support for Docker & Heroku. It’s almost dependency-free and only uses net/http native package without additional abstractions for better performance.

Creating TAR Archives in Go

A short program to show how to write TAR-GZ and TAR-XZ (LZMA) archives. Note that I have not included an example for TAR-BZ2 because there is no easily-findable public library for doing so.

package main

import (
    "archive/tar"
    "compress/gzip"

    "fmt"
    "os"
    "io"
    "time"

    "github.com/ulikunitz/xz"
)

func addFile(tw *tar.Writer, filepath string) {
    data := fmt.Sprintf("I am data: %s\n", filepath)

    h := new(tar.Header)
    h.Name = filepath
    h.Size = int64(len(data))
    h.Mode =  int64(0666)
    h.ModTime = time.Now()

    // write the header to the tarball archive
    if err := tw.WriteHeader(h); err != nil {
        panic(err)
    }

    // copy the file data to the tarball 
    if _, err := io.WriteString(tw, data); err != nil {
        panic(err)
    }
}

func createTarGz() {
    f, err := os.Create("output.tar.gz")
    if err != nil {
        panic(err)
    }

    defer f.Close()

    gw := gzip.NewWriter(f)
    defer gw.Close()

    tw := tar.NewWriter(gw)
    defer tw.Close()

    addFile(tw, "aa")
    addFile(tw, "bb/cc")
}

func createTarXz() {
    f, err := os.Create("output.tar.xz")
    if err != nil {
        panic(err)
    }

    defer f.Close()

    xw, err := xz.NewWriter(f)
    if err != nil {
        panic(err)
    }

    defer xw.Close()

    tw := tar.NewWriter(xw)
    defer tw.Close()

    addFile(tw, "dd")
    addFile(tw, "ee/ff")
}

func main() {
    createTarGz()
    createTarXz()
}

Examine the outputs:

$ tar tzf output.tar.gz 
aa
bb/cc
$ tar xz -O - -f output.tar.gz aa
I am data: aa
$ tar xz -O - -f output.tar.gz bb/cc
I am data: bb/cc

$ tar tJf output.tar.xz
dd
ee/ff
I am data: bb/cc
$ tar xJ -O - -f output.tar.xz dd
I am data: dd
$ tar xJ -O - -f output.tar.xz ee/ff
I am data: ee/ff

Using the Google Maps Client Library for Go in AppEngine

The default HTTP transport implementation for Go isn’t supported when running in AppEngine. Trying to use it will result in the following error:

http.DefaultTransport and http.DefaultClient are not available in App Engine. See https://cloud.google.com/appengine/docs/go/urlfetch/

To fix this, you need to use the http.Client implementation from AppEngine’s urlfetch package (imported from google.golang.org/appengine/urlfetch).

uc := urlfetch.Client(ctx)

options := []maps.ClientOption {
    maps.WithHTTPClient(uc),
    maps.WithAPIKey(GoogleApiKey),
}

c, err := maps.NewClient(options...)
if err != nil {
    panic(err)
}

nsr := &maps.NearbySearchRequest{
    Location: &maps.LatLng {
        Lat: latitude,
        Lng: longitude,
    },
    Radius: radius,
    OpenNow: true,
    RankBy: maps.RankByProminence,
    Type: maps.PlaceTypeRestaurant,
}

psr, err := c.NearbySearch(ctx, nsr)
if err != nil {
    panic(err)
}