go

Go: Build-Time Variables

Go allows you to override global variables at build time during the linker stage:

$ go build -o /tmp/prog-custom -ldflags "-X main.overrideableValuePhrase=123456" ./cmd/prog

Requirements:

  • It must be a global variable and not a constant.
  • It might be in the package that you are telling it to build. This definitely works for executables and, with the removal of support for binary-only packages (BOPs), it probably doesn’t apply whatsoever to intermediate packages.
  • It must be a string (so, process it from the init() function).

It doesn’t matter whether it is an exported or unexported symbol.

 

Go: Write RPC-Connected Plugins

Plugins are Go’s system for developing shared-libraries. However, this is backed by a general system that can also have alternative implementations. In this case, you can write a plugin in Go that runs from one system and load that plugin in Go, on the fly, from a million other systems. Courtesy of Hashicorp.

https://github.com/hashicorp/go-plugin

 

 

Go: An In-Memory ReadWriteSeeker

The SeekableBuffer type is similar to a bytes.Buffer in that you can perform both reads and writes against it, but has the following two main differences:

  1. You can seek on it.
  2. After writing to it, the current position will be on the byte following whatever you wrote. This is the typical behavior for a file resource on almost any platform but not for a bytes.Buffer.

Eccentric usage of seek, read, and write behavior, such as the following, will work as expected:

  • seeking but not writing
  • seeking past the end of the file and writing
  • seeking past the end of the file and reading
  • writing N+M bytes when positioned only N bytes from the end of the file

Usage (from the unit-tests):

sb := NewSeekableBuffer()

// Write first string.

data := []byte("word1-word2")

_, err := sb.Write(data)
log.PanicIf(err)

// Seek and replace partway through, and replace more data than we
// currently have.

_, err = sb.Seek(6, os.SEEK_SET)
log.PanicIf(err)

data2 := []byte("word3-word4")

_, err = sb.Write(data2)
log.PanicIf(err)

// Read contents.

_, err = sb.Seek(0, os.SEEK_SET)
log.PanicIf(err)

buffer := make([]byte, 20)

_, err = sb.Read(buffer)
log.PanicIf(err)

// `buffer` currently has "word1-word3-word4".

Go: Parsing Time Expressions

go-time-parse will parse time expressions into time.Duration quantities. From the example:

actualDuration, phraseType, err := ParseDuration("24 days from now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour/24, phraseType)

actualDuration, phraseType, err = ParseDuration("now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration, phraseType)

actualDuration, phraseType, err = ParseDuration("12m")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Minute, phraseType)

actualDuration, phraseType, err = ParseDuration("every 6 hours")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour, phraseType)

Output:

24 [time]
0 [time]
12 [interval]
6 [interval]

Identifying Nearest Major Cities

go-geographic-attractor is a new project that indexes world city and population data and can match a given coordinate to either the nearest major city or the nearest city (if no major city is near) in near-instantaneous time.

From the example:

// Load countries.

countryDataFilepath := path.Join(appPath, "test", "asset", "countryInfo.txt")

f, err := os.Open(countryDataFilepath)
log.PanicIf(err)

defer f.Close()

countries, err := geoattractorparse.BuildGeonamesCountryMapping(f)
log.PanicIf(err)

// Load cities.

gp := geoattractorparse.NewGeonamesParser(countries)

cityDataFilepath := path.Join(appPath, "index", "test", "asset", "allCountries.txt.detroit_area_handpicked")
g, err := os.Open(cityDataFilepath)
log.PanicIf(err)

defer g.Close()

ci := NewCityIndex()

err = ci.Load(gp, g)
log.PanicIf(err)

// Do the query.

clawsonCoordinates := []float64{42.53667, -83.15041}

sourceName, visits, cr, err := ci.Nearest(clawsonCoordinates[0], clawsonCoordinates[1])
log.PanicIf(err)

// Print the results.

for _, vhi := range visits {
    fmt.Printf("%s: %s\n", vhi.Token, vhi.City)
}

fmt.Printf("\n")

fmt.Printf("Source: %s\n", sourceName)
fmt.Printf("ID: %s\n", cr.Id)
fmt.Printf("Country: %s\n", cr.Country)
fmt.Printf("City: %s\n", cr.City)
fmt.Printf("Population: %d\n", cr.Population)
fmt.Printf("Latitude: %.10f\n", cr.Latitude)
fmt.Printf("Longitude: %.10f\n", cr.Longitude)

Go: Generating KML

Use go-kml to build a KML structure. Then, marshal using the xml package.

An example using code from the project:

package main

import (
    "encoding/xml"
    "os"

    "github.com/twpayne/go-kml"
)

func main() {
    x := kml.KML(
        kml.Placemark(
            kml.Name("The Pentagon"),
            kml.Polygon(
                kml.Extrude(true),
                kml.AltitudeMode("relativeToGround"),
                kml.OuterBoundaryIs(
                    kml.LinearRing(
                        kml.Coordinates([]kml.Coordinate{
                            {-77.05788457660967, 38.87253259892824, 100},
                            {-77.05465973756702, 38.87291016281703, 100},
                            {-77.05315536854791, 38.87053267794386, 100},
                            {-77.05552622493516, 38.868757801256, 100},
                            {-77.05844056290393, 38.86996206506943, 100},
                            {-77.05788457660967, 38.87253259892824, 100},
                        }...),
                    ),
                ),
            ),
        ),
    )

    e := xml.NewEncoder(os.Stdout)
    e.Indent("", "  ")

    err := e.Encode(x)
    if err != nil {
        panic(err)
    }
}

This prints:

<kml xmlns="http://www.opengis.net/kml/2.2">
  <Placemark>
    <name>The Pentagon</name>
    <Polygon>
      <extrude>1</extrude>
      <altitudeMode>relativeToGround</altitudeMode>
      <outerBoundaryIs>
        <LinearRing>
          <coordinates>-77.05788457660967,38.87253259892824,100 -77.05465973756702,38.87291016281703,100 -77.0531553685479,38.87053267794386,100 -77.05552622493516,38.868757801256,100 -77.05844056290393,38.86996206506943,100 -77.05788457660967,38.87253259892824,100</coordinates>
        </LinearRing>
      </outerBoundaryIs>
    </Polygon>
  </Placemark>
</kml>

..which renders as this when using Google Earth:

Google Earth Pro_004

Go: Read and Browse Ext4 Filesystems in User-Space

go-ext4 is a pure Go implementation of an Ext4 reader with support for reading the journal. An example of how to walk the file-structure:

inodeNumber := InodeRootDirectory

filepath := path.Join(assetsPath, "hierarchy_32.ext4")

f, err := os.Open(filepath)
log.PanicIf(err)

defer f.Close()

_, err = f.Seek(Superblock0Offset, io.SeekStart)
log.PanicIf(err)

sb, err := NewSuperblockWithReader(f)
log.PanicIf(err)

bgdl, err := NewBlockGroupDescriptorListWithReadSeeker(f, sb)
log.PanicIf(err)

bgd, err := bgdl.GetWithAbsoluteInode(inodeNumber)
log.PanicIf(err)

dw, err := NewDirectoryWalk(f, bgd, inodeNumber)
log.PanicIf(err)

allEntries := make([]string, 0)

for {
    fullPath, de, err := dw.Next()
    if err == io.EOF {
        break
    } else if err != nil {
        log.Panic(err)
    }

    description := fmt.Sprintf("%s: %s", fullPath, de.String())
    allEntries = append(allEntries, description)
}

sort.Strings(allEntries)

for _, entryDescription := range allEntries {
    fmt.Println(entryDescription)
}

// Output:
//
// directory1/fortune1: DirectoryEntry
// directory1/fortune2: DirectoryEntry
// directory1/fortune5: DirectoryEntry
// directory1/fortune6: DirectoryEntry
// directory1/subdirectory1/fortune3: DirectoryEntry
// directory1/subdirectory1/fortune4: DirectoryEntry
// directory1/subdirectory1: DirectoryEntry
// directory1/subdirectory2/fortune7: DirectoryEntry
// directory1/subdirectory2/fortune8: DirectoryEntry
// directory1/subdirectory2: DirectoryEntry
// directory1: DirectoryEntry
// directory2/fortune10: DirectoryEntry
// directory2/fortune9: DirectoryEntry
// directory2: DirectoryEntry
// lost+found: DirectoryEntry
// thejungle.txt: DirectoryEntry

This project is used to directly read the filesystem, file, and journal data without the support of kernel or the FUSE interface. Therefore, no elevated privileges are required.

go-exif-knife: One Exif Command-Line Tool to [Nearly] Rule Them All

go-exif-knife is a tool that will allow you to parse Exif from JPEG and PNG images and to do a brute-force parse of Exif embedded in any other format. You can cherry-pick specific IFDs or tags to print, and print them both as normal and JSON-formatted text. You can can also print parsed GPS data and timestamps and even produce a Google S2 geohash from the GPS data, and dump the thumbnail. If using JPEG or PNG, you can also update or add new Exif data.

This project is built on top of go-jpeg-image-structure, go-png-image-structure, and go-exif. PNG added support for Exif only in the last year, and this project was in service of providing useful Exif support for PNG.

Binary downloads are available here.

 

 

Go: Exif Reader/Writer Library

The go-exif project is now available. It allows you to parse and enumerate/visit/search/dump the existing IFDs/tags in an Exif blob, instantiate a builder to create and construct a new Exif blob, and create a builder from existing IFDs/tags (so you can add/remove starting from what you have). There are also utility functions to make the GPS data manageable.

There are currently 140 unit-tests in the CI process and tested examples covering enumeration, building, thumbnails, GPS, etc…

I have also published go-jpeg-image-structure and go-png-image-structure to actually implement reading/writing Exif in those corresponding formats. PNG adopted Exif support in 2017 and this project was primarily meant to provide PNG with fully-featured Exif-writer support both via library and via command-line tool.

go-exif includes a command-line utility to generally find and parse Exif data in any blob of data. This works for TIFF right off the bat (TIFF is the underlying format of Exif), which I did not specifically write a wrapper implementation for.