Go: Write RPC-Connected Plugins

Plugins are Go’s system for developing shared-libraries. However, this is backed by a general system that can also have alternative implementations. In this case, you can write a plugin in Go that runs from one system and load that plugin in Go, on the fly, from a million other systems. Courtesy of Hashicorp.

https://github.com/hashicorp/go-plugin

 

 

Go: An In-Memory ReadWriteSeeker

The SeekableBuffer type is similar to a bytes.Buffer in that you can perform both reads and writes against it, but has the following two main differences:

  1. You can seek on it.
  2. After writing to it, the current position will be on the byte following whatever you wrote. This is the typical behavior for a file resource on almost any platform but not for a bytes.Buffer.

Eccentric usage of seek, read, and write behavior, such as the following, will work as expected:

  • seeking but not writing
  • seeking past the end of the file and writing
  • seeking past the end of the file and reading
  • writing N+M bytes when positioned only N bytes from the end of the file

Usage (from the unit-tests):

sb := NewSeekableBuffer()

// Write first string.

data := []byte("word1-word2")

_, err := sb.Write(data)
log.PanicIf(err)

// Seek and replace partway through, and replace more data than we
// currently have.

_, err = sb.Seek(6, os.SEEK_SET)
log.PanicIf(err)

data2 := []byte("word3-word4")

_, err = sb.Write(data2)
log.PanicIf(err)

// Read contents.

_, err = sb.Seek(0, os.SEEK_SET)
log.PanicIf(err)

buffer := make([]byte, 20)

_, err = sb.Read(buffer)
log.PanicIf(err)

// `buffer` currently has "word1-word3-word4".

Go: Parsing Time Expressions

go-time-parse will parse time expressions into time.Duration quantities. From the example:

actualDuration, phraseType, err := ParseDuration("24 days from now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour/24, phraseType)

actualDuration, phraseType, err = ParseDuration("now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration, phraseType)

actualDuration, phraseType, err = ParseDuration("12m")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Minute, phraseType)

actualDuration, phraseType, err = ParseDuration("every 6 hours")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour, phraseType)

Output:

24 [time]
0 [time]
12 [interval]
6 [interval]

Identifying Nearest Major Cities

go-geographic-attractor is a new project that indexes world city and population data and can match a given coordinate to either the nearest major city or the nearest city (if no major city is near) in near-instantaneous time.

From the example:

// Load countries.

countryDataFilepath := path.Join(appPath, "test", "asset", "countryInfo.txt")

f, err := os.Open(countryDataFilepath)
log.PanicIf(err)

defer f.Close()

countries, err := geoattractorparse.BuildGeonamesCountryMapping(f)
log.PanicIf(err)

// Load cities.

gp := geoattractorparse.NewGeonamesParser(countries)

cityDataFilepath := path.Join(appPath, "index", "test", "asset", "allCountries.txt.detroit_area_handpicked")
g, err := os.Open(cityDataFilepath)
log.PanicIf(err)

defer g.Close()

ci := NewCityIndex()

err = ci.Load(gp, g)
log.PanicIf(err)

// Do the query.

clawsonCoordinates := []float64{42.53667, -83.15041}

sourceName, visits, cr, err := ci.Nearest(clawsonCoordinates[0], clawsonCoordinates[1])
log.PanicIf(err)

// Print the results.

for _, vhi := range visits {
    fmt.Printf("%s: %s\n", vhi.Token, vhi.City)
}

fmt.Printf("\n")

fmt.Printf("Source: %s\n", sourceName)
fmt.Printf("ID: %s\n", cr.Id)
fmt.Printf("Country: %s\n", cr.Country)
fmt.Printf("City: %s\n", cr.City)
fmt.Printf("Population: %d\n", cr.Population)
fmt.Printf("Latitude: %.10f\n", cr.Latitude)
fmt.Printf("Longitude: %.10f\n", cr.Longitude)

Go: Generating KML

Use go-kml to build a KML structure. Then, marshal using the xml package.

An example using code from the project:

package main

import (
    "encoding/xml"
    "os"

    "github.com/twpayne/go-kml"
)

func main() {
    x := kml.KML(
        kml.Placemark(
            kml.Name("The Pentagon"),
            kml.Polygon(
                kml.Extrude(true),
                kml.AltitudeMode("relativeToGround"),
                kml.OuterBoundaryIs(
                    kml.LinearRing(
                        kml.Coordinates([]kml.Coordinate{
                            {-77.05788457660967, 38.87253259892824, 100},
                            {-77.05465973756702, 38.87291016281703, 100},
                            {-77.05315536854791, 38.87053267794386, 100},
                            {-77.05552622493516, 38.868757801256, 100},
                            {-77.05844056290393, 38.86996206506943, 100},
                            {-77.05788457660967, 38.87253259892824, 100},
                        }...),
                    ),
                ),
            ),
        ),
    )

    e := xml.NewEncoder(os.Stdout)
    e.Indent("", "  ")

    err := e.Encode(x)
    if err != nil {
        panic(err)
    }
}

This prints:

<kml xmlns="http://www.opengis.net/kml/2.2">
  <Placemark>
    <name>The Pentagon</name>
    <Polygon>
      <extrude>1</extrude>
      <altitudeMode>relativeToGround</altitudeMode>
      <outerBoundaryIs>
        <LinearRing>
          <coordinates>-77.05788457660967,38.87253259892824,100 -77.05465973756702,38.87291016281703,100 -77.0531553685479,38.87053267794386,100 -77.05552622493516,38.868757801256,100 -77.05844056290393,38.86996206506943,100 -77.05788457660967,38.87253259892824,100</coordinates>
        </LinearRing>
      </outerBoundaryIs>
    </Polygon>
  </Placemark>
</kml>

..which renders as this when using Google Earth:

Google Earth Pro_004

Go: Read and Browse Ext4 Filesystems in User-Space

go-ext4 is a pure Go implementation of an Ext4 reader with support for reading the journal. An example of how to walk the file-structure:

inodeNumber := InodeRootDirectory

filepath := path.Join(assetsPath, "hierarchy_32.ext4")

f, err := os.Open(filepath)
log.PanicIf(err)

defer f.Close()

_, err = f.Seek(Superblock0Offset, io.SeekStart)
log.PanicIf(err)

sb, err := NewSuperblockWithReader(f)
log.PanicIf(err)

bgdl, err := NewBlockGroupDescriptorListWithReadSeeker(f, sb)
log.PanicIf(err)

bgd, err := bgdl.GetWithAbsoluteInode(inodeNumber)
log.PanicIf(err)

dw, err := NewDirectoryWalk(f, bgd, inodeNumber)
log.PanicIf(err)

allEntries := make([]string, 0)

for {
    fullPath, de, err := dw.Next()
    if err == io.EOF {
        break
    } else if err != nil {
        log.Panic(err)
    }

    description := fmt.Sprintf("%s: %s", fullPath, de.String())
    allEntries = append(allEntries, description)
}

sort.Strings(allEntries)

for _, entryDescription := range allEntries {
    fmt.Println(entryDescription)
}

// Output:
//
// directory1/fortune1: DirectoryEntry
// directory1/fortune2: DirectoryEntry
// directory1/fortune5: DirectoryEntry
// directory1/fortune6: DirectoryEntry
// directory1/subdirectory1/fortune3: DirectoryEntry
// directory1/subdirectory1/fortune4: DirectoryEntry
// directory1/subdirectory1: DirectoryEntry
// directory1/subdirectory2/fortune7: DirectoryEntry
// directory1/subdirectory2/fortune8: DirectoryEntry
// directory1/subdirectory2: DirectoryEntry
// directory1: DirectoryEntry
// directory2/fortune10: DirectoryEntry
// directory2/fortune9: DirectoryEntry
// directory2: DirectoryEntry
// lost+found: DirectoryEntry
// thejungle.txt: DirectoryEntry

This project is used to directly read the filesystem, file, and journal data without the support of kernel or the FUSE interface. Therefore, no elevated privileges are required.