Implementing a Sandboxed Go Environment

Google’s “Native Client” (NaCl) is used to produce a restricted execution environment. It’s used in Google Chrome (to run C/C++ apps) and Google Playground (to run Go code safely):

Native Client

Note that this project is not to be confused with the Sodium (NaCL) cryptopgraphic library (project Go package).

Advertisements

Implementing Sessions Under AppEngine With Go

A simple and intuitive package named cascadestore provided by the go-appengine-sessioncascade project to implement and combine Memcache, Datastore, the request context, or any combination of them, as session backends under AppEngine.

Example:

package handlers

import (
    "net/http"

    "google.golang.org/appengine"
    "google.golang.org/appengine/log"

    "github.com/dsoprea/goappenginesessioncascade"
)

const (
    sessionName = "MainSession"
)

var (
    sessionSecret = []byte("SessionSecret")
    sessionStore  = cascadestore.NewCascadeStore(cascadestore.DistributedBackends, sessionSecret)
)

func HandleRequest(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)

    if session, err := sessionStore.Get(r, sessionName); err != nil {
        panic(err)
    } else {
        if vRaw, found := session.Values["ExistingKey"]; found == false {
            log.Debugf(ctx, "Existing value not found.")
        } else {
            v := vRaw.(string)
            log.Debugf(ctx, "Existing value: [%s]", v)
        }

        session.Values["NewKey"] = "NewValue"
        if err := session.Save(r, w); err != nil {
         panic(err)
        }
    }
}

Efficiently Processing GPX Files in Go

Use gpxreader to process a GPX file of any size without reading the whole thing into memory. This also avoids Go’s issue where the decoder can decode one node at a time, but, when you do that, it implicitly ignores all child nodes (because it automatically seeks to the matching close tag for validation without any ability to disable this behavior).

An excerpt of the test-script from the project:

//...

func (gv *gpxVisitor) GpxOpen(gpx *gpxreader.Gpx) error {
    fmt.Printf("GPX: %s\n", gpx)

    return nil
}

func (gv *gpxVisitor) GpxClose(gpx *gpxreader.Gpx) error {
    return nil
}

func (gv *gpxVisitor) TrackOpen(track *gpxreader.Track) error {
    fmt.Printf("Track: %s\n", track)

    return nil
}

func (gv *gpxVisitor) TrackClose(track *gpxreader.Track) error {
    return nil
}

func (gv *gpxVisitor) TrackSegmentOpen(trackSegment *gpxreader.TrackSegment) error {
    fmt.Printf("Track segment: %s\n", trackSegment)

    return nil
}

func (gv *gpxVisitor) TrackSegmentClose(trackSegment *gpxreader.TrackSegment) error {
    return nil
}

func (gv *gpxVisitor) TrackPointOpen(trackPoint *gpxreader.TrackPoint) error {
    return nil
}

func (gv *gpxVisitor) TrackPointClose(trackPoint *gpxreader.TrackPoint) error {
    fmt.Printf("Point: %s\n", trackPoint)

    return nil
}

//...

func main() {
    var gpxFilepath string

    o := readOptions()

    gpxFilepath = o.GpxFilepath

    f, err := os.Open(gpxFilepath)
    if err != nil {
        panic(err)
    }

    defer f.Close()

    gv := newGpxVisitor()
    gp := gpxreader.NewGpxParser(f, gv)

    err = gp.Parse()
    if err != nil {
        print("Error: %s\n", err.Error())
        os.Exit(1)
    }
}

Output:

$ gpxreadertest -f 20140909.gpx
GPX: GPX
Track: Track
Track segment: TrackSegment
Point: TrackPoint
Point: TrackPoint
Point: TrackPoint

Calculating a Hash for a Path (Recursively)

PathFingerprint allows you to recursively generate hashes for a directory structure. While doing this, it builds a catalog in a separate directory to serve as a cache. Subsequent runs of large directories will run much quicker. You can also do simple lookups against an existing catalog and generate/print a report of what has changed since the last run.

Build a test directory:

$ mkdir -p scan_path/subdir1
$ mkdir -p scan_path/subdir2
$ touch scan_path/subdir1/aa
$ touch scan_path/subdir1/bb

Calculate the hash (with reporting enabled):

$ pfhash -s scan_path -c catalog_path -R - 
create file subdir1/aa
create file subdir1/bb
create path subdir1
create path subdir2
create path .
0df9bc5a7657b7d481c219656441f10d21fd5668

Run again with a couple of changes (with reporting enabled):

$ touch scan_path/subdir1/aa
$ touch scan_path/subdir2/new_file

$ pfhash -s scan_path -c catalog_path -R - 
update file subdir1/aa
create file subdir2/new_file
update path subdir2
update path .
e700843c1b5c2f40a68098e1df96ef08b6081fe8

Lookup the hash using the lookup tool:

$ pflookup -c catalog_path
e700843c1b5c2f40a68098e1df96ef08b6081fe8

$ pflookup -c catalog_path -r subdir1
426a98d313a0a740b8445daa5102b3ed6dd7f4ed

$ pflookup -c catalog_path -r subdir1/aa
da39a3ee5e6b4b0d3255bfef95601890afd80709

Create a Service Catalog with consul.io

The requirement for service-discovery comes with the territory when you’re dealing with large farms containing multiple large components that might be themselves load-balanced clusters. Service discovery becomes a useful abstraction to map the specific designations and port numbers of the your services/load-balancers to nice, semantic names. The utility of this is being able to refer to things by semantic names instead of IP address, or even hostnames.

However, such a useful layer gets completely passed-over in medium-sized networks.

Here enters consul.io. It has a handful of useful characteristics, and it’s very easy to get started. I’ll cover the process here, and reiterate some of the examples from their homepage, below.

Overview

An instance of the Consul agent runs on every machine of the services that you want to publish. The instances of Consul form a Consul cluster. Each machine has a directory in which are stored “service definition” files for each service you wish to announce. You then hit either a REST endpoint or do a DNS query to render an IP. They’re especially proud of the DNS-compatible interface, and it provides for automatic caching.

Multiple machines can announce themselves for the same services, and they’ll all be enumerated in the result. In fact, Consul will use a load-balancing strategy similar to round-robin when it returns DNS answers.

consul.io-Specific Features

Generally speaking, service-discovery is a relatively simple concept to both understand and even implement. However, the following things are especially cool/interesting/fun:

  • You can assign a set of organizational tags to each service-definition (“laser” or “color” printer, “development” or “production” database). You can then query by either semantic name or tag.
  • It’s written in Go. That means that it’s inherently skilled at parallezing, it’s multiplatform, and you won’t have to worry about library dependencies (Go programs are statically linked).
  • It uses the RAFT consensus protocol. RAFT is the hottest thing going right now when it comes to strongly-consistent clusters (simple to understand and implement, and self-organizing).
  • You can access it via both DNS and HTTP.

Getting Started

We’re only going to do a quick reiteration of the more obvious/useful functionalities.

Configuring Agents

We’ll only be configuring one server (thus completely open to data loss).

  1. Set-up a Go build environment, and cd into $GOPATH.
  2. Clone the consul.io source:

    $ git clone git@github.com:hashicorp/consul.git src/github.com/hashicorp/consul
    Cloning into 'src/github.com/hashicorp/consul'...
    remote: Counting objects: 5701, done.
    remote: Compressing objects: 100% (1839/1839), done.
    remote: Total 5701 (delta 3989), reused 5433 (delta 3803)
    Receiving objects: 100% (5701/5701), 4.69 MiB | 1.18 MiB/s, done.
    Resolving deltas: 100% (3989/3989), done.
    Checking connectivity... done.
    
  3. Build it:

    $ cd src/github.com/hashicorp/consul
    $ make
    $ make
    --> Installing build dependencies
    github.com/armon/circbuf (download)
    github.com/armon/go-metrics (download)
    github.com/armon/gomdb (download)
    github.com/ugorji/go (download)
    github.com/hashicorp/memberlist (download)
    github.com/hashicorp/raft (download)
    github.com/hashicorp/raft-mdb (download)
    github.com/hashicorp/serf (download)
    github.com/inconshreveable/muxado (download)
    github.com/hashicorp/go-syslog (download)
    github.com/hashicorp/logutils (download)
    github.com/miekg/dns (download)
    github.com/mitchellh/cli (download)
    github.com/mitchellh/mapstructure (download)
    github.com/ryanuber/columnize (download)
    --> Running go fmt
    --> Installing dependencies to speed up builds...
    # github.com/armon/gomdb
    ../../armon/gomdb/mdb.c:8513:46: warning: data argument not used by format string [-Wformat-extra-args]
    /usr/include/secure/_stdio.h:47:56: note: expanded from macro 'sprintf'
    --> Building...
    github.com/hashicorp/consul
    
  4. Create a dummy service-definition as web.json in /etc/consul.d (the recommended path):

    {"service": {"name": "web", "tags": ["rails"], "port": 80}}
    
  5. Boot the agent and use a temporary directory for data:

    $ bin/consul agent -server -bootstrap -data-dir /tmp/consul -config-dir /etc/consul.d
    
  6. Querying Consul

    Receive a highly efficient but eventually-consistent list of agent/service nodes:

    $ bin/consul members
    dustinsilver.local  192.168.10.16:8301  alive  role=consul,dc=dc1,vsn=1,vsn_min=1,vsn_max=1,port=8300,bootstrap=1
    

    Get a complete list of current nodes:

    $ curl localhost:8500/v1/catalog/nodes
    [{"Node":"dustinsilver.local","Address":"192.168.10.16"}]
    

    Verify membership:

    $ dig @127.0.0.1 -p 8600 dustinsilver.local.node.consul
    
    ; <> DiG 9.8.3-P1 <> @127.0.0.1 -p 8600 dustinsilver.local.node.consul
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46780
    ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
    ;; WARNING: recursion requested but not available
    
    ;; QUESTION SECTION:
    ;dustinsilver.local.node.consul.	IN	A
    
    ;; ANSWER SECTION:
    dustinsilver.local.node.consul.	0 IN	A	192.168.10.16
    
    ;; Query time: 0 msec
    ;; SERVER: 127.0.0.1#8600(127.0.0.1)
    ;; WHEN: Sun May 25 03:17:49 2014
    ;; MSG SIZE  rcvd: 94
    

    Pull an IP for the service named “web” using DNS (they’ll always have the “service.consul” suffix):

    $ dig @127.0.0.1 -p 8600 web.service.consul
    
    ; <> DiG 9.8.3-P1 <> @127.0.0.1 -p 8600 web.service.consul
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42343
    ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
    ;; WARNING: recursion requested but not available
    
    ;; QUESTION SECTION:
    ;web.service.consul.		IN	A
    
    ;; ANSWER SECTION:
    web.service.consul.	0	IN	A	192.168.10.16
    
    ;; Query time: 1 msec
    ;; SERVER: 127.0.0.1#8600(127.0.0.1)
    ;; WHEN: Sun May 25 03:20:33 2014
    ;; MSG SIZE  rcvd: 70
    

    or, HTTP:

    $ curl http://localhost:8500/v1/catalog/service/web
    [{"Node":"dustinsilver.local","Address":"192.168.10.16","ServiceID":"web","ServiceName":"web","ServiceTags":["rails"],"ServicePort":80}]
    

    Get the port-number, too, using DNS:

    $ dig @127.0.0.1 -p 8600 web.service.consul SRV
    
    ; <> DiG 9.8.3-P1 <> @127.0.0.1 -p 8600 web.service.consul SRV
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44722
    ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
    ;; WARNING: recursion requested but not available
    
    ;; QUESTION SECTION:
    ;web.service.consul.		IN	SRV
    
    ;; ANSWER SECTION:
    web.service.consul.	0	IN	SRV	1 1 80 dustinsilver.local.node.dc1.consul.
    
    ;; ADDITIONAL SECTION:
    dustinsilver.local.node.dc1.consul. 0 IN A	192.168.10.16
    
    ;; Query time: 0 msec
    ;; SERVER: 127.0.0.1#8600(127.0.0.1)
    ;; WHEN: Sun May 25 03:21:00 2014
    ;; MSG SIZE  rcvd: 158
    

    Search by tag:

    $ dig @127.0.0.1 -p 8600 rails.web.service.consul
    
    ; <> DiG 9.8.3-P1 <> @127.0.0.1 -p 8600 rails.web.service.consul
    ; (1 server found)
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16867
    ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
    ;; WARNING: recursion requested but not available
    
    ;; QUESTION SECTION:
    ;rails.web.service.consul.	IN	A
    
    ;; ANSWER SECTION:
    rails.web.service.consul. 0	IN	A	192.168.10.16
    
    ;; Query time: 0 msec
    ;; SERVER: 127.0.0.1#8600(127.0.0.1)
    ;; WHEN: Sun May 25 03:21:26 2014
    ;; MSG SIZE  rcvd: 82