Go: An In-Memory ReadWriteSeeker

The SeekableBuffer type is similar to a bytes.Buffer in that you can perform both reads and writes against it, but has the following two main differences:

  1. You can seek on it.
  2. After writing to it, the current position will be on the byte following whatever you wrote. This is the typical behavior for a file resource on almost any platform but not for a bytes.Buffer.

Eccentric usage of seek, read, and write behavior, such as the following, will work as expected:

  • seeking but not writing
  • seeking past the end of the file and writing
  • seeking past the end of the file and reading
  • writing N+M bytes when positioned only N bytes from the end of the file

Usage (from the unit-tests):

sb := NewSeekableBuffer()

// Write first string.

data := []byte("word1-word2")

_, err := sb.Write(data)
log.PanicIf(err)

// Seek and replace partway through, and replace more data than we
// currently have.

_, err = sb.Seek(6, os.SEEK_SET)
log.PanicIf(err)

data2 := []byte("word3-word4")

_, err = sb.Write(data2)
log.PanicIf(err)

// Read contents.

_, err = sb.Seek(0, os.SEEK_SET)
log.PanicIf(err)

buffer := make([]byte, 20)

_, err = sb.Read(buffer)
log.PanicIf(err)

// `buffer` currently has "word1-word3-word4".
Advertisements

Image Processing: Identifying Groups of Bracketed Images

A new project to recursively scan through a tree of images and determine which were captured as part of an auto-bracketing image capture operation.

bracketed_image_finder

Example:

$ bif_find tests/assets/images --json
[
    {
        "entries": [
            {
                "exposure_value": 0.0,
                "rel_filepath": "DSC08196.JPG",
                "timestamp": "2019-02-13T00:31:50"
            },
            {
                "exposure_value": -0.7,
                "rel_filepath": "DSC08197.JPG",
                "timestamp": "2019-02-13T00:31:50"
            },
            {
                "exposure_value": 0.7,
                "rel_filepath": "DSC08198.JPG",
                "timestamp": "2019-02-13T00:31:50"
            },
            {
                "exposure_value": -1.3,
                "rel_filepath": "DSC08199.JPG",
                "timestamp": "2019-02-13T00:31:50"
            },
            {
                "exposure_value": 1.3,
                "rel_filepath": "DSC08200.JPG",
                "timestamp": "2019-02-13T00:31:51"
            }
        ],
        "type": "periodic"
    },
    {
        "entries": [
            {
                "exposure_value": 0.0,
                "rel_filepath": "DSC08201.JPG",
                "timestamp": "2019-02-13T00:32:09"
            },
            {
                "exposure_value": -0.7,
                "rel_filepath": "DSC08202.JPG",
                "timestamp": "2019-02-13T00:32:09"
            },
            {
                "exposure_value": 0.7,
                "rel_filepath": "DSC08203.JPG",
                "timestamp": "2019-02-13T00:32:10"
            },
            {
                "exposure_value": -1.3,
                "rel_filepath": "DSC08204.JPG",
                "timestamp": "2019-02-13T00:32:10"
            },
            {
                "exposure_value": 1.3,
                "rel_filepath": "DSC08205.JPG",
                "timestamp": "2019-02-13T00:32:10"
            }
        ],
        "type": "periodic"
    }
]

Python: Parsing XML and Retaining the Comments

By default, Python’s built-in ElementTree module strips comments as it reads them. The solution is just obscure enough to be hard to find.

import xml.etree.ElementTree as ET

class _CommentedTreeBuilder(ET.TreeBuilder):
    def comment(self, data):
        self.start('!comment', {})
        self.data(data)
        self.end('!comment')

def parse(filepath):
    ctb = _CommentedTreeBuilder()
    xp = ET.XMLParser(target=ctb)
    tree = ET.parse(filepath, parser=xp)

    root = tree.getroot()
    # ...

When enumerating the parsed nodes, the comments will have a tag-name of “!comment”.

ssl: Promoting Existing Client Socket to SSL in C/C++

You may be in a situation where something else produces the sockets for you (such as an event-loop) or you otherwise need to manage the socket rather then allowing something else to.

#include <stdio.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <openssl/ssl.h>

int main(int argc, char *argv[])
{
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    if (!sockfd) {
        printf("Error creating socket.\n");
        return -1;
    }

    struct sockaddr_in sa;
    memset (&sa, 0, sizeof(sa));

    sa.sin_family = AF_INET;
    sa.sin_addr.s_addr = inet_addr("172.217.2.196");
    sa.sin_port = htons (443); 

    socklen_t socklen = sizeof(sa);
    if (connect(sockfd, (struct sockaddr *)&sa, socklen)) {
        printf("Error connecting to server.\n");
        return -1;
    }

    SSL_library_init();
    SSLeay_add_ssl_algorithms();
    SSL_load_error_strings();

    const SSL_METHOD *meth = TLSv1_2_client_method();
    SSL_CTX *ctx = SSL_CTX_new (meth);

    SSL *ssl = SSL_new (ctx);
    if (ssl == NULL) {
        printf("Could not create SSL context.\n");
        return -1;
    }

    SSL_set_fd(ssl, sockfd);

    int err = SSL_connect(ssl);
    if (err <= 0) {
        printf("Could not connect.\n");
        return -1;
    }

    printf ("SSL connection using %s\n", SSL_get_cipher (ssl));

    // Do send/receive here.

    return 0;
}

Adapted from openssl-in-c-socket-connection-https-client, and works with both OpenSSL and BoringSSL.

Go: Parsing Time Expressions

go-time-parse will parse time expressions into time.Duration quantities. From the example:

actualDuration, phraseType, err := ParseDuration("24 days from now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour/24, phraseType)

actualDuration, phraseType, err = ParseDuration("now")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration, phraseType)

actualDuration, phraseType, err = ParseDuration("12m")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Minute, phraseType)

actualDuration, phraseType, err = ParseDuration("every 6 hours")
log.PanicIf(err)

fmt.Printf("%d [%s]\n", actualDuration/time.Hour, phraseType)

Output:

24 [time]
0 [time]
12 [interval]
6 [interval]

Identifying Nearest Major Cities

go-geographic-attractor is a new project that indexes world city and population data and can match a given coordinate to either the nearest major city or the nearest city (if no major city is near) in near-instantaneous time.

From the example:

// Load countries.

countryDataFilepath := path.Join(appPath, "test", "asset", "countryInfo.txt")

f, err := os.Open(countryDataFilepath)
log.PanicIf(err)

defer f.Close()

countries, err := geoattractorparse.BuildGeonamesCountryMapping(f)
log.PanicIf(err)

// Load cities.

gp := geoattractorparse.NewGeonamesParser(countries)

cityDataFilepath := path.Join(appPath, "index", "test", "asset", "allCountries.txt.detroit_area_handpicked")
g, err := os.Open(cityDataFilepath)
log.PanicIf(err)

defer g.Close()

ci := NewCityIndex()

err = ci.Load(gp, g)
log.PanicIf(err)

// Do the query.

clawsonCoordinates := []float64{42.53667, -83.15041}

sourceName, visits, cr, err := ci.Nearest(clawsonCoordinates[0], clawsonCoordinates[1])
log.PanicIf(err)

// Print the results.

for _, vhi := range visits {
    fmt.Printf("%s: %s\n", vhi.Token, vhi.City)
}

fmt.Printf("\n")

fmt.Printf("Source: %s\n", sourceName)
fmt.Printf("ID: %s\n", cr.Id)
fmt.Printf("Country: %s\n", cr.Country)
fmt.Printf("City: %s\n", cr.City)
fmt.Printf("Population: %d\n", cr.Population)
fmt.Printf("Latitude: %.10f\n", cr.Latitude)
fmt.Printf("Longitude: %.10f\n", cr.Longitude)