Go: Encoding Maps with Non-String Keys to JSON

I ran into some issues with 1.6.2 encoding a map to a JSON structure because it had int/int64 keys. Sure, JSON prescribes string-only keys, but I incorrectly made the reasonable assumption that Go would coerce these to strings. It turns out that this change was made in the very recent past and that, as of the last release (1.7.1) (and probably earlier ones) this should no longer be a problem.

Unfortunately, AppEngine still runs on 1.6.2, for now. So, if you had the same problem, you will have to go the conventional routeĀ and translate these keys to strings, yourself, prior to marshaling.

Convert Syslog Events to a JSON Stream

The syslog-ng project serves as a general replacement for rsyslog (the default syslog daemon incarnation on Ubuntu and other distributions). It allows you to simply defining syslog sources, defining filters, defining destinations, and mapping them together. It also provides the ability to apply a pattern-tree to messages for classification (see “Processing message content with a pattern database” in the “Administrator Guide” PDF, below), as well as translating log messages into different formats.

It’s the latter that we’re concerned with. We can take Syslog output and use the JSON template-plugin to send JSON into a pipe, network destination, etc..

For this example, we’ll simply translate the system syslog events into JSON.


  1. Install packages:
    $ sudo apt-get install syslog-ng-core
    $ sudo apt-get install syslog-ng-mod-json
  2. Modify /etc/syslog-ng/syslog-ng.conf:
    destination d_json {
       file("/var/log/messages.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
    log {
       source(s_src); destination(d_json);
  3. Restart the service:
    $ sudo service syslog-ng restart

Now, you’ll see a /var/log/messages.json. Mine shows the following:

{"TAGS":".source.s_src","SOURCEIP":"","PROGRAM":"sudo","PRIORITY":"notice","MESSAGE":"  dustin : TTY=pts/23 ; PWD=/home/dustin ; USER=root ; COMMAND=/usr/sbin/service syslog-ng restart","LEGACY_MSGHDR":"sudo: ","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"authpriv","DATE":"Sep 16 04:51:41"}
{"TAGS":".source.s_src","SOURCEIP":"","PROGRAM":"sudo","PRIORITY":"info","MESSAGE":"pam_unix(sudo:session): session opened for user root by dustin(uid=0)","LEGACY_MSGHDR":"sudo: ","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"authpriv","DATE":"Sep 16 04:51:41"}
{"TAGS":".source.s_src","SOURCEIP":"","SOURCE":"s_src","PROGRAM":"syslog-ng","PRIORITY":"notice","PID":"15800","MESSAGE":"syslog-ng shutting down; version='3.5.3'","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"syslog","DATE":"Sep 16 04:51:41"}
{"TAGS":".source.s_src","SOURCEIP":"","SOURCE":"s_src","PROGRAM":"syslog-ng","PRIORITY":"notice","PID":"15889","MESSAGE":"syslog-ng starting up; version='3.5.3'","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"syslog","DATE":"Sep 16 04:51:41"}
{"TAGS":".source.s_src","SOURCEIP":"","SOURCE":"s_src","PROGRAM":"syslog-ng","PRIORITY":"notice","PID":"15889","MESSAGE":"EOF on control channel, closing connection;","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"syslog","DATE":"Sep 16 04:51:41"}
{"TAGS":".source.s_src","SOURCEIP":"","PROGRAM":"sudo","PRIORITY":"info","MESSAGE":"pam_unix(sudo:session): session closed for user root","LEGACY_MSGHDR":"sudo: ","HOST_FROM":"dustinhub","HOST":"dustinhub","FACILITY":"authpriv","DATE":"Sep 16 04:51:41"}


This all enables you to build your own filter in your favorite programming language by using a socket-server and a set of rules. You don’t have to be concerned with parsing the Syslog protocol or the semantics of file-parsing and message formats, and you can avoid the age-old paradigm of parsing log-files, after the fact, by chunks of time, and start to process them in real-time.


The syslog-ng website has some “Administrator Guide” PDFs, but the site has very little other usefulness, and, though everyone loves syslog-ng, there is little more than configurations snippets in forum posts. However, those PDFs are thorough, and the configuration file is easy to understand (essentially different incarnations of the commands above).

Inspecting JSON at the Command-Line

This is a simple tool to pull specific values out of JSON, or to pull JSON from JSON, at the command-line:


It’s useful to pull configuration values from within a Bash script.

Example data:

{"a": [9, 6, {"b": [99, 88, 77, "text", 55]}]}

Example commands:

$ cat example.json | jp a.2.b.3

$ cat example.json | jp a.2 | jp b.3

$ cat example.json | jp a.2 | jp -p b.3