Hint/Trick

Python: Substitute Values Into YAML During Load

There’s definitely a case to be made for automatically and efficiently applying environment variables or another set of replacements into a YAML-based config or set of instructions on load. This example uses PyYAML. We’ll use Python’s built-in string templating to replace tokens like “$NAME” with values from a dictionary. It will fail, as it should, if the name is not in the given dictionary.

import string
import yaml

def load_yaml(f, context):

    def string_constructor(loader, node):

        t = string.Template(node.value)
        value = t.substitute(context)

        return value


    l = yaml.SafeLoader
    l.add_constructor('tag:yaml.org,2002:str', string_constructor)

    token_re = string.Template.pattern
    l.add_implicit_resolver('tag:yaml.org,2002:str', token_re, None)

    x = yaml.load(f, Loader=l)
    return x

y = """\
aa: bb
cc: dd $EE ff
"""

context = {
    'EE': '123',
}

d = waw.utility.load_yaml(y, context)
print(d)

Output:

{'aa': 'bb', 'cc': 'dd 123 ff'}

Installing VMWare Remote Console (VMRC) On Arch/Manjaro

When you download the Remote Console bundle installer from VMWare and run it, the UI will start and simply fail with “Installation was unsuccessful” error. There is no console output, seems to be no log, and no apparent option to enable verbosity. Since I’m using Manjaro, I’d rather not use the package, which is only available in AUR.

A successful install is not terribly difficult. You can start the install using a command-line installer rather than the GUI:

sudo ./VMware-Remote-Console-11.2.0-16492666.x86_64.bundle --console

There will be some prompting that isn’t present in the GUI. When it asks you:

System service scripts directory (commonly /etc/init.d).:

…enter an alternative directory. That’s it. In my case, it didn’t even have to exist.

The install will take a minute expanding things, and then deposit the “vmrc” executable within the executable search-path. The browser should now be able to find it when the remote console wants to open the viewer.

Go: Build-Time Variables

Go allows you to override global variables at build time during the linker stage:

$ go build -o /tmp/prog-custom -ldflags "-X main.overrideableValuePhrase=123456" ./cmd/prog

Requirements:

  • It must be a global variable and not a constant.
  • It might be in the package that you are telling it to build. This definitely works for executables and, with the removal of support for binary-only packages (BOPs), it probably doesn’t apply whatsoever to intermediate packages.
  • It must be a string (so, process it from the init() function).

It doesn’t matter whether it is an exported or unexported symbol.

 

Go: Write RPC-Connected Plugins

Plugins are Go’s system for developing shared-libraries. However, this is backed by a general system that can also have alternative implementations. In this case, you can write a plugin in Go that runs from one system and load that plugin in Go, on the fly, from a million other systems. Courtesy of Hashicorp.

https://github.com/hashicorp/go-plugin

 

 

Python: Command-Line Completion for argparse

argcomplete provides very useful functionality that you will basically get for free with just a couple of steps.

Implementation

Put some markup below the shebang of your frontend script in the form of a comment:

#!/usr/bin/env python
# PYTHON_ARGCOMPLETE_OK

The BASH-completion script argcomplete will basically identify and scan any script with a Python shebang that is used with BASH-completion. This entails actually running the script. In order to minimize how much time is spent loading scripts that don’t actually use argcomplete, the completion script will ignore anything that does not have this comment directly following the shebang.

Next, add and import for the argcomplete package and run argcomplete.autocomplete(parser) after you have configured your command-line parameters but before your call to parser.parse_args() (where parser is an instance of argparse.ArgumentParser). This function will produce command-line configuration metadata and then terminate.

That is it. Note that it is not practical to assume that everyone who uses your script will have argcomplete installed. They may not be using BASH (BASH is the only well-supported shell at this time), they may not be using a supported OS, and/or any commercial environments that adopt your tools may be server environments that have no use for command-line completion and refuse to support it. Therefore, you should wrap the import with a try-except for ImportError and then only call argcomplete.autocomplete if you were able to import the package.

Installation

To install autocomplete, the simplest route is to merely do a “sudo pip install argcomplete” and then call “activate-global-python-argcomplete” (this is a physical script likely installed to /usr/local/bin. This only has to be done once and will install a non-project-specific script that will work for any script that is equipped to use argcomplete. For other configuration and semantics, see the project page.

Custom String Template Format in Python

This might be necessary if you, for example, want to apply your own set of replacements to a string argument that will be passed to you by another mechanism that applies its own set of replacements.

This example supposes that you might want to use square-brackets instead of the standard curly-brackets.

import string
import re

_FIELD_RE = re.compile(r'\[([a-zA-Z0-9_]+)\]')

class CustomReplacer(string.Formatter):
    def parse(self, s):
        last_stop_index = None
        for m in _FIELD_RE.finditer(s):
            token_name = m.group(1)

            start_index, stop_index = m.span()

            if start_index == 0:
                prefix_fragment = ''
            elif last_stop_index is None:
                prefix_fragment = s[:start_index]
            else:
                prefix_fragment = s[last_stop_index:start_index]

            last_stop_index = stop_index

            yield prefix_fragment, token_name, '', None

cr = CustomReplacer()
template = 'aa [name]      bb [name2] cc dd [name3]'

replacements = {
    'name': 'howard',
    'name2': 'mark',
    'name3': 'james',
}

output = cr.format(template, **replacements)
print(output)

Output:

aa howard      bb mark cc dd james

Note that no formatting is supported with our custom replacer (though it could be added, with more work). If any formatting specifiers are provided, they will fail the regular-expression match and be ignored.

Measure Internet Speed from CLI

Use speedtest-cli:

$ curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Comcast Cable (73.1.128.16)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Broadwave (Fort Lauderdale, FL) [43.78 km]: 22.155 ms
Testing download speed................................................................................
Download: 232.72 Mbit/s
Testing upload speed......................................................................................................
Upload: 10.07 Mbit/s