Retrieving Your Shopify Store’s GraphQL Schema

Using Python and SGQLC with Shopify’s latest API (2025-01):

$ python3 -m sgqlc.introspection https://<subdomain>.myshopify.com/admin/api/2025-01/graphql.json -H "X-Shopify-Access-Token: <secret>"

{
  "data": {
    "__schema": {
      "directives": [
        {
          "args": [
            {
              "defaultValue": "null",
...

Incidentally, Shopify’s Python client establishes the schema’s URL here:

https://github.com/Shopify/shopify_python_api/blob/f58c9910e5699fbce950766772c6ac30634f85ea/shopify/resources/graphql.py#L9

Register New Executable Formats In The Linux Kernel

If the kernel identifies that a file you’re trying to execute has a known sequence of magic bytes, it will happily execute it. However, you might want to run certain formats without needing to pass them to another tool. In as much as these are usually binary tools, you can prepend a magic-bytes preamble (or use an existing sequence, if your binary format already declares one) and then associate that with a certain loader in the running kernel using binfmt_misc. Access is usually via procfs (currently /proc/sys/fs/binfmt_misc/register). Cross-execution with a virtualized process is a common use-case for this.

Note that this doesn’t really apply to text files (it’s all data) because you’d need to inject gibberish at the top of your file to use this functionality when using a shebang (which is meant for this case) should work just fine.

This is a walkthrough of how to do this for compiled Python or LUA source-code: https://twdev.blog/2024/01/docker_multi_platform

Since both of the example compilation formats include magic bytes, you can just write a tool to acquire those from libraries (at least with Python) and call binfmt_misc in one step.

You can also write configs to be automatically loaded at boot. There appears to be no way to list registered formats, with the only recourse being to sift the config paths (probably empty by default) or to check your kernel config to see which supported formats were included in that build.

Check YAML Dictionary Key Uniqueness with PyYAML

If you are dealing with very large YAML files that are curated over time by hand, it is not inconceivable that someone will inadvertently introduce a duplicate. The problem is that, as PyYAML is just dutifully enumerating the nodes and loading a dictionary and has no requirements, knowledge, or authority to do anything else, you will already have lost the duplicates by the time you’ve received the dictionary back.

This implementation is only concerned with dictionaries under duplicate keys, and not integers, strings, lists, etc.. under duplicate keys. Note the corresponding comment. It was unnecessary in my situation and something you’ll have to account for if/when modifying this routine for your purposes.

The following code overrides the YAML loader and the map construction to do this. This source-code is also available as a gist.

import yaml

def load_and_assert_uniqueness(x):

    # We'd like to detect duplicates. Since PyYAML both loads things depth-first
    # *and* doesn't give us the parent when processing a child node, we'll index
    # of all of the object IDs as we're constructing them, and then see which
    # are disappeared from the final hierarchy. Since all we can do is pass a
    # class, we need to inline the class in order to load into an index within
    # our scope.
    #
    # We're only concerned about dictionary keys with dictionary values because
    # a) this is our use-case, and b) we can stash additional information as
    # dictionary keys without having to override any types.

    nodes_by_id = {}


    class _UniqueCheckedLoader(yaml.SafeLoader):

        def construct_yaml_map(self, node):
            data = {}

            id_ = id(data)
            data['_id'] = id_
            nodes_by_id[id_] = data

            yield data

            value = self.construct_mapping(node)
            data.update(value)


    _UniqueCheckedLoader.add_constructor(
        'tag:yaml.org,2002:map',
        _UniqueCheckedLoader.construct_yaml_map
    )


    # Load

    blob = yaml.load(x, Loader=_UniqueCheckedLoader)


    # Remove all nodes in the final dictionary from the by-ID index

    q = [blob]
    while q:
        d, q = q[0], q[1:]

        id_ = d.pop('_id')
        del nodes_by_id[id_]

        for v in d.values():

            # We're only concern with dictionary nodes
            if v.__class__ is not dict:
                continue

            q.append(v)


    # We've visited all referencesd nodes. Everything still indexed must've been
    # pruned due to nonuniqueness. As mentioned above, we really don't have any
    # hierarchical context, by we can just search out occurrences of the
    # attributes from the node(s) in the data in order to find the duplicates.

    if nodes_by_id:

        # Cleanup representation before displaying

        nodes = []
        for node in nodes_by_id.values():
            del node['_id']
            nodes.append(node)

        # Error out

        raise \
            Exception(
                "({}) nodes were duplicates:\n{}".format(
                    len(nodes), nodes))


    return blob

Embedding Python in PostgreSQL Functions

Example:

CREATE OR REPLACE FUNCTION url_quote (url text)
RETURNS TEXT
AS $$
    from urllib.parse import quote
    return quote(url)

$$
LANGUAGE 'plpython3u';

SELECT url_quote('https://www.postgresql.org/docs/12/plpython-data.html#id-1.8.11.11.3');

Getting Started with Postgres Functions in PL/Python

How to Render Django Templates Without Loading Django (or Its Configuration System)

#!/usr/bin/env python3

import django.template
import django.template.engine

def _main():
    e = django.template.engine.Engine()

    body = """\
aa {{ test_token }} cc
"""

    t = django.template.Template(body, engine=e)

    context = {
        'test_token': 'bb',
    }

    c = django.template.Context(context)
    r = t.render(c)

    print(r)


_main()

Output:

$ ./test_render.py 
aa bb cc

Python: Substitute Values Into YAML During Load

There’s definitely a case to be made for automatically and efficiently applying environment variables or another set of replacements into a YAML-based config or set of instructions on load. This example uses PyYAML. We’ll use Python’s built-in string templating to replace tokens like “$NAME” with values from a dictionary. It will fail, as it should, if the name is not in the given dictionary.

import string
import yaml

def load_yaml(f, context):

    def string_constructor(loader, node):

        t = string.Template(node.value)
        value = t.substitute(context)

        return value


    l = yaml.SafeLoader
    l.add_constructor('tag:yaml.org,2002:str', string_constructor)

    token_re = string.Template.pattern
    l.add_implicit_resolver('tag:yaml.org,2002:str', token_re, None)

    x = yaml.load(f, Loader=l)
    return x

y = """\
aa: bb
cc: dd $EE ff
"""

context = {
    'EE': '123',
}

d = waw.utility.load_yaml(y, context)
print(d)

Output:

{'aa': 'bb', 'cc': 'dd 123 ff'}

Python: Command-Line Completion for argparse

argcomplete provides very useful functionality that you will basically get for free with just a couple of steps.

Implementation

Put some markup below the shebang of your frontend script in the form of a comment:

#!/usr/bin/env python
# PYTHON_ARGCOMPLETE_OK

The BASH-completion script argcomplete will basically identify and scan any script with a Python shebang that is used with BASH-completion. This entails actually running the script. In order to minimize how much time is spent loading scripts that don’t actually use argcomplete, the completion script will ignore anything that does not have this comment directly following the shebang.

Next, add and import for the argcomplete package and run argcomplete.autocomplete(parser) after you have configured your command-line parameters but before your call to parser.parse_args() (where parser is an instance of argparse.ArgumentParser). This function will produce command-line configuration metadata and then terminate.

That is it. Note that it is not practical to assume that everyone who uses your script will have argcomplete installed. They may not be using BASH (BASH is the only well-supported shell at this time), they may not be using a supported OS, and/or any commercial environments that adopt your tools may be server environments that have no use for command-line completion and refuse to support it. Therefore, you should wrap the import with a try-except for ImportError and then only call argcomplete.autocomplete if you were able to import the package.

Installation

To install autocomplete, the simplest route is to merely do a “sudo pip install argcomplete” and then call “activate-global-python-argcomplete” (this is a physical script likely installed to /usr/local/bin. This only has to be done once and will install a non-project-specific script that will work for any script that is equipped to use argcomplete. For other configuration and semantics, see the project page.

Custom String Template Format in Python

This might be necessary if you, for example, want to apply your own set of replacements to a string argument that will be passed to you by another mechanism that applies its own set of replacements.

This example supposes that you might want to use square-brackets instead of the standard curly-brackets.

import string
import re

_FIELD_RE = re.compile(r'\[([a-zA-Z0-9_]+)\]')

class CustomReplacer(string.Formatter):
    def parse(self, s):
        last_stop_index = None
        for m in _FIELD_RE.finditer(s):
            token_name = m.group(1)

            start_index, stop_index = m.span()

            if start_index == 0:
                prefix_fragment = ''
            elif last_stop_index is None:
                prefix_fragment = s[:start_index]
            else:
                prefix_fragment = s[last_stop_index:start_index]

            last_stop_index = stop_index

            yield prefix_fragment, token_name, '', None

cr = CustomReplacer()
template = 'aa [name]      bb [name2] cc dd [name3]'

replacements = {
    'name': 'howard',
    'name2': 'mark',
    'name3': 'james',
}

output = cr.format(template, **replacements)
print(output)

Output:

aa howard      bb mark cc dd james

Note that no formatting is supported with our custom replacer (though it could be added, with more work). If any formatting specifiers are provided, they will fail the regular-expression match and be ignored.

Measure Internet Speed from CLI

Use speedtest-cli:

$ curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Comcast Cable (73.1.128.16)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Broadwave (Fort Lauderdale, FL) [43.78 km]: 22.155 ms
Testing download speed................................................................................
Download: 232.72 Mbit/s
Testing upload speed......................................................................................................
Upload: 10.07 Mbit/s

Python: Parsing XML and Retaining the Comments

By default, Python’s built-in ElementTree module strips comments as it reads them. The solution is just obscure enough to be hard to find.

import xml.etree.ElementTree as ET

class _CommentedTreeBuilder(ET.TreeBuilder):
    def comment(self, data):
        self.start('!comment', {})
        self.data(data)
        self.end('!comment')

def parse(filepath):
    ctb = _CommentedTreeBuilder()
    xp = ET.XMLParser(target=ctb)
    tree = ET.parse(filepath, parser=xp)

    root = tree.getroot()
    # ...

When enumerating the parsed nodes, the comments will have a tag-name of “!comment”.