Use ADB to Connect to Your Android Device From a Docker Container

You may have a use-case where you want to write software to manipulate an Android device using a system or set of tools that are not natively available from your current system. However, you might be able to expose this as a Docker image. For example, your device is (or will be) connected to a Windows machine and you really want to or need to use Linux tools.

No problem. ADB implicitly uses a client-server model: The ADB tool (on your system) connects to the ADB server (runs in the background on your system) which interacts with the ADB daemon (runs on your device). This means that we can forward requests from ADB on the command-line in the guest container in Docker to the ADB server on the host system.

The ADB client and server have to be at the same version, or the client will indiscriminately kill/restart your ADB server. So, as I am currently running Ubuntu 14.04 on my host system, I will do the same in Docker.

First, I will make sure the ADB server is running on my host system. Most of the subcommands that will automatically start the local server, but I will start it directly:

$ adb start-server
* daemon not running. starting it now on port 5037 *
* daemon started successfully *

Now, I will start a container in Docker with Ubuntu 14.04 and automatically install ADB before dropping to a prompt. Note that we are passing “–network=host” in order to share the host’s network identity:

$ docker run -i -t --network=host ubuntu:14.04 /bin/bash -c "sudo apt-get update && sudo apt-get install -y android-tools-adb && /bin/bash"

Eventually, you will end-up at the prompt. Just do something simple like enumerating the devices:

root@mlll2664:/# adb devices
List of devices attached 
05157df572841820 device

The “mlll2664” hostname, represented in the prompt in the Docker container, is, actually, the same hostname as my host system.

So, there you go. Not too painful.

 

Advertisements

C++: Embedding the V8 JavaScript Engine

V8 is Chromium’s JavaScript interpreter. Not only can you use the included D8 utility to open a console and execute JavaScript code directly, but it is not as tough as you would think to integrate your native functions from your JavaScript and your JavaScript functions from your native functions. Of course, you have to get your head around all of the layers of isolates, context, and scoping.

For a walkthrough of embedding V8, start here.

In order to build a quick example of how to implement a native function for consumption from your JavaScript, go ahead and build V8. You will have to choose how to build the project. You’ll provide the name of a particular build configuration, which will write a few build parameters (called “args”) to the “out.gn//args.gn” file. This is a GN thing. GN is a build tool from the Chromium depot-tools project that generates Ninja scripts for doing the actual build.

For a list of build configurations, run:

$ tools/dev/v8gen.py list
...
s390.optdebug
s390.optdebug.sim
s390.release
s390.release.sim
s390x.debug
s390x.debug.sim
s390x.optdebug
s390x.optdebug.sim
s390x.release
s390x.release.sim
x64.debug
x64.optdebug
x64.release
x64.release.sample

(list shortened for simplicity)

We’ve used the “x64.release.sample” configuration for our example. It is straightforward for the purpose of this example. Once you have V8 downloaded and the dependencies installed, the actual project builds in about fifteen-minutes (with sixteen threads).

Drop this example into a file named “native_function.cpp” (or update the build command below to whatever you go with). This is an amalgam of excerpts from the samples in the project, examples in the documentation, and some customization on our part.

#include <sstream>

#include "libplatform/libplatform.h"
#include "v8.h"

// This is the function that we'll register into JS as a global function.
static void TestCallback(const v8::FunctionCallbackInfo<v8::Value>& args) {
  v8::Isolate* isolate = args.GetIsolate();
  v8::HandleScope scope(isolate);

  // Return [the arbitrary integer] 55 as our result.
  auto result = v8::Integer::New(isolate, 55);

  // Set the result.
  args.GetReturnValue().Set(result);
}

int main(int argc, char* argv[]) {

  // Initialize V8.
  v8::V8::InitializeICUDefaultLocation(argv[0]);
  v8::V8::InitializeExternalStartupData(argv[0]);
  std::unique_ptr<v8::Platform> platform = v8::platform::NewDefaultPlatform();
  v8::V8::InitializePlatform(platform.get());
  v8::V8::Initialize();

  // Create a new Isolate and make it the current one.
  v8::Isolate::CreateParams create_params;
  create_params.array_buffer_allocator =
      v8::ArrayBuffer::Allocator::NewDefaultAllocator();

  v8::Isolate* isolate = v8::Isolate::New(create_params);

  // Scoping in the C++ is tightly related to scoping in the JS. So, we'll 
  // retain the organizational blocks from the samples.
  {
    v8::Isolate::Scope isolate_scope(isolate);

    // Create a stack-allocated handle scope.
    v8::HandleScope handle_scope(isolate);

    // Create a global object to install our function into.
    v8::Local<v8::ObjectTemplate> global = v8::ObjectTemplate::New(isolate);

    // Register our global function.
    global->Set(
      v8::String::NewFromUtf8(isolate, "testCall", v8::NewStringType::kNormal).ToLocalChecked(),
      v8::FunctionTemplate::New(isolate, TestCallback)
    );

    // Create a new context and apply the global object.
    v8::Local<v8::Context> context = v8::Context::New(isolate, NULL, global);

    // Enter the context for compiling and running the hello world script.
    v8::Context::Scope context_scope(context);

    // Load the script.

    std::stringstream sourceStream;

    sourceStream
      << "testCall();" << std::endl;

    v8::Local<v8::String> source =
      v8::String::NewFromUtf8(
          isolate, 
          sourceStream.str().c_str(), 
          v8::NewStringType::kNormal)
        .ToLocalChecked();

    // Compile the source code.
    v8::Local<v8::Script> script =
      v8::Script::Compile(context, source)
        .ToLocalChecked();

    // Run the script.
    v8::Local<v8::Value> result = script->Run(context).ToLocalChecked();

    // Print the screen output.
    v8::String::Utf8Value output(isolate, result);
    printf("%s\n", *output);
  }

  // Dispose the isolate and tear down V8.
  isolate->Dispose();
  v8::V8::Dispose();
  v8::V8::ShutdownPlatform();
  delete create_params.array_buffer_allocator;

  return 0;
}

To build, set V8PATH to the absolute path of your “v8/v8” directory (the main V8 path established in the getting-started document above), and run the following:

$ g++ "-I${V8PATH}/include" -o native_function native_function.cpp -lv8_monolith "-L${V8PATH}/out.gn/x64.release.sample/obj" -pthread -std=c++0x

The example implements a simple, native function that just returns (55), registers it as a global JS function, creates a simple JS script that calls it, and then runs the script and prints the screen output. The screen output is just the result of that function printed to the screen since it was not otherwise assigned to a variable:

$ ./native_function 
55

For a couple of more examples, see the project repository.

Go: Read and Browse Ext4 Filesystems in User-Space

go-ext4 is a pure Go implementation of an Ext4 reader with support for reading the journal. An example of how to walk the file-structure:

inodeNumber := InodeRootDirectory

filepath := path.Join(assetsPath, "hierarchy_32.ext4")

f, err := os.Open(filepath)
log.PanicIf(err)

defer f.Close()

_, err = f.Seek(Superblock0Offset, io.SeekStart)
log.PanicIf(err)

sb, err := NewSuperblockWithReader(f)
log.PanicIf(err)

bgdl, err := NewBlockGroupDescriptorListWithReadSeeker(f, sb)
log.PanicIf(err)

bgd, err := bgdl.GetWithAbsoluteInode(inodeNumber)
log.PanicIf(err)

dw, err := NewDirectoryWalk(f, bgd, inodeNumber)
log.PanicIf(err)

allEntries := make([]string, 0)

for {
    fullPath, de, err := dw.Next()
    if err == io.EOF {
        break
    } else if err != nil {
        log.Panic(err)
    }

    description := fmt.Sprintf("%s: %s", fullPath, de.String())
    allEntries = append(allEntries, description)
}

sort.Strings(allEntries)

for _, entryDescription := range allEntries {
    fmt.Println(entryDescription)
}

// Output:
//
// directory1/fortune1: DirectoryEntry
// directory1/fortune2: DirectoryEntry
// directory1/fortune5: DirectoryEntry
// directory1/fortune6: DirectoryEntry
// directory1/subdirectory1/fortune3: DirectoryEntry
// directory1/subdirectory1/fortune4: DirectoryEntry
// directory1/subdirectory1: DirectoryEntry
// directory1/subdirectory2/fortune7: DirectoryEntry
// directory1/subdirectory2/fortune8: DirectoryEntry
// directory1/subdirectory2: DirectoryEntry
// directory1: DirectoryEntry
// directory2/fortune10: DirectoryEntry
// directory2/fortune9: DirectoryEntry
// directory2: DirectoryEntry
// lost+found: DirectoryEntry
// thejungle.txt: DirectoryEntry

This project is used to directly read the filesystem, file, and journal data without the support of kernel or the FUSE interface. Therefore, no elevated privileges are required.

Verifying Gerrit CRs to Your Jenkins’ Pipeline’s Shared Libraries

Jenkins’ pipelines represent a totally different direction from traditional, script-based jobs. Instead of specifying your SCM configuration and other build semantics in your job, you mostly script them out via a pipeline (“Jenkinsfile”) file, which is a heterogeneous script/declarative mess. Although you can be purely declarative, this is sometimes too strict to be useful, e.g. not being able to have traditional variable assignments in order to pass information between steps. Even though there are drawbacks, your whole workflow is largely version-controlled.

One of the drawbacks is the complexity of managing shared-library dependencies that you might have in order to make some of your Java/Groovy logic reusable. You can define these in your project (or, the case of multibranch pipelines, the folder) or at the admin level. You can also define these on the fly in the code.

Gerrit change-requests are applied essentially by fetching on a pseudo-refspec location (refs/changes/), and then cherry-picking it in. Therefore, in order to use one, you need to 1) clone, 2) fetch, and 3) either cherry-pick or checkout (or a couple of other methods). Although you can do this with a little effort with your actual Jenkinsfile (which is configured in the job; you can take the refspec from the environment during a verification and then use “FETCH_HEAD” as your branch), these are not intuitively available for the shared-libraries that you might be importing into your pipeline.

It turns out that you can massage the on-the-fly library loader to do this for you.

if (env.GERRIT_PATCHSET_REVISION) {
  echo("Using shared-library for verification.")

  library([
    identifier: 'myLibrary@' + env.GERRIT_PATCHSET_REVISION,
    retriever: modernSCM([
      $class: 'GitSCMSource',
      remote: 'https://repo.host/pipeline/library',
      traits: [
        [$class: 'jenkins.plugins.git.traits.BranchDiscoveryTrait'],
        [
          $class: 'RefSpecsSCMSourceTrait',
          templates: [
            [value: '+refs/heads/*:refs/remotes/@{remote}/*'], 
            [value: "+refs/changes/*:refs/remotes/@{remote}/*"]
          ]
        ]
      ]
    ])
  ])
} else {
  echo("Using shared-library from branch (not a verification).")

  library("myLibrary@" + env.BRANCH_NAME)
}

The principal things to notice are:

  1. We are telling it to bring all of the change-requests into scope (“+refs/changes/:refs/remotes/@{remote}/“).
  2. We are telling Jenkins to import exactly the library version tied to the change (“‘myLibrary@’ + env.GERRIT_PATCHSET_REVISION”). This wouldn’t be accessible without (1).

It works great.

I generated the original version of the code by using the Snippet Generator with the “library” step and then modifying according to the above.

Note that this pipeline can be used both in a multibranch pipeline job context as well as in the normal [single-branch] pipeline job used for verification (because we would only want to kick-off verification jobs just for the branch of the change). env.BRANCH_NAME will automatically be defined in the multibranch context.

Python: Retrieve User Info from LDAP

Supports returning the full DN for the user as well as a list of groups that they are a member of:

import logging
import ldap
import collections

# Install:
#
# apt: libsasl2-dev
# pip: python-ldap
#

_USER_QUERY = '(&(objectClass=USER)(sAMAccountName={username}))'
_GROUP_QUERY = '(&(objectClass=GROUP)(cn={group_name}))'

_LOGGER = logging.getLogger(__name__)

_USER = \
    collections.namedtuple(
        '_USER', [
            'dn',
            'attributes',
            'groups',
        ])


class NotFoundException(Exception):
    pass


class LdapAdapter(object):
    def __init__(self, host_and_port, username, password, base_dn):
        self.__host_and_port = host_and_port
        self.__username = username
        self.__password = password
        self.__base_dn = base_dn

        self.__raw_resource = None

    @property
    def __resource(self):
        if self.__raw_resource is None:
            self.__raw_resource = self.__auth()

        return self.__raw_resource

    def __auth(self):
        conn = ldap.initialize('ldap://' + self.__host_and_port)
        conn.protocol_version = 3
        conn.set_option(ldap.OPT_REFERRALS, 0)

        try:
            conn.simple_bind_s(self.__username, self.__password)
        except ldap.INVALID_CREDENTIALS:
            # Pinned, for the future.

            raise
        except ldap.SERVER_DOWN:
            # Pinned, for the future.

            raise
        except ldap.LDAPError:
            _LOGGER.error("LDAP error content:\n{}".format(e.message))

            if issubclass(e.message.__class__, dict) is True and \
               'desc' in e.message:
                raise Exception("LDAP: {}".format(e.message['desc']))

            raise
        else:
            return conn

    def get_dn_by_username(self, username):
        """Return user information. See _USER."""

        uf = _USER_QUERY.format(username=username)
        results_raw = \
            self.__resource.search_s(
                self.__base_dn,
                ldap.SCOPE_SUBTREE,
                uf)

        if not results_raw:
            raise NotFoundException(username)

        results = []
        for dn, attributes in results_raw:
            if dn is None:
                continue

            u = _USER(
                    dn=dn,
                    attributes=attributes,
                    groups=attributes['memberOf'])

            results.append(u)

        assert \
            len(results) == 1, \
            "More than one result was found for user [{}], which doesn't " \
            "make sense: {}".format(username, results)

        return results[0]

    def get_group_members(self, group_name):
        """Return a list of DNs."""

        gf = _GROUP_QUERY.format(group_name=group_name)
        results_raw = \
            self.__resource.search_s(
                self.__base_dn,
                ldap.SCOPE_SUBTREE,
                gf)

        if not results_raw:
            raise NotFoundException(group_name)

        collections = []
        for dn, attributes in results_raw:
            if dn is None:
                continue

            collections.append(attributes['member'])

        if not collections:
            raise NotFoundException(group_name)

        assert \
            len(collections) == 1, \
            "Too many sets of results returned: {}".format(collections)

        return collections[0]

go-exif-knife: One Exif Command-Line Tool to [Nearly] Rule Them All

go-exif-knife is a tool that will allow you to parse Exif from JPEG and PNG images and to do a brute-force parse of Exif embedded in any other format. You can cherry-pick specific IFDs or tags to print, and print them both as normal and JSON-formatted text. You can can also print parsed GPS data and timestamps and even produce a Google S2 geohash from the GPS data, and dump the thumbnail. If using JPEG or PNG, you can also update or add new Exif data.

This project is built on top of go-jpeg-image-structure, go-png-image-structure, and go-exif. PNG added support for Exif only in the last year, and this project was in service of providing useful Exif support for PNG.

Binary downloads are available here.