Using Tokenization (Token Replacement) for Builds/Releases in vNext/TFS 2015

Microsoft has provided support for doing token replacements into build artifacts. Since this is provided as a “utility” task there’s no bias in how it’s used but you’ll likely find it most useful from your release workflow.

Inexplicably, Microsoft doesn’t actually provide this with TFS. Rather, you need to use their marketplace to download and install it. Still, as far as boneheaded designs go, it’s not so bad.

There are three things that it allows you to do:

  1. Do string-replacements using the environment.
  2. Do string-replacements using the environment-specific dictionary from the configuration file.
  3. Do environment-specific XPath replacements.

Note that:

  1. (1) and (2) will expect to find tokens that look like “[A-Za-z0-9._-]*” (e.g. “XYZ“, “ABC_DEF“, “GHI.JKL“, “MNO-PQR“), remove the underscores on the margins, and translate the periods to underscores.
  2. (2) and (3) are only possible if you provide a configuration file and if the source file is XML (obviously).

Installation

Download the Release Management Utility tasks extension (as a VSIX file). Install it using the instructions from my other post. You’ll now see a “Tokenize with XPath/Regular expressions” utility task.

For reference, this is the original project URL:

Extension-UtilitiesPack

Usage

The configuration file looks like:

{
   "Default Environment":{
      "CustomVariables":{
         "Token2":"value_from_custom2",
         "Token3":"value_from_custom3"
      },
      "ConfigChanges":[
         {
            "KeyName":"/configuration/appSettings/add[@key='TestKey1']",
            "Attribute":"value",
            "Value":"value_from_xpath"
         }
      ]
   }
}

If we’re going to do token-replacements over an App.settings file, the source file might look like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
      <add key="TestKey1" value="__Token1__" />
      <add key="TestKey2" value="__Token2__" />
      <add key="TestKey3" value="__Token3__" />
      <add key="TestKey4" value="__Token4__" />
    </appSettings>
    <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
    </startup>
</configuration>

Since TestKey4 is not otherwise defined, it will either be replaced with itself (no net difference) or the value for an environment-variable named “Token4” if defined.

In my test situation, my build definition’s “Copy Files” task is copying the contents (“**\*”) of “$(Build.SourcesDirectory)\TestConsoleApplication\bin\Release” to “$(Build.ArtifactStagingDirectory)”. This is an example of the arguments to the “Tokenize with XPath/Regular expressions” task that I’ve added in my release-definition:

  • Source filename: “$(Build.ArtifactStagingDirectory)\drop\App.config.template”
  • Destination filename: “$(Build.ArtifactStagingDirectory)\drop\TestConsoleApplication.exe.config”
  • Configuration Json filename: “$(Build.ArtifactStagingDirectory)\drop\TokenEnvironmentConfig.json”

Personally, I’d prefer that parsing this be a little stricter: fail if can’t parse as JSON, fail if any tokens can not be resolved, and optionally allow it to fail if any XPath specifications don’t match anything.

Additional Notes

  • The RELEASE_ENVIRONMENTNAME variable is usually defined and used to determine the current environment. However, if it isn’t defined then the default environment searched in the configuration file is “default”.

  • The CustomVariables and ConfigChanges blocks don’t have to exist in the configuration file (won’t fail validation). What is there will be used.

  • For debugging, I’ve had issues enabling logging verbosity in PowerShell (with VerbosePreference). It would be easier adjusting the tasks\Tokenizer\x.y.z\tokenize.ps1 module in your build-agent’s directory to change all the instances of “Write-Verbose” to “Write-Host”. The module provides a lot of useful messaging if you can get it to show.

  • In the current version of the task, it provides a PowerShell 3 script in addition to the original PS script. I get the following message on my system: ##[error]Index operation failed; the array index evaluated to null. I downloaded the source-code to the VSIX that is available in the Marketplace, adjusted it to not use the PS3 version of the script, repackaged the VSIX, uploaded it to my on-premise TFS, and, bam, the token-replacement worked perfectly. The PowerShell 3 script is broken.

Uploading Release Management Tasks With a VSIX Extension Package

TFS 2015 lets you define build and release workflows comprised of discrete tasks. Although it provides a handful to you (build-specific, release-specific, general utility, etc..), you can also write your own “Extension Tasks” and upload them. I previously described how to use Microsoft’s tfx tool to upload a single task from the command-line.

You might also need to upload one or more tasks that have been packaged into a VSIX file (Microsoft’s Visual Studio Extension packaging format). These are essentially zip-files that have a manifest and one or more directories. In this case, those directories represent tasks (they each have a task.json file). Although you can probably do this from the command-line using the tfx tool, you can also use your TFS’s web UI to do it. Note that this process will need to be repeated for each collection that you’ll want the extension to be available in.

Click the “Shop” icon to open the corresponding menu and then click on “Browse of Extensions”:

tfsbrowseforextensions.png

Click on the “Manage extensions” button in the middle of the page. It’s not the same link/button as the previous one even if they have the same text:

release-management-extension-upload-3-manage-extensions-2.png

Click on “Upload New Extension”. Browse to the VSIX file and upload it.

release-management-extension-upload-4-extension-upload.png

After, you’re presented with a list of installed extensions:

release-management-extension-upload-5-extension-uploaded.png

Observe that the extension populated on to the previous local-management screen. Click on it:

release-management-extension-upload-6.png

Now, click on “Install”:

release-management-extension-upload-7.png

After installation, the new tasks will be available within the lists of available tasks.

Adding a Task to Your On-Premise TFS 2015 Server

TFS 2015 embeds previous vNext functionality to provide you workflow-driven, scriptable builds. This is based on establishing discrete tasks that can be reused between your build definitions and release definitions as relevant.

You can extend the basic set of tasks, which, though sufficient for very simple builds and releases, might quickly become insufficient. New tasks are called “extension tasks”. Discussing how to create one is out of scope for this post so we’re assuming that you already have a path that describes a task. At the very least, there will be a task.json file in it.

What comes next?

  1. Install Microsoft’s “tfx” tool.
  2. Added support for “Basic Authentication” to your on-premise TFS’ IIS server.
  3. Enable Basic Authentication on that IIS server.
  4. Login to your TFS using the tfx tool.
  5. Upload task.

Instructions

1. Install Microsoft’s “tfx” tool

This is a cross-platform project that requires NodeJS to be installed locally. The TFS-CLI project is hosted here: https://github.com/Microsoft/tfs-cli.

To install:

npm install -g tfx-cli

2. Add support for “Basic Authentication” to your on-premise TFS’ IIS server

On a workstation flavor of Windows (in my case, Windows 8.1):

  1. Go to “Programs and Features” -> “Turn Windows features on or off”.
  2. Navigate down through the tree of features: “Internet Information Services” -> “World Wide Web Services” -> “Security”.
  3. Enable “Basic Authentication” and click “OK”.

If you’re running a server flavor of Windows you’ll have to update the “Web Server” role instead:

(https://github.com/Microsoft/tfs-cli/blob/master/docs/configureBasicAuth.md)

windowsserverconfigurebasicauthfeature.png

Make sure to close and re-open IIS Manager afterward.

3. Enable Basic Authentication in IIS

Under the application, click “Authentication”, right-click on “Basic Authentication”, and click “Enable”. Make sure you enable “Basic Authentication” under the application, not the site. Setting it on the site will be ineffectual.

4. Login to your TFS using the tfx tool

C:\>tfx login --auth-type basic
TFS Cross Platform Command Line Interface v0.3.20
Copyright Microsoft Corporation
> Service URL: http://localhost:8181/tfs/DefaultCollection
> Username: yourdomain\dustin.oprea
> Password:
Logged in successfully

5. Upload task

I tested using the “XLD” task, here:

XL Deploy

Unzip and install by passing the directory path:

tfx build tasks upload --task-path xld

Now, if you go to your list of release tasks (where the XLD task ends-up), you’ll see it:

windowsserverconfigurebasicauthfeature.png