TFS Tasks Do Not Appear When Uploading a VSIX

I ran into a weird issue. I was experimenting with TFS tasks and constructing VSIX files because I have been away for a little while. I started with an existing, open-source project (which had a vss-extension.json file) and slowly morphed it from what was there to what I want it. However, I found that, when it came time to change the name of the “publisher” field in the VSIX manifest and the “author” field in the task manifest, I would do an upload and, although the VSIX uploaded and installed perfectly fine, the tasks would not show up.

Things I tried:

  • Using the publisher name off another project located in the Marketplace.
  • Using *my* personal publisher name (I am registered in the Marketplace).
  • Using a random string definitely not registered as an existing publisher.
  • Using a second, different project and trying the same changes to the publisher name, just in case there was something magical about the publisher in the first one (Microsoft, incidentally). There were some implied correlations between the publisher names, project names, scopes, and/or targets, and, most of those started with “ms” and I didn’t know whether this was significant and the real issue.
  • Since I was experiementing on an on-premise TFS 2015 instance colocated on my laptop, I cut my Internet connection and installed the VSIX of a brand-new project in order to determine whether it could run or if it potentially did publisher lookups against the Marketplace. The latter was not the case. It installed fine.

It turns out that, if you are going to change the publisher, you need to change the version as well. When you uninstall a VSIX, the TFS still remains polluted with prior knowledge of that VSIX and/or those tasks.

 

Converting TFS 2015 Build Definition Names to Their Directory IDs

A TFS 2015 agent maintains a directory of build-space directories, where each corresponds to a single build definition. Each of these has a source-directory (“s”; usually automatically checked-out), binaries-directory (“b”; for intermediate binaries prior to publish-oriented cherry-picking), and artifact-staging directory (“a”; for artifacts to be published).

Not only is each of the build-space directories an integer with no clear mapping to the build-definition but this integer is different from one agent to the next. It turns out that there is a meta-information directory whose children are collection GUIDs. Furthermore, the children of those directories are additional directories with integer names. Each of these has a JSON file that contains build-definition information.

I quickly wrote a Python script to search for a given build definition and print the ID. The basic functionality is split into succinct, easily-callable methods for whatever other tasks you might have.

import logging
import os.path
import json

_LOGGER = logging.getLogger(__name__)


class Tfs(object):
    def __init__(self, agent_path):
        self.__agent_path = agent_path
        self.__srm_path = \
            os.path.join(
                self.__agent_path, 
                '_work', 
                'SourceRootMapping')

        _LOGGER.debug("Agent SRM path: [%s]", self.__srm_path)

    def collection_gen(self):
        for filename in os.listdir(self.__srm_path):
            filepath = os.path.join(self.__srm_path, filename)
            if os.path.isdir(filepath) is False:
                continue

            _LOGGER.debug("Collection GUID: [%s]", filename)
            yield filename, filepath

    def definition_gen(self, collection_path):
        for srm_id_raw in os.listdir(collection_path):
            _LOGGER.debug("SRM ID: [%s]", srm_id_raw)

            definition_info_filepath = \
                os.path.join(
                    collection_path, 
                    srm_id_raw, 
                    'SourceFolder.json')

            with open(definition_info_filepath) as f:
                yield json.load(f)

    def lookup_definition(self, definition_name):
        for collection_guid, collection_path in self.collection_gen():
            for definition in self.definition_gen(collection_path):
                current_definition_name = definition['definitionName']
                _LOGGER.debug("Definition: [%s]", current_definition_name)

                if current_definition_name != definition_name:
                    continue

                return definition

        raise ValueError("Could not find definition: {}".format(
                         definition_name))


def _configure_logging():
    sh = logging.StreamHandler()
    formatter = logging.Formatter('%(asctime)s [%(name)s %(levelname)s] %(message)s')
    sh.setFormatter(formatter)

    _LOGGER.addHandler(sh)
    _LOGGER.setLevel(logging.DEBUG)

if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()

    parser.add_argument(
        'agent_path', 
        help="Agent path")

    parser.add_argument(
        'definition_name', 
        help="Build-definition name")

    args = parser.parse_args()

    _configure_logging()

    t = Tfs(args.agent_path)
    definition = t.lookup_definition(args.definition_name)

    print(definition['agent_builddirectory'])

Output:

D:\development\python>python tfs.py c:\tfs_build_agent ConsoleProject.Dev
2016-07-07 23:32:02,756 [__main__ DEBUG] Agent SRM path: [c:\tfs_build_agent\_work\SourceRootMapping]
2016-07-07 23:32:02,756 [__main__ DEBUG] Collection GUID: [2cf8d3cb-b8d4-49e1-bbdb-2aacf02f48c4]
2016-07-07 23:32:02,757 [__main__ DEBUG] SRM ID: [1]
2016-07-07 23:32:02,757 [__main__ DEBUG] Definition: [ConsoleProject.Dev]
1

Important Notes

  • The child directories of the SourceRootMapping directory will not exactly reflect the current build definitions. Some of them may represent build definitions that no longer exist.

  • One of the current work directories may be using an ID used by an old build-definition at some point in the past. So, if you are building a dictionary of work-directory IDs to build-definitions, you will have to use the build-definition’s ID or one of the timestamps described in the its JSON file to reconcile the latest build-definition to use that directory.

TFS 2015 Database Structure Investigation

A TFS 2015 build agent maintains a list of directories where builds occur. Each of these directories is assigned a monotonically-incrementing number and has three children: “s” (“source” directory where your files are, usually, automatically checked-out for you), “b” (“binaries” directory where you can choose to stash files that aren’t automatically cleaned-up), and “a” (“artifact-staging” directory where you push files/directories that will be published). Each agent assigns its own IDs for each build definition.

I sought to figure out 1) how to lookup the mapping between the build-definitions and these agent-ID pairs, and 2) how to lookup the paths when you publish a build to be stored by the TFS server (rather than a fileshare). Every so often I wander into the TFS database to chip away at its elusive topology. I’ll briefly document my observations here for the benefit of myself and others. I’ll add on new information as I might encounter it in the future. I’m only currently concerned with build-definitions but any information that can be provided for release-definitions would also be welcomes.

There is a main database (e.g. “Tfs_Configuration”) and subordinate collection-specific database (e.e. “Tfs_DefaultCollection”).

[Build].[tbl_Build]

Information on individual builds including build numbers (e.g. 20160512.1) and “definition ID”.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
,[DataspaceId]
,[BuildId]
,[DefinitionId]
,[DefinitionVersion]
,[BuildNumber]
,[BuildNumberRevision]
,[RepositoryId]
,[BranchId]
,[SourceVersion]
,[Parameters]
,[Status]
,[QueueId]
,[QueueTime]
,[Priority]
,[StartTime]
,[FinishTime]
,[Reason]
,[Result]
,[RequestedFor]
,[RequestedBy]
,[ChangedOn]
,[ChangedBy]
,[OrchestrationId]
,[Deleted]
,[ValidationIssues]
,[QueueOptions]
,[KeepForever]
,[ChangesCalculated]
,[DeletedOn]
FROM [Tfs_DefaultCollection].[Build].[tbl_Build]

Record looks like:

PartitionId DataspaceId BuildId DefinitionId    DefinitionVersion   BuildNumber BuildNumberRevision RepositoryId    BranchId    SourceVersion   Parameters  Status  QueueId QueueTime   Priority    StartTime   FinishTime  Reason  Result  RequestedFor    RequestedBy ChangedOn   ChangedBy   OrchestrationId Deleted ValidationIssues    QueueOptions    KeepForever ChangesCalculated   DeletedOn

1   22  168 1   58  20160701.1  1   1   2   C31 {"system.debug":"false","BuildConfiguration":"release","BuildPlatform":"any cpu"}   2   1   2016-07-01 16:11:01.2247845 3   2016-07-01 16:11:03.5025120 2016-07-01 16:11:23.2632483 1   2   FD456772-708D-496C-9259-32596770CD73    FD456772-708D-496C-9259-32596770CD73    2016-07-01 16:11:23.450 02442124-655A-49B2-A6FB-20269EDEBEF6    E6ABEC05-179C-4406-91CB-E96B9B6CFD7C    0   NULL    NULL    0   1   NULL

[Build].[tbl_Definition]

Describes individual build-definitions, presumably. Includes “definition ID”, “definition version” (probably a pointer to the head version), “definition name”.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[DataspaceId]
      ,[DefinitionId]
      ,[DefinitionVersion]
      ,[DefinitionName]
      ,[Quality]
      ,[QueueId]
      ,[QueueStatus]
      ,[RepositoryId]
      ,[DefaultBranchId]
      ,[TriggerTypes]
      ,[Description]
      ,[BuildNumberFormat]
      ,[JobAuthorizationScope]
      ,[JobTimeout]
      ,[Comment]
      ,[Author]
      ,[CreatedOn]
      ,[ParentDefinitionId]
      ,[Options]
      ,[Repository]
      ,[Triggers]
      ,[Steps]
      ,[Variables]
      ,[Demands]
      ,[RetentionPolicy]
      ,[BadgeEnabled]
      ,[Deleted]
  FROM [Tfs_DefaultCollection].[Build].[tbl_Definition]

Record looks like:

PartitionId DataspaceId DefinitionId    DefinitionVersion   DefinitionName  Quality QueueId QueueStatus RepositoryId    DefaultBranchId TriggerTypes    Description BuildNumberFormat   JobAuthorizationScope   JobTimeout  Comment Author  CreatedOn   ParentDefinitionId  Options Repository  Triggers    Steps   Variables   Demands RetentionPolicy BadgeEnabled    Deleted

1   22  1   58  ConsoleProject.Dev  1   1   0   1   1   1   NULL    $(date:yyyyMMdd)$(rev:.r)   1   60  Added publish-to-server step.   FD456772-708D-496C-9259-32596770CD73    2016-07-01 16:10:57.070 NULL    [{"enabled":false,"definition":{"id":"7c555368-ca64-4199-add6-9ebaf0b0137d"},"inputs":{"multipliers":"[]","parallel":"false","continueOnError":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"a9db38f9-9fdc-478c-b0f9-464221e58316"},"inputs":{"workItemType":"1","assignToRequestor":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"57578776-4c22-4526-aeb0-86b6da17ee9c"},"inputs":{"additionalFields":"{}"}}]    {"properties":{"labelSources":"0","tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db\",\"mappingType\":\"map\",\"localPath\":\"\\\\\"},{\"serverPath\":\"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/Drops\",\"mappingType\":\"cloak\",\"localPath\":\"\\\\\"}]}"},"id":"$/","type":"TfsVersionControl","name":"d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","url":"http://dustin-pc:8181/tfs/DefaultCollection/","defaultBranch":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","rootFolder":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","clean":"false","checkoutSubmodules":false}  NULL    [{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution $/TestProject/TestConsoleApplication.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestConsoleApplication.sln","msbuildArgs":"/target:publish /p:ApplicationVersion=1.0.2.2 /p:InstallUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:UpdateUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:PublishUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:UpdateEnabled=true /p:UpdateMode=Foreground /p:ProductName=TestConsoleApplication /p:IsWebBootstrapper=false","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution $/TestProject/TestDatabase.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestDatabase.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Run script $/TestProject/TestScripts/TestEcho.cmd","task":{"id":"bfc8bf76-e7ac-4a8c-9a55-a944a9f632fd","versionSpec":"*"},"inputs":{"filename":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestScripts/TestEcho.cmd","arguments":"","modifyEnvironment":"false","workingFolder":"","failOnStandardError":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\TestConsoleApplication\\bin\\Release","Contents":"**\\*","TargetFolder":"$(Build.ArtifactStagingDirectory)","CleanTargetFolder":"true","OverWrite":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)\\deployment\\Release\\Utility","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\Release\\Utility","Contents":"**\\*","TargetFolder":"$(Build.ArtifactStagingDirectory)\\deployment\\Release\\Utility","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)/TestDatabase1","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\TestDatabase1\\bin\\Release","Contents":"TestDatabase1.dacpac","TargetFolder":"$(Build.ArtifactStagingDirectory)/TestDatabase1","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Publish Artifact: drop","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"drop","ArtifactType":"FilePath","TargetPath":"\\\\localhost\\build_publish\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Publish Artifact: Published_$(Build.BuildNumber)","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"Published_$(Build.BuildNumber)","ArtifactType":"Container","TargetPath":"\\\\my\\share\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}}]  {"system.debug":{"value":"false","allowOverride":true},"BuildConfiguration":{"value":"release","allowOverride":true},"BuildPlatform":{"value":"any cpu","allowOverride":true},"DUSTINPASSWORD":{"value":null,"allowOverride":true,"isSecret":true}} NULL    [{"branches":["+refs/heads/*"],"artifacts":["build.SourceLabel"],"daysToKeep":10,"minimumToKeep":1,"deleteBuildRecord":true,"deleteTestResults":true}]  0   0

[Build].[tbl_DefinitionHistory]

Describes the history of changes to the definition. Includes “definition ID”, “definition version”, “definition name”, and several JSON blocks.

Note that the record I provided is the most recent available (in my local sandbox environment). Though I had just made a change to a definition, it was, curiously, not represented in this table (though many other, previous, changes ones were).

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[DataspaceId]
      ,[DefinitionId]
      ,[DefinitionVersion]
      ,[DefinitionName]
      ,[QueueId]
      ,[QueueStatus]
      ,[RepositoryId]
      ,[DefaultBranchId]
      ,[Description]
      ,[BuildNumberFormat]
      ,[JobAuthorizationScope]
      ,[JobTimeout]
      ,[Comment]
      ,[Author]
      ,[CreatedOn]
      ,[Options]
      ,[Repository]
      ,[Triggers]
      ,[Steps]
      ,[Variables]
      ,[Demands]
      ,[RetentionPolicy]
      ,[BadgeEnabled]
      ,[Deleted]
  FROM [Tfs_DefaultCollection].[Build].[tbl_DefinitionHistory]

Record looks like:

PartitionId DataspaceId DefinitionId    DefinitionVersion   DefinitionName  QueueId QueueStatus RepositoryId    DefaultBranchId Description BuildNumberFormat   JobAuthorizationScope   JobTimeout  Comment Author  CreatedOn   Options Repository  Triggers    Steps   Variables   Demands RetentionPolicy BadgeEnabled    Deleted

1   32  12  18  PWC Audit360 Simulation Build   1   0   2   9   NULL    $(date:yyyyMMdd)$(rev:.r)   1   60  NULL    FD456772-708D-496C-9259-32596770CD73    2016-05-27 18:45:03.117 [{"enabled":false,"definition":{"id":"7c555368-ca64-4199-add6-9ebaf0b0137d"},"inputs":{"multipliers":"[]","parallel":"false","continueOnError":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"a9db38f9-9fdc-478c-b0f9-464221e58316"},"inputs":{"workItemType":"16","assignToRequestor":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"57578776-4c22-4526-aeb0-86b6da17ee9c"},"inputs":{"additionalFields":"{}"}}]   {"properties":{"labelSources":"0","tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/d0942daa-ac39-462f-8115-fada54d8f780\",\"mappingType\":\"map\",\"localPath\":\"\\\\\"},{\"serverPath\":\"$/d0942daa-ac39-462f-8115-fada54d8f780/Drops\",\"mappingType\":\"cloak\",\"localPath\":\"\\\\\"}]}"},"id":"$/","type":"TfsVersionControl","name":"d0942daa-ac39-462f-8115-fada54d8f780","url":"http://dustin-pc:8181/tfs/DefaultCollection/","defaultBranch":"$/d0942daa-ac39-462f-8115-fada54d8f780","rootFolder":"$/d0942daa-ac39-462f-8115-fada54d8f780","clean":"false","checkoutSubmodules":false}  NULL    [{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Server.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Server.sln","nugetConfigPath":"$/d0942daa-ac39-462f-8115-fada54d8f780/nuget.config","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.SilverlightBuild.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.SilverlightBuild.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Database.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Database.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Queue.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Queue.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore MarketData.Database.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"MarketData.Database.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Server.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Server.sln","msbuildArgs":"/t:build;publish /p:DeployOnBuild=True /m:1 /p:DeployPrefix=F1","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.SilverlightBuild.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.SilverlightBuild.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Database.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Database.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Queue.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Queue.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution MarketData.Database.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"MarketData.Database.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Test Assemblies **\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**","task":{"id":"ef087383-ee5e-42c7-9a53-ab56c98420f9","versionSpec":"*"},"inputs":{"testAssembly":"**\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**","testFiltercriteria":"","runSettingsFile":"","overrideTestrunParameters":"","codeCoverageEnabled":"false","runInParallel":"false","vsTestVersion":"14.0","pathtoCustomTestAdapters":"","otherConsoleOptions":"","testRunTitle":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","publishRunAttachments":"true"}},{"enabled":false,"continueOnError":true,"alwaysRun":false,"displayName":"Publish symbols path: ","task":{"id":"0675668a-7bba-4ccb-901d-5ad6554ca653","versionSpec":"*"},"inputs":{"SymbolsPath":"","SearchPattern":"**\\bin\\**\\*.pdb","SymbolsFolder":"","SkipIndexing":"false","TreatNotIndexedAsWarning":"false","SymbolsMaximumWaitTime":"","SymbolsProduct":"","SymbolsVersion":"","SymbolsArtifactName":"Symbols_$(BuildConfiguration)"}},{"enabled":false,"continueOnError":false,"alwaysRun":true,"displayName":"Copy Files to: $(build.artifactstagingdirectory)","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(build.sourcesdirectory)","Contents":"**\\bin\\$(BuildConfiguration)\\**","TargetFolder":"$(build.artifactstagingdirectory)","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":false,"continueOnError":false,"alwaysRun":true,"displayName":"Publish Artifact: drop","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"drop","ArtifactType":"Container","TargetPath":"\\\\my\\share\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Run script $/PwcSimulation/Build/RMPostBuild.cmd","task":{"id":"bfc8bf76-e7ac-4a8c-9a55-a944a9f632fd","versionSpec":"*"},"inputs":{"filename":"$/d0942daa-ac39-462f-8115-fada54d8f780/Build/RMPostBuild.cmd","arguments":"/C:FB /E:F1","modifyEnvironment":"false","workingFolder":"","failOnStandardError":"false"}}] {"system.debug":{"value":"false","allowOverride":true},"BuildConfiguration":{"value":"FB","allowOverride":true},"BuildPlatform":{"value":"any cpu","allowOverride":true}}   NULL    [{"branches":["+refs/heads/*"],"artifacts":["build.SourceLabel"],"daysToKeep":10,"minimumToKeep":1,"deleteBuildRecord":true,"deleteTestResults":true}]  0   0

[dbo].[tbl_Workspace]

Appears to describe the workspaces that are currently defined for users, builds, etc..

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[OwnerId]
      ,[WorkspaceName]
      ,[WorkspaceVersion]
      ,[Type]
      ,[Comment]
      ,[CreationDate]
      ,[Computer]
      ,[PolicyOverrideComment]
      ,[LastAccessDate]
      ,[CheckInNoteId]
      ,[RefreshRecursive]
      ,[HasDeletedChanges]
      ,[SequentialId]
      ,[IsLocal]
      ,[PendingChangeSig]
      ,[FileTime]
      ,[LastMappingsUpdate]
      ,[ProjectNotificationId]
      ,[ItemIdCounter]
      ,[VersionStamp]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_Workspace]

Record looks like :

PartitionId WorkspaceId OwnerId WorkspaceName   WorkspaceVersion    Type    Comment CreationDate    Computer    PolicyOverrideComment   LastAccessDate  CheckInNoteId   RefreshRecursive    HasDeletedChanges   SequentialId    IsLocal PendingChangeSig    FileTime    LastMappingsUpdate  ProjectNotificationId   ItemIdCounter   VersionStamp

1   420000001   65CED833-E406-4B14-BDB2-51590FE7569C    ws_5_1  1   0   Created by Distributed Task - getCode   2016-06-29 03:47:10.363 DUSTIN-PC   NULL    2016-06-29 03:47:10.363 NULL    0   0   22  1   B2140B25-F70A-4B4D-BFB1-184703037010    0   2016-06-29 03:47:10.370 37  -1025   2.00

[dbo].[tbl_WorkingFolder]

This use of this table is unclear. It describes the paths on the disk where the build-processes for each definition are hosted, but they also include a “workspace ID”. This would make sense for individual builds but not for latent build-definitions (workspaces for builds are ephemeral, unless the build fails, and are only created once the build starts). This seems promising for where the mappings for the build directories are stored, except that there appears to be potentially many records for the same build-definitions.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[ItemDataspaceId]
      ,[ProjectName]
      ,[ServerItem]
      ,[LocalItem]
      ,[MappingType]
      ,[CreationDate]
      ,[Depth]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_WorkingFolder]

Record looks like:

PartitionId WorkspaceId ItemDataspaceId ProjectName ServerItem  LocalItem   MappingType CreationDate    Depth

1   320000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\1\s\  1   2016-07-01 16:11:07.500 120
1   320000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-07-01 16:11:07.500 120
1   360000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\3\s\  1   2016-05-24 18:22:09.760 120
1   360000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-05-24 18:22:09.760 120
1   420000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\5\s\  1   2016-06-29 03:47:10.363 120
1   420000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-06-29 03:47:10.363 120

[dbo].[tbl_WorkingFolderHistory]

The purpose of this table is completely unclear, though it includes build location paths (just like WorkingFolder).

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[ItemDataspaceId]
      ,[ServerItem]
      ,[LocalItem]
      ,[Active]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_WorkingFolderHistory]

Record looks like:

PartitionId WorkspaceId ItemDataspaceId ServerItem  LocalItem   Active

1   320000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\1\s\  1
1   320000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   C:\tfs>build>agent\>work\1\s\Drops\    0
1   360000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\3\s\  1
1   400000001   31  $\d0942daa"ac39"462f"8115"fada54d8f780\ C:\tfs>build>agent\>work\7\s\  0
1   400000001   31  $\d0942daa"ac39"462f"8115"fada54d8f780\Drops\   C:\tfs>build>agent\>work\7\s\Drops\    0
1   420000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\5\s\  1

“Invalid Credentials” When Using TFX Tool to Login to TFS

The cross-platform TFX tool is, among other things, what you use to create new boiler-plate VSIX extensions (to populate with your own information) as well as to upload individual Release Management (vNext) tasks to TFS 2015. Previously, I’ve logged in using a fully-qualified UNC-formatted username (\domain\dustin.oprea) but this started, suddenly, not working for me. I have not applied any noteworthy updates. I did recently have to change my password (per group policy), but that may or may not have affected it.

As my web-based TFS UI indicated that I was logged-in as “DOMAIN\dustin.oprea”, I unsuccessfully tried logging in as that at the command-line. Eventually, I opened up a private session within the browser in order to log-in using that same username but the UI refused to log me out (even though it was a private session, I was always logged-in). I tried clearing web credential in Windows, but, alas, nothing was stored there (if that could’ve even helped).

I also restarted the system just in case there was credentials-related caching in the system, in or outside of TFS’ application pool.

It turns out that now logging-in with a fully-qualified email-account as my username works just fine.

For example:

C:\>tfx login --auth-type basic --service-url http://localhost:8181/tfs/DefaultCollection --username dustin@email.com
TFS Cross Platform Command Line Interface v0.3.20
Copyright Microsoft Corporation
> Password:
Logged in successfully

Thanks, TFS. Do you think you might unify how users are logging-in both at the command-line and in the UI? Maybe?

Thanks to Could Not Connect, Invalid Credentials, Bad Username/ID/Password for drawing my eye to this potential option.

Publishing with ClickOnce via Visual Studio and TFS 2015 (vNext)

ClickOnce equips your published application to be able to check for updates and to, optionally, do the update automatically. See this post for more information.

To invoke this functionality from either the command-line or your build definition/process, you need to pass some additional project parameters to MSBuild.

ApplicationVersion (e.g. 1.0.1.6)
InstallUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
UpdateUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
PublishUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
UpdateEnabled (e.g. true)
UpdateMode (e.g. Foreground)
ProductName (e.g. TestConsoleApplication)

There is some disagreement as to whether the “ProductName” parameter is necessary.

The corresponding set of MSBuild arguments using the example values:

/target:publish /p:ApplicationVersion=1.0.1.6 /p:InstallUrl=http://clickonce.localhost/consoletestfrombuild/ /p:UpdateUrl=http://clickonce.localhost/consoletestfrombuild/ /p:PublishUrl=http://clickonce.localhost/consoletestfrombuild/ /p:UpdateEnabled=true /p:UpdateMode=Foreground /p:ProductName=TestConsoleApplication

If your polling locations are going to be UNCs (“\\path\\file”) rather than URLs then you’ll have to pass an additional argument:

/p:IsWebBootstrapper=false

To configure this under a TFS 2015 build-definition, simply add the parameters to the “MSBuild Arguments” field:

ClickOnce Settings in TFS 2015

References

Building ClickOnce Applications from the Command Line (Microsoft)

Deploying a click-once application with build vNext in Azure Blob Storage

Using Tokenization (Token Replacement) for Builds/Releases in vNext/TFS 2015

Microsoft has provided support for doing token replacements into build artifacts. Since this is provided as a “utility” task there’s no bias in how it’s used but you’ll likely find it most useful from your release workflow.

Inexplicably, Microsoft doesn’t actually provide this with TFS. Rather, you need to use their marketplace to download and install it. Still, as far as boneheaded designs go, it’s not so bad.

There are three things that it allows you to do:

  1. Do string-replacements using the environment.
  2. Do string-replacements using the environment-specific dictionary from the configuration file.
  3. Do environment-specific XPath replacements.

Note that:

  1. (1) and (2) will expect to find tokens that look like “[A-Za-z0-9._-]*” (e.g. “XYZ“, “ABC_DEF“, “GHI.JKL“, “MNO-PQR“), remove the underscores on the margins, and translate the periods to underscores.
  2. (2) and (3) are only possible if you provide a configuration file and if the source file is XML (obviously).

Installation

Download the Release Management Utility tasks extension (as a VSIX file). Install it using the instructions from my other post. You’ll now see a “Tokenize with XPath/Regular expressions” utility task.

For reference, this is the original project URL:

Extension-UtilitiesPack

Usage

The configuration file looks like:

{
   "Default Environment":{
      "CustomVariables":{
         "Token2":"value_from_custom2",
         "Token3":"value_from_custom3"
      },
      "ConfigChanges":[
         {
            "KeyName":"/configuration/appSettings/add[@key='TestKey1']",
            "Attribute":"value",
            "Value":"value_from_xpath"
         }
      ]
   }
}

If we’re going to do token-replacements over an App.settings file, the source file might look like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
      <add key="TestKey1" value="__Token1__" />
      <add key="TestKey2" value="__Token2__" />
      <add key="TestKey3" value="__Token3__" />
      <add key="TestKey4" value="__Token4__" />
    </appSettings>
    <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
    </startup>
</configuration>

Since TestKey4 is not otherwise defined, it will either be replaced with itself (no net difference) or the value for an environment-variable named “Token4” if defined.

In my test situation, my build definition’s “Copy Files” task is copying the contents (“**\*”) of “$(Build.SourcesDirectory)\TestConsoleApplication\bin\Release” to “$(Build.ArtifactStagingDirectory)”. This is an example of the arguments to the “Tokenize with XPath/Regular expressions” task that I’ve added in my release-definition:

  • Source filename: “$(Build.ArtifactStagingDirectory)\drop\App.config.template”
  • Destination filename: “$(Build.ArtifactStagingDirectory)\drop\TestConsoleApplication.exe.config”
  • Configuration Json filename: “$(Build.ArtifactStagingDirectory)\drop\TokenEnvironmentConfig.json”

Personally, I’d prefer that parsing this be a little stricter: fail if can’t parse as JSON, fail if any tokens can not be resolved, and optionally allow it to fail if any XPath specifications don’t match anything.

Additional Notes

  • The RELEASE_ENVIRONMENTNAME variable is usually defined and used to determine the current environment. However, if it isn’t defined then the default environment searched in the configuration file is “default”.

  • The CustomVariables and ConfigChanges blocks don’t have to exist in the configuration file (won’t fail validation). What is there will be used.

  • For debugging, I’ve had issues enabling logging verbosity in PowerShell (with VerbosePreference). It would be easier adjusting the tasks\Tokenizer\x.y.z\tokenize.ps1 module in your build-agent’s directory to change all the instances of “Write-Verbose” to “Write-Host”. The module provides a lot of useful messaging if you can get it to show.

  • In the current version of the task, it provides a PowerShell 3 script in addition to the original PS script. I get the following message on my system: ##[error]Index operation failed; the array index evaluated to null. I downloaded the source-code to the VSIX that is available in the Marketplace, adjusted it to not use the PS3 version of the script, repackaged the VSIX, uploaded it to my on-premise TFS, and, bam, the token-replacement worked perfectly. The PowerShell 3 script is broken.