TFS Tasks Do Not Appear When Uploading a VSIX

September 16, 2016dustin

I ran into a weird issue. I was experimenting with TFS tasks and constructing VSIX files because I have been away for a little while. I started with an existing, open-source project (which had a vss-extension.json file) and slowly morphed it from what was there to what I want it. However, I found that, when it came time to change the name of the “publisher” field in the VSIX manifest and the “author” field in the task manifest, I would do an upload and, although the VSIX uploaded and installed perfectly fine, the tasks would not show up.

Things I tried:

Using the publisher name off another project located in the Marketplace.
Using *my* personal publisher name (I am registered in the Marketplace).
Using a random string definitely not registered as an existing publisher.
Using a second, different project and trying the same changes to the publisher name, just in case there was something magical about the publisher in the first one (Microsoft, incidentally). There were some implied correlations between the publisher names, project names, scopes, and/or targets, and, most of those started with “ms” and I didn’t know whether this was significant and the real issue.
Since I was experiementing on an on-premise TFS 2015 instance colocated on my laptop, I cut my Internet connection and installed the VSIX of a brand-new project in order to determine whether it could run or if it potentially did publisher lookups against the Marketplace. The latter was not the case. It installed fine.

It turns out that, if you are going to change the publisher, you need to change the version as well. When you uninstall a VSIX, the TFS still remains polluted with prior knowledge of that VSIX and/or those tasks.

Setting Environment Variables From PowerShell Tasks Under TFS 2015

July 25, 2016dustin

This is less than intuitive, but you can hotwire some underlying functionality to set your own environment variables that will persist from task to task. Here, we take a value from the environment and set another value derived from it.

Write-Host "##vso[task.setvariable variable=ACTIONSFOLDER;]$env:BUILD_DROPFOLDER\Scripts\Steps"

Converting TFS 2015 Build Definition Names to Their Directory IDs

July 7, 2016dustin

A TFS 2015 agent maintains a directory of build-space directories, where each corresponds to a single build definition. Each of these has a source-directory (“s”; usually automatically checked-out), binaries-directory (“b”; for intermediate binaries prior to publish-oriented cherry-picking), and artifact-staging directory (“a”; for artifacts to be published).

Not only is each of the build-space directories an integer with no clear mapping to the build-definition but this integer is different from one agent to the next. It turns out that there is a meta-information directory whose children are collection GUIDs. Furthermore, the children of those directories are additional directories with integer names. Each of these has a JSON file that contains build-definition information.

I quickly wrote a Python script to search for a given build definition and print the ID. The basic functionality is split into succinct, easily-callable methods for whatever other tasks you might have.

import logging
import os.path
import json

_LOGGER = logging.getLogger(__name__)


class Tfs(object):
    def __init__(self, agent_path):
        self.__agent_path = agent_path
        self.__srm_path = \
            os.path.join(
                self.__agent_path, 
                '_work', 
                'SourceRootMapping')

        _LOGGER.debug("Agent SRM path: [%s]", self.__srm_path)

    def collection_gen(self):
        for filename in os.listdir(self.__srm_path):
            filepath = os.path.join(self.__srm_path, filename)
            if os.path.isdir(filepath) is False:
                continue

            _LOGGER.debug("Collection GUID: [%s]", filename)
            yield filename, filepath

    def definition_gen(self, collection_path):
        for srm_id_raw in os.listdir(collection_path):
            _LOGGER.debug("SRM ID: [%s]", srm_id_raw)

            definition_info_filepath = \
                os.path.join(
                    collection_path, 
                    srm_id_raw, 
                    'SourceFolder.json')

            with open(definition_info_filepath) as f:
                yield json.load(f)

    def lookup_definition(self, definition_name):
        for collection_guid, collection_path in self.collection_gen():
            for definition in self.definition_gen(collection_path):
                current_definition_name = definition['definitionName']
                _LOGGER.debug("Definition: [%s]", current_definition_name)

                if current_definition_name != definition_name:
                    continue

                return definition

        raise ValueError("Could not find definition: {}".format(
                         definition_name))


def _configure_logging():
    sh = logging.StreamHandler()
    formatter = logging.Formatter('%(asctime)s [%(name)s %(levelname)s] %(message)s')
    sh.setFormatter(formatter)

    _LOGGER.addHandler(sh)
    _LOGGER.setLevel(logging.DEBUG)

if __name__ == '__main__':
    import argparse

    parser = argparse.ArgumentParser()

    parser.add_argument(
        'agent_path', 
        help="Agent path")

    parser.add_argument(
        'definition_name', 
        help="Build-definition name")

    args = parser.parse_args()

    _configure_logging()

    t = Tfs(args.agent_path)
    definition = t.lookup_definition(args.definition_name)

    print(definition['agent_builddirectory'])

Output:

D:\development\python>python tfs.py c:\tfs_build_agent ConsoleProject.Dev
2016-07-07 23:32:02,756 [__main__ DEBUG] Agent SRM path: [c:\tfs_build_agent\_work\SourceRootMapping]
2016-07-07 23:32:02,756 [__main__ DEBUG] Collection GUID: [2cf8d3cb-b8d4-49e1-bbdb-2aacf02f48c4]
2016-07-07 23:32:02,757 [__main__ DEBUG] SRM ID: [1]
2016-07-07 23:32:02,757 [__main__ DEBUG] Definition: [ConsoleProject.Dev]
1

Important Notes

The child directories of the SourceRootMapping directory will not exactly reflect the current build definitions. Some of them may represent build definitions that no longer exist.
One of the current work directories may be using an ID used by an old build-definition at some point in the past. So, if you are building a dictionary of work-directory IDs to build-definitions, you will have to use the build-definition’s ID or one of the timestamps described in the its JSON file to reconcile the latest build-definition to use that directory.

TFS 2015 Database Structure Investigation

July 1, 2016dustin

A TFS 2015 build agent maintains a list of directories where builds occur. Each of these directories is assigned a monotonically-incrementing number and has three children: “s” (“source” directory where your files are, usually, automatically checked-out for you), “b” (“binaries” directory where you can choose to stash files that aren’t automatically cleaned-up), and “a” (“artifact-staging” directory where you push files/directories that will be published). Each agent assigns its own IDs for each build definition.

I sought to figure out 1) how to lookup the mapping between the build-definitions and these agent-ID pairs, and 2) how to lookup the paths when you publish a build to be stored by the TFS server (rather than a fileshare). Every so often I wander into the TFS database to chip away at its elusive topology. I’ll briefly document my observations here for the benefit of myself and others. I’ll add on new information as I might encounter it in the future. I’m only currently concerned with build-definitions but any information that can be provided for release-definitions would also be welcomes.

There is a main database (e.g. “Tfs_Configuration”) and subordinate collection-specific database (e.e. “Tfs_DefaultCollection”).

[Build].[tbl_Build]

Information on individual builds including build numbers (e.g. 20160512.1) and “definition ID”.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
,[DataspaceId]
,[BuildId]
,[DefinitionId]
,[DefinitionVersion]
,[BuildNumber]
,[BuildNumberRevision]
,[RepositoryId]
,[BranchId]
,[SourceVersion]
,[Parameters]
,[Status]
,[QueueId]
,[QueueTime]
,[Priority]
,[StartTime]
,[FinishTime]
,[Reason]
,[Result]
,[RequestedFor]
,[RequestedBy]
,[ChangedOn]
,[ChangedBy]
,[OrchestrationId]
,[Deleted]
,[ValidationIssues]
,[QueueOptions]
,[KeepForever]
,[ChangesCalculated]
,[DeletedOn]
FROM [Tfs_DefaultCollection].[Build].[tbl_Build]

Record looks like:

PartitionId DataspaceId BuildId DefinitionId    DefinitionVersion   BuildNumber BuildNumberRevision RepositoryId    BranchId    SourceVersion   Parameters  Status  QueueId QueueTime   Priority    StartTime   FinishTime  Reason  Result  RequestedFor    RequestedBy ChangedOn   ChangedBy   OrchestrationId Deleted ValidationIssues    QueueOptions    KeepForever ChangesCalculated   DeletedOn

1   22  168 1   58  20160701.1  1   1   2   C31 {"system.debug":"false","BuildConfiguration":"release","BuildPlatform":"any cpu"}   2   1   2016-07-01 16:11:01.2247845 3   2016-07-01 16:11:03.5025120 2016-07-01 16:11:23.2632483 1   2   FD456772-708D-496C-9259-32596770CD73    FD456772-708D-496C-9259-32596770CD73    2016-07-01 16:11:23.450 02442124-655A-49B2-A6FB-20269EDEBEF6    E6ABEC05-179C-4406-91CB-E96B9B6CFD7C    0   NULL    NULL    0   1   NULL

[Build].[tbl_Definition]

Describes individual build-definitions, presumably. Includes “definition ID”, “definition version” (probably a pointer to the head version), “definition name”.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[DataspaceId]
      ,[DefinitionId]
      ,[DefinitionVersion]
      ,[DefinitionName]
      ,[Quality]
      ,[QueueId]
      ,[QueueStatus]
      ,[RepositoryId]
      ,[DefaultBranchId]
      ,[TriggerTypes]
      ,[Description]
      ,[BuildNumberFormat]
      ,[JobAuthorizationScope]
      ,[JobTimeout]
      ,[Comment]
      ,[Author]
      ,[CreatedOn]
      ,[ParentDefinitionId]
      ,[Options]
      ,[Repository]
      ,[Triggers]
      ,[Steps]
      ,[Variables]
      ,[Demands]
      ,[RetentionPolicy]
      ,[BadgeEnabled]
      ,[Deleted]
  FROM [Tfs_DefaultCollection].[Build].[tbl_Definition]

Record looks like:

PartitionId DataspaceId DefinitionId    DefinitionVersion   DefinitionName  Quality QueueId QueueStatus RepositoryId    DefaultBranchId TriggerTypes    Description BuildNumberFormat   JobAuthorizationScope   JobTimeout  Comment Author  CreatedOn   ParentDefinitionId  Options Repository  Triggers    Steps   Variables   Demands RetentionPolicy BadgeEnabled    Deleted

1   22  1   58  ConsoleProject.Dev  1   1   0   1   1   1   NULL    $(date:yyyyMMdd)$(rev:.r)   1   60  Added publish-to-server step.   FD456772-708D-496C-9259-32596770CD73    2016-07-01 16:10:57.070 NULL    [{"enabled":false,"definition":{"id":"7c555368-ca64-4199-add6-9ebaf0b0137d"},"inputs":{"multipliers":"[]","parallel":"false","continueOnError":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"a9db38f9-9fdc-478c-b0f9-464221e58316"},"inputs":{"workItemType":"1","assignToRequestor":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"57578776-4c22-4526-aeb0-86b6da17ee9c"},"inputs":{"additionalFields":"{}"}}]    {"properties":{"labelSources":"0","tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db\",\"mappingType\":\"map\",\"localPath\":\"\\\\\"},{\"serverPath\":\"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/Drops\",\"mappingType\":\"cloak\",\"localPath\":\"\\\\\"}]}"},"id":"$/","type":"TfsVersionControl","name":"d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","url":"http://dustin-pc:8181/tfs/DefaultCollection/","defaultBranch":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","rootFolder":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db","clean":"false","checkoutSubmodules":false}  NULL    [{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution $/TestProject/TestConsoleApplication.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestConsoleApplication.sln","msbuildArgs":"/target:publish /p:ApplicationVersion=1.0.2.2 /p:InstallUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:UpdateUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:PublishUrl=\\\\localhost\\clickonce\\consoletestfrombuild\\ /p:UpdateEnabled=true /p:UpdateMode=Foreground /p:ProductName=TestConsoleApplication /p:IsWebBootstrapper=false","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution $/TestProject/TestDatabase.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestDatabase.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Run script $/TestProject/TestScripts/TestEcho.cmd","task":{"id":"bfc8bf76-e7ac-4a8c-9a55-a944a9f632fd","versionSpec":"*"},"inputs":{"filename":"$/d67f9d95-2f6c-43f0-aa2f-6f7804fde7db/TestScripts/TestEcho.cmd","arguments":"","modifyEnvironment":"false","workingFolder":"","failOnStandardError":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\TestConsoleApplication\\bin\\Release","Contents":"**\\*","TargetFolder":"$(Build.ArtifactStagingDirectory)","CleanTargetFolder":"true","OverWrite":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)\\deployment\\Release\\Utility","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\Release\\Utility","Contents":"**\\*","TargetFolder":"$(Build.ArtifactStagingDirectory)\\deployment\\Release\\Utility","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Copy Files to: $(Build.ArtifactStagingDirectory)/TestDatabase1","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(Build.SourcesDirectory)\\TestDatabase1\\bin\\Release","Contents":"TestDatabase1.dacpac","TargetFolder":"$(Build.ArtifactStagingDirectory)/TestDatabase1","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Publish Artifact: drop","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"drop","ArtifactType":"FilePath","TargetPath":"\\\\localhost\\build_publish\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Publish Artifact: Published_$(Build.BuildNumber)","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"Published_$(Build.BuildNumber)","ArtifactType":"Container","TargetPath":"\\\\my\\share\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}}]  {"system.debug":{"value":"false","allowOverride":true},"BuildConfiguration":{"value":"release","allowOverride":true},"BuildPlatform":{"value":"any cpu","allowOverride":true},"DUSTINPASSWORD":{"value":null,"allowOverride":true,"isSecret":true}} NULL    [{"branches":["+refs/heads/*"],"artifacts":["build.SourceLabel"],"daysToKeep":10,"minimumToKeep":1,"deleteBuildRecord":true,"deleteTestResults":true}]  0   0

[Build].[tbl_DefinitionHistory]

Describes the history of changes to the definition. Includes “definition ID”, “definition version”, “definition name”, and several JSON blocks.

Note that the record I provided is the most recent available (in my local sandbox environment). Though I had just made a change to a definition, it was, curiously, not represented in this table (though many other, previous, changes ones were).

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[DataspaceId]
      ,[DefinitionId]
      ,[DefinitionVersion]
      ,[DefinitionName]
      ,[QueueId]
      ,[QueueStatus]
      ,[RepositoryId]
      ,[DefaultBranchId]
      ,[Description]
      ,[BuildNumberFormat]
      ,[JobAuthorizationScope]
      ,[JobTimeout]
      ,[Comment]
      ,[Author]
      ,[CreatedOn]
      ,[Options]
      ,[Repository]
      ,[Triggers]
      ,[Steps]
      ,[Variables]
      ,[Demands]
      ,[RetentionPolicy]
      ,[BadgeEnabled]
      ,[Deleted]
  FROM [Tfs_DefaultCollection].[Build].[tbl_DefinitionHistory]

Record looks like:

PartitionId DataspaceId DefinitionId    DefinitionVersion   DefinitionName  QueueId QueueStatus RepositoryId    DefaultBranchId Description BuildNumberFormat   JobAuthorizationScope   JobTimeout  Comment Author  CreatedOn   Options Repository  Triggers    Steps   Variables   Demands RetentionPolicy BadgeEnabled    Deleted

1   32  12  18  PWC Audit360 Simulation Build   1   0   2   9   NULL    $(date:yyyyMMdd)$(rev:.r)   1   60  NULL    FD456772-708D-496C-9259-32596770CD73    2016-05-27 18:45:03.117 [{"enabled":false,"definition":{"id":"7c555368-ca64-4199-add6-9ebaf0b0137d"},"inputs":{"multipliers":"[]","parallel":"false","continueOnError":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"a9db38f9-9fdc-478c-b0f9-464221e58316"},"inputs":{"workItemType":"16","assignToRequestor":"true","additionalFields":"{}"}},{"enabled":false,"definition":{"id":"57578776-4c22-4526-aeb0-86b6da17ee9c"},"inputs":{"additionalFields":"{}"}}]   {"properties":{"labelSources":"0","tfvcMapping":"{\"mappings\":[{\"serverPath\":\"$/d0942daa-ac39-462f-8115-fada54d8f780\",\"mappingType\":\"map\",\"localPath\":\"\\\\\"},{\"serverPath\":\"$/d0942daa-ac39-462f-8115-fada54d8f780/Drops\",\"mappingType\":\"cloak\",\"localPath\":\"\\\\\"}]}"},"id":"$/","type":"TfsVersionControl","name":"d0942daa-ac39-462f-8115-fada54d8f780","url":"http://dustin-pc:8181/tfs/DefaultCollection/","defaultBranch":"$/d0942daa-ac39-462f-8115-fada54d8f780","rootFolder":"$/d0942daa-ac39-462f-8115-fada54d8f780","clean":"false","checkoutSubmodules":false}  NULL    [{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Server.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Server.sln","nugetConfigPath":"$/d0942daa-ac39-462f-8115-fada54d8f780/nuget.config","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.SilverlightBuild.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.SilverlightBuild.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Database.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Database.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore Audit360.Queue.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"Audit360.Queue.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"NuGet restore MarketData.Database.sln","task":{"id":"333b11bd-d341-40d9-afcf-b32d5ce6f23b","versionSpec":"*"},"inputs":{"solution":"MarketData.Database.sln","nugetConfigPath":"","noCache":"false","nuGetRestoreArgs":"","nuGetPath":""}},{"enabled":true,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Server.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Server.sln","msbuildArgs":"/t:build;publish /p:DeployOnBuild=True /m:1 /p:DeployPrefix=F1","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.SilverlightBuild.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.SilverlightBuild.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Database.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Database.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution Audit360.Queue.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"Audit360.Queue.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Build solution MarketData.Database.sln","task":{"id":"71a9a2d3-a98a-4caa-96ab-affca411ecda","versionSpec":"*"},"inputs":{"solution":"MarketData.Database.sln","msbuildArgs":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","clean":"true","restoreNugetPackages":"true","vsVersion":"14.0","msbuildArchitecture":"x86","logProjectEvents":"true"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Test Assemblies **\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**","task":{"id":"ef087383-ee5e-42c7-9a53-ab56c98420f9","versionSpec":"*"},"inputs":{"testAssembly":"**\\$(BuildConfiguration)\\*test*.dll;-:**\\obj\\**","testFiltercriteria":"","runSettingsFile":"","overrideTestrunParameters":"","codeCoverageEnabled":"false","runInParallel":"false","vsTestVersion":"14.0","pathtoCustomTestAdapters":"","otherConsoleOptions":"","testRunTitle":"","platform":"$(BuildPlatform)","configuration":"$(BuildConfiguration)","publishRunAttachments":"true"}},{"enabled":false,"continueOnError":true,"alwaysRun":false,"displayName":"Publish symbols path: ","task":{"id":"0675668a-7bba-4ccb-901d-5ad6554ca653","versionSpec":"*"},"inputs":{"SymbolsPath":"","SearchPattern":"**\\bin\\**\\*.pdb","SymbolsFolder":"","SkipIndexing":"false","TreatNotIndexedAsWarning":"false","SymbolsMaximumWaitTime":"","SymbolsProduct":"","SymbolsVersion":"","SymbolsArtifactName":"Symbols_$(BuildConfiguration)"}},{"enabled":false,"continueOnError":false,"alwaysRun":true,"displayName":"Copy Files to: $(build.artifactstagingdirectory)","task":{"id":"5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c","versionSpec":"*"},"inputs":{"SourceFolder":"$(build.sourcesdirectory)","Contents":"**\\bin\\$(BuildConfiguration)\\**","TargetFolder":"$(build.artifactstagingdirectory)","CleanTargetFolder":"false","OverWrite":"false"}},{"enabled":false,"continueOnError":false,"alwaysRun":true,"displayName":"Publish Artifact: drop","task":{"id":"2ff763a7-ce83-4e1f-bc89-0ae63477cebe","versionSpec":"*"},"inputs":{"PathtoPublish":"$(build.artifactstagingdirectory)","ArtifactName":"drop","ArtifactType":"Container","TargetPath":"\\\\my\\share\\$(Build.DefinitionName)\\$(Build.BuildNumber)"}},{"enabled":false,"continueOnError":false,"alwaysRun":false,"displayName":"Run script $/PwcSimulation/Build/RMPostBuild.cmd","task":{"id":"bfc8bf76-e7ac-4a8c-9a55-a944a9f632fd","versionSpec":"*"},"inputs":{"filename":"$/d0942daa-ac39-462f-8115-fada54d8f780/Build/RMPostBuild.cmd","arguments":"/C:FB /E:F1","modifyEnvironment":"false","workingFolder":"","failOnStandardError":"false"}}] {"system.debug":{"value":"false","allowOverride":true},"BuildConfiguration":{"value":"FB","allowOverride":true},"BuildPlatform":{"value":"any cpu","allowOverride":true}}   NULL    [{"branches":["+refs/heads/*"],"artifacts":["build.SourceLabel"],"daysToKeep":10,"minimumToKeep":1,"deleteBuildRecord":true,"deleteTestResults":true}]  0   0

[dbo].[tbl_Workspace]

Appears to describe the workspaces that are currently defined for users, builds, etc..

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[OwnerId]
      ,[WorkspaceName]
      ,[WorkspaceVersion]
      ,[Type]
      ,[Comment]
      ,[CreationDate]
      ,[Computer]
      ,[PolicyOverrideComment]
      ,[LastAccessDate]
      ,[CheckInNoteId]
      ,[RefreshRecursive]
      ,[HasDeletedChanges]
      ,[SequentialId]
      ,[IsLocal]
      ,[PendingChangeSig]
      ,[FileTime]
      ,[LastMappingsUpdate]
      ,[ProjectNotificationId]
      ,[ItemIdCounter]
      ,[VersionStamp]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_Workspace]

Record looks like :

PartitionId WorkspaceId OwnerId WorkspaceName   WorkspaceVersion    Type    Comment CreationDate    Computer    PolicyOverrideComment   LastAccessDate  CheckInNoteId   RefreshRecursive    HasDeletedChanges   SequentialId    IsLocal PendingChangeSig    FileTime    LastMappingsUpdate  ProjectNotificationId   ItemIdCounter   VersionStamp

1   420000001   65CED833-E406-4B14-BDB2-51590FE7569C    ws_5_1  1   0   Created by Distributed Task - getCode   2016-06-29 03:47:10.363 DUSTIN-PC   NULL    2016-06-29 03:47:10.363 NULL    0   0   22  1   B2140B25-F70A-4B4D-BFB1-184703037010    0   2016-06-29 03:47:10.370 37  -1025   2.00

[dbo].[tbl_WorkingFolder]

This use of this table is unclear. It describes the paths on the disk where the build-processes for each definition are hosted, but they also include a “workspace ID”. This would make sense for individual builds but not for latent build-definitions (workspaces for builds are ephemeral, unless the build fails, and are only created once the build starts). This seems promising for where the mappings for the build directories are stored, except that there appears to be potentially many records for the same build-definitions.

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[ItemDataspaceId]
      ,[ProjectName]
      ,[ServerItem]
      ,[LocalItem]
      ,[MappingType]
      ,[CreationDate]
      ,[Depth]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_WorkingFolder]

Record looks like:

PartitionId WorkspaceId ItemDataspaceId ProjectName ServerItem  LocalItem   MappingType CreationDate    Depth

1   320000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\1\s\  1   2016-07-01 16:11:07.500 120
1   320000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-07-01 16:11:07.500 120
1   360000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\3\s\  1   2016-05-24 18:22:09.760 120
1   360000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-05-24 18:22:09.760 120
1   420000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\5\s\  1   2016-06-29 03:47:10.363 120
1   420000001   21  TestProject $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   NULL    0   2016-06-29 03:47:10.363 120

[dbo].[tbl_WorkingFolderHistory]

The purpose of this table is completely unclear, though it includes build location paths (just like WorkingFolder).

SELECT looks like:

SELECT TOP 1000 [PartitionId]
      ,[WorkspaceId]
      ,[ItemDataspaceId]
      ,[ServerItem]
      ,[LocalItem]
      ,[Active]
  FROM [Tfs_DefaultCollection].[dbo].[tbl_WorkingFolderHistory]

Record looks like:

PartitionId WorkspaceId ItemDataspaceId ServerItem  LocalItem   Active

1   320000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\1\s\  1
1   320000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\Drops\   C:\tfs>build>agent\>work\1\s\Drops\    0
1   360000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\3\s\  1
1   400000001   31  $\d0942daa"ac39"462f"8115"fada54d8f780\ C:\tfs>build>agent\>work\7\s\  0
1   400000001   31  $\d0942daa"ac39"462f"8115"fada54d8f780\Drops\   C:\tfs>build>agent\>work\7\s\Drops\    0
1   420000001   21  $\d67f9d95"2f6c"43f0"aa2f"6f7804fde7db\ C:\tfs>build>agent\>work\5\s\  1

“Invalid Credentials” When Using TFX Tool to Login to TFS

May 20, 2016dustin

The cross-platform TFX tool is, among other things, what you use to create new boiler-plate VSIX extensions (to populate with your own information) as well as to upload individual Release Management (vNext) tasks to TFS 2015. Previously, I’ve logged in using a fully-qualified UNC-formatted username (\domain\dustin.oprea) but this started, suddenly, not working for me. I have not applied any noteworthy updates. I did recently have to change my password (per group policy), but that may or may not have affected it.

As my web-based TFS UI indicated that I was logged-in as “DOMAIN\dustin.oprea”, I unsuccessfully tried logging in as that at the command-line. Eventually, I opened up a private session within the browser in order to log-in using that same username but the UI refused to log me out (even though it was a private session, I was always logged-in). I tried clearing web credential in Windows, but, alas, nothing was stored there (if that could’ve even helped).

I also restarted the system just in case there was credentials-related caching in the system, in or outside of TFS’ application pool.

It turns out that now logging-in with a fully-qualified email-account as my username works just fine.

For example:

C:\>tfx login --auth-type basic --service-url http://localhost:8181/tfs/DefaultCollection --username dustin@email.com
TFS Cross Platform Command Line Interface v0.3.20
Copyright Microsoft Corporation
> Password:
Logged in successfully

Thanks, TFS. Do you think you might unify how users are logging-in both at the command-line and in the UI? Maybe?

Thanks to Could Not Connect, Invalid Credentials, Bad Username/ID/Password for drawing my eye to this potential option.

Publishing with ClickOnce via Visual Studio and TFS 2015 (vNext)

May 9, 2016dustin

ClickOnce equips your published application to be able to check for updates and to, optionally, do the update automatically. See this post for more information.

To invoke this functionality from either the command-line or your build definition/process, you need to pass some additional project parameters to MSBuild.

ApplicationVersion (e.g. 1.0.1.6)
InstallUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
UpdateUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
PublishUrl (e.g. http://clickonce.localhost/consoletestfrombuild/)
UpdateEnabled (e.g. true)
UpdateMode (e.g. Foreground)
ProductName (e.g. TestConsoleApplication)

There is some disagreement as to whether the “ProductName” parameter is necessary.

The corresponding set of MSBuild arguments using the example values:

/target:publish /p:ApplicationVersion=1.0.1.6 /p:InstallUrl=http://clickonce.localhost/consoletestfrombuild/ /p:UpdateUrl=http://clickonce.localhost/consoletestfrombuild/ /p:PublishUrl=http://clickonce.localhost/consoletestfrombuild/ /p:UpdateEnabled=true /p:UpdateMode=Foreground /p:ProductName=TestConsoleApplication

If your polling locations are going to be UNCs (“\\path\\file”) rather than URLs then you’ll have to pass an additional argument:

/p:IsWebBootstrapper=false

To configure this under a TFS 2015 build-definition, simply add the parameters to the “MSBuild Arguments” field:

ClickOnce Settings in TFS 2015

References

Building ClickOnce Applications from the Command Line (Microsoft)

Deploying a click-once application with build vNext in Azure Blob Storage

Using Tokenization (Token Replacement) for Builds/Releases in vNext/TFS 2015

May 6, 2016dustin

Microsoft has provided support for doing token replacements into build artifacts. Since this is provided as a “utility” task there’s no bias in how it’s used but you’ll likely find it most useful from your release workflow.

Inexplicably, Microsoft doesn’t actually provide this with TFS. Rather, you need to use their marketplace to download and install it. Still, as far as boneheaded designs go, it’s not so bad.

There are three things that it allows you to do:

Do string-replacements using the environment.
Do string-replacements using the environment-specific dictionary from the configuration file.
Do environment-specific XPath replacements.

Note that:

(1) and (2) will expect to find tokens that look like “[A-Za-z0-9._-]*” (e.g. “XYZ“, “ABC_DEF“, “GHI.JKL“, “MNO-PQR“), remove the underscores on the margins, and translate the periods to underscores.
(2) and (3) are only possible if you provide a configuration file and if the source file is XML (obviously).

Installation

Download the Release Management Utility tasks extension (as a VSIX file). Install it using the instructions from my other post. You’ll now see a “Tokenize with XPath/Regular expressions” utility task.

For reference, this is the original project URL:

Extension-UtilitiesPack

Usage

The configuration file looks like:

{
   "Default Environment":{
      "CustomVariables":{
         "Token2":"value_from_custom2",
         "Token3":"value_from_custom3"
      },
      "ConfigChanges":[
         {
            "KeyName":"/configuration/appSettings/add[@key='TestKey1']",
            "Attribute":"value",
            "Value":"value_from_xpath"
         }
      ]
   }
}

If we’re going to do token-replacements over an App.settings file, the source file might look like:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <appSettings>
      <add key="TestKey1" value="__Token1__" />
      <add key="TestKey2" value="__Token2__" />
      <add key="TestKey3" value="__Token3__" />
      <add key="TestKey4" value="__Token4__" />
    </appSettings>
    <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5.2" />
    </startup>
</configuration>

Since TestKey4 is not otherwise defined, it will either be replaced with itself (no net difference) or the value for an environment-variable named “Token4” if defined.

In my test situation, my build definition’s “Copy Files” task is copying the contents (“**\*”) of “$(Build.SourcesDirectory)\TestConsoleApplication\bin\Release” to “$(Build.ArtifactStagingDirectory)”. This is an example of the arguments to the “Tokenize with XPath/Regular expressions” task that I’ve added in my release-definition:

Source filename: “$(Build.ArtifactStagingDirectory)\drop\App.config.template”
Destination filename: “$(Build.ArtifactStagingDirectory)\drop\TestConsoleApplication.exe.config”
Configuration Json filename: “$(Build.ArtifactStagingDirectory)\drop\TokenEnvironmentConfig.json”

Personally, I’d prefer that parsing this be a little stricter: fail if can’t parse as JSON, fail if any tokens can not be resolved, and optionally allow it to fail if any XPath specifications don’t match anything.

Additional Notes

The RELEASE_ENVIRONMENTNAME variable is usually defined and used to determine the current environment. However, if it isn’t defined then the default environment searched in the configuration file is “default”.
The CustomVariables and ConfigChanges blocks don’t have to exist in the configuration file (won’t fail validation). What is there will be used.
For debugging, I’ve had issues enabling logging verbosity in PowerShell (with VerbosePreference). It would be easier adjusting the tasks\Tokenizer\x.y.z\tokenize.ps1 module in your build-agent’s directory to change all the instances of “Write-Verbose” to “Write-Host”. The module provides a lot of useful messaging if you can get it to show.
In the current version of the task, it provides a PowerShell 3 script in addition to the original PS script. I get the following message on my system: ##[error]Index operation failed; the array index evaluated to null. I downloaded the source-code to the VSIX that is available in the Marketplace, adjusted it to not use the PS3 version of the script, repackaged the VSIX, uploaded it to my on-premise TFS, and, bam, the token-replacement worked perfectly. The PowerShell 3 script is broken.

TFS Release Management Build Agent Doesn’t Pickup Jobs

May 6, 2016dustin

If your build tells you “Build Not started”, “Waiting for an available agent”, or “Waiting for an agent to be requested”, make sure that your agent is running (if as a service, then it’s called “VSO Agent”) as well as the “Visual Studio Team Foundation Background Job Agent” service. The latter feeds the former. In my case, I am using TFS 2015 and the latter appeared to be configured as “Automatic (Delayed Start)” but had to be configured as “Automatic” instead.

Uploading Release Management Tasks With a VSIX Extension Package

May 5, 2016dustin 1 Comment

TFS 2015 lets you define build and release workflows comprised of discrete tasks. Although it provides a handful to you (build-specific, release-specific, general utility, etc..), you can also write your own “Extension Tasks” and upload them. I previously described how to use Microsoft’s tfx tool to upload a single task from the command-line.

You might also need to upload one or more tasks that have been packaged into a VSIX file (Microsoft’s Visual Studio Extension packaging format). These are essentially zip-files that have a manifest and one or more directories. In this case, those directories represent tasks (they each have a task.json file). Although you can probably do this from the command-line using the tfx tool, you can also use your TFS’s web UI to do it. Note that this process will need to be repeated for each collection that you’ll want the extension to be available in.

Click the “Shop” icon to open the corresponding menu and then click on “Browse of Extensions”:

Click on the “Manage extensions” button in the middle of the page. It’s not the same link/button as the previous one even if they have the same text:

release-management-extension-upload-3-manage-extensions-2.png

Click on “Upload New Extension”. Browse to the VSIX file and upload it.

release-management-extension-upload-4-extension-upload.png

After, you’re presented with a list of installed extensions:

release-management-extension-upload-5-extension-uploaded.png

Observe that the extension populated on to the previous local-management screen. Click on it:

Now, click on “Install”:

After installation, the new tasks will be available within the lists of available tasks.

Adding a Task to Your On-Premise TFS 2015 Server

May 5, 2016dustin 1 Comment

TFS 2015 embeds previous vNext functionality to provide you workflow-driven, scriptable builds. This is based on establishing discrete tasks that can be reused between your build definitions and release definitions as relevant.

You can extend the basic set of tasks, which, though sufficient for very simple builds and releases, might quickly become insufficient. New tasks are called “extension tasks”. Discussing how to create one is out of scope for this post so we’re assuming that you already have a path that describes a task. At the very least, there will be a task.json file in it.

What comes next?

Install Microsoft’s “tfx” tool.
Added support for “Basic Authentication” to your on-premise TFS’ IIS server.
Enable Basic Authentication on that IIS server.
Login to your TFS using the tfx tool.
Upload task.

Instructions

1. Install Microsoft’s “tfx” tool

This is a cross-platform project that requires NodeJS to be installed locally. The TFS-CLI project is hosted here: https://github.com/Microsoft/tfs-cli.

To install:

npm install -g tfx-cli

2. Add support for “Basic Authentication” to your on-premise TFS’ IIS server

On a workstation flavor of Windows (in my case, Windows 8.1):

Go to “Programs and Features” -> “Turn Windows features on or off”.
Navigate down through the tree of features: “Internet Information Services” -> “World Wide Web Services” -> “Security”.
Enable “Basic Authentication” and click “OK”.

If you’re running a server flavor of Windows you’ll have to update the “Web Server” role instead:

(https://github.com/Microsoft/tfs-cli/blob/master/docs/configureBasicAuth.md)

Make sure to close and re-open IIS Manager afterward.

3. Enable Basic Authentication in IIS

Under the application, click “Authentication”, right-click on “Basic Authentication”, and click “Enable”. Make sure you enable “Basic Authentication” under the application, not the site. Setting it on the site will be ineffectual.

4. Login to your TFS using the tfx tool

C:\>tfx login --auth-type basic
TFS Cross Platform Command Line Interface v0.3.20
Copyright Microsoft Corporation
> Service URL: http://localhost:8181/tfs/DefaultCollection
> Username: yourdomain\dustin.oprea
> Password:
Logged in successfully

5. Upload task

I tested using the “XLD” task, here:

XL Deploy

Unzip and install by passing the directory path:

tfx build tasks upload --task-path xld

Now, if you go to your list of release tasks (where the XLD task ends-up), you’ll see it:

Random Engineering

Gotta figure that out.

Widgets

Search

tfs 2015

TFS Tasks Do Not Appear When Uploading a VSIX

Setting Environment Variables From PowerShell Tasks Under TFS 2015

Converting TFS 2015 Build Definition Names to Their Directory IDs

TFS 2015 Database Structure Investigation

[Build].[tbl_Build]

[Build].[tbl_Definition]

[Build].[tbl_DefinitionHistory]

[dbo].[tbl_Workspace]

[dbo].[tbl_WorkingFolder]

[dbo].[tbl_WorkingFolderHistory]

“Invalid Credentials” When Using TFX Tool to Login to TFS

Publishing with ClickOnce via Visual Studio and TFS 2015 (vNext)

References

Using Tokenization (Token Replacement) for Builds/Releases in vNext/TFS 2015

Installation

Usage

Additional Notes

TFS Release Management Build Agent Doesn’t Pickup Jobs

Uploading Release Management Tasks With a VSIX Extension Package

Adding a Task to Your On-Premise TFS 2015 Server

Instructions

1. Install Microsoft’s “tfx” tool

2. Add support for “Basic Authentication” to your on-premise TFS’ IIS server

3. Enable Basic Authentication in IIS

4. Login to your TFS using the tfx tool

5. Upload task

Share this:

Share this:

Share this:

[Build].[tbl_Build]

[Build].[tbl_Definition]

[Build].[tbl_DefinitionHistory]

[dbo].[tbl_Workspace]

[dbo].[tbl_WorkingFolder]

[dbo].[tbl_WorkingFolderHistory]

Share this:

Share this:

References

Share this:

Installation

Usage

Additional Notes

Share this:

Share this:

Share this:

Instructions

1. Install Microsoft’s “tfx” tool

2. Add support for “Basic Authentication” to your on-premise TFS’ IIS server

3. Enable Basic Authentication in IIS

4. Login to your TFS using the tfx tool

5. Upload task

Share this: