Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for private git repositories #21

Closed
nhhagen opened this issue Oct 16, 2013 · 14 comments
Closed

Support for private git repositories #21

nhhagen opened this issue Oct 16, 2013 · 14 comments

Comments

@nhhagen
Copy link
Contributor

nhhagen commented Oct 16, 2013

When building from private git repos and the master branch is not yet tracked or up to date the builds fails with HTTP 401 (not authenticated) when trying to fetch the branch (TC only fetches the branch it is going to build)

For quick solution user/pass could be read from TC parameters before passing to libgit2sharp

Hopefully libgit2sharp can get propper ssh support and this will not be a problem.

@nhhagen
Copy link
Contributor Author

nhhagen commented Oct 16, 2013

Here is the build log for a private build over https

[20:31:07]Checking for changes
[20:31:09]Publishing internal artifacts
[20:31:09][Publishing internal artifacts] Sending build.start.properties.gz file
[20:31:09]Clearing temporary directory: D:\TeamCity\buildAgent\temp\buildTmp
[20:31:09]Checkout directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09]Updating sources: agent side checkout (4s)
[20:31:09][Updating sources] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist
[20:31:09][Updating sources] Cleaning D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09][Updating sources] VCS Root: [email protected]:comperio/git-flow-test.git (4s)
[20:31:09][VCS Root: [email protected]:comperio/git-flow-test.git] revision: 4e3405146e9e577271b6d81753fcf330a5ef49e7
[20:31:09][VCS Root: [email protected]:comperio/git-flow-test.git] Cleaning D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:09][VCS Root: [email protected]:comperio/git-flow-test.git] The .git directory is missing in 'D:\TeamCity\buildAgent\work\1fb6f465af877b6d'. Running 'git init'...
[20:31:09][VCS Root: [email protected]:comperio/git-flow-test.git] Commit '4e3405146e9e577271b6d81753fcf330a5ef49e7' is not found in repository. Running 'git fetch'...
[20:31:13][VCS Root: [email protected]:comperio/git-flow-test.git] Checking out branch refs/heads/develop in [email protected]:comperio/git-flow-test.git in D:\TeamCity\buildAgent\work\1fb6f465af877b6d with revision 4e3405146e9e577271b6d81753fcf330a5ef49e7
[20:31:13][VCS Root: [email protected]:comperio/git-flow-test.git] Cleaning [email protected]:comperio/git-flow-test.git in D:\TeamCity\buildAgent\work\1fb6f465af877b6d the file set ALL_UNTRACKED
[20:31:13]Step 1/3: NuGet Installer
[20:31:13][Step 1/3] scan: Searching for nuget.config files
[20:31:13][scan] Found packages folder: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages
[20:31:13][scan] Found list of packages.config files: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages\repositories.config
[20:31:13][scan] Scanning projects in solution file: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test.sln
[20:31:13][scan] Found project D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj
[20:31:13][Step 1/3] install: Installing NuGet packages for git-flow-test\packages.config
[20:31:13][install] NuGet command: D:\TeamCity\buildAgent\tools\NuGet.CommandLine.DEFAULT.nupkg\tools\NuGet.exe install D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\packages.config -OutputDirectory D:\TeamCity\buildAgent\work\1fb6f465af877b6d\packages -Source https://nuget.org/api/v2/ -Source https://comperio.myget.org/F/comperio/
[20:31:13][install] Starting: D:\TeamCity\buildAgent\temp\agentTmp\custom_script8657142140777516706.cmd
[20:31:13][install] in directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test
[20:31:14][install] Installing 'GitFlowVersionTask 0.4.0'.
[20:31:14][install] Successfully installed 'GitFlowVersionTask 0.4.0'.
[20:31:14][install] Process exited with code 0
[20:31:14]Step 2/3: Visual Studio (sln) (2s)
[20:31:14][Step 2/3] Starting: D:\TeamCity\buildAgent\plugins\dotnetPlugin\bin\JetBrains.BuildServer.MsBuildBootstrap.exe /workdir:D:\TeamCity\buildAgent\work\1fb6f465af877b6d /msbuildPath:C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe
[20:31:14][Step 2/3] in directory: D:\TeamCity\buildAgent\work\1fb6f465af877b6d
[20:31:15][Step 2/3] git-flow-test\git-flow-test.csproj.teamcity: Build target: Rebuild
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] CoreClean
[20:31:15][CoreClean] MakeDir
[20:31:15][MakeDir] Creating directory "obj\Debug\".
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] PrepareForBuild
[20:31:15][PrepareForBuild] MakeDir
[20:31:15][MakeDir] Creating directory "bin\Debug\".
[20:31:15][git-flow-test\git-flow-test.csproj.teamcity] UpdateAssemblyInfo
[20:31:15][UpdateAssemblyInfo] GitFlowVersionTask.UpdateAssemblyInfo
[20:31:16][GitFlowVersionTask.UpdateAssemblyInfo] Version not in cache. Calculating version.
[20:31:16][GitFlowVersionTask.UpdateAssemblyInfo] No local branch with name master found, going to try on the remote origin(https://[email protected]/comperio/git-flow-test.git)
[20:31:16]
[GitFlowVersionTask.UpdateAssemblyInfo] Error occurred: LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error).
Request failed with status code: 401
   at LibGit2Sharp.Core.Ensure.HandleError(Int32 result)
   at LibGit2Sharp.Core.Proxy.git_remote_connect(RemoteSafeHandle remote, GitDirection direction)
   at LibGit2Sharp.Network.Fetch(Remote remote, Nullable`1 tagFetchMode, ProgressHandler onProgress, CompletionHandler onCompletion, UpdateTipsHandler onUpdateTips, TransferProgressHandler onTransferProgress, Credentials credentials)
   at GitFlowVersion.LibGitExtensions.GetBranch(IRepository repository, String name) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\LibGitExtensions.cs:line 129
   at GitFlowVersion.VersionOnMasterFinder.<VersionsOnMaster>d__3.MoveNext() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 71
   at GitFlowVersion.VersionOnMasterFinder.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 37
   at GitFlowVersion.DevelopVersionFinder.GetSemanticVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 36
   at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13
   at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44
   at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18
   at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39
   at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57
[20:31:16][git-flow-test\git-flow-test.csproj.teamcity] Project git-flow-test\git-flow-test.csproj.teamcity failed.
[20:31:16][Step 2/3] Process exited with code 1
[20:31:16][Step 2/3] MSBuild output
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] Done Building Project "D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity" (Rebuild target(s)) -- FAILED.
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] Build FAILED.
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] "D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity" (Rebuild target) (1) ->
[20:31:16][MSBuild output] (UpdateAssemblyInfo target) -> 
[20:31:16][MSBuild output]   MSBUILD : error : Error occurred: LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error). [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error : Request failed with status code: 401 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Core.Ensure.HandleError(Int32 result) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Core.Proxy.git_remote_connect(RemoteSafeHandle remote, GitDirection direction) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at LibGit2Sharp.Network.Fetch(Remote remote, Nullable`1 tagFetchMode, ProgressHandler onProgress, CompletionHandler onCompletion, UpdateTipsHandler onUpdateTips, TransferProgressHandler onTransferProgress, Credentials credentials) [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.LibGitExtensions.GetBranch(IRepository repository, String name) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\LibGitExtensions.cs:line 129 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionOnMasterFinder.<VersionsOnMaster>d__3.MoveNext() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 71 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionOnMasterFinder.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionOnMasterFinder.cs:line 37 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.GetSemanticVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 36 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.DevelopVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\BranchFinders\DevelopVersionFinder.cs:line 13 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.GitFlowVersionFinder.FindVersion() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\GitFlowVersionFinder.cs:line 44 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionForRepositoryFinder.GetVersion(Repository repository) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionForDirectoryFinder.cs:line 18 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersion.VersionCache.GetVersion(String gitDirectory) in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersion\VersionCache.cs:line 39 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] MSBUILD : error :    at GitFlowVersionTask.UpdateAssemblyInfo.Execute() in c:\BuildAgent\work\5e0d79a4e3943b17\GitFlowVersionTask\UpdateAssemblyInfo.cs:line 57 [D:\TeamCity\buildAgent\work\1fb6f465af877b6d\git-flow-test\git-flow-test.csproj.teamcity]
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output]     0 Warning(s)
[20:31:16][MSBuild output]     1 Error(s)
[20:31:16][MSBuild output] 
[20:31:16][MSBuild output] Time Elapsed 00:00:00.68
[20:31:16][Step 2/3] Step Visual Studio (sln) failed
[20:31:16]Step 3/3: NuGet Pack
[20:31:16][Step 3/3] Build step NuGet Pack is skipped because the previous step has failed
[20:31:16]Publishing artifacts
[20:31:16][Publishing artifacts] Collecting files to publish: [D:\TeamCity\buildAgent\temp\buildTmp\nuget24305130608310186packages\nuget.xml => .teamcity/nuget]
[20:31:16][Publishing artifacts] Sending D:/TeamCity/buildAgent/temp/buildTmp/nuget24305130608310186packages/nuget.xml
[20:31:16]Publishing internal artifacts
[20:31:16][Publishing internal artifacts] Sending build.finish.properties.gz file
[20:31:16]Build finished

@JustinThirkell
Copy link

i've come across exactly the same error.
@SimonCropp, @andreasohlund , how are you handling this on the Particular TC server?

for the moment i'm hacking in a post-versioncontrol, pre-build step to ensure the master branch exists for a develop branch build.
This basically does the following (i'm yet to handle case where the master branch already exists on the build server from a previous build):

  • call git remote set-branches origin --add master
  • call git fetch origin
  • call git branch --track master origin/master
    (we've got a private github enterprise and using ssh exclusively to get to it from our TC)

i'm guessing since you guys (Particular) are using this for your develop branch builds you must have found a way to ensure master branch is available at build time...?

thanks

@andreasohlund
Copy link
Contributor

We automatically pull down the master branch and that works on TC as well (you need agent checkout enabled)

@JustinThirkell
Copy link

thanks for quick reply.

yep, i've got agent checkout enabled. (also found out how to disable shallow fetch for a repo which is our TC default!!)
from the logs it appears TC is issuing the following:
"C:\Program Files (x86)\Git\bin\git.exe" fetch --progress origin +refs/heads/develop:refs/heads/develop

how do you ensure master branch is auto pulled down too?

btw, i'm loving gitflowversion - very very cool idea.

@andreasohlund
Copy link
Contributor

We auto pull master as well

On Tue, Oct 22, 2013 at 8:36 AM, Justin Thirkell
[email protected]:

thanks for quick reply.

yep, i've got agent checkout enabled. (also found out how to disable
shallow fetch for a repo which is our TC default!!)
from the logs it appears TC is issuing the following:
"C:\Program Files (x86)\Git\bin\git.exe" fetch --progress origin
+refs/heads/develop:refs/heads/develop

how do you ensure master branch is auto pulled down too?

btw, i'm loving gitflowversion - very very cool idea.


Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-26780255
.

@JustinThirkell
Copy link

sorry, being dense here.
I understand you auto pull master branch down onto the build server (and how that makes it all work)

what i don't understand is how exactly you're doing that.
are you using a custom git step to checkout or using the builtin jetbrains.git vcs?

our setup is that we tell TC to checkout develop branch - can't see anywhere to tell TC to auto pull master as well...

we're using the jetbrains.git vcs, and fwiw, our vcs settings are:
fetch url = [email protected]:mycompany/MyRepo.git
default branch = %branch% (which is set as a per-buildconfig parameter)
branch specification = [currently empty]

@andreasohlund
Copy link
Contributor

This is how we do it

https://github.com/Particular/GitFlowVersion/blob/4a879843194ad858e76f9bb0502517e90a9d675d/GitFlowVersion/LibGitExtensions.cs#L43

So you don't need anything extra.

fetch url = [email protected]:mycompany/MyRepo.git
Think this might be the issue, can you try https instead?

On Tue, Oct 22, 2013 at 8:45 AM, Justin Thirkell
[email protected]:

sorry, being dense here.
I understand you auto pull master branch down onto the build server (and
how that makes it all work)

what i don't understand is how exactly you're doing that.
are you using a custom git step to checkout or using the builtin
jetbrains.git vcs?

our setup is that we tell TC to checkout develop branch - can't see
anywhere to tell TC to auto pull master as well...

we're using the jetbrains.git vcs, and fwiw, our vcs settings are:
fetch url = [email protected]:mycompany/MyRepo.git
default branch = %branch% (which is set as a per-buildconfig parameter)
branch specification = [currently empty]


Reply to this email directly or view it on GitHubhttps://github.com//issues/21#issuecomment-26780624
.

@JustinThirkell
Copy link

right, ok, that explains it.
yep, this in previous build logs:

[GitFlowVersionTask.UpdateAssemblyInfo] Error occurred: GitFlowVersion.MissingBranchException: Could not fetch from '[email protected]:Xero/Bus.git' since LibGit2 does not support the transport. You have most likely cloned using SSH. If there is a remote branch named 'master' then fetch it manually, otherwise please create a local branch named 'master'. ---> LibGit2Sharp.LibGit2SharpException: An error was raised by libgit2. Category = Net (Error).
This transport isn't implemented. Sorry

currently we're not allowed to use https - it's all ssh here. sigh.
above error message is why i'm having to manually pull down master branch via ssh.

so the answer is that Particular are using https to enable fetching master via LibGitExtensions.

think I'll be asking Ops to turn on https tomorrow. ;)
thanks for your help

@JustinThirkell
Copy link

oh, and hunting down ssh status for libgit2, look who turns up in the PR comments...
libgit2/libgit2#1558 (comment)
😄

@SimonCropp
Copy link
Contributor

I was just about to post that link.

So you need to ask the libgit2sharp guys about ssh on windows. Since we (particular) are happy using https we are not going to push it too hard.

@nhhagen
Copy link
Contributor Author

nhhagen commented Oct 22, 2013

https has not been a problem for me either, but a password protected https repo (e.g. private github/bitbucket repo) does not work on version 0.4.0 (as far as I've gotten).

I still use ssh and are using this workaround for now.

Add a new PowerShell step (the is a plugin for TC)

add this script:

$branch = . git symbolic-ref --short -q HEAD;. git pull; foreach ($remote in . git branch -r) { . git checkout $remote.Trim().Replace("origin/", ""); . git pull } . git checkout $branch

It will track and update all branches.

My set-up is currently

TC with 6 agents and private bitbucket repos accessed over ssh.

@JustinThirkell
Copy link

that's really helpful - thanks very much!

@JakeGinnivan
Copy link
Contributor

The command line version supports username and password being passed.

Are we happy to close this issue, also do we need an issue to track adding username/password support in the MSBuild task?
@andreasohlund @SimonCropp

@ghost
Copy link

ghost commented Aug 24, 2015

When running TeamCity on Windows it is better to replace

  • git

with

  • "%env.TEAMCITY_GIT_PATH%"

Or agents do not have git in the PATH variable.

However - SSH connections are a problem (unknown host key or missing rsa private key).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants