Getting Git submodules in private Azure DevOps repositories to work in a pipeline

Introduction

Recently, I embarked on a journey to tackle some Infrastructure as Code and Azure DevOps challenges. My solution? Git submodules. This approach proved practical and efficient, making it a fitting choice for my objectives.

My journey with Azure DevOps and Git submodules was not without its hurdles. However, overcoming these challenges led to a functional pipeline and ensured all security options were intact. With some perseverance and problem-solving, you can accomplish this as well.

A quick overview of the setup

I have a main repository where our pipeline lives. That repo has two Git submodules added, which link to their repositories in the same Azure DevOps project.

Adding the submodules to the main repos was done as follows while in the root of our main Git repo.

git submodule add
https://workinghardinit@dev.azure.com/workinghardinit/InfraAsCode/_git/AzureFwChildPolMarShip
.\IAC\Up\Shared\bicep\AzureFwChildPolMarShip

git submodule add ../AzureFwChildPolMarShip
.\IAC\Up\Shared\bicep\AzureFwChildPolMarShip

That is as documented in Git – git-submodule Documentation (git-scm.com), but doing this led to my first problem.

The folder structure of the repositories looks like the one in the picture below when cloned to my workstation.

It shows up in the main Azure DevOps repo as a reference to the submodules repo with a commit ID.

Since I need the files in the submodules to be cloned and made available during our deployment of Azure infrastructure, I added the “submodules: true” key to the checkout step in the checkout stage of my YAML pipeline.

Those are the relevant details of the repositories and the pipeline so far. At least, that’s what I started with. It was not enough.

Problem 1 – The checkout stage fails with an error when cloning the submodules

While running my pipeline, I encountered a problem with some errors that were confusing to me. The pipeline is trying to prompt me for a password, but it can’t. But I authenticated already, and it should not prompt me.

Cloning into ‘D:/a/1/s/IAC/Up/Shared/bicep/AzureFwChildPolFleetMgnt’…
fatal: Cannot prompt because user interactivity has been disabled.
fatal: Cannot prompt because user interactivity has been disabled.
fatal: could not read Password for ‘https://workinghardinit@dev.azure.com/workinghardinit/InfraAsCode/_git/AzureFwChildPolFleetMgnt’: terminal prompts disabled
fatal: clone of
‘https://workinghardinit@dev.azure.com/workinghardinit/InfraAsCode/_git/AzureFwChildPolFleetMgnt’
into submodule path ‘D:/a/1/s/IAC/Up/Shared/bicep/AzureFwChildPolFleetMgnt’ failed
Failed to clone ‘IAC/Up/Shared/bicep/AzureFwChildPolFleetMgnt’. Retry scheduled

This one had me scratching my head. I mean, my repo and the repositories of the submodules are all in the same Azure DevOps project. That means there should not be an issue of permissions. If the submodules Azure Repos Git repositories are one or more different projects than your pipeline, and “Limit job authorization scope to current project for non-release pipelines” is set to On (the default) for your YAML, you must grant permission to the build service identity for your pipeline to the second project. See Understand job access tokens – Azure Pipelines | Microsoft Learn. But we are OK here.

So what gives? After a lot of trial and error, it hit me. These are all private Azure DevOps repositories, and I read something about that somewhere. So, I waded through the docs again, found the nugget of information I needed, and only now started to understand. Read along in Pipeline options for Git repositories under the Checkout submodules section.

Yes, I added the submodules following examples I read in many git examples and documents, which works fine for public repositories without authentication.

Sure, I was authenticated and did not leave my Azure DevOps organization, not even my project. As I said, all the repositories are part of the same project.

But my URLs for the submodules in the file .gitmodules were pointing to the external ones. I don’t have access as authentication is not even requested, and the access fails with the above error in my pipeline. So, I edited my .gitmodule file and changed the URLs to relative paths, as the documentation states.

How can we avoid this? Well, instead of adding a submodule using the full URL path you find in Azure DevOps “Clone Repository” button like below …

git submodule add https://workinghardinit@dev.azure.com/workinghardinit/InfraAsCode/_git/AzureFwChildPolMarShip .\IAC\Up\Shared\bicep\AzureFwChildPolMarShip

use the relative path

git submodule add ../AzureFwChildPolMarShip .\IAC\Up\Shared\bicep\AzureFwChildPolMarShip

Doing so means you get it right when working with private repositories the first time. After I did that, the checkout stage in my pipeline started working. Hurrah!

BONUS TIP. What if the submodule’s remote repository lives in the same Azure DevOps organization but in a different project? With the full URL path, you’d have the same issue. Well, you can still use a relative path, but you need to specify more of it. Instead of just using ../SubModuleRepoInSameProject, use ../../../OtherProject/_git/SubModuleInOtherProject for the url.

See the example and the image below.

git submodule add ../../../ProjectTwo/_git/SubRepoProjectTwo .\MySubModules\SubRepoProjectTwo

Problem 2 – The checkout stage fails with an error when cloning the submodules

So, we solved problem one, only to reveal problem two. The checkout kept failing but with a different error. Cloning the submodules fails, stating it does not exist or lacks proper permissions,

TF401019: The Git repository with name or identifier AzureFwChildPolMarShip does not exist or you do not have permissions for the operation you are attempting.

Below is what it looks like in the pipeline.

When googling this error, I found that I could solve this problem by changing the project settings for our pipelines.

• Navigate to Project Setting
• Under Pipelines, go to Settings
• Set Protect access to repositories in YAML pipelines to Off

When I reran the pipeline, the checkout succeeded. Cool, but I do not want to do this! I like that “Set Protect access to repositories in YAML pipelines” is enabled. So how do I achieve that?

Well, you need to do a couple of things.

Use a repository resource to reference an additional repository in your pipeline

The explanation is in resources.repositories.repository definition | Microsoft Learn, but it took me a while to find and understand it all. The repository keyword lets you specify an external repository. Use a repository resource to reference an additional repository in your pipeline.

Add a uses statement to your checkout stage

In Build Azure Repos Git repositories – Azure Pipelines | Microsoft Learn, you read another part of the solution. Your pipeline code must have a reference to the submodule repositories as we defined them in resources.repositories.repository in stages.stage.jobs.job.uses.repositories.

The latter is critical. With the “Protect access to repositories in YAML pipelines” enabled, we must explicitly reference the Azure Repo Git repositories of those submodules. That way, we can clone the code we want to use in the pipeline as a checkout step for the job that uses the repository. Git commands such as cloning fail if we do not do this.

Note what Microsoft has to say about this.

Protect access to repositories in YAML pipelines is enabled by default for new organizations and projects created after May 2020. When Protect access to repositories in YAML pipelines is enabled, your YAML pipelines must explicitly reference any Azure Repos Git repositories you want to use in the pipeline as a checkout step in the job that uses the repository. You won’t be able to fetch code using scripting tasks and git commands for an Azure Repos Git repository unless that repo is first explicitly referenced.

Note the conditions when you don’t need to do this and realize that we do need it here:

If your pipeline does not have an explicit checkout step, it behaves as if it has a “checkout: self” step and the self repository has checked out.

Suppose you are using a script to perform read-only operations on a repository in a public project. In that case, you don’t need to reference the public project repository during checkout.

If you use a script that provides authentication to the repo, such as a PAT, you don’t need to reference that repository in a checkout step.

Also note that instead of a uses statement, we could check out the Git repositories of the submodules, but I chose this route. Go through motions of Git add ., git commit -m “fixed references to submodule repositories in pipeline” and git push. Once we have done that, we set back “Protect access to repositories in YAML pipelines” to On to enable it.

Now, rerun the pipeline. Watch the output window, as you need to permit access to both submodule repositories.

Once you permit access, the pipeline run can be completed successfully with your security settings intact.

That’s it! That concludes our demo on getting Git submodules in private Azure DevOps repositories to work in a pipeline.

Overview

To recapitulate, these are the steps we needed to make things work:

1. The submodules and their Git repos exist in the same project as the Git repo to which you added the submodules. If these are in different projects, we need to take care of security settings in DevOps to make this work.
2. We add the submodules as a URL relative to the main repository.
3. Your pipeline code must have the submodules key set to true under stages: – stage: checkout jobs: – job: steps: – checkout: self.
4. Your pipeline must reference the submodule repositories in resources.repositories.repository.
5. You need to leverage the use statement and refer to the submodule repositories you defined in step 4.

BONUS TIP. Remember that a submodule’s remote repository lives in the same Azure DevOps organization but in a different project? We used a relative path like this while adding the submodule ../../../OtherProject/_git/SubModuleInOtherProject for the url. Well in your pipeline reference the name with the Project/RepoName and you’ll be fine.

See the example in the image below.

In your main repository’s project setting, you must set the option “Limit job authorization scope to the current project for non-release pipelines” to Off to avoid permission issues!

Conclusion

This article has become my public documentation on getting the checkout stage in my Azure DevOps pipeline to work for a private repo with submodules pointing to other private repos. In this case, all repositories are part of the same project. Figuring out how to clone a submodule in an Azure DevOps pipeline took me an extremely long day.

So, to never forget and to find it again if I ever forget it, I wrote it all down in this article and shared it with you.

What can I say? Writing documentation about Git and pipelines is hard. Reading and understanding that documentation is also challenging. Finding, combining., linking, and understanding various pieces of information everywhere is even more difficult. It requires time and persistence combined with trial and error. That sucks until you find the solution. Then, you are delighted. Don’t linger on the effort, and document your findings! Maybe, one day, I can help a reader of this article save time and effort. And yes, that reader could be me.