If you are looking for a way to avoid running the Azure pipeline agents with administrator privileges, I hope this short blog post might be of use to you.
The situation
Azure DevOps is a great tool and consists of a lot of tools. Today I’ll be talking about the Release Pipelines. More specifically, I’ll investigate the possibilities of locking down the pipeline agents.
Visually, the pipelines agents work as follows:
These pipeline agents are registered using a Powershell script that is generated when you create a new deployment group in Azure Devops. The problem with this Powershell command is that it registers the pipeline agent as a service running as NT AUTHORITY\SYSTEM. Something we should try to avoid, right?
Let’s see what we can do about it.
The requirements
In my case, these were the requiremnts:
- Able to use deployment groups using the autogenerated registration script
- Able to use the IIS web app manage and IIS web app deploy task
- Able to set environment variables using Powershell
- Able to schedule tasks using Powershell
The goal was to lock down the permissions of the Azure Pipeline Agent to the minimum required to execute the above listed tasks. I started Googling, pretty sure I would find a way to achieve this goal. I didn’t. It’s surprising to not find an answer about this. It seems like the principle of least privilege does not apply anymore in a devops world.
Locking it down
In the remainder of this post, I’ll go over three steps to lock down your agent:
- Inventorize the permissions your agent requires
- Create a devopsagent user and assign it the required permissions
- Lower the permissions of the Pipelines service to ‘Local Service’ and have it activate the devopsagent user upon each release
The last step is perhaps the most important one. You could skip over the first two and really just lower the permissions of your pipeline agent to NT Authority\Local Service. Step 3 will give you a way to elevate your agent when you need it: during release time.
Step 1: Inventorize the permissions your agent requires
I tried to make an overview of the permissions that a pipeline agent really requires, given my requirements. Don’t start applying them just yet, there’s a script at the end to do this all for you.
Any agent
- logon as a service permission
- full access to the ‘azagent’ folder
Agent modifying system environment variables
- full access to
Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment
IIS web app manage & IIS web app deploy agent:
- full access to
C:\Windows\System32\inetsrv\Config - full acccess to
C:\inetpub - read access to three keys in
C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\- 6de9cb26d2b98c01ec4e9e8b34824aa2_GUID (iisConfigurationKey)
- d6d986f09a1ee04e24c949879fdb506c_GUID (NetFrameworkConfigurationKey)
- 76944fb33636aeddb9590521c2e8815a_GUID (iisWasKey)
Agent scheduling tasks using Powershell (Register-ScheduledTask cmdlet)
- full access to C:\Windows\Tasks
- full access to the working directory of the task you’re scheduling (duh)
Step 2: Create a devopsagent user and assign it the required permissions
In the script, I create a ‘devopsagent’ user and assign it the permissions listed above. I also assign the Local Service user the rights to start and stop the pipelines service since we need that in step 3.
The script can be found on GitHub.
Step 3: Lower the permissions of the Pipelines service to ‘Local Service’ and have it activate the devopsagent user upon each release
You should create three new stages in your pipeline:
- elevate agent permissions, which will ask for a password at release time
- drop agent permissions
- abandon release
The abandonment is required to avoid risking that someone simply ‘redeploys’ your elevation stage without having to enter the elevation password.
By default, I will drop the user to Local Service but you could change that by playing with the pipelineLowPrivUser_name variable.
Elevate agent permissions stage
This stage will assume the pipeline agent is currently running as local service, and will elevate it to the user defined in the pipelineElevatedUser_name variable. Three tasks are defined in this stage:
- the elevate permissions task
- a simple sleep task to give the agent service time to restart and reconnect
- the elevation verification task
Visually, your stage should look like this:
Drop agent permissions stage
This stage will drop the permissions of the elevated pipeline agent to a user of your choosing, by default the local service user. Again, three tasks are defined:
- the drop permissions task
- a simple sleep task to give the agent service time to restart and reconnect
- the drop verification task
Visually, your stage should look like this:
<img class=”leftalign src=”/images/2019-07-23-18-56-26.png”>
Abandon release stage
This stage will abandon the release by calling the Azure Devops API. Make sure to grant this deployment group task access to the OAuth Access Token. This stage will contain one task:
Visually, this stage should look like this:
About TLS certificates and HTTPS bindings
A pipeline agent creating a HTTPS binding MUST have administrator privileges. There is no way to lock that down. I personally made the choice to not use HTTPs bindings in the release, and to manually add them using https://github.com/PKISharp/win-acme after the first release. These bindings will not be overwritten and you can keep on doing releases without having to ever worry about them again.
Alternatively, you could simply follow step 3 as explained above and elevate the pipeline user to an administrative user. Then, drop the permissions again after the release has finished.
Troubleshooting permissions
The logs in the Azure release pipeline are pretty obvious sometimes, and more interestingly contain the commands that the agent is executing. Should you run into a failure because of some other requirement, I recommend you do the following to troubleshoot it:
- Download https://docs.microsoft.com/en-us/sysinternals/downloads/procmon, extract it, and run it
- Browse to the Azure Release Pipeline page and download the log
- Copy the command that caused a failure from the log
- Run cmd.exe as the locked down user on the webserver (run as)
- Paste the command copied in step 2.
- Make sure you get the same error as in the Azure Release pipeline logs.
Then, you should start filtering the procmon output. One obvious thing is to filter for all ‘ACCESS_DENIED’ entries in the ‘Result’ column. Another good way to filter is to filter based on the process. If your command copied in step 2 starts with‘appcmd.exe’ for example, you can simply filter for that process name.
Keep safe, and if you need help contact me at michael(dot)boeynaems(at)portasecura(dot)com!