Problematic UseNewEnvironment switch of Start-Process cmdlet in PowerShell

Have you ever tried using UseNewEnvironment switch when invoking Start-Process cmdlet? Intuitively, turning it on makes the newly created process use the default environment instead of that of the parent process. However, its semantics is way more complicated, obscure and buggy than the intuition. TL; DR: You rarely want to use this switch.

To read a succinct summary and my suggested solution, go there directly. The following will be the process of discovering the bug and the solution.

If you run Start-Process powershell -UseNewEnvironment in Windows PowerShell 5.1, you will see a console window flash to crash. If you execute Start-Process powershell -NoNewWindow -UseNewEnvironment, you will see the error message produced by PowerShell:

Internal Windows PowerShell error.  Loading managed Windows PowerShell failed with error 8009001d.

For PowerShell Core 6.0, the error message is

The shell cannot be started. A failure occurred during initialization:
The static PrimaryRunspace property can only be set once, and has already been set.

Why is that? I explored the current implementation of Start-Process cmdlet in Windows PowerShell 5.1 with ILSpy. Similar problems exist in PowerShell Core 6.0.

The class implementing Start-Process cmdlet is Microsoft.PowerShell.Commands.StartProcessCommand, and processes the command in its overridden BeginProcessing method. It packages the parameters into a ProcessStartInfo before delegating the logic of starting a new process to its start method, which in turn calls either StartWithShellExecute or StartWithCreateProcess, depending on the resolved parameter set. Our code path will be the one with StartWithCreateProcess.

The parts where UseNewEnvironment is involved are:

1. In BeginProcessing, if UseNewEnvironment is on, ClrFacade.GetProcessEnvironment and LoadEnvironmentVariable will be called to build an environment block.
2. In StartWithCreateProcess, the environment block is retrieved with ClrFacade.GetProcessEnvironment and is converted to a pinned byte array for consumption by CreateProcessXxx APIs.

ClrFacade.GetProcessEnvironment(ProcessStartInfo psi) simply returns psi.EnvironmentVariables. Note that calling this method guarantees that the environment block (inheriting that of the calling process) for this ProcessStartInfo is created (see the getter of the EnvironmentVariables property). BeginProcessing calls Clear on the returned dictionary, then calls LoadEnvironmentVariable to load the machine-wide environment variables into the dictionary, and finally calls LoadEnvironmentVariable to load the user-wide environment variables into it. The relevant excerpt is copied here for reference (for PowerShell Core 6.0, see here):

// psi is the ProcessStartInfo object being built.
ClrFacade.GetProcessEnvironment(psi).Clear();
LoadEnvironmentVariable(psi, Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine));
LoadEnvironmentVariable(psi, Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User));

How does LoadEnvironmentVariable combine the variables? Let’s read recursively into that method. ILSpy says (or GitHub says)

// Rewritten by me for easier understanding.
void LoadEnvironmentVariable(ProcessStartInfo psi, IDictionary toLoad)
{
    var target = psi.EnvironmentVariables;
    foreach (var entry in toLoad)
    {
        var key = entry.Key.ToString();
        if (target.ContainsKey(key))
        {
            target.Remove(key);
        }
        if (key.Equals("PATH")) // *
        {
            var machinePath = Environment.GetEnvironmentVariable(key, EnvironmentVariableTarget.Machine);
            var userPath = Environment.GetEnvironmentVariable(key, EnvironmentVariableTarget.Machine);
            target.Add(key, machinePath + ";" + userPath);
        }
        else
        {
            target.Add(key, entry.Value.ToString());
        }
    }
}

Now we have found a bug (I didn’t notice that until I read the code). The line with // * is wrong, because it turns out that the key for PATH environment variable can actually be "Path" or have any other casing, and that key is a string, whose Equals method is always case-sensitive. Indeed, on my Surface Book 2, the name of PATH is Path, as indicated by the dictionaries returned by [Environment]::GetEnvironmentVariables('Machine') and [Environment]::GetEnvironmentVariables('User'). The correct ways to detect PATH are

// Use another overload of Equals.
if (key.Equals("PATH", System.StringComparison.InvariantCultureIgnoreCase))
    ;
// Use ToLowerInvariant.
if (key.ToLowerInvariant() == "PATH".ToLowerInvariant())
    ;
// Avoid begin loquacious.
if (key.ToLowerInvariant() == "path")
    ;

Okay, but that’s not too bad. Except for PATH, everything seems okay, no? No. There are two more problems.

You normally start PowerShell from File Explorer or Cortana. It turns out that the environment blocks of theirs are more than just the combination of machine-wide and user-wide environment blocks! If you do the melting yourself, and compare that to that of a newly launched PowerShell’s, you will find some environment variables are missing from the combined dictionary. For me, the following environment variables found in PowerShell launched from File Explorer are missing in the combined dictionary:

ALLUSERSPROFILE
APPDATA
CommonProgramFiles
CommonProgramFiles(x86)
CommonProgramW6432
COMPUTERNAME
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
ProgramData
ProgramFiles
ProgramFiles(x86)
ProgramW6432
PUBLIC
SESSIONNAME
SystemDrive
SystemRoot
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERPROFILE

For whatever reason, PowerShell doesn’t seem to be able to handle the missing of them. Neither do I expect it to handle them, because normally, you will have these environment variables.

The other thing is that bad things happen if a credential is supplied. In this case, the code will eventually call CreateProcessWithLogonW function. If UseNewEnvironment is on, the cmdlet passes the environment block (wrongly) combined for the current user to CreateProcessWithLogonW. If the current user is A, and the credential is for user B, the resulting process will be created in the security context of B, but will have partial environment of A. I won’t expect a program to expect getting a subset of some other user’s environment when running as some user. Interestingly, you might always want to not specify UseNewEnvironment if you are creating a process as another user, in which case the environment block pointer passed to CreateProcessWithLogonW will be the null pointer. In that case, CreateProcessWithLogonW creates the process with the environment created from that user’s profile, which is the initial environment of wininit.exe (or File Explorer) for a newly logged-on user.

Summary and Solution

PowerShell Start-Process creates the environment block in a stupid way, and generally you should avoid using UseNewEnvironment switch before Microsoft fixes it. The current behaviour is listed in the following table:

The pattern: turning on UseNewEnvironment results bad behaviour.

To fix the problem on Windows, Microsoft should:

1. Stop building the environment block manually, whose algorithm is wrong and which results in missing environment variables. Instead, call CreateEnvironmentBlock function. The code path for CreateProcessWithLogonW might be changed to use CreateProcessAsUser and LogonUser. After all, you need a handle to the user’s token to create an environment block.
2. Resolve the problem of mixing different users’ environment. Using CreateEnvironmentBlock function should partially solve the problem. The philosophy is that you cannot do much when creating the process as another user. The simplest fix is to completely ignore UseNewEnvironment and just use a new environment of the specified user when Credential is supplied.

Having written this entry, I find issues #3545 and #4671 have already brought the problems up to Microsoft.