Tests can't be run in existing AppDomain

Code doesn't exist in nunit-3.0... bug must be intended for nunitv2

Are you editing the xml file to create these settings? The gui settings page doesn't support them.

I'm creating my own standalone application, which uses NUnit framework to make tests internally.
Necessary options was set using this code:

Services.UserSettings.SaveSetting("Options.TestLoader.DomainUsage", DomainUsage.None);
Services.UserSettings.SaveSetting("Options.TestLoader.ProcessModel", ProcessModel.Single);

When TestLoader was created, it creates new test-domain. I set a breakpoint inside and found a place in MakeTestPackage, where options was rewritten to Default.

You are right that the code in question is incorrect. NUnit only uses TestLoader for the Gui, which cannot run with DomainUsage.None, so this was never noticed before.

OTOH, the code you suggest doesn't work for all cases. What's intended is to prevent both ProcessModel and DomainUsage from being set to Multiple, which requires

if ( (processModel != ProcessModel.Multiple || domainUsage != DomainUsage.Multiple)
                && !package.Settings.Contains("DomainUsage"))
                package.Settings["DomainUsage"] = domainUsage;

I'm torn here. On the one hand, I'd like to fix this for you but on the other, I'm tempted to add 'domainUsage != domainUsage.None' to the statement, so that the Gui won't crash. I'll look to see if I can put that test in the Gui itself.

This is now fixed (in the source) in a way that will work for you. However, I suggest you avoid using the Settings to control options for your use of NUnit. That approach will cause NUnit itself, both the Gui and Console runners, to use the settings you leave behind. A better approach for programmatic control is to place the options in the TestPackage you create. TestPackage settings will override anything in the general NUnit settings.