Domain and server isolation with IPsec

Firewall rules based on IP addresses answer the question "where does this packet come from?". Attackers are good at arriving from the right place: a compromised workstation in the right subnet, or a laptop plugged into a meeting room port. IPsec connection security rules add a stronger question: "which authenticated computer, or user, is on the other end?"

With domain isolation, domain members only accept inbound traffic from other authenticated domain members. With server isolation, a sensitive server only accepts traffic from specific computers or users, identified by Kerberos, regardless of IP address. Both are built from the same Windows Firewall components you already manage through Group Policy at scale, and they build on the scoping principles in the Windows Firewall hardening pillar.

This is an advanced control. It changes how every connection is established, so the rollout matters as much as the configuration.

The building blocks

Connection security rules do not allow traffic on their own. They make sure traffic is authenticated; the firewall rules then decide whether authenticated traffic from that peer is allowed.

Step 1: set IPsec defaults

GPO path: Computer Configuration > Policies > Windows Settings > Security Settings > Windows Defender Firewall with Advanced Security, properties, IPsec Settings tab, Customize:

In PowerShell, crypto sets are separate objects referenced by rules:

$mm = New-NetIPsecMainModeCryptoProposal -Encryption AES256 -Hash SHA256 -KeyExchange DH19
New-NetIPsecMainModeCryptoSet -Name 'MM-Strong' -DisplayName 'MM-Strong' -Proposal $mm -PolicyStore 'corp.example.com\FW-IPsec'

$qm = New-NetIPsecQuickModeCryptoProposal -Encapsulation ESP -ESPHash SHA256
New-NetIPsecQuickModeCryptoSet -Name 'QM-Integrity' -DisplayName 'QM-Integrity' -Proposal $qm -PolicyStore 'corp.example.com\FW-IPsec'

$auth = New-NetIPsecAuthProposal -Machine -Kerberos
New-NetIPsecPhase1AuthSet -Name 'P1-ComputerKerberos' -DisplayName 'P1-ComputerKerberos' -Proposal $auth -PolicyStore 'corp.example.com\FW-IPsec'

Rules reference these sets by their Name, not their display name, which is why each set gets an explicit -Name; without it, Windows generates a GUID and the rule below cannot find the set.

Step 2: domain isolation in request mode

Create an Isolation connection security rule in a new GPO linked to a pilot OU:

• Requirements: Request authentication for inbound and outbound connections.
• Authentication method: Computer (Kerberos V5).
• Profile: Domain.

New-NetIPsecRule -DisplayName 'Domain isolation' -InboundSecurity Request -OutboundSecurity Request `
  -Phase1AuthSet 'P1-ComputerKerberos' -Profile Domain -PolicyStore 'corp.example.com\FW-IPsec'

Request mode never blocks anything. Domain members that both have the rule negotiate security associations; everything else falls back to clear traffic. This phase shows you which peers can authenticate and which cannot, without an outage.

Add authentication exemptions for infrastructure that cannot or should not negotiate IPsec:

• Domain controllers, so clients can obtain the Kerberos tickets that IPsec itself needs. Leave DCs in request mode rather than require.
• DHCP and DNS servers if they are not DCs.
• Network devices, printers, IoT, and non-Windows systems that talk to domain members.

Keep the exemption list small, documented and based on addresses you control; each exemption is a hole in the model.

Step 3: require inbound authentication

Once monitoring shows that nearly all domain traffic is protected (see the monitoring section below), switch servers first, then workstations, to Require inbound and request outbound. Devices without a domain identity can no longer open connections to them. Outbound stays request so that domain members can still reach exempted and non-Windows systems.

Avoid Require inbound and outbound except for tightly controlled server groups: it breaks every outbound connection to a peer that cannot negotiate IPsec.

Step 4: server isolation for sensitive systems

For a sensitive server, such as a finance database or a Tier 0 management server, create inbound firewall rules that only accept authenticated traffic from authorised computers:

1. Create a computer security group, for example IPsec-FinanceDB-Clients, containing the application servers that may connect.
2. On the server, create an inbound allow rule for the service port with Allow the connection if it is secure, and on the Remote Computers tab authorise only that group. Optionally add Remote Users for user-level authorisation (this needs second authentication with user Kerberos).
3. Remove any other allow rule for that port.

$sid = (Get-ADGroup 'IPsec-FinanceDB-Clients').SID.Value
New-NetFirewallRule -DisplayName 'SQL 1433 - authorised computers only' -Direction Inbound `
  -Protocol TCP -LocalPort 1433 -Action Allow -Authentication Required `
  -RemoteMachine "D:(A;;CC;;;$sid)" -Profile Domain -PolicyStore 'corp.example.com\FW-FinanceDB'

Add -Encryption Required where the protocol does not already encrypt. Computer group membership is carried in the Kerberos ticket, so adding a computer to the group takes effect only after that computer's ticket is renewed (a reboot or klist purge -li 0x3e7).

The same pattern gives you identity-based management access: RDP and WinRM on Tier 0 servers accepted only from computers in a Tier0-PAW group, which aligns with admin tiering and the tier model.

What IPsec does and does not stop

IPsec isolation makes an unmanaged device on the network largely useless for reaching domain members, and it makes IP spoofing and simple network relays ineffective against isolated servers. It complements, rather than replaces, protocol hardening such as SMB signing and NTLM restrictions from the legacy protocols guide: an attacker who controls a domain-joined computer authenticates as that computer. Server isolation limits which domain computers qualify; NTLM relay protections still matter.

What breaks

• Load balancers and NAT between clients and servers: IPsec transport mode is negotiated end to end, so a load balancer that terminates or rewrites connections breaks authentication. Exempt the VIP or protect traffic only behind the load balancer.
• Network monitoring and IDS lose payload visibility when you require encryption. Integrity-only ESP keeps packets readable.
• Non-domain and cross-forest systems need certificates or exemptions.
• Kerberos problems become connectivity problems: clock skew, broken SPNs or unreachable DCs now show up as failed connections.
• Performance: CPU overhead is small on modern hardware with NIC offload, but test high-throughput servers.
• Troubleshooting is harder for teams not used to IPsec. Train the network and server teams before enforcing.

Monitor security associations

Enable IPsec auditing under Advanced Audit Policy Configuration > Audit Policies > Logon/Logoff: Audit IPsec Main Mode and Audit IPsec Quick Mode (failure at minimum), plus System > Audit IPsec Driver. Useful Security log events:

A rise in 4653 after moving to require mode points to peers without a valid identity: exactly the devices you need to exempt or fix. Success events are high-volume; collect failures centrally and sample successes.

Verify it

On a server in scope:

Get-NetIPsecRule -PolicyStore ActiveStore | Select-Object DisplayName, InboundSecurity, OutboundSecurity, Enabled
Get-NetIPsecMainModeSA | Select-Object RemoteEndpoint, LocalFirstId, RemoteFirstId
Get-NetIPsecQuickModeSA | Select-Object RemoteEndpoint, RemotePort, LocalPort

RemoteFirstId shows the authenticated computer identity of each peer. In wf.msc > Monitoring > Security Associations you see the same data graphically. Then test negatively: from a non-domain laptop on the same subnet, Test-NetConnection <server> -Port 445 should fail once the server requires inbound authentication, while a domain member in the authorised group still connects.

Checklist

• IPsec defaults set: no SHA-1 or DH Group 2, ICMP exempt
• Domain isolation rule deployed in request mode to a pilot, then broadly
• DCs, DHCP, DNS and non-Windows infrastructure exempted and documented
• Servers moved to require inbound, request outbound, after SA coverage was measured
• Sensitive servers restricted to authorised computer groups with "allow if secure"
• Encryption required only where the protocol does not protect itself
• IPsec main and quick mode failures audited and forwarded

See the Windows Firewall topic for the full series, including logging and policy management.