RDP certificate retrieval has an extra step that HTTPS does not have. Before TLSleuth can start a TLS handshake with an RDP server, it has to participate in the initial RDP security negotiation.

This post looks at that negotiation path in more detail: the request TLSleuth sends, how it interprets Ssl, Hybrid, and HybridEx, where NLA fits, and what server failure responses mean.

The RDP Pre-TLS Step

An RDP connection starts with protocol negotiation before the TLS handshake. TLSleuth handles that in Invoke-RdpTlsNegotiation.

At a high level, the helper does this:

validate stream -> build negotiation request -> send bytes -> read response -> decide whether TLS can start

The implementation deliberately keeps that binary protocol work out of the public commands. Get-TLSleuthCertificate and Test-TLSleuthProtocol still call the shared transport dispatcher, and the dispatcher routes RdpTls to the RDP adapter.

Security Protocol Names

TLSleuth requests three RDP security protocol values:

Name	Mask value
Ssl	0x00000001
Hybrid	0x00000002
HybridEx	0x00000008

The Ssl name here is the RDP negotiation label. It does not mean TLSleuth is trying to force SSL 3.0. The actual TLS handshake is still performed by .NET SslStream using the TLS protocol options selected for the TLSleuth command.

The default requested set is:

@('Ssl','Hybrid','HybridEx')

ConvertTo-RdpSecurityProtocolMask converts those names into a bitmask for the RDP Negotiation Request.

Request Construction

New-RdpNegotiationRequest builds a byte array shaped as an RDP X.224 Connection Request with an RDP Negotiation Request attached.

The request includes the security protocol bitmask in little-endian byte order:

[byte]($protocolMask -band 0xff),
[byte](($protocolMask -shr 8) -band 0xff),
[byte](($protocolMask -shr 16) -band 0xff),
[byte](($protocolMask -shr 24) -band 0xff)

That packet is written to the network stream by Send-BinaryProtocolData. TLSleuth uses the binary helper path here, not the text protocol helpers used for SMTP, IMAP, and POP3.

Response Parsing

Read-RdpNegotiationResponse reads the response in two stages:

1. read the 4-byte TPKT header
2. read the remaining payload length from the TPKT length field

The helper validates that:

• the response starts with a TPKT header beginning 0x03 0x00
• the TPKT length is large enough to contain the expected RDP response
• the payload includes an X.224 connection confirm
• the payload includes an RDP negotiation response or failure structure

The parser looks for a negotiation structure with one of these types:

Type	Meaning in TLSleuth
0x02	negotiation response
0x03	negotiation failure

For a successful negotiation response, TLSleuth converts the selected protocol mask back to names with ConvertFrom-RdpSecurityProtocolMask.

Negotiation Flow

The full decision path looks like this:

flowchart TD
    Start["Invoke-RdpTlsNegotiation"]
    Stream["Connection has NetworkStream?"]
    Request["New-RdpNegotiationRequest"]
    Send["Send-BinaryProtocolData"]
    Read["Read-RdpNegotiationResponse"]
    Failure{"Response type is Failure?"}
    TlsCapable{"Selected mask includes Ssl, Hybrid, or HybridEx?"}
    ThrowFailure["Throw negotiation failure"]
    ThrowNonTls["Throw non-TLS RDP security error"]
    Continue["Return response; TLS handshake may start"]

    Start --> Stream
    Stream --> Request
    Request --> Send
    Send --> Read
    Read --> Failure
    Failure -->|yes| ThrowFailure
    Failure -->|no| TlsCapable
    TlsCapable -->|no| ThrowNonTls
    TlsCapable -->|yes| Continue

The selected protocol mask must include at least one of:

0x00000001 Ssl
0x00000002 Hybrid
0x00000008 HybridEx

In code, TLSleuth checks the combined mask:

0x0000000b

If none of those bits are present, TLSleuth treats the selected mode as non-TLS-capable for certificate retrieval.

Standard RDP Security

ConvertFrom-RdpSecurityProtocolMask reports a zero mask as:

StandardRdp

That is useful diagnostic information, but it is not a path TLSleuth can use to retrieve a TLS certificate. If the server selects standard RDP security, TLSleuth stops with an error explaining that TLS certificate retrieval requires TLS-capable RDP security.

Where NLA Fits

NLA is commonly associated with Hybrid and HybridEx RDP security modes. From TLSleuth’s perspective, the important boundary is simpler:

• negotiate a TLS-capable RDP security mode
• start the TLS handshake
• retrieve certificate and TLS/session details
• stop

TLSleuth does not perform CredSSP, does not authenticate to the server, and does not log on to a desktop session.

That means RdpTls can inspect the certificate used by the TLS channel, but it does not prove that a user could successfully authenticate through NLA. It also means that if server policy rejects negotiation before TLS starts, TLSleuth cannot continue to certificate retrieval.

Failure Responses

When the server returns an RDP negotiation failure structure, TLSleuth maps the failure code to a readable reason:

Code	Reason
0x00000001	SSL_REQUIRED_BY_SERVER
0x00000002	SSL_NOT_ALLOWED_BY_SERVER
0x00000003	SSL_CERT_NOT_ON_SERVER
0x00000004	INCONSISTENT_FLAGS
0x00000005	HYBRID_REQUIRED_BY_SERVER
0x00000006	SSL_WITH_USER_AUTH_REQUIRED_BY_SERVER

Unknown failure codes are reported as:

UNKNOWN_FAILURE_0x########

For example:

RDP server rejected TLS negotiation: HYBRID_REQUIRED_BY_SERVER (0x00000005).

That error happens before TLS starts. At that point TLSleuth has not received a TLS certificate.

Malformed Or Unexpected Responses

TLSleuth also fails clearly when the response is not shaped like the expected RDP negotiation response.

Representative errors include:

• the response did not start with a valid TPKT header
• the TPKT length was too short
• the response did not include an X.224 connection confirm
• the response did not include a negotiation response or failure structure
• the stream ended before the requested byte count was read

These usually point to one of three situations:

• the endpoint is not really RDP
• a proxy or gateway returned something unexpected
• the server or network closed the connection before negotiation completed

After Negotiation

When RDP negotiation succeeds, Invoke-RdpTlsTransportAdapter returns an internal TLSleuth.TransportNegotiationResult containing:

• requested security protocols
• selected protocol mask
• TPKT length
• selected protocol name

The public commands intentionally do not expose that internal object in their normal output. They continue into:

Start-TlsHandshake -> Get-TlsHandshakeDetails -> Get-RemoteCertificate

That keeps the public output contract stable while still letting the transport layer keep useful internal diagnostics.

Testing The Parser And Failure Paths

The RDP negotiation code is covered by unit tests for:

• request construction
• security protocol mask conversion
• successful negotiation response parsing
• failure response parsing
• rejected negotiation errors
• non-TLS selected security mode errors
• dispatcher and adapter routing

The live RDP test is opt-in because it needs a reachable RDP endpoint:

$env:TLSLEUTH_RDP_HOST = 'rdp.example.com'
$env:TLSLEUTH_RDP_PORT = '3389'
$env:TLSLEUTH_RDP_TARGETHOST = 'rdp.example.com'
Invoke-Build IntegrationTest

Without TLSLEUTH_RDP_HOST, that test is skipped so normal CI does not depend on an external RDP server.

Summary

RDP certificate retrieval is not just “open port 3389 and start TLS.” TLSleuth has to:

1. send an RDP negotiation request
2. request TLS-capable security protocols
3. parse the server response or failure
4. reject non-TLS selected security modes
5. start the normal TLS handshake only after negotiation succeeds

That boundary is why RDP support lives behind the RdpTls transport. It keeps RDP’s binary pre-TLS negotiation separate from the shared TLS handshake and certificate inspection pipeline.