Use your RD Gateway only when your host is unreachable.
In your Microsoft Remote Desktop (RDP) entry, under Connection, you can enable Open gateway only when unable to ping host, connectivity with the remote host will be tested to see if it is reachable, if not, the RD Gateway will be used.