Diagnosing network connectivity issues in a .NET application; or ‘How JavaScript really has taken over everything, including your network configuration’
tldr; JavaScript really has taken over everything, it may be configuring your proxy server settings, via a PAC file, and if you use the full .NET Framework you may find your application cannot reconnect to the internet automatically after a network connection is restored. You have to redetect the settings with GetSystemWebProxy when your application is back online.
Note : This does not affect .NET Core.
Stanley Kubrick once said,
‘If it can be written, or thought, it can be filmed’
Which was possibly borrowed by Jeff Atwood to coin a new Atwood’s Law,
‘any application that can be written in JavaScript,
will eventually be written in JavaScript’
While debugging network connectivity issues with the Huddle Desktop for Windows application, I experienced just how true this is.
We found that when Huddle Desktop lost network connectivity; for instance, when being removed from a docking station and switching from wired network to WiFi, or moving between different WiFi networks, it detected the loss of the network but did not reconnect when it became available again.
This was only happening on companies with custom proxy server configurations and was initially hard to reproduce, and even harder to then track down.
Clearly I needed to learn more about proxy servers …
Network proxies
So what is a proxy server? A proxy server is a computer system or an application that acts as an intermediary for requests from clients seeking resources from other servers (source: Wikipedia).
When you have a network proxy configured, every time you request a resource from the local network or the internet it is processed by the proxy server which will inform you where to look for that resource. This may even include delivering a modified or cached version of a resource from the proxy server itself. In the past, mobile phone operators have even used proxy servers to resize images on the fly to reduce data usage on their networks.
If you are a developer, and have used Fiddler or Charles to inspect network traffic, you are using that application as a proxy server. When you are inspecting traffic from your local system then all the proxy server configuration happens in the background when you start Fiddler.
A proxy server can be configured manually to a specific IP address (this is how Fiddler achieves it), configured automatically at the same time as obtaining a network address via DHCP, or it can be determined by using a Proxy auto-configuration (PAC) file.
Here is how the proxy configuration dialog looks in Internet Explorer, via Tools –> Internet Options –> Connections (tab) –> LAN settings (button)
The PAC file option here is that check box, Use automatic configuration script, followed by the location of the script to be used.
I’m sure you mentioned JavaScript ages ago
A PAC file contains a JavaScript function “FindProxyForURL(url, host)”. This function returns a string with one or more access method specifications. These specifications cause the user agent to use a particular proxy server or to connect directly (source: WikiPedia).
So every time you request a URI in your browser, it calls the JavaScript method FindProxyForURL which returns the proxy server (or to not use a proxy server at all) for that URI request. For one of our clients that JavaScript file has over 700 lines, mainly handling network segmentation across multiple countries.
Importantly, the PAC file cannot be a local file path, but must be served over HTTP or HTTPS.
The .NET Framework is special
A rifle through some .NET framework pages also revealed that the full .NET framework (not .NET Core) handles proxy servers separately to the underlying Windows operating system. These even includes processing PAC files with a JavaScript VM that has subtle differences to that used by Windows.
As the .NET framework maintains its own state of the network configuration, separate to that of the underlying operating system, they can be out of step with one another. This was precisely what our customers were seeing; our application (and some other .NET applications) could not recover from network disconnections, but most applications, and the operating system could recover successfully.
In .NET the proxy settings can be read from WebRequest.DefaultWebProxy, or from WebRequest.GetSystemWebProxy. The difference is subtle; DefaultWebProxy supports overriding the proxy settings via the app.config file, but will use system settings where no such override exists. GetSystemWebProxy only reads proxy settings as configured in Internet Explorer. At Huddle we support the app.config file model for customers wishing to implement custom settings specific to our application.
It should be noted that .NET Core has changed this model. It relies solely on the underlying operating system for proxy configuration. Also, there are no standard app.config settings for overriding this configuration.
Reproducing the bug
So after this period of research, we could finally reproduce the connectivity issue. We placed a simple PAC file into Amazon AWS S3 , that pointed to a local proxy server on our internal network.
function FindProxyForURL(url, host) {
return “PROXY 10.50.0.26:3128”;
}We had spent some time setting the local proxy server directly, but this never reproduced the connectivity issue as it was the act of failing to load the PAC file which was the root cause. Once we could reproduce the issue it was down to debugging our desktop application to pinpoint what was happening.
It became clear that on a network disconnect, the configuration to use a PAC file was lost, and never recovered. It appears that when the network fails, the PAC file is reloaded in some manner, and when it too cannot be found, the network stack removes the PAC file as a proxy option, and defaults to use direct access only with no proxy server. The PAC file option is not tested again when network connectivity returns.
We even took a look at the source code for the .NET Framework over at https://github.com/Microsoft/referencesource/ to see what was happening.
Our code was only using the DefaultWebProxy property, to respect any app.config setting, but recreating this did not restore the PAC file configuration. However, a call to GetSystemWebProxy did detect the PAC file configuration when the network connectivity returned.
The Solution
When we are validating network connectivity returning, we query for proxy settings just as we do when starting the application.
If DefaultWebProxy provides a proxy, that must have come from either the app.config override or from a manually configured proxy server in system settings, so we use that.
If DefaultWebProxy does not return a proxy (which occurs after a network disconnect when a PAC file has been configured), but GetSystemWebProxy does return a proxy, we use that. If no proxy is returned by either method then we must have direct access with no proxy servers.
By making a call to GetSystemWebProxy to obtain a new instance of the WebProxy class and using this for future network calls, it processes the PAC file as expected to return the correct proxy server.
We also get to have happy customers.
Originally published at blog.liamwestley.co.uk on June 7, 2018.
