Simple view on how proxies work (inc DNS)

[Paul]

Since it seems to confuse people a lot I thought I'd try and explain it in simple terms;

Techies - feel free to point out any mistakes. :)


First of all - without any proxies - when a client wants a web page ;

1. It does a dns lookup of the required domain using its own (locally) defined dns servers.
2. It sends the http request on tcp port 80 to the destination servers IP address.
3. This request is routed over the network direct to the specified server IP.
4. The requested page is then returned, by the server, direct to the client.


Now if you have a manually defined proxy (such as in IE, on port 8080) - when a client wants a web page ;

1. It does a dns lookup of the proxy servers IP using its own (locally) defined dns servers.
2. It sends the http request on tcp port 8080 to the proxy servers IP address.
3. This request is routed over the network direct to the specified proxy server IP.
4. The proxy server checks if it has a valid copy of the requested page in its cache.
5. If the proxy has the page it returns it directly back to the client, Job done.
6. If the proxy does not have the page it does a dns lookup of the requested domain using it's own defined dns servers.
7. The proxy sends the http request on tcp port 80 to the destination servers IP address.
8. This request is routed over the network direct to the destination servers IP.
9. The requested page is then returned, by the destination server, directly back to the proxy server.
10. The requested page is then returned, by the proxy server, direct to the client.


Now if you have no manually defined proxy, but NTL's proxies are active - when a client wants a web page ;

1. It does a dns lookup of the destination servers IP using its own (locally) defined dns servers.
2. It sends the http request on tcp port 80 to the destination servers IP address.
3. This request is intercepted by NTL's routers and re-directed to one of a group of transparent proxy [cache] servers - the decision on which proxy server gets used is (usually) based on the requested domain.
4. The chosen proxy server checks if it has a valid copy of the requested page in its cache.
5. If the proxy has the page it returns it directly back to the client, Job done.
6. If the proxy does not have the page it does a dns lookup of the requested domain using it's own defined dns servers.
7. The proxy sends the http request on tcp port 80 to the destination servers IP address.
8. This request is routed over the network direct to the destination servers IP.
9. The requested page is then returned, by the destination server, directly back to the proxy server.
10. The requested page is then returned, by the proxy server, direct to the client.

The final scenario is called transparent because the client is unaware that its request as been "hijacked" and still thinks it is talking directly to the destination server.

From the above it can be seen that if you are having web browsing problems and proxies are in use (transparent or otherwise) then ;

1. The DNS servers the proxy is using are just as important as yours - if they are duff then you are in trouble. (However, even if the proxy cannot resolve or contact the destination server - you can still access the requested pages if the proxy has a valid copy of them in its cache)

2. Doing a tracert to the destination server will not show you the route your request is taking as it won't go via any proxy in use.

This is not a precise and exact description of how it all works but I hope it makes it clearer. :D

To workout if you are using an NTL transparent proxy - use this simple link (http://romulas.zmnt.co.uk/forum/vars.asp)

Pem.

[danielf]

Cheers for that. I think that makes sense :) I have a question though. What is a 'valid copy' of a page. I.e. How does the proxy decide if the page it has cached is recent enough? Do all pages carry a time stamp (last modified date/time), and does the proxy query the requested site for the time stamp of the requested page?

[Paul]

All pages effectivley have an expiry date & time.

[danielf]

All pages effectivley have an expiry date & time.

Ah, that makes sense, as I was wondering if the proxy had to query for a more recent page, which would increase traffic and defeat the purpose of the proxy unless you only have to retrieve the time stamp of the most recent page.

I presume this expiry time/date is set differently depending on the type of page? (very low for a forum like this) and high(er) for less dynamic pages)?

Or is it just based on recent 'refresh rates'?

[Chrysalis]

pem even when a port is specified my pages are still going through the transperent proxies, the guy who told us this in the other thread was right.

[Paul]

Ah, that makes sense, as I was wondering if the proxy had to query for a more recent page, which would increase traffic and defeat the purpose of the proxy unless you only have to retrieve the time stamp of the most recent page.

I presume this expiry time/date is set differently depending on the type of page? (very low for a forum like this) and high(er) for less dynamic pages)?

Or is it just based on recent 'refresh rates'?

Believe me - you would not believe how complicated it is for a proxy to determine if it's copy is current ("fresh"). There are pages and pages of specifications and http headers to control it. :D


pem even when a port is specified my pages are still going through the transperent proxies, the guy who told us this in the other thread was right.

Sorry, I don't follow what you are saying. Can you explain ?

[zendawn]

Does CTRL+Refresh Overide the proxy in order to get an updated refresh??

[Paul]

Does CTRL+Refresh Overide the proxy in order to get an updated refresh??
Erm ........ if client and proxy follow the rules - Yes.

Do they all follow the rules ...... do they **** :banghead:

[danielf]

Believe me - you would not believe how complicated it is for a proxy to determine if it's copy is current ("fresh"). There are pages and pages of specifications and http headers to control it. :D

Ah, the joys of optimisation/efficiency :)

[tkiely]

just ditch them, the're not there for your convenience!

mind you, i like playing with them at work. but that's not for here!!!!!!!!!!

[ntl customer]

How long to the proxies keep stored pages for?

[tkiely]

How long to the proxies keep stored pages for?

generally, until they change or expire. Some will make checks on a site for change on a daily basis to see if there have been changes that they should cache

[Chrysalis]

well right now pem I am running through my own proxy on port 3333, but the ntl transperent proxy is still intercepting my pages, I will look for the post that explains this and paste it here.

[Chrysalis]

http://forum.nthellworld.co.uk/showpost.php?p=163902&postcount=89

shows the info

but right after i retrieved this post the ip shown in my sig is no longer ntl proxy server.

I think it is working after all

[Chrysalis]

when I use IE or firefox it doesnt use ntl proxy but if I use MYIE2 it does, very weird especially as MYIE2 uses IE core.

--edit--

fixed it :)

[mr-b]

Ah, that makes sense, as I was wondering if the proxy had to query for a more recent page, which would increase traffic and defeat the purpose of the proxy unless you only have to retrieve the time stamp of the most recent page.

I presume this expiry time/date is set differently depending on the type of page? (very low for a forum like this) and high(er) for less dynamic pages)?

Or is it just based on recent 'refresh rates'?
One important thing to note is that a web page is made up of several objects, each of which can have different cacheability HTTP headers:

- Expires (the best since they are explictly set by the content publisher)

- Cache-Control

- Last-Modified

So a forum page should have the frequently updated part as having either a very recently modified date, a short expiry time or else marked as non-cacheable and the relatively static parts (logos, banners etc.) as being older or having longer expiry times.
This will make the page much more responsive than making the whole page uncacheable. Note that these settings are what your local browser cache uses as well, so it's not just for the benefit of proxies).

In fact checking this page via the Cacheability Engine shows this to be true (takes some time to load):
http://www.web-caching.com/cgi-web-caching/cacheability.py?version=1.30&descend=on&query=http%3A//forum.nthellworld.co.uk/showthread.php%3Ft%3D9354

There's some more info about how this all works (without requiring knowledge of the RFCs) and how to optimise web servers at:
http://www.mnot.net/cache_docs/

[Paul]

well right now pem I am running through my own proxy on port 3333, but the ntl transperent proxy is still intercepting my pages, I will look for the post that explains this and paste it here.
Running a proxy on your own machine is a complete waste of time - it is certainly not going to bypass NTL's caches because as far as the above explanation goes, your proxy is still the local client.

[Florence]

The bottom line is these transparent proxies slow down your surfing as if they havent got the page you want it now has to look it up which you had already done on the proxy. In real time doubling the time till the page arrives if its one not stored.

If I have it wrong then I am willing to learn so come on techies teach me the ideas behind proxies and what benefit they give customers!

[danielf]

I presume (but I might be wrong), the reason for having proxies is not to speed up browsing but to reduce the bandwith used outside the NTL network, thereby reducing cost (and as a result reducing what we need to pay for our connection)?

[Chrysalis]

pem you might have misread my previous posts, the proxy is not on my local machine, it is on a remote box. It is a pass through proxy it doesnt cache pages.

this link no longer shows any ntl proxy servers.

http://romulas.zmnt.co.uk/forum/vars.asp

[Paul]

pem you might have misread my previous posts, the proxy is not on my local machine, it is on a remote box. It is a pass through proxy it doesnt cache pages.

this link no longer shows any ntl proxy servers.

http://romulas.zmnt.co.uk/forum/vars.asp

Fair enough, where is this box located then ?

[Chrysalis]

dedicated box I got in a datacentre in the us.

I got some minor issues with cookies and ssl at the moment needs some tweaking, but overall I am pleased with the improvements.

[John Doe]

http://www.cisco.com/warp/public/732/Tech/switching/wccp/

hohum

[Florence]

http://www.cisco.com/warp/public/732/Tech/switching/wccp/

hohum

Thank you for the link unfortunately what that says isn't happening.. I was quicker when proxies was off that now on my general surfing I.E. when researching for college course visiting websites I have never been to before..

Also the one website I visit the most the forums I moderate is slower this is hosted in Germany. I have more page errors on proxies even though the address is correct to get to these pages I have to go via Google.

Infect when the proxies were turned off that was the best internet I had and the DNS propagated quicker. I have had to waste downloads as I have sometimes started a download and itâ₠¬ÃƒÂ¢Ã¢â‚¬Å¾Ã‚¢s stopped partway through and would not restart. Yet if I started again from it worked.

Suppose its the proxies but I find the information on that link very limited about proxies and not at all reasuring after all the problems with mail servers how can we be sure the proxies will stay up. The more different ways we have to ask to get to our final website the more bandwidth we end up using.... IMPO........

[mr-b]

The bottom line is these transparent proxies slow down your surfing as if they havent got the page you want it now has to look it up which you had already done on the proxy. In real time doubling the time till the page arrives if its one not stored.

If I have it wrong then I am willing to learn so come on techies teach me the ideas behind proxies and what benefit they give customers!
Not sure exactly what the above is saying, but if your question is about the latency that proxies add to internet requests, then (assuming that the proxy is working correctly, running within specification and there are no network problems) it is negligible compared to the response time via the internet and remote web server. Caches are designed to reduce the latency of internet requests as a whole.
A cache 'hit' (where it satisfies the request from local disk) should take tens of milliseconds (ms). A 'miss' (where it has to fetch the object from the remote web server via the internet) takes several hundreds to thousands of ms.

A slightly older doc on the problems of latency (but the principles still hold true today) is:
http://rescomp.stanford.edu/~cheshire/rants/Latency.html

[Chrysalis]

well ntl rang me up today, concerning my proxy complaint, I ran some tests for him downloaded a file from download.com and it was fine. I told him of the hotmail problems and on the firefox site as well as my browsing slowdown, he told me what I believed to be true tho, these proxies were turned on after ntl decided nationwide they are all to be on, he said its ok for me to use my own proxy, so I will carry on using it.

[Florence]

Not sure exactly what the above is saying, but if your question is about the latency that proxies add to internet requests, then (assuming that the proxy is working correctly, running within specification and there are no network problems) it is negligible compared to the response time via the internet and remote web server. Caches are designed to reduce the latency of internet requests as a whole.
A cache 'hit' (where it satisfies the request from local disk) should take tens of milliseconds (ms). A 'miss' (where it has to fetch the object from the remote web server via the internet) takes several hundreds to thousands of ms.

A slightly older doc on the problems of latency (but the principles still hold true today) is:
http://rescomp.stanford.edu/~cheshire/rants/Latency.html


Thank you for that link I will read it later tonight when I have peace. I must say that the proxies was working ok recently but last night they blew it big time. As soon as it was too late to call in the proxies for me died completely. Unable to access any website.. MSN, Yahoo, ICQ and AIM alsom MIRC chatroom all still worked but anything that needed the http was dead.... this lasted about 30 mins...

Need some sort of way to communicate these problems when faults is closed or 24 hour cover..

[SMHarman]

Ah, that makes sense, as I was wondering if the proxy had to query for a more recent page, which would increase traffic and defeat the purpose of the proxy unless you only have to retrieve the time stamp of the most recent page.

I presume this expiry time/date is set differently depending on the type of page? (very low for a forum like this) and high(er) for less dynamic pages)?

Or is it just based on recent 'refresh rates'?

Http also has a HEAD command and a GET command. GET asks for the whole page, HEAD asks for the headers, which can be used to compare to the cached headers. Head uses a lot less bandwidth as a result.

[iadom]

http://forum.nthellworld.co.uk/showpost.php?p=163902&postcount=89

shows the info

but right after i retrieved this post the ip shown in my sig is no longer ntl proxy server.

I think it is working after all
The proxies have only just been turned on again in this area and I have just noticed that your sig is now giving out my proxy and not my true IP.

[Chrysalis]

yeah I got this sig from someone else I added it purely as a diagnostic to see if using a ntl proxy.

[mr-b]

Http also has a HEAD command and a GET command. GET asks for the whole page, HEAD asks for the headers, which can be used to compare to the cached headers. Head uses a lot less bandwidth as a result.
Correct - but there's a way of using even less bandwidth.

If you issue an HTTP GET and also include an If-Modified-Since header then the web server will only send you the object if it has been modified since the date that you specified. If it hasn't, it replies with HTTP/1.1 304 Not Modified and then you can mark your content as being fresh again having only transferred a few bytes.

You can view this in a trace by seeing lots of 304 responses typically coming back after you hit browser Refresh.

But now we're getting away from the 'simple' aspect of this thread ... :)

[SMHarman]

Correct - but there's a way of using even less bandwidth.

If you issue an HTTP GET and also include an If-Modified-Since header then the web server will only send you the object if it has been modified since the date that you specified. If it hasn't, it replies with HTTP/1.1 304 Not Modified and then you can mark your content as being fresh again having only transferred a few bytes.

You can view this in a trace by seeing lots of 304 responses typically coming back after you hit browser Refresh.

But now we're getting away from the 'simple' aspect of this thread ... :)

There are some well formatted caches and browsers out there. Looking at my site stats about 25% of traffic is 304'd. That saves me a good bit in bandwidth costs.

[mr-b]

There are some well formatted caches and browsers out there. Looking at my site stats about 25% of traffic is 304'd. That saves me a good bit in bandwidth costs.
Absolutely, and if you specify explicit expiry times e.g. by using a header such as:
Cache-Control: max-age=3600
on appropriate images etc. then it optimises your web performance even further since the browsers/caches will know exactly when to refresh the content and won't need to guess by sending periodic IMS requests.

For instance Google's logo.gif expires in Jan 2038! Of course if they want to change it then they just change the name.

[Chrysalis]

what would happen if the page was updated

and Cache-Control: max-age=3600 was after the update?

[mr-b]

what would happen if the page was updated

and Cache-Control: max-age=3600 was after the update?
A simplistic approach to this is to consider a web page as a collection of objects referenced by an html object. As the content publisher you apply appropriate expiry times to your objects e.g. images tend not to change very much, so you might expire them in a month. Your html might change more often so give html objects expiry times of 3600s.

If you apply inappropriate expiry times then you risk browsers and caches storing your content for longer than is correct. The usual rule of thumb is to keep the html objects small and with short expiry times (or mark it uncacheable) and mark the larger objects with longer expiry times. The larger objects are the ones that you don't want being downloaded all the time anyway, since it will both load your web server and give users the impression that your page is slow.

If you want to change the page and objects completely then change all the objects' names and then you can start afresh.
More details are at http://www.web-caching.com/mnot_tutorial/ (might have posted that link before)

[asdf]

Yup, I noticed proxies had been turned on in both Leicester and Nottingham in the past week or so. Previously I'd pretty much always been proxy free.

Seems okay so far.

[Chrysalis]

mr-b that would explain why I had so many problems on the development pages I used, the first time I downloaded the file probably cached it and because its a large file it is probably cached for a long time, but the file is updated at the server end numerous times a day, the cache made it pretty much unusable for me to use. This was before I setup my own proxy of course.

[mr-b]

mr-b that would explain why I had so many problems on the development pages I used, the first time I downloaded the file probably cached it and because its a large file it is probably cached for a long time, but the file is updated at the server end numerous times a day, the cache made it pretty much unusable for me to use. This was before I setup my own proxy of course.
Size of file has nothing to do with how long browsers or proxies cache the object for. They cache the object until they run out of allocated disk space and something else replaces it.

Just run your dev server's URL through the Cacheablity Engine http://www.ircache.net/cgi-bin/cacheability.py and it'll tell you exactly what's going on (Definitely given that link out before).

If the file is being updated frequently then its Last Modified header will be more recent which will tell your browser or proxy to verify its freshness more frequently.

[Chrysalis]

"The usual rule of thumb is to keep the html objects small and with short expiry times (or mark it uncacheable) and mark the larger objects with longer expiry times. The larger objects are the ones that you don't want being downloaded all the time anyway, since it will both load your web server and give users the impression that your page is slow. "

I thought by that you was reffering to size of file?

I ran a link through that url and it seems to confirm my proxy server is only proxying and doing no caching which is what I want :)

[theman]

i just ran ntlworld though that site and it said they are stale

[mr-b]

i just ran ntlworld though that site and it said they are stale
The analysis is not saying that the actual object is stale, more accurately that browsers and caches will treat the homepage html object as being stale and so always fetch a new copy every time it is accessed. ntl have done this by setting a cookie on the homepage which effectively makes it uncacheable.
The gifs have Last-Modified headers on and so will be cached and verified according to age.

[erol]

The problem with proxies is they work on a flase premise. The false premise they work on is that external bandwidth is expensive - and thus it is worth spending time money and resource on reducing external bandwidth requirments. It is very like the false premise that 'storage' was expensive - that led to people coding year date fields as 2 numbers and not 4 - well into the 80's and beyond, thus creating the y2k problems.

I would challenge anyone to prove to me that the cost of running and maintaining the NTL proxies is less than the amount saved on external bandwidth costs. If you add in a cost for the problems these proxies cause for users as well then the figures make even less sense. Of course you can also use such proxies for other objectives - like limiting what and how much people can browse but that is apparently not the objective for NTL.

PS there was an onlinbe petition setup a while back - but it's not had much support so far so maybe NTL's forced proxies are not that much of a problem to most users?
http://www.petitiononline.com/ntlhpp/petition.html

[SMHarman]

Erol - you also need to factor in the bandwidth costs that the supplier of the data pays. A site pays its host bandwidth cost. NTLs (and others proxies) reduce the bandwidth I use at my host by about 25%, meaning I stay within my prepaid bandwidth.

NTLs proxies generally work well for me so I don't complain. Ctrl F5 addresses any problems. Occasionally I need to specify as the transparant Luton/Ware has gone screwy.

I like to think of proxys like warehouses. Just because a pair of shoes is made in China this doesn't mean the entire stock stays there until its called for. They are shipped out to the other continents for quicker delivery.

[erol]

Erol - you also need to factor in the bandwidth costs that the supplier of the data pays. A site pays its host bandwidth cost. NTLs (and others proxies) reduce the bandwidth I use at my host by about 25%, meaning I stay within my prepaid bandwidth.

This is true but I doubt NTL run the proxies to save your web hosters external bandwidth. For me if you are doing a cost evaluation from NTL's perspective of how much the proxies save vs how much they cost - then what is relevant is how mcuh they save for NTL (and not how much they save for NTL + third parties)


NTLs proxies generally work well for me so I don't complain. Ctrl F5 addresses any problems. Occasionally I need to specify as the transparant Luton/Ware has gone screwy.


Well I am not an NTL user and never have been so my personal experience is zero. However having hung around this forum and other NTL related forums it seems to me that 'try changing your proxy' is one of the most common pieces of advise given and thus it would appear that NTL's forced proxies do create significant problems for significant numbers of users. For me it is all about choice. I do not think NTL should necessarily scapt its proxies but just that it should allow people to chooses whether they use them or not.

[SMHarman]

This is true but I doubt NTL run the proxies to save your web hosters external bandwidth. For me if you are doing a cost evaluation from NTL's perspective of how much the proxies save vs how much they cost - then what is relevant is how mcuh they save for NTL (and not how much they save for NTL + third parties).

True however this is indicative of the additional bandwidth of a small site, what would it be like for Amazon or other popular sites? Ultimately it could put up the cost of doing business on line as the bandwidth costs need to be recovered in the selling price.

This is also a reciprocal thing. NTL host sites. If all the proxies on the internet were switched off, then the bandwidth hitting NTLs web servers (their own ntlworld and all their customers) would increase and require more hardware and bandwidth. Bandwidth will become a more used resource, more scarse and more expensive.

Well I am not an NTL user and never have been so my personal experience is zero. However having hung around this forum and other NTL related forums it seems to me that 'try changing your proxy' is one of the most common pieces of advise given and thus it would appear that NTL's forced proxies do create significant problems for significant numbers of users. For me it is all about choice. I do not think NTL should necessarily scapt its proxies but just that it should allow people to chooses whether they use them or not.

Yes they screw up for me too. I can get mail, message, but not browse - its the proxys that are the problem in that instance.

[erol]

This is also a reciprocal thing. NTL host sites. If all the proxies on the internet were switched off, then the bandwidth hitting NTLs web servers (their own ntlworld and all their customers) would increase and require more hardware and bandwidth. Bandwidth will become a more used resource, more scarse and more expensive.


Bandwidth will only become more expensive as it is more used if the rate at which bandwidth cost is falling is slower than the rate at usage indreases, and it is not. Yes it is reciprocal but then so is the cost of running proxies (need software, needs hardware, needs people to configue and manage them). I still do not believe that the amount of money saved on external bandwidth (at either end) is greater than the cost of running the proxies in the first place (at either end). This is only going to become truer and truer over time as bandwidth costs continue to drop - yet costs in staff to maintain the proxies rise.

[Chrysalis]

If the proxies were turned of it wouldnt make any extra external traffic to the ntl hosted websites, all that traffic would be internal if their network is setup properly, for instance I leeched a file of a website onto a box on the same isp as the website and it zommed across at 1000mbit on the internal network.

[SMHarman]

If the proxies were turned of it wouldnt make any extra external traffic to the ntl hosted websites, all that traffic would be internal if their network is setup properly, for instance I leeched a file of a website onto a box on the same isp as the website and it zommed across at 1000mbit on the internal network.

Yes traffic over the NTL subnet of the internet will all whizz around missing thier proxys as internal traffic.

If NTL switch their proxies off then every request to sites hosted on Thus, Tiscali, Telewest and all other external subnets will go to those subnets and none of the traffic will be caught by proxies. A resultant increase in traffic for the other sites.

If all ISPs switched off their proxies, then all traffic from say Blueyonder or Demon to NTL hosted sites would be directed to the server not a proxy, resulting in an increase in traffic on NTLs subnet.

[erol]

If all ISPs switched off their proxies, then all traffic from say Blueyonder or Demon to NTL hosted sites would be directed to the server not a proxy, resulting in an increase in traffic on NTLs subnet.

Well many many ISP (I would even venture most) already do not use _forced_ proxies anyway. BT Openworld used to a long time back - and abondond the idea for example.

If any that did 'turned off' their proxies overnight then yes their external traffic would increase - but if this increase in external traffic cost _less_ than running the proxies in the first place then they would _all_ be better off. That's the (same) point made nefore in relation to NTL. Introducing 'both ends' makes no difference to the basic common sense of the point.

[Rik]

Bravo
Lets have some more threads like this, makes a nice change from moaning posts!
Very informative thanks very much :)

[Paul]

Bravo
Lets have some more threads like this, makes a nice change from moaning posts!
Very informative thanks very much :)

Thank you - it's nice to be appreciated. :)

[Lee]

So how would I go about reporting a problem with a particular 'transparent' proxy?

The one (or more?) in Kirklees is messing up web pages, causing lots of script errors and most importantly stopping eBay from working!! (Have verified thatusing a different proxy solves these issues)

[Paul]

Contact ntl Technical Support would be the best way.