GLossary Pages

Sunday, April 24, 2011

A little about how email Clients get Email from the server

I am writing this because there appears to be considerable misunderstanding about just how the process works. I see in support forums comments like

"My mail works just fine, I have logged into it from my providers web site"

All this does is prove that you know the current account password and the login name your provider has supplied. It also gives a reassurance that their mail system is still working. Other than that it does not help with the process of making your mail client work.

Before we can talk about how, a little about the protocols involved

Mail is really a collection of communication protocols. Mail clients 'talk' to the mail server to get your mail, and unless they talk to same language as the mail server nothing works. There are three common communication protocols in use that most of us use at one time or another.

EMAPI
This is a closed protocol owned by Microsoft and is the native language that outlook uses to talk to Microsoft Exchange servers. Because it is proprietary very few mail clients even attempt to use it. A large part of the reason is that Home users (the biggest segment of mail users who have control of their own mail) don't, as a general rule, have an exchange server running in the corner of the lounge room to supply their own mail domain.

IMAP
A sychronised mail access method which is designed from the ground up to keep your messages on the mail server. This is ideal for the folks that like to get their mail on a computer a phone and other devices and have things just synchronise between them so that every device shows the same thing. The draw back of this protocol is that if the provider deletes your account you may loose everything because of the synchronisation.

POP
The oldest of the mail protocols, and by far the one most people have been using until the last few years. This protocol is designed to act in a very similar way to the snail mail service. You address a letter (email) and it is delivered to the recipients mail box where they download it and it is deleted from the server.

POP does not keep messages anywhere but in the mail client that downloaded them. It is very difficult do synchronise messages between devices but is quite reliable, for a protocol that does not guarantee delivery.


Connecting the pieces

When you get your email client, it knows nothing of you or your mail. You need to tell it about you and your mail provider and in the process tell it which of these protocols are to be used and exactly how to find your mail out there in the internet. There are about a dozen different setting and methods of connecting. These options are set by the mail provider and the client MUST match them exactly for the process to work.

The authors of mail clients understand that the average person is somewhat daunted by the intricacies of these setting and try and streamline the process to expose their newest convert to as little of the process as possible. Hence wizards appear to guide us through what is a veritable mine field of accounts, password protocols and ports.

So how does it go so wrong?

There are a number of reasons for this. From good old fashioned human error right through to computer programs that think they know better and use the wrong settings.

It is the computer programs that think they know better that is my interest here. With the advent of ubiquitous anti virus programs and firewalls downloading your mail has just become a lot harder because now we have a couple of other hurdles to step through before the mail client can actually talk to the server.

Have you ever wondered how your anti virus program manages to 'scan' your incoming and outgoing mail?
It is a shame that most people never even consider this as it is something of a wonder, and a nasty imposition at the same time. Almost all anti virus programs set themselves up as what is known as a proxy server. As a proxy, ALL communications are routed through that proxy. This is very good for the anti virus program (and you in not getting virii) but it can be bad for the email client. Those protocols I mentioned earlier rely on things happening in a set order in a timely fashion. If the proxy spends to much time scanning and re routing messages, things go to hell in a hand basket. You get a message from your mail client that simply says timeout. No mention of the fact that it is the anti virus program is causing it. Your mail client does not even know that the anti virus has insinuated itself into the process.

Other problems can occur if the proxy is in any way less than perfect. Messages can be truncated, not delivered to the mail server or simply just disappear into the ether. Again your mail client will have no knowledge of this occurring and will continue to think it has done it's job or that the mail server has a problem and give you erroneous error messages about the mail server.

Firewalls.

A firewall is supposed to control message traffic to and from your computer and restrict it to what you have approved. Very simple and quite effective in principle. Very complex and exceedingly tiresome to set up in practice. It used to be a specialty in computing and still is in big organisations and data centres. The average person however is not going to pay a specialist to come in an set up their firewall, so again with the wizards. The vendors of firewalls have done a huge amount of work to take what is a complex process and turn it into a point and click process. It is this work that lays the trap for the unsuspecting customer to stop everything in it's tracks.

You are typing away and your firewall helpfully pops up a message telling you that some program is trying to access the internet. Would you like to allow or disallow this. While really good in theory many bad firewall decisions are made based on the default button on that dialog.

Many people when typing look at the keyboard when they type, not at the screen. So they have no idea that they just clicked yes to the dialog that they did not see. Suddenly their shiny new email program does not work right. It can't send mail or some other feature does not look right. The first obvious point of call is the email program. You have changed nothing have you, so it must be the email program.