Struggling with e-mail systems

As a perfectionist, I have been struggling to have a perfect PIM solution. For the personal part, I currently use (Exchange-based) Outlook.com, with which I am much content. For the professional (institutional) part, things have been so hard. This entry records some of the problems I have encountered so far, and for certain ones, a solution. Expect this entry to be a mess and full of complaints.

Earlier entries regarding e-mailing include A bug of Outlook 2016, Outlook 2016 GUI creates… and Course management with Outlook object model (in Chinese).

Programmatic access

There are a lot of reasons why I choose Outlook.com. One of the most important reasons is that it allows a rich set of APIs to access its data. The two sets I use most often are Microsoft Graph and Component Object Model (the link is for VBA but COM works with any Win32-compatible programming language), of which the latter is preferred.

If you are using Gmail, the only set of API of which I am aware is their cloud API (here, there might be two versions?). The same reason I prefer COM to Graph applies here: you can use COM to create data you want and export them (if you are using it for mailboxes you are not synching to Outlook), and you don’t have to be online, and the exported data are applicable to a wide range of services (but the cloud API only works with specific products).

Label v.s. folder

There is another reason I don’t primarily use Gmail. It doesn’t have a ‘perfect’ client for Windows. (There might be one, e.g., Gmail for iOS, on iOS. But that’s another story.) The client is perfect if it maps well with the concepts found in the service and gives you access most functionalities available in that product. Outlook is a ‘perfect’ client for Exchange-based mailboxes, and is ‘perfect’ in more aspects, e.g., programmatic access.

The main problem here is the concept of Label. Gmail maps Label to IMAP folder when you sync Gmail using IMAP. That’s a good but failed try. Messages in different IMAP folders are supposed to be different messages (i.e., have different identities). A Gmail message with multiple labels appears in multiple IMAP folders and are still a single message.

There is an another problem, which is not Google’s fault. I don’t know whose fault it is, but Outlook seems to not sync Sent Items folder after a message is sent. If you don’t sync the folder manually, you will temporarily have an incomplete view of the conversations. It could be Outlook or the way IMAP works.

Yet another problem, which I think is a bug in Outlook for iOS, is that Outlook for iOS doesn’t seem to sync a Gmail inbox at all if you use ‘Google’ as the provider (not using IMAP). The hierarchy for a defective Gmail account set up in Outlook for iOS resembles that for an Exchange-based mailbox, for which the tell-tale is the folder ‘Conversation History’. The only time I could sync Gmail with Outlook for iOS, I couldn’t send messages from that account.

The ideal solution is to have the imaginary, native Gmail client for each platform. Ideally, there should be a relation among Gmail client, Gmail backend and Gmail.com, analogous to that among Outlook, Exchange and Outlook Web Access.

Label has created more problems, especially when Google itself doesn’t manipulate labels on messages carefully. See Spam filtering.

Incoming mail tagging

I maintain a repository of e-mail addresses from which automated e-mails are sent. These are used to sort incoming mails. I even have a dedicated script (with GUI for OAuth) to deploy (update) the list to my inbox rules, which is currently not open-sourced.

Many argue that artificial-intelligence-based sorting is already good enough. My point is that tagging using the list of automated addresses is more reliable and totally under the user’s control.

Spam filtering

I don’t like it when spam filter sucks. And yes, Gmail’s spam filter sucks no less than Outlook.com (or Exchange Online) does. Gmail classified a job offer to my friend as spam. See also Dual deployment.

I like the ability to completely disable the inbox-level spam filter. This is available for Gmail users (personal account and organizational account). Here’s how I tried it: Create a rule that Never sends mails to Spam if the size is larger than 0 MB. This is wrong as it has the side-effect of putting every mail you send to Inbox. The reason behind this is that the rule works by applying Inbox label to any message newly put into your mailbox, where ‘newly put into’ includes those put into your mailbox because you just sent them. Gmail doesn’t use the concept of folders, and having the Inbox label means the mail appears in your inbox. The very, very stupid thing here is that the rule applies to messages you did not receive (that you sent).

To fix this, change the rule to Never send mails to Spam if the sender does not include yourself. Cite this StackExchange post.

Dual deployment

As cloud services become ubiquitous, many universities have adopted e-mail systems from Microsoft (Exchange) and Google (Gmail). Some even have dual deployment, like UW CSE. However, there are new problems related with dual deployment.

First let’s talk about another school. When I was first enrolled to UCSB, the mail system was Office 365. I was happy because I could reuse my incoming mail tagging script. But hope was broken in just a few months, as they decided to move to G Suite. My complaint was that the school IT should have deployed the transition before new students arrive. It is very unreasonable to ask newcomers to migrate within, like, 3 months of setting up their account. G Suite is okay for me, as I have found the way to live with it.

Now for the UW story. Initially, I chose Office 365 as my UW CSE mailbox. However, I wasn’t able to reply to my job offer. To work around this, I switched to G Suite and resent the reply. Of course, I also did some observation and opened a ticket with the support.

As a veteran in Microsoft products, I noticed that the problem could be Exchange delivering mails internally (since every user has two mailboxes, in Exchange and in G Suite), so the reply never reached UW CSE mail server that transports the message to the system selected by the recipient. But how come I could send messages to certain, other users using G Suite and in the same domain? They also have an Exchange mailbox. It turns out that there is a forwarding rule set in my Exchange inbox that redirects the messages to UW CSE mail server, which will select the correct destination. I bet the same for other users, but somehow the recipients who did not receive my reply disabled forwarding accidentally.

I sent the above observation and theory to support. After more investigation, it turned out that the affected people have forwarding set, but Gmail somehow classified my message as spam before the message reached their inbox. Gmail sucks because:

• The behavior is consistent for the message. I sent two replies before switching to G Suite, and both replies got the same classification result.
• The behavior is inconsistent for the recipients. All recipients of the reply use G Suite, but only a proper subset of them failed to receive the mail. (Not even in their Spam box.)
• UW CSE IT has already set our own domain as a safe domain, which should stop Gmail from filtering our own mails.

The problem could have been transient. But the mere fear that silently failed delivery might happen again deprives my freedom to choose Exchange. Also, we don’t have a stable reproduction yet, so we cannot escalate the issue with Microsoft and Google.