So there’s a lot of speculation currently about what Trump was doing with a mail server that (from DNS traffic analysis only) appears to only have been used to talk to a select set of servers, mainly at a Russian bank. Who set it up? What was it for? Was Trump even aware of it?
I’m in no place to answer those questions. I’m just someone who’s been in IT a while, by no means an expert, but… I do know when things aren’t set up right. That’s my job — I fix servers that aren’t working as expected, which includes diagnosing why they’re not working as expected, based on logs of their activity and current non-working behavior. I’ve also been a technical instructor, and a good one, because I’m very apt at turning techie concepts into layman’s terms.
So, when I read the Slate article alleging that Trump had an email server set up to talk only to a Russian bank’s email server, I had to think about how it would behave. What follows is technical stuff, but I’ll try to render it in layman’s terms.
First off, this is how email is supposed to work. There are two protocols:
- Simple Mail Transfer Protocol (SMTP) — otherwise known as a “mail drop”
- Internet Messaging Access Protocol (IMAP) — otherwise known as a “post office box”.
Like the real world counterparts, SMTP accepts inbound and outbound mail to be delivered, and IMAP stores it for the user to read/download. Pretty simple so far.
Now, SMTP has “Ways To Set This Up To Not Piss People Off”. Back in the early days of the Interwebs, a man known as Sanford Wallace set himself up as the Email Mass Mailing Go-To Guy. That’s right. This fellow invented spam email. And because of his rather prolific methods (and attempts to get around spam blockage by using open relays), most SMTP servers were set up to allow only the following:
- Accept email from world+dog, but only to addresses in the local domain
- Accept email from addresses in the local domain, to be sent to world+dog
(This works as an anti-spam measure, because a spammer would either A) only be able to target customers one ISP per mass mailing at a time, slowing him down, or B) using his own ISP’s servers to send spam, which would get his mass-marketing tuckus kicked off the ISP.)
So, again with the real world analogy. If I send a postcard from Houston to Seattle, it has a Houston postmark on the stamp. If I travel to Chicago to send the same postcard, an open relay post office drop would take my postcard, stamp it with the O’Hare Airport postmark, and gladly deliver it to Seattle with a Houston return address. The latter is an example of an Open Relay. If instead, the postmaster at O’Hare said, “Sorry Mr. Blues, you can’t send that from here, you’re not a Chicago resident, you need to send that from Houston when you get back” then it would be a Closed Relay.
So, now, out of the box, most SMTP systems are designed to be operated in this Closed Relay mode. They’ll send mail anywhere, and accept it from anywhere, but their own home turf determines what actually gets sent or delivered, and what gets rejected.
And the findings of Tea Leaves and his cadre of researchers did indeed find that, as far back as 2009, there was an email server within Trump’s real estate domain, sending out “marketing email”. However, they found that this specific server changed roles at some point. Instead of sending to any and all domains that serviced email accounts on Trump’s mass-mailing lists, it only talked to a few other computers.
----—
[redacted MAC section — not really relevant now that I’ve slept a bit since writing this out]
Now, we look at how this one-time spam email server gets repurposed to talk to a healthcare company run by the Amway family and a bank in Russia… and no one else.
Trump’s SMTP mail system worked off a “white list”, which means, it had to know beforehand which systems were allowed to ask to talk to it. If you tried to establish a connection to the SMTP protocol, unlike most mail servers that would respond with “HELO”, acknowledging there was indeed a mail server running, this one fed back an error. But, this only happened if you weren’t on the list. A DNS lookup for the server would respond with an IP, but trying to talk to that IP would get nothing. However, there was DNS traffic from specific machines that showed a lot of lookups to this box, and lookups from this box back to those machines.
Moreover, they weren’t evenly distributed and constant as you’d expect from a botnet, or one-way as you would expect from a “zombie” or compromised system used by a hacker. Instead the times of these lookups corresponded to a human-like conversation cadence — message goes one way, and a few moments later, another message goes the other way. These transfers also happened during business hours in their respective time zones, such that someone (likely at work) was initiating the process… and then the conversation would convene again from the other direction, again while someone was at work checking email.
The question is, why would someone change the settings of a mass-mailing server to turn it into a “hotline” email relay, if not to set up this arrangement? It’s not like it was hacked — otherwise one would expect the traffic to fit machine schedules aka constant 24/7 activity OR only responding from one time zone, not both.
---—
Now, what about the specific hostnames, server disappearance and name change? How did that work?
DNS stands for Domain Name System. It’s a vast collection of databases, that together map domain hostnames such as www.dailykos.com to respective IP addresses. Think of it as an Internet Phone Book. You feed it a name, and it responds in one of two ways:
- I don’t know that host, here’s the (next) authority, ask there.
- I know that host, here’s the IP address you want.
This is called a Recursive Query. It’s like looking up a phone number for a Denver address. “I don’t have a listing. Let’s find a Colorado phone book.” The Colorado phone book person says “I don’t have a listing for that specific address, but this phone book covers Denver, check in there.” The Denver phone book reply is “Oh, yes. I have that, here’s the number.” And then you connect to the system directly.
For this system to work, each level of the DNS hierarchy needs to know who to ask next. For the example www.dailykos.com:
- The ROOT authority sees “.COM” and refers to that namespace.
- The .COM authority looks for dailykos.com, and refers to that namespace.
- The DailyKos.com DNS server says “Yah, WWW is our web server, here’s the IP address.”
Once you’ve set up a DNS listing, it has a Time To Live (TTL). This is a value within the DNS table (and sent as part of the lookup response) that tells the receiving system how long it can be kept, or cached, before asking again. Think of it as a sell-by date. If a DNS record gets too old, it’s purged and a new one is retrieved from the root domain level on down at the next request. This is needed because, as we’ve seen, domains do change from time to time. Computers get repurposed, mail and web hosts move from in-house to a contract firm, etc etc. So you want your DNS lookups to be relatively fresh.
At some point soon after the Russian bank, Alfa, was asked about this weird traffic pattern, the DNS records for Trump’s mail server got taken down. However, they weren’t really taken down correctly. Two of the three resolvers in Trump’s domain had the host records for this server deleted, which meant they had NO information about the host anymore. Inquiries to those nameservers came back with “that host doesn’t exist here”. Keep in mind that it’s this last group that tells you about specific servers in a domain, so this behavior is normal for when a host gets ‘decomissioned’.
However, nameserver #3 only had the A record deleted. This is like leaving the phone listing in the phone book, but erasing the actual phone number in the listing. This resulted in an answer of “Yes, that host’s IP is <null>”. Which is, of course, broken, but it proves the existence of the host, if not its current status.
When the system came back up, it came up under another name as part of a different domain. Now, this is where another “odd coincidence” pops up. Normally if you do this, the original server disappears, and a completely new one takes its place. It’s virtually impossible to guess at the new host’s name. However, in this case, the Alfa bank server was the first system to lookup the new hostname.
Ponder this: It didn’t attach directly via IP, using a cached DNS record from the previous name (and the assumption here is that the new server kept the old IP address). It asked for the new host name from DNS.
This can only mean one thing: someone had to relay the new name of the server to the Alfa bank personnel, so that the conversation could be resumed. As was pointed out in the comments, this is like two people (with unlisted numbers) who converse regularly to each other but no one else via telephone, and then one of them gets a new unlisted number. How does person A get Person B’s new number? They can’t ask around, they can’t resume an old conversation on the old number… so Person B must communicate that name to Person A.
Soon after this, the traffic slowed to nil… likely because both parties realized their arrangement had been noticed.
This is the end of the saga of the Russian bank and the 7-year-old spam server, that had a short but increasingly infamous life as a spy thriller dead drop. (Were it my server, it would be named WalterMitty.trump.com )
Methinks Director Comey, or some other enterprising and yet not-GOP-infected FBI official, should request a warrant for that server. They know the name. They know the service address and administrative contact. And there’s a DNS record of authority saying who owns it, and who administers DNS for it. Anyone can look it up, using an Internet “reverse lookup” service called WHOIS.
As for Trump or his cronies getting to it to destroy evidence, this is likely a leased system from a service bureau, and I doubt anyone in Trump’s employ have physical access to it.
---—
Material for study:
WHOIS trump-email.com (the old name)
WHOIS contact-client.com (the org names have changed, but look at the admin/tech addresses)
(NOTE: the phone numbers listed are likely for IT personnel who work for GoDaddy and/or the IT firm contracted by Trump to do his mass mail stuff. I doubt they’d respond well to DKOS people calling, so please let the pros do that part. Thanks from a fellow IT grunt.)
For those that are interested, here is the site Tea Leaves set up to detail all of the findings.
--—
EDIT : Based on corrections by fellow IT pros that actually remembered that Layer 2 doesn’t persist past a router (I should have remembered that) I’ve edited that section out — it’d only be relevant if someone were using VPLS or MPLS tunneling, that mimics a “virtual” Layer 2 on top of Layer 3. It is something I’ve had to work with in the past, and that may be how the misconception started. But, as was stated in comments, not relevant here, at least not at first glance.
---—
Wait, REC list? no way. I need to clean up my writing, this was a 2 am braindump and I’m still feeling bleh from lack of sleep. Thank you.