uber
UberGlobal Business provides the confidence of business class hosting with Australian 24/7 phone support and a variety of dedicated hosting and managed services.
UberGlobal Enterprise offers reliability and secruity through managed infrastructure, platform and software responses for clients seeking enterprise grade solutions.
UberGlobal Wholesale provides whitelabel reseller services via our SaaS and IaaS platform. Generate revenue for your business using our products and services.

Mail Delay Incident Report

Background

Outbound email from Uber Shared Hosting customers using mail clients (ie Outlook etc, not Webmail) normally works as follows:

  • The User creates an email on their personal computer, mobile device etc using their mail client (Outlook, Mail etc).
  • The email leaves the user’s computer/device and travels to the Shared Hosting Server account provided by Uber (both Plesk and cPanel shared hosting servers) via smtp.
  • The Shared Hosting Server sends this email to the recipient via Uber’s internet connections.  Uber filters outbound emails as they leave our network for spam and viruses to ensure that the Uber network is not a source of malicious traffic to the internet.

 

The one-to-one relationship between the User and the Shared Hosting Server account is authenticated by the email address and mail password.

 

There are many ways for spammers to exploit this system.  One common method is to obtain or guess the user’s email addresses and then attempt to guess the password.  The user’s email address can be obtained from many different places on the internet, including straight off the user’s website itself.  An email address can be guessed because shared web accounts typically set up common email addresses such as admin@ or support@.  Weak passwords can be guessed using brute force dictionary attacks.

 

Web servers have features built in to prevent such attacks:

  • Brute force attacks are sometime defeated by locking accounts if too many failed logins occur from the same IP address.  Spammers can overcome this approach by using a large number of coordinated servers to ensure that each guess comes from a different IP address.
  • Testing passwords for strength by checking for non-standard characters or running the password against a dictionary before accepting it.  These measures are often effective but unfortunately they only been implemented seriously by a number of Shared Hosting control panel developers in the last few years.  Accounts that have been in existence for many years are likely to have been established before rigorous password testing was implemented by control panel vendors.

 

Finally when people hear that passwords are tested against a ‘dictionary’, they may think ‘Oxford’, ‘Webster’ or ‘Macquarie’.  The password dictionary we used to conduct our audit for weak passwords has over fourteen million entries.  It includes variations on ‘password’ such as ‘P@5sw0rd!’ and many non-words such as ‘1234567’.  Unless your password contains a mixture of upper and lower case characters, special characters and numbers, then it is likely to not be very strong.

 

The Incident

Commencing on Monday 12 March, Engineers noticed an increase in spam being generated from both Plesk and cPanel Shared Hosting Servers. Due to the nature of low cost hosting there are often minor spam outbreaks generated from within the Uber shared hosting fleet.  Uber employ outbound mail filtering to catch spam before it leaves our network and alert our Engineers who then rectify the source accounts. However the increase on Monday was significantly higher than normal and began to overload our outbound mail filters.

 

Initial investigations into the spam source revealed a large number of spamming email accounts. Engineers investigated and discovered that the spamming accounts were located on both cPanel and Plesk Shared Servers.  It appeared that the spammers had obtained copies of both the email address and passwords of the email accounts.  As the spamming accounts were identified the passwords were changed, thereby stopping the spam, and the customers notified.

 

Over the week more and more accounts began spamming causing the number of spam emails to increase dramatically to the point that the Mail Transfer Agents employed to forward email to the internet were at times overwhelmed with the volume of mail, causing delays in sending and receiving mail.  Later in the week Uber began to discover scripts loaded on a small number of accounts that were also starting to generate large volumes of email traffic.  As each account is discovered the password is reset (and if relevant the script removed) so the account can no longer be used to send spam.

 

Investigations have shown that the only common factor in all these events is the use of weak account passwords.

 

Analysis of logs shows:

  • No obvious password brute-forcing in progress presently which indicates the reconnaissance phase is probably over and now the spammers are in exploit phase.
  • A very large number of servers initiating spamming smtp connections, suggesting that a large netbot is probably being used to target the Uber infrastructure.

Unfortunately these two facts means there are no obvious high-threat nodes that Uber can block to reduce the volume of the event.

 

Analysis of historical logs shows no obvious high volume brute-force IP addresses which reinforces our view that the source of the traffic is a large netbot, indicating a sophisticated spammer.

 

Root Cause

The root cause of the delays in email delivery over the last week is the use by spammers of Share Hosting email and other accounts with weak passwords to send large volumes of outbound spam. This spam clogs the mail filters, slowing the email at points during the day.

 

Risk to Customers

The passwords that the spammers are guessing are mail passwords, not account passwords.  There is no evidence of spammers gaining access to customer’s billing details.  However with access to the email address/password combination it possible to read a user’s email.  Based on our analysis of the logs we do not see any evidence of spammers accessing customers email, but this is always a possibility.  However this event highlights the importance for internet users to ensure they use strong passwords in their dealings on the web.

 

Short Term Fix

A number of methods to aid with rapid detection and email delivery management have now been employed to minimise the impact to customers allowing Uber to very quickly identify and remediate accounts that are in use by spammers.

 

Medium Term Fix

Planning is underway to perform a large-scale password update for customers identified as having weak passwords.  We will be shortly contacting these customers.  If you believe that you have a weak password for any of your accounts presently held with AussieHQ or Jumba it is suggested you change it immediately.

 

Long Term Fix

There are at least two possible long term fixes:

  • Investigate methods for retrofitting password strength testing tools to old control panel distributions (upgrading the control panel software will not retrospectively force the passwords to be re-tested).
  • Implement routine password strength audits to identify Users who have entered weak passwords and contact them to upgrade their strength.

 

Other options continued to be investigated.

This entry was posted in Customers, Technology. Bookmark the permalink. Trackbacks are closed, but you can post a comment.

3 Comments

  1. twz
    Posted March 16, 2012 at 5:50 pm | Permalink

    Couple of questions..

    Is/was this problem affecting outbound email only? ie was there any delay if sending via our ISP mail server to an address hosted by AHQ/Jumba?

    Are passwords in cPanel/Plesk stored as plain text? How else could you identify users with weak passwords?

  2. Bob Tatus
    Posted March 18, 2012 at 2:51 am | Permalink

    I’m pretty sure cPanel stores them using md5(unix) hashing, and Plesk 9 stores them in the database in plain text, I think with Plesk 10 they are encrypted, so the root user could always extract them from the MySQL database, and with cPanel you can enforce password policy upon log in if your password isn’t strong enough.

  3. Posted March 19, 2012 at 9:09 am | Permalink

    Hi twz,

    Email passwords in Plesk are stored in the clear. Email passwords in cPanel are stored as a hash however by taking the password dictionary and passing it through the same hash function it is possible to make a pretty good guess about what the email passwords are. NB: Server root passwords are more strongly protected by measures built into the operating system.

    Also we have done some more analysis work over the weekend and it appears the majority of the botnet addresses are originating from China.

    Cheers,
    Heather

Post a Comment

You must be logged in to post a comment.

UberGlobal RSS