Domain reputation and blacklists
In the beginning of June, we introduced a new antivirus by Revisium in ISPmanager. One of its key features is monitoring domain reputation. Grigoriy Zemskov, the head of Revisium website security department, tells about domain blacklists: what they really are, how getting blacklisted influences website traffic, and how to protect the reputation of a web page.
Usually domain or IP address reputation is considered in the meaning of email delivery when messages from a particular domain or a server are meant to be safe or spam. But in fact, the concept of reputation has a much broader meaning. In general, reputation can be defined as the level of trust to your domain from a security perspective.
Blacklists of domains
Domain's reputation is monitored by search engines, web browsers, and antivirus solutions so that they could warn users about entering a suspicious web page. They get information from so-called blacklists, databases of dangerous domains and IP-addresses. The logic is simple: if a website is blacklisted, it might be insecure. That means all blacklisted websites used to have — or still do — both security problems and, as a consequence, a negative reputation.
Many antivirus and online services maintain their own lists of dangerous websites. They register those resources that spread viruses or malicious code or redirect visitors to phishing or a malware site, etc. Let’s look at the most credible and authoritative sources for domain verification.
Google Safe Browsing
Google and other search engines maintain their databases of suspicious and dangerous domains, for example, Google Safe Browsing. These databases are formed as a result of scanning the indexed websites by an antivirus bot. The bot checks websites for phishing, hidden redirects, malicious code or dangerous files. If it detects a threat, the domain gets blacklisted by a search engine. After that, it can appear in blacklists of one or more antivirus services.
Blacklisted domains do not necessarily belong to intruders. This list may include, for example, a website of an online store or a corporate portal that was hacked and began to spread malicious code.
Chrome, Firefox, and Safari use Google Safe Browsing to restrict access to websites that are banned by the search engine.
You can find out about getting blacklisted by search engines, for example, from the Google console, if you add your website in advance.
VirusTotal is an aggregator of information about viruses and dangerous websites. It collects data from 67 services, including Kaspersky, Dr.Web, ESET, Trustwave, CleanMX, PhishLab, etc. These services upload their databases of threats and infections into the VirusTotal database. Therefore, if a domain or an IP-address gets blacklisted by at least one of these 67 antivirus services, all users of VirusTotal database will find out about it.
With the help of VirusTotal, it is easy to check the domain status in different antiviruses. To save the reputation you can quickly detect that your website is blocked by a specific antivirus
Problems of blacklisted websites
The main problem of a blacklisted website that visitors cannot access it, which leads to traffic drop. There are several reasons for this:
- If a website faces sanctions by the search engine, forget about search traffic. In search results, your site is marked as unsafe. In some cases, its positions might be lowered or even excluded from search results.
- In browsers that use Safe Browsing API (Google Chrome, Opera, Firefox, and Safari), when switching to a banned site, the browser displays a red lock screen and a warning about viruses or fraud.
- When the site is banned by antivirus, anyone who uses the same solution and tries to reach the web page will see the proactive protection thread warning, or the site will be simply get blocked.
In addition to losing traffic, getting blacklisted threatens the company reputation. The warning that a site is fraudulent and threatens security is likely to leave a negative impression of the company. What is more, information about the blocking can quickly spread through social networks.
Why do websites get blacklisted
There are not so many reasons for websites to get blacklisted:
Hacking and malware infection.Through vulnerabilities in the code, a hacker or an attacking bot can upload malicious code to the website: for redirection, theft of data or spread of viruses. This website will be dangerous for users, search engines and antiviruses will quickly detect it and restrict access to the website.
Customer complaints . Website visitors may complain about illegal content, indecent advertising or security risks via special forms on the websites of antivirus companies. If a complaint is confirmed, a website will be banned.
Spamming. If there is a spam mail from the domain or IP-address, they are likely to be listed in SpamCop or Spamhaus database and other services. Messages from the domain will be put into in the "spam" folder or won’t reach the addressee at all.
Spam from the website can be caused not only by server hacking but also by mass registration of bots with email notifications (for example, online registration). Or because of the scripting vulnerability, which allows you to send email to arbitrary recipients without authorization (as it happened with VirtueMart component in Joomla CMS).
How to remove a website from blacklists
If the domain or IP-address faces penalties, blacklist removal process can take from one day to several weeks. Google allows for the fastest removal process: usually, it takes one day if the website site has not faced manual penalties. It may take a long time to remove from the blacklist of antivirus services. Some of them (especially little-known and new ones) may detect the site as a threat by mistake, for example, as a result of so-called false triggering. If the service does not allow to remove the website block automatically, you will have to send, you will have to send an application, and its consideration can take several weeks.
To exclude a site from the blacklist of search engines, you need to complete three steps:
- Eliminate the cause of penalties (source of the problem).
- Add the site to Google Webmaster Console
- Send a request for the removal through the panel.
To remove a website from the blacklist of antivirus programs, use the application form on their site. Usually, the application submitted in English. It is necessary to briefly describe the essence of the appeal and note that the domain got into the database of malware by mistake. If you can select the type of address in the form, you should specify "False Positive."
How to avoid being blacklisted
Monitoring website security will help you avoid domain blocking, save your reputation or at least reduce the risks of a site getting blacklisted. Its goal is to detect the problem before the search engine bot or antivirus service does. Once it is detected, the specialist will be able to quickly remove it and, thus, protect the site from getting banned.
However, it is better to take care of your website security in advance. Install protection tools against web attacks and hacking, work out security measures when working with the site. Ask specialists to help you with this.
A simpler version is to use the Revisium Antivirus module for ISPmanager Lite (from version 5.155). It supports automatic checking for infection and penalties. Both free and premium versions are available.