Loading IP address geographic location...
Offsore Cloud Services

Am I under DDoS attack?

uk ru

When you are not able to visit your website, web application or server you can assume that the reason is a DDoS attack (distributed denial-of-service attack to make server or network to be unavailable). Sometimes it's true but sometimes it's not.

There are quick and advanced steps to determine if you are under attack or you have another type of issue. Some methods can be different depends on your infrastructure, but it's introduced as a general approach.

1. Quick steps

When you are faced to web application's failure or get an alert from a monitoring system you can try to make some steps to analyze a root of the issue quickly:

  1. read email:

    there can be a message from your hosting provider or datacenter related to the issue

  2. visit hosting provider's billing panel and website:

    check server's status (it can be suspended, stopped and etc.), check bandwidth limits and overdue invoice, connect to the server over VNC if it's a cloud server

  3. check website/application availability:

    using machines in different countries and over VPN/proxy/Tor (make HTTP(s) request or TCP port checking to be sure that it's not available only for you, check timeouts and response time, SSL certificate)

  4. ping the server:

    send ICMP requests using terminal or another machine or ping service to be sure that the server is responded and no packets loss (note: the server can work, but ICMP packets are blocked); ping nearby IPs and gateway to check that the issue is related to other servers in the datacenter as well (ping 1.2.3.5 and 1.2.3.1 if server's IP address is 1.2.3.4)

  5. connect over ssh (or VNC, RDP, KVM-over-IP) and analyze the system:

    check load average over uptime, top/htop, netstat, iostat, analyze dmesg output, check /var/log/syslog file, check hostnames resolving, check outgoing connections availability...

  6. analyze web server/application logs:

    identify any abnormal requests to web server/application

  7. check domain and DNS settings:

    domain expiration date, DNS records and etc.

During the quick research you can find out that the server is not available for you only, the server has a temporary packet loss and so on.

But if it's not obvious to determine a source of the issue quickly open a support ticket and continue your research.

2. Advanced steps

2.1 Hosting / datacenter issue

  1. maintenance period: check email for any message from support team about server maintenance
  2. server's status: check status in the control panel, it can be "Suspended" or "Stopped" (try to "Power on" it), check bandwidth limits, look for overdue invoices, try to connect over VNC/KVM-over-IP to the server and etc.
  3. twitter: visit accounts of hosting provider, datacenter and CDN for any news regarding the potential incident (example: Linode twitter status account)
  4. status page: visit status page of hosting provider (example: OVH status page), datacenter and CDN (example: Cloudflare status page) if you use it

2.2 Domain issue

  1. domain expiration: get whois data and check expiry date ("Updated Date", "Creation Date"...), domain can be not renewed, there can be some suspicious changes, EPP status code is serverHold and etc.
  2. nameservers (NS): nameservers do not response with a required IP address or there is no answer, use dig or nslookup commands: 
    dig @nameserver yourdomain.com
nslookup yourdomain.com nameserver
  3. domain can be taken over or seized by law enforcement: domain is not expired, but nameserver are changed, or nameservers reply with not your set up records, access to domain's registrant user panel is not available, SSL certificate is new or invalid/self-signed, domain status is clientDeleteProhibited

2.3 DNS issue

There can be some DNS issues like:

  • Wrong A record
  • Wrong AAAA record
  • A and AAAA points to different servers
  • hostname has AAAA record, but server cannot be reached over IPv6, only over IPv4
  • hostname has A record, but server cannot be reached over IPv4, only over IPv6

Use utilities like host, dig, nslookup for analyzing.

2.4 Network issue

  1. make HTTP(S) and TCP port(s) checking from different locations, analyze response time, timeouts, web server errors and etc., it can be your ISP issue or government restrictions (search at OONI Explorer)
  2. ping (send ICMP requests to server):
    • ping the server to identify packets loss (over other machine, VPN, check-host.net and alternatives...)
    • ping gateway and nearby IP addresses
    • diagnose where packets are lost using traceroute or mtr commands

    ICMP can be blocked (by default or as a new network protection rule)

  3. connect over VNC/KVM-over-IP via hosting control panel and check outgoing TCP and UDP packets (for resolving hostnames) at least

For partial packets loss or high latency there can be:

  • DDoS attack targeting to bandwidth
  • temporarily network issue in the datacenter or between you and your server
  • firewall (or packet filter) strict configuration

For no ICMP answer and no any port connection there can be:

  • datacenter/hosting network issue
  • server is down (it can be powered off, hardware issue, kernel panic...)
  • IP address is in a black hole route (null route) by datacenter because of a huge DDoS attack or no DDoS protection by hosting provider (black hole route rule can be enabled up for a few hours or permanently)
  • server is taken over or seized by police (support department keeps silence usually in this case) or erased by hackers

2.5 Hardware issue

For dedicated servers it can be:

  • memory issue (use memtester or jump to memtest86+ checking)
  • HDD issue (smartctl, ioping, check dmesg for I/O error, analyze /var/log/syslog)
  • RAID issue (cat /proc/mdstat)
  • network card (NIC) issue (ip link show to check card status)
  • CPU issue
  • power supply, cable and other issues

Sometimes it's available to reboot to a rescue system / mode.

A ticket to support team can be a solution to diagnose the hardware's issue.

2.6 Software issue

So you are able to join the server. And it means that the potential DDoS attack is not so huge or has a periodical nature or it's not an attack.

What kind of checks you can do next?

System:

  1. Hostname resolving (analyze /etc/resolv.conf, check with host, nslookup and dig) - UDP outgoing packets can be blocked or nameservers are down
  2. System limits: analyze ulimit, sysctl, lsof, /etc/security/limits.conf
  3. System logs: /var/log/syslog, /var/log/messages, dmesg, uptime, history, last (for the last entries to server), analyze /etc/shadow, ~/.ssh/authorized_keys, ~/.bash_history to be sure that the server is not used as a part of botnet impacting to server's resources
    dmesg output examples with issues:
    TCP: request_sock_TCP: Possible SYN flooding on port 443. Sending cookies. Check SNMP counters.
    nf_conntrack: nf_conntrack: table full, dropping packet
    Out of memory: Kill process 28725 (postgres) score 216 or sacrifice child Killed process 28725 (postgres) total-vm:36365660kB,anon-rss:32344260kB, file-rss:0kB, shmem-rss:0kB
  4. File system (check file system for errors over fsck)
  5. Resources limits: CPU load avg (uptime), disk space (df -h), memory and swap (top/htop), processes (ps aux), I/O speed and latency (iostat, ioping)
  6. Latest software updates: the last updates can be a source of the problem, or it's required to update some packages (check out the last lines of history command output)

Network:

  1. TCP connections to 80 or 443 ports (or other important server's port)
  2. Network connections and bandwidth (netstat -ant | wc -l, bmon, lsof, nload, iftop, ifstat, iptraf-ng, vnstat)
  3. Firewall/packet filter/fail2ban rules
  4. Network monitoring system charts and alerts (zabbix, prometeus and etc.)
  5. Deep network analysis using tcpdump/wireshark, packet filter's logs

Application:

  1. Containers health (docker ps)
  2. SSL certificate (it can be expired or misconfigured and normal users are not able to visit the website)
  3. Database (service process existent, memory usage, logs)
  4. Web server or application process in the list of system processes
  5. Web server's blacklist rules
  6. Web server's logs (live logs with tail -f, grep 500-503 errors, analyze logs for the time of web service unavailability)
    nginx errors example:
    accept4() failed (24: Too many open files) socket() failed (24: Too many open files)
  7. Web application logs
  8. Deep applications analysis over strace

3. Conclusion

So first of all determine the source of the issue:
hosting/datacenter, domain, DNS, network, software, hardware or a real incoming DDoS attack

I'm under DDoS attack:

  • keep calm
  • do not pay to attackers
  • detect a type of the attack (analyze logs and network activity, get information related to a malicious traffic type from hosting or datacenter):
    • network layer (L3) - Gbps
    • a transport layer (L4) - SYN/SYN-ACK/UDP floods, NTP/DNS amplification, IP fragmentation
    • application layer (L7)

DDoS is not huge (or periodic) and server can be accessed remotely or over fail over IP address:

  • system tuning over sysctl, ulimit and etc. (against server resources attack, SYN flood,..)
  • packet filter configuration (iptables, nftables, pf, ipfw) or hosting firewall
  • web server/application connection rates and additional WAF (Web Application Firewall) rules, caching
  • web application improvements, optimization, caching
  • web application protection on frontend: JavaScript and cookie challenges, CAPTCHA (self-hosted or CDN's solution)
  • web server log analysis and blocking IP addresses according web requests patterns
  • automatic abuses to ISPs of IP addresses which involved in the attack

DDoS is huge, permanent and bandwidth (transit) capacity is not enough:

  • contact support team for any help or suggestions
  • access to the server over fail over IP address, change IP address (if it makes sense or for a test) for the hostname of your website/application
  • enable DDoS protection by current hosting provider or change hosting to another one which have a DDoS protection (give a try to JVPS.hosting)
  • switch to CDN (CloudFlare for example) and change IP address of the server, deny any connections to the server except IPs from CDN's IP ranges
  • use load balancers
  • switch to a full-featured DDoS protection service

DDoS attacks are not endless, attacks costs money and resources for attackers. But do not wait the end of attack, act as soon as you can (as it depends on your business).

In most cases DDoS attacks can be mitigated without any help by paid third-party solutions, it depends on some system and network administration skills and experience.

2023.12.08 © Check-Host.net