About the attack
In the late February 2018 enormously high-volume DDoS attack was observed. It was UDP based memcached service reflection-and-amplification attack, reported as 1.4Tbpos strength.
The 1.4Tbps it is more than entire network capacity of many SP. The big SP of the world has network with higher capacity, but even for them – why to waste 10-20-30% of their resources to carry malicious traffic? So – DROP IT! And do this as early as possible.
More about attack could be find here:
- https://blog.cloudflare.com/memcrashed-major-amplification-attacks-from-port-11211/
- http://www.zdnet.com/article/memcached-ddos-the-biggest-baddest-denial-of-service-attacker-yet/
- https://githubengineering.com/ddos-incident-report/
It is also worth to read about reflection-and-amplification here:
- https://blog.cloudflare.com/reflections-on-reflections/
- https://blog.cloudflare.com/deep-inside-a-dns-amplification-ddos-attack/
The memcached is a network service providing in-memory database. Than they are stored as key-value pairs where value can have arbitrary(large) size. Principle of attack is to plant single (a few) very-large objects into memcached server, and then generate read requests for this object at high rate using spoofed source address. As read requests carry only small key and response carry entire object, the amplification factor is expected to be even 1:50.000. So 1Gbps of requests could potentially trigger 50Tbps of attack…
The memcached service reflection-and-amplification attack is no different than other of the class. It is UDP base, sourced on abused and not-so-well managed servers and use source port 11211 (user port registered by IANA for memcached service). The destination ports are random and destination IP is a victim’s IP.
Attack vector identification
First of all – use of network service providing in-memory database is not something expected over internet. But who is SP to judge end-users practices? As SP, one should carry all traffic with valid destination IP.
Furthermore UDP port 11211 belong to IANA “user port” space, what mean that any user-space program can bind to this port. Even if IANA registered port 11211 for memcached service, it is not normative allocation (in contrast to system ports – below value 1024). Any user may enable any program to use port 11211 – and this is perfectly legal and normal. (e.g. one may run SIP proxy on UDP port 11211)
So, simple discarding traffic form UDP port 11211 is not an option.
However when attack that harms is detected, something need to be done.
Unfortunately, if attack is detected (rather ‘suspected’ I should say) based on fact that particular destination IP is exposed to enormously high traffic form UDP port 11211 (e.g. by IPFIX collection and analytics), doesn’t mean that all this traffic it is memcached reflection-and-amplification attack for sure. It could be mix of legitimate memcached traffic, other application legitimate traffic and memcached attack.
Discarding on router all traffic sourced from UDP port 11211 and destined to victim could lead to discard of legitimate traffic, effectively completing DDoS.
However, there is a light at the end of tunnel.
- First, we can look for memcached specific pattern in payload to prevent discarding packet of other applications that by coincidence use same UDP port. (see example below)
- Second – do you recall that attack started with planting one large object in memcached, which is then requested multiple times? This mean that many responses will carry the same payload. This open-up opportunity to more accurate discarding malicious traffic on router itself.
For brevity let look at case A. alone – DDoS detection engine identify attack vector as follow:
- Src IP: random, possibly multiple 1000’s
- Protoco: UDP
- Packet length: 1442 OR 1446
- Src port: 11211
- Dst I: victim IP
- Payload: 0x0000 starting on 14th byte of UDP payload. (reserved bits in memcached protocol)
Let also assume that DDOS detection engine can tell the router to discard traffic as above by some way (Say Netconf for simplicity)
Attack mitigation at MX.
So, we have to discard traffic that match above characteristic. It mean that for every packet we need to check values it carry’s in certain L3 and L4 header fields… and in payload. Not every peering router can do this, but Juniper MX can. And it is the most popular large-scale peering router used.
All currently supported JUNOS versions support ‘flexible-match-filter’ construct. Also all version of MPC line-cards, even oldest one – 16x10GE MPC – supports this match type. So, what we need to do, is just configure MX with something like:
Please note that we have two terms here – MEMCACHD-AMP-RSPD – matches all memcached traffic send by server, as identified by DDoS detection engine. The MEMCACHD-AMP-RQ matches on memcached request that triggers above amplified responses. Depending on position of attacker in Internet, the request may bit not have to traverse given router, but it is worth to apply this term just in case.
Obviously, nobody expect NOC operator to put it on multiple MX manually by typing into console. Again, JUNOS – the most popular network OS for peering routers – is here for help. You can use extraordinary JUNOS automation capability and load above short-living configuration into ephemeral DB via standard NetConf protocol. The ephemeral configuration DB brings following benefits:
- allow changes to be committed very fast, without parsing and auditing entire configuration,
- provides separation of short-leaving changes done by controlled (e.g. DDoS protection rules – it live time is hours) from baseline permanent configuration (that lives years).
Below simple python/PyEZ script is loading and committing above filter rule into ephemeral DB:
Conclusion
Combination of MX/JUNOS filtering and automation capabilities allows SP for surgical mitigation of massive amplification attacks, without need to procure terabits of capacity of DPI/purpose build scrubbers. The terabit attack is mitigated just on entry to network – on the peering router.
