Problem:

When trying to ban brute force login attempts at roundcube webpage. It's possible to successfully ban imap brute force attempts at dovecot. But for roundcube it won't work.

Let's say an user 'testuser' wants to login at my roundcube front end using brute force. His public IP is 80.111.100.1 and my server public IP is 70.111.100.1 (Sorry if this aren't good example IPs)

Roundcube logs fail attempts like this:

[19-Aug-2016 15:01:05 +0200]: <ea7sisp2> IMAP Error: Login failed for testuser from 127.0.0.1. AUTHENTICATE PLAIN: Authentication failed. in /var/www/html/rc/program/lib/Roundcube/rcube_imap.php on line 193 (POST /rc/?_task=login&_action=login)

Dovecot loggs it this way:

2016-08-19 15:01 imap-login: Info: Disconnected (auth failed, 1 attempts in 2 secs): user=<testuser>, method=PLAIN, rip=70.111.100.1, lip=192.168.1.143, TLS, session=<7+xfSmt62gBQgCQz>

Normally, all auth failed attemps in this log file with fail2ban. But because roundcube somehow writes own IP in rip, block main server and not testuser's puplic IP.

Solution:

We can resolve this issue by fail2ban's whitelisting function:

Whitelisting is setup in the jail.conf file using a space separated list.

[DEFAULT]
# "ignoreip" can be an IP address, a CIDR mask or a DNS host. Fail2ban will not
# ban a host which matches an address in this list. Several addresses can be
# defined using space separator.

ignoreip = 127.0.0.1 192.168.1.0/24 8.8.8.8

# This will ignore connection coming from common private networks.
# Note that local connections can come from other than just 127.0.0.1, so
# this needs CIDR range too.
ignoreip = 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16

Whats is this round cube is actually?
I'm a newbie of Linux platform. Would you like to discuss bit more?