libapache2-mod-gnutls: mod_gnutls consumes 100% cpu (CVE-2023-25824)

Debian Bug report logs - #942737
libapache2-mod-gnutls: mod_gnutls consumes 100% cpu (CVE-2023-25824)

Reply or subscribe to this bug.

Toggle useless messages

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

From: Nicolas <nicolas@progweb.com>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Subject: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Sun, 20 Oct 2019 20:16:32 +0200

Package: libapache2-mod-gnutls
Version: 0.9.0-1
Severity: grave
Tags: upstream
Justification: renders package unusable

Dear Maintainer,

I have updated apache2 & mod gnutls application. In testing with a basic html page,
I notice that apache2 process loops infinitly, "ps" output:

www-data  6103 92.8  0.1  29696  9708 ?        R    19:37  28:24 /usr/sbin/apache2 -k start

I try with ssl module:

<IfModule mod_ssl.c>
	SSLEngine On
	SSLCertificateFile /etc/apache2/ssl/domain.com.crt
	SSLCertificateKeyFile /etc/apache2/ssl/domain.com.key
</IfModule>

<IfModule mod_gnutls.c>
	GnuTLSEnable on
	GnuTLSCertificateFile /etc/apache2/ssl/domain.com.crt
	GnuTLSKeyFile /etc/apache2/ssl/domain.com.key
	GnuTLSPriorities PFS:-VERS-ALL:+VERS-TLS1.3:+VERS-TLS1.2:%SERVER_PRECEDENCE
</IfModule>

I have the issue only with gnutls module.

To check, I attach a gdb session:

(gdb) bt
#0  0x00007f78b4cfb92f in ?? () from target:/usr/lib/x86_64-linux-gnu/libgnutls.so.30
#1  0x00007f78b4cfdf7c in ?? () from target:/usr/lib/x86_64-linux-gnu/libgnutls.so.30
#2  0x00007f78b4e90f38 in ?? () from target:/usr/lib/apache2/modules/mod_gnutls.so
#3  0x00007f78b4e91ad2 in mgs_filter_input () from target:/usr/lib/apache2/modules/mod_gnutls.so
#4  0x000055c220cd08e1 in ap_rgetline_core ()
#5  0x000055c220cd336c in ap_read_request ()
#6  0x000055c220cfe7a8 in ?? ()
#7  0x000055c220cf38b0 in ap_run_process_connection ()
#8  0x00007f78b3bd23df in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so
#9  0x00007f78b3bd26d4 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so
#10 0x00007f78b3bd272f in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so
#11 0x00007f78b3bd32f3 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_prefork.so
#12 0x000055c220ccc67e in ap_run_mpm ()
#13 0x000055c220cc4f57 in main ()

I don't know how to fix the issue and how to help you.

-- System Information:
Debian Release: 10.1
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-6-amd64 (SMP w/4 CPU cores)
Kernel taint flags: TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages libapache2-mod-gnutls depends on:
ii  apache2-bin [apache2-api-20120211]  2.4.38-3+deb10u3
ii  libc6                               2.28-10
ii  libgnutls30                         3.6.7-4
ii  libmsv1                             1.1.1-3

libapache2-mod-gnutls recommends no packages.

libapache2-mod-gnutls suggests no packages.

-- no debconf information

Available to do tests and more.

Nico

Message #10 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Fiona Klute <fiona.klute@gmx.de>

To: 942737@bugs.debian.org

Subject: Can't reproduce, please try dbgsym packages for backtrace

Date: Sun, 24 Nov 2019 13:31:00 +0100

Hi Nico,

I tried to reproduce the issue but couldn't. Exactly how are you sending
the requests? I've tried both curl and hand-typing over gnutls-cli.

If you still have this issue, could you try enabling the appropriate
debug repositories (see https://wiki.debian.org/AutomaticDebugPackages)
and installing the relevant *-dbgsym packages to see which functions
loop in the backtrace? I assume you'll need at least these:
apache2-bin-dbgsym libapache2-mod-gnutls-dbgsym libgnutls30-dbgsym

Best regards,
Fiona

Message #15 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Bernhard Übelacker <bernhardu@mailbox.org>

To: 942737@bugs.debian.org

Cc: Fiona Klute <fiona.klute@gmx.de>, Nicolas <nicolas@progweb.com>

Subject: Re: Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Mon, 16 Dec 2019 02:37:50 +0100

Dear Maintainer,
tried to reconstruct the given backtrace with debug symbols
in a gdb session and came to following, maybe it could be
of some help.
(Still a proper backtrace with dbgsym packages
installed would be better.)

Kind regards,
Bernhard


Reconstructed:
#0  0x00007f78b4cfb92f in gnutls_assert_val_int at ../../lib/errors.h:139 from libgnutls.so.30
#1  0x00007f78b4cfdf7c in _gnutls_recv_int at ../../lib/record.c:1773 from libgnutls.so.30
                       in gnutls_record_recv at ../../lib/record.c:2281
#2  0x00007f78b4e90f38 in gnutls_io_input_read at gnutls_io.c:246 from mod_gnutls.so
#3  0x00007f78b4e91ad2 in gnutls_io_input_getline at gnutls_io.c:342 from mod_gnutls.so
                       in mgs_filter_input at gnutls_io.c:595
                       in ap_get_brigade at util_filter.c:553
#4  0x000055c220cd08e1 in ap_rgetline_core at protocol.c:246
#5  0x000055c220cd336c in read_request_line at protocol.c:682
                       in ap_read_request at protocol.c:1322
#6  0x000055c220cfe7a8 in ap_process_http_sync_connection at http_core.c:192
#7  0x000055c220cf38b0 in ap_run_process_connection at connection.c:42
                       in ap_process_connection at connection.c:219
#8  0x00007f78b3bd23df in child_main at prefork.c:615 from mod_mpm_prefork.so
#9  0x00007f78b3bd26d4 in make_child at prefork.c:717 from mod_mpm_prefork.so
#10 0x00007f78b3bd272f in startup_children at prefork.c:735 from mod_mpm_prefork.so
#11 0x00007f78b3bd32f3 in prefork_run at prefork.c:897 from mod_mpm_prefork.so
#12 0x000055c220ccc67e in ap_run_mpm at mpm_common.c:94
#13 0x000055c220cc4f57 in main at main.c:819

Message #20 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Emanuel Larios <anubes19@yahoo.com>

To: 942737@bugs.debian.org

Subject: Bug

Date: Sun, 1 Nov 2020 00:30:51 -0700

Sent from my iPhone

Message #25 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Félix Arreola Rodríguez <fgatuno.123@gmail.com>

To: 942737@bugs.debian.org

Subject: Re: Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Sat, 28 Nov 2020 14:56:51 -0600

[Message part 1 (text/plain, inline)]

On Mon, 16 Dec 2019 02:37:50 +0100 =?UTF-8?Q?Bernhard_=c3=9cbelacker?=
<bernhardu@mailbox.org> wrote:
> Dear Maintainer,
> tried to reconstruct the given backtrace with debug symbols
> in a gdb session and came to following, maybe it could be
> of some help.
> (Still a proper backtrace with dbgsym packages
> installed would be better.)
> 
> Kind regards,
> Bernhard
> 
> 

Dear Maintainer:

I came across this bugs too. The problem is a infinite loop between
mod-reqtimeout and mod-gnutls. The mod-gnutls (and underlaying gnutls)
tries to read some bytes when the client hasn't sent any (like an empty
tcp conn) and the mod-reqtimeout returns a timeout, causing mod-gnutls
to loop endless trying to read more. Either mod-reqtimeout is broken,
or either mod-gnutls doesn't handle correctly the timeouts.

After debugging a lot with gdb, I can reconstruct the loop as follow:

The code starts at function gnutls_io_input_read from gnutls_io.c from
mod_gnutls, line 246, the "while(1)" loop.

The code calls gnutls_record_recv, from gnutls, which in turns calls
_gnutls_recv_int. gnutls_recv_int is called with
type=GNUTLS_APPLICATION_DATA asking for 8192 bytes. Next it calls
check_session_status, which checks if there is an EOF, and there is
NO EOF. Also, check_session_status says that the
session->internals.recv_state is RECV_STATE_0, whichs makes
check_session_status return 1.

Next, it calls get_data_from_buffers, which returns 0, which means
there is no data in the buffer to consume. Next, it
calls _gnutls_recv_in_buffers. _gnutls_recv_in_buffers calls
recv_headers, which tries to read the header (which is 5). it calls
_gnutls_io_read_buffered for 5 bytes. _gnutls_io_read_buffered ask for
5 bytes to _gnutls_read, which in turn uses _gnutls_stream_read to
complete the request.

_gnutls_stream_read returns back to mod-gnutls using the "pull_func"
mgs_transport_read. msg_transport_read is located at gnutls_io.c:824
mgs_transport_read tries to use ap_get_brigade to get some bytes from
the underlaying socket.

ap_get_brigade tries to read from the "next" ap_filter_t, which (in my
case) is "reqtimeout". reqtimeout_filter function calls
check_time_left, which returns APR_TIMEUP. When it returns APR_TIMEUP,
it logs "Request %s read timeout". So, back to mgs_transport_read, it
reads APR_TIMEUP from the ap_get_brigade. BUT, it handles wrong this
timeout and converts it to EAGAIN, which makes this whole stack run
back.

This is a big, 'nice' nasty bug, I have to say.

As a workout around, I disabled module reqtimeout. Seems to solve the
cpu usage issue. But, the bug is way more big than 'disabling'
reqtimeout. Because, the problem is between mod-gnutls and
mod-reqtimeout

Steps to reproduce:
Enable apache2 with the modules mod-gnutls, and mod-reqtimeout. Setup a
reqtimeout like: RequestReadTimeout header=20-40,minrate=500 and open
an openssl s_client:

openssl s_client -connect IP:port

Don't send any data over the openssl connect. Just wait for the timeout
to happen. After the timeout, the CPU usage will increase. Also, you
can quit the openssl s_client and the apache process will be stuck in
the endless loop.


-- 
Atte. Félix Arreola
Firmado con GPG 0x1e249ee4

[Message part 2 (application/pgp-signature, inline)]

Message #30 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Félix Arreola Rodríguez <fgatuno.123@gmail.com>

To: 942737@bugs.debian.org

Subject: Re: Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Sat, 28 Nov 2020 15:08:07 -0600

[Message part 1 (text/plain, inline)]

Now I think this bug, this could be used as DOS, should we call the
security team to handle this?

-- 
Atte. Félix Arreola
Firmado con GPG 0x1e249ee4

[Message part 2 (application/pgp-signature, inline)]

Message #35 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Félix Arreola Rodríguez <fgatuno.123@gmail.com>

To: 942737@bugs.debian.org

Subject: Re: Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Sat, 28 Nov 2020 21:16:23 -0600

[Message part 1 (text/plain, inline)]

Tags 942737 security
thanks

-- 
Atte. Félix Arreola
Firmado con GPG 0x1e249ee4

[Message part 2 (application/pgp-signature, inline)]

Message #42 received at 942737@bugs.debian.org (full text, mbox, reply):

From: Félix Arreola Rodríguez <fgatuno.123@gmail.com>

To: 942737@bugs.debian.org

Subject: Re: Bug#942737: libapache2-mod-gnutls: mod_gnutls consumes 100% cpu

Date: Sun, 5 Jun 2022 15:06:10 -0500

[Message part 1 (text/plain, inline)]

Finally managed to write a patch. This patch applies ok on mod-gnutls
0.9.0 and not sure if it will work for buster.

-- 
Atte. Félix Arreola
Firmado con GPG 0x1e249ee4

[fix_timeout.patch (text/x-patch, attachment)]

[Message part 3 (application/pgp-signature, inline)]

Send a report that this bug log contains spam.