avformat/tls_mbedtls: fix DTLS handshake failure when receiving non-D…#77
Open
JackLau1222 wants to merge 1 commit into
Open
avformat/tls_mbedtls: fix DTLS handshake failure when receiving non-D…#77JackLau1222 wants to merge 1 commit into
JackLau1222 wants to merge 1 commit into
Conversation
…TLS packets Some WebRTC servers such as Pion send STUN packets concurrently during the DTLS handshake. Unlike OpenSSL and GnuTLS which filter non-DTLS packets internally, mbedtls passes all received UDP packets directly to its DTLS state machine, causing the handshake to fail. Fix this by adding ff_is_dtls_packet() to tls.c to check the DTLS record layer header, and using it in mbedtls_recv to discard non-DTLS packets such as STUN by returning WANT_READ, as specified by RFC 5764 Section 5.1.2. This function is also used in whip.c. Signed-off-by: Nariman-Sayed <narimansayed28@gmail.com>
github-actions
Bot
force-pushed
the
workflows
branch
7 times, most recently
from
May 11, 2026 00:39
2a8238f to
6c06cf8
Compare
github-actions
Bot
force-pushed
the
workflows
branch
7 times, most recently
from
May 18, 2026 00:42
ea01160 to
ef1af6f
Compare
github-actions
Bot
force-pushed
the
workflows
branch
8 times, most recently
from
May 26, 2026 00:42
57bf5b8 to
7cfa7c0
Compare
github-actions
Bot
force-pushed
the
workflows
branch
7 times, most recently
from
June 2, 2026 00:49
436b8ba to
f7516d1
Compare
github-actions
Bot
force-pushed
the
workflows
branch
7 times, most recently
from
June 20, 2026 00:46
433b564 to
e3a07fc
Compare
github-actions
Bot
force-pushed
the
workflows
branch
8 times, most recently
from
June 28, 2026 00:45
5e533d1 to
2525990
Compare
github-actions
Bot
force-pushed
the
workflows
branch
7 times, most recently
from
July 5, 2026 02:43
05910b9 to
5db5ac9
Compare
github-actions
Bot
force-pushed
the
workflows
branch
8 times, most recently
from
July 13, 2026 02:26
4ac181a to
b7d0383
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…TLS packets
Some WebRTC servers such as Pion send STUN packets concurrently during the DTLS handshake. Unlike OpenSSL and GnuTLS which filter non-DTLS packets internally, mbedtls passes all received UDP packets directly to its DTLS state machine, causing the handshake to fail.
Fix this by adding ff_is_dtls_packet() to tls.c to check the DTLS record layer header, and using it in mbedtls_recv to discard non-DTLS packets such as STUN by returning WANT_READ, as specified by RFC 5764 Section 5.1.2. This function is also used in whip.c.