RTMP-E and Red5 - Need a client to debug

[mondain]

If anyone knows of a test app or service I can use to debug my RTMP-E (enhanced-rtmp not RTMPE) implementation in Red5, feel free to share. My search thus far has come up short and I cannot use the rtmfp-cpp testing tools since they don't support regular RTMP as a client. Neither OBS nor builds of ffmpeg appear to send any codec information in the connect call.

[zenomt]

it's true that tcconn and tcpublish in rtmfp-cpp only make outbound RTMFP connections. however, tcrelay can accept an RTMFP or RTWebSocket connection, and then forward that as RTMP to a server. so i have two multi-hop client setups you could use:

• tcconn or tcpublish (RTMFP) -> tcrelay (RTMP) -> Red5
• play-demo.html (RTWebSocket) -> tcrelay (RTMP) -> Red5

the latter might even play some video & audio for you. :) i know that AV1 and HEVC work in play-demo.html with E-RTMP. while "enhanced audio" (AAC, MP3, Opus, FLAC) should work for playback, i haven't tested it yet, because there's no sample media. Chrome (or a Chromium-based browser like Edge) is required for playback because it uses WebCodecs.

oh yeah, i also don't support E-RTMP "Multitrack", though tcconn and tcrelay should pass multitrack messages through.

let me know if you need sample command lines to set up tcrelay for the above cases.

[zenomt]

One of these days I'll kill all that reading numbers as doubles, but for now its usable.

to be fair, AMF Numbers are always floating point. that's a "printing" problem. if it was C/C++ i'd say "use %g in your printf". i don't know about Java. :)

note that tcconn will send the wildcard * for fourCcList, audioFourCcInfoMap, and videoFourCcInfoMap with value 5 (saying that it can forward and decode all codecs), plus send 65535 for videoCodecs and audioCodecs (saying that it can accept any traditional codec). it should be obvious from the source for both play-demo.html and tcconn.cpp where to change those, if you want to experiment with how Red5 behaves if you do or don't send a particular codec, bit flag, wildcard, etc.

[zenomt]

2024-06-22 15:10:15,290 [NioProcessor-3] DEBUG org.red5.server.net.rtmp.codec.RTMPProtocolDecoder - Num params: 0

i assume "Num params: 0" is referring to the number of connect arguments after the Command Object. FYI with tcconn, tcpublish, and play-demo.html you can add extra String arguments to connect by putting them in the userinfo of the URI:

rtmfp://arg1:arg2:arg3@localhost/live#streamName
ws://arg1:arg2::arg4@localhost:8080/live#streamName

more info about the userinfo and fragment productions at https://zenomt.github.io/rtmp-errata-addenda/rtmp-uris

[mondain]

Yes, those are the extra args/params one can pass via the uri.

[veovera]

Thanks, for all your help, I sorta got it working,

Great work to all involved. @mondain can you describe your setup in more detail so perhaps others can leverage your work? thx!

[mondain]

@veovera I'll be sure to post once my implementation in Red5 is ready to go; I commend you on your easy-to-follow specification. I would however like to see more low level detail on the multichannel section, specifically, but not limited to codec per track determination.

[zenomt]

note that tcrelay can turn any RTMP server into an RTMFP (or RTWebSocket) server. :)

[zenomt]

oh, i noticed i hadn't updated play-demo.html to send the latest fourCcs in fourCcList, nor to send audioFourCcInfoMap and videoFourCcInfoMap. it does now.

[mondain]

@veovera which version of FFMPEG supports the current spec?

[mondain]

I have found that OBS beta 30.2.0 rc1 supports v2, but I don't yet know whether or not it will multi-track.

[mondain]

To answer my own question; OBS beta does not do multitrack nor Opus yet and is feature frozen.

[mondain]

@veovera @zenomt Quick question, when I have incoming enhanced audio or video that is not multitrack, will I always get the four byte fourcc code preceding every packet type's data contents? SequenceStart, SequenceEnd, CodedFrames, CodedFramesX, and Metadata; the last is the one I haven't seen yet, so it seems quicker to ask.

[zenomt]

that's my reading of the spec. my implementation treats a message that's marked as "enhanced", but that's not at least 5 bytes long, as "not enhanced": /~https://github.com/zenomt/rtmfp-cpp/blob/63ee4f3f98a6e1372af7e25453d388463bf592fd/src/TCMessage.cpp#L143-L146

[veovera]

when I have incoming enhanced audio or video that is not multitrack, will I always get the four byte fourcc code preceding every packet type's data contents?

In most cases that is correct. Please see the pseudocode

• for audio
• for video

As you can see when in enhanced mode and fetching video metadata you don't fetch a FOURCC value.

[zenomt]

@veovera two things:

first: i didn't notice until now that the treatment for "enhanced video with frametype 'command'" is different from E-RTMP v1. i wish i had noticed this before; i would have had discussion items. i assume this change is so that smart-seeking a multi-track stream (which might have different codecs for different tracks) can only have all tracks start/end smart-seek together.

second: my reading is that only an enhanced video message that's both not metadata and is also a video command will not have a FourCC. a message that is metadata (regardless of if it's a video command), or a message that isn't metadata but also isn't a command, will have a FourCC.

[veovera]

E-RTMP v2 is currently in alpha, allowing for potential breaking changes. This flexibility enables us to refine and evolve the protocol during this phase. However, as we move towards beta, such changes will be stabilized. Note: so far there have not been any breaking changes in E-RTMP v2 alpha.

Regarding the treatment of "enhanced video with frametype 'command,'" it's worth noting that in both E-RTMP v1 and E-RTMP v2, our approach remains consistent. When videoFrameType is VideoFrameType.Command (i.e., 5), the payload does not contain video data. Instead, the VideoTagHeader is followed by a UI8 indicating:

• 0 for the start of a client-side seeking video frame sequence
• 1 for the end of such a sequence

This approach ensures uniformity across versions. While our pseudocode may not always cover every detail exhaustively, it aims to comprehensively address scenarios involving VideoFrameType.Command in E-RTMP v2.

In legacy RTMP, if FrameType is command, instead of the video payload the message stream includes a UI8 indicating the command type, similar to the approach described above.

[mondain]

@veovera the more I implement to the spec, the more I like it; thanks for your efforts to extend the life of RTMP.

[veovera]

Thank you for the feedback, this makes my day! I am looking forward to seeing your resulting service. Keep us posted...

[mondain]

@veovera I grabbed FFMPEG latest and built with all the appropriate codecs, but I cannot get VP8 to work; the v2 spec says the fourcc is generated with vp08 and I've tried that and vp80 with neither working: Video: none (vp80 / 0x30387076), none I also checked the VLC header source for clues and they're using uppercase ASCII as VP80 who's the most correct here? /~https://github.com/videolan/vlc/blob/master/src/misc/fourcc_list.h
Lastly, seems I'll run into the same issue with VP9 and AV1..

[mondain]

It would seem that ffmpeg doesn't support vp8 in flv; [flv @ 0x55ce4143dbc0] Video codec vp8 not compatible with flv

[veovera]

who's the most correct here?

As far as FOURCC values we use the values defined in the specs below

• https://www.webmproject.org/vp9/mp4/ (for "vp08" and "vp09")
• https://aomediacodec.github.io/av1-isobmff/v1.3.0.html#av1codecconfigurationbox-definition (for "av01")

FFmpeg uses the same values for e-rtmp/flv see <link>

[veovera]

It would seem that ffmpeg doesn't support vp8 in flv; [flv @ 0x55ce4143dbc0] Video codec vp8 not compatible with flv

It looks like FFmpeg chose not to support vp8. Today I don't believe vp8 is a popular codec. We chose to add vp8 to the spec because webRTC officially supports this codec.

[mondain]

It may not be all that popular, but a lot of Android devices still have it and its a requirement in the WebRTC spec. I'll work around it and maybe do a PR for ffmpeg later; thanks for the updates.

[veovera]

its a requirement in the WebRTC spec

Agree 100%, that's why we included it! Post your PR link here if you get a chance. thx!

[mondain]

@veovera here are my current example FLV files for AV1 and VP8; this repo also includes the FFmpeg patch that I'm waiting to submit, once they let me on the list. /~https://github.com/mondain/mediafiles

[zenomt]

i'm able to play your AV1 sample; however, i need a little bit of playback buffer or the framerate looks very low (i suspect from out-of-order frames or something). that is, playback with almost 0 buffer looks like about 5 frames per second, but if i play back with a half-second buffer, the video looks smooth.

i need to go add support for VP8 at least on my playback side. it looks like Chrome has WebCodecs support for that, so it should just be a few lines of code. i should be able to confirm playback pretty soon.

[zenomt]

i can play the VP8 sample now (changes not pushed to github yet). however, it looks like audio and video are a little bit out-of-sync in the VP8 sample, with audio maybe 100-200 ms behind (just based on my eye/ear, not actually measuring).

[zenomt]

VP8 support (+3 lines of code) added to /~https://github.com/zenomt/rtwebsocket . i also added the codec ID to /~https://github.com/zenomt/rtmfp-cpp but that's not necessary for tcpublish/tcconn/tcserver to handle it.

[zenomt]

actually, it looks like the AV1 sample might also have the same A/V sync issue; however, that sample is much shorter and there's only one good part (the neck crack) to judge sync (although the "ahhh" after smelling the flowers is also off, but that one it's harder to tell).

[mondain]

These are just base recordings until I get all the other stuff sorted; I'm not using the comp offset timing data at the moment for those codecs that supply it.

[koolscooby]

A few resources for folks :

• ertmp-toolbox: Simple rust utility that can read and push eFLV files /~https://github.com/kc5nra/ertmp-toolbox
• FFmpeg: Branch that has Enhanced RTMP Multitrack FLV support /~https://github.com/BtbN/FFmpeg/tree/enhanced-flv
• Wireshark: v4.4+ support dissecting/parsing Enhanced RTMP Multitrack connections
• OBS Studio: v30.2+ support Twitch Enhanced Broadcasting powered by Enhanced RTMP Multitrack Feature. A CLI option --enable-multitrack-video-dev lets you experiment with custom OBS configurations for testing streaming to services other than Twitch and provides an optional checkbox to record an eFLV file.

[veovera]

@koolscooby Thank you for sharing!