Why Unreal Engine Can't Play RTSP Streams (IPStreamMedia #1)

Prologue

Before 2023, when 360° VR broadcast wasn’t a part of my life yet, I was just a chill guy whose knowledge of livestreaming was as much as the next person. I just knew that one can use OBS to stream content to YouTube and Twitch, and based on what you are streaming you need to have a PC that can handle that kind of load.

But during the RnD phase of the 360° broadcast workflow I got to learn about new abbrevations and jargons like RTMP, RTSP, SRT, HLS, etc. Back then I didn’t have much time to properly study each one of these concepts because execution was more important than education, what I got to know was that every streaming protocol has it’s pros and cons which need to be weighed against each other per use-case.

When I started working on ReFrame 360 I wanted to receive the 360 camera’s live video stream inside Unreal Engine for the whole concept to work. Unfortunately I realised that Unreal Engine does not have native support for IP-based streams except for HLS which is supported by the Electra Player plugin. HLS is good but it was no use to me, I needed RTMP or RTSP.

In Unreal Engine 4 you could also stream RTMP and RTSP using the VlcMedia plugin, but last I checked it wasn’t being maintained for UE5.

The Idea

While I was trying to think about ideas for a personal project this problem came to my mind and I thought “Why not create a custom Unreal media player plugin which can handle RTMP, RTSP, SRT, HLSL, everything ?!” After spending a few minutes imagining the world-wide glory that I would get for making this media player 🤩, I curbed my expectations (and my goals) and decided to create a custom media player plugin that can receive an RTSP stream inside Unreal Engine with as low latency as possible.

I also decided that I’m going to put education before execution this time around. Instead of using Claude Code to just write the code for me blindly I’m going to use it to learn and understand each and every concept behind the code that I’m writing. While I’m not shying away from the fact that I’m using GenAI to create this plugin, I want to be very clear that I’m using it as a guide and a teacher rather than like a freelance programmer.

In today’s day I don’t think there is any point in spending your time searching for relevant research and study material only using Google and other traditional methods. I believe that LLM models are basically the librarians of the entire world’s knowledge base, and as long as you are using them to train your brain you should be fine.

What’s next ?

The next practical step is to learn some theoretical concepts about IP Streaming, particularly focusing on RTSP and running some experiments to see these concepts in real-life.

I will update my progress in the next devlog. Till then, Cheers! 🍻