I made a video about this topic on Odysee (I would also upload to YouTube, but because it mentions youtube-dl and yt-dlp, it's not allowed on YouTube). The video is long, rambly, and terrible, though, so you're probably better off just reading this page if you want to know how RSS works.
A YouTube video from 2007 about RSS feeds.
RSS is an old web technology that allows someone with a piece of software called an RSS reader (or news reader) to get a feed from their favorite sites supporting RSS with the latest news or updates. It's the progenitor to modern systems like YouTube subscriptions, Twitter feeds, Facebook's home page, and so on. Basically, you go to sites that have a little RSS icon and click it, and it will take you to a page with a bunch of code on it. You take that link that links to the code and put it as the link to the RSS feed in your reader, and then you can refresh your reader whenever you would like to check all (or just one) of the sites you're subscribed to for new content. In effect, it's a decentralized home page that you can customize with the sites you want and order in whatever way you would like to personalize and prioritize what's going on with the websites that you subscribe to.
The RSS icon looks like this (you've probably seen it a million times and never knew what it meant):
In essence, all that's happening with RSS is that the person running the website you're subscribing to generates a simple .xml file with a few of the latest entries on the site, and when your RSS reader reloads your subscriptions, it just downloads the latest copy of the .xml text file from the site and parses it into posts that you can read, with links attached that let you navigate from your RSS reader directly to the site if you prefer to read it there.
There's technically two standards for RSS, that being RSS and Atom, but I believe all readers should take both and they function on the user end completely identically. I would personally avoid JSON RSS feeds though (like those offered by Peertube later) for compatibility reasons.
As niche as RSS is becoming nowadays (except for perhaps a slight resurgence recently from tech enthusiasts), it's actually surprisingly well supported by many sites. Podcasts, for example, are entirely RSS feeds that just link to an .mp3 or other audio file format. Most Linux sites, especially for Arch-based distros, have RSS feeds as the primary method of keeping up to date with updates to the software (Gentoo should have this, in my opinion, but it uses its own system built into portage for news instead). And the main drawing point of RSS versus other proprietary walled-garden systems such as Facebook or Twitter is that, instead of having all your interests on a centralized, external system which tracks you, instead you locally have an RSS reader that only calls out to the sites for updates when you ask it to, and does so directly with no middleman. Therefore, the only people who know you connected to a specific site are the web hosts of that particular site you connected to; there's no cross-referencing with your other interests.
RSS was mainly developed for websites with some kind of news or any site with new entries in a list, like blogs or podcasts. Most of these sites still support RSS, and all you have to do is click on the RSS icon on the page to get the link to the RSS feed for the site. But if you're like me and you like watching videos on YouTube, Odysee, Peertube, or the occasional Twitch vod, RSS is just as useful for these sites as it is for news.
Let's start with the biggest one first: YouTube. Watching YouTube videos without a YouTube account can be a pain because there's no way to keep subscribed to channels you like and see updates from them - instead you have to navigate out to each channel and check them one-by-one for new videos if you're interested. This is where RSS comes in, as a way to have a decentralized, external "subscription" to a channel without needing any Google account or really ever navigating to the YouTube site except the first time to get the channel's RSS feed, and then when you want to watch the video. However, this is where we start to enter the Matrix a little bit. YouTube doesn't seem like it even has RSS feeds - in fact, if someone didn't tell you this information, you would never even be able to find out they exist. But, they do exist.
What you need to do is navigate to the channel you would like to get an RSS feed for in your web browser, and then right click somewhere on the page and click View page source (or, in Chromium, you can press Ctrl+U). The page source for YouTube pages is absolutely massive, so what you want to do is press Ctrl+F to open up search in your web browser, and search for the string rss. Keep stepping through results and eventually you'll end up at a part that looks like this:
<link rel="alternate" type="application/rss+xml" title="RSS" href="https://www.youtube.com/feeds/videos.xml?channel_id=UC1ydE9gDHTdvbNVIgEKIKzw">
The link after href is the link you're looking for. It should actually be an active hyperlink in the page's source code that you're viewing, so you can right click it and click Copy link address and that will be the link for the RSS feed for the YouTube channel you're looking at (the above link is for VWestlife's channel).
For Twitch vods, there is a third party tool that can be used to generate RSS feeds for Twitch channels. This tool can be found at https://twitchrss.appspot.com/. To create an RSS feed for only vods, just append vodonly/channel_name to the end of the link. For example, to create an RSS feed for Tech Tangents vods, the following would be your RSS link: https://twitchrss.appspot.com/vodonly/techtangents
For Odysee, if you go to a channel that you want to subscribe to and press the little three dots next to the "Follow" button, there's a button to press to automatically copy the RSS feed link to your clipboard. Simple enough, except there's a few caveats. At the time of writing this, Odysee has been having weirdly huge load times with its RSS feeds, easily 10x longer than the other feeds listed here. As well, if you are going to use Thunderbird as your RSS reader, Thunderbird tries to verify all feeds before adding them to your reader, and Odysee feeds can fail this validation due to the way that they link to the origin source for the feed - they need a link which is an exact, proper link with https:// and everything else, and they use the link provided by the channel which may or may not be a full, legitimate link. In fact, Odysee encourages you in its settings where you set this URL to use a simplified link instead of a full link, which will fail this test.
Peertube should also automatically generate RSS feeds that you can follow. Just go to the homepage of the Peertube instance you're interested in and click the three little waves (like the image above) to come up with different RSS options. Either pure RSS or Atom is fine, I believe all readers will take both (but I wouldn't pick JSON, that's not standard). You can just pick regular RSS and that should be fine.
Now you need an RSS reader to feed these links so that you can start browsing through them. The main RSS reader that we will be talking about today is newsboat, which should be in the repositories for most Linux distributions. It is a terminal program, if that scares you (it's not that hard to use, you just run it in a terminal window and navigate it with the arrow keys to go up and down, the Enter key to select, and the Q key to quit out of a feed/post or exit). If you'd prefer a different RSS reader, there are plenty out there like Thunderbird and the like, however newsboat is by far the most lightweight and easy to use, and newsboat has some bonus features we'll get into shortly. If you're just going to use RSS for your news, and maybe for your video streaming channels as well, but plan on watching the videos if you do by navigating to the website, then any RSS reader will now work. They're not too scary, just give it the RSS feed link from a website and refresh all the feeds every time you open the program and you'll be able to figure it out from there.
However, we're going to start really going one step beyond and I'm going to show you how to watch videos from your streaming sites locally on your computer without ever having to visit Youtube/Odysee/Peertube/Twitch in your browser, the benefit of which being the ability to limit the amount of tracking data is sent to these sites and truly start looking at the internet in terms of the value of the videos you're watching with no influence from the platform you're watching on - it's kind of therapeutic. Instead of the site (like YouTube) knowing how long you watched a video (or if you watched it at all), what parts of the video you watched and what parts you skipped through, what thumbnails in the recommendations you moused over, what comments you read, etc. instead, the site is limited to only getting two pieces of data: knowing your IP downloaded the RSS feeds for these specific channels, and that you downloaded specific videos to potentially watch at some point. That's still kind of a lot to cross-reference who you are, but it's not as much as they usually get, and since there's no YouTube account being created, any association would have to be done through a heuristic algorithm of some sort.
First, you're going to want to install newsboat (obviously) as well as yt-dlp if available in your distribution's repositories (if not, use youtube-dl instead, but note that it has dodgy support for Peertube, and the fixes were only merged into yt-dlp and not mainline youtube-dl).
The way that your feed works in newsboat is that the links to your RSS feeds you want to follow need to be put on separate lines in a configuration file called urls at the location ~/.newsboat/urls. To create different categories, you need to separate the videos into the categorization you would like, and you can put " " on a line to create a blank space, and "===Some Title===" on a line (replacing Some Title with the title of a section) in order to create sections in your RSS feed. For example, this is what my urls file looks like (click to enlarge):
And this is what the resulting feed looks like in newsboat:
As mentioned earlier, navigation is as easy as using the arrow keys, Enter, and Q to move around. The line at the bottom gives good hints about how to do things; the most useful is if you hit Shift-R (or capital R, if you want to think of it like that), that will refresh all of the RSS feeds one by one with the latest entries.
So, we have newsboat working with all our feeds, but let's tie newsboat into yt-dlp so that we can download videos from streaming platforms directly from newsboat to watch at our convenience later. (If you only have youtube-dl available, just replace all the mentions of yt-dlp from this point onwards with youtube-dl.)
First, if you want your downloads from yt-dlp to always go to a certain folder instead of to wherever you're currently navigated when you open newsboat (recommended), create a config file at ~/.config/yt-dlp/config and add the following to it:
-o /path/to/video/folder/%(title)s.%(ext)s
Change the /path/to/video/folder with where you would like the videos to go; I like putting them in ~/Videos/downloads. Anything put in this config file will be passed as an option to yt-dlp every time it is run. You could instead put this into our newsboat macro later on, but it's a little better to set it up like this in case you want to download a one-off video from a YouTube link directly and have it go to the same place. The %(title)s.%(ext)s just means that the resulting video file created by yt-dlp should be named "Title of the video"."extension" (such as Kryflux Low Level Floppy Disk Imaging First Tests.webm).
Next, create a config file for newsboat at the location ~/.newsboat/config where we will put our macros to download videos. Place the following text in the file:
#set browser browser "chromium %u" #map ,d to download video macro d set browser "yt-dlp %u"; open-in-browser; set browser "chromium %u"
Change the two chromiums with the name of your web browser in Linux. To open the web page for a news entry in newsboat, hover over or open an entry in a feed and press o. This macro also adds the ability to press , and then d to download the video linked to if the link is to a video on a video streaming platform.
newsboat is a weird piece of software in that it doesn't really let you script it, but it does let you set a "browser" to whatever you want, open the "browser", then change the browser back. So what we're doing is we've made a macro such that we can change the browser to yt-dlp, run yt-dlp passing it the link from the RSS feed (i.e. the video link), yt-dlp will download the video, and then we clean up by changing the browser back to what it was originally so we can still open links like normal.
And there you go! Now you have a completely decentralized system for watching videos from YouTube and other video streaming services on your computer without ever having to visit YouTube after the first time to get your channel RSS feed link.
There's one small problem with the video downloads that we're doing above, or, more accurately, a potential problem. If you're using an older computer to run Linux that isn't able to watch videos in high quality consistently or well (say, anything above 720p), then this method will download videos at their highest quality, and you won't be able to actually watch them. Never fear though, because one of the prime uses of this technique of RSS-ifying video streaming services is that it actually lowers the system requirements to enjoy YouTube, by allowing you to download videos to watch in an external video player, at the highest quality able to be played on your device, without ever having to navigate through the actual YouTube website - which is a nightmare on older devices.
What we're going to do is go back to ~/.newsboat/config and change it to look like this:
#set browser browser "chromium %u" #map ,d to download video macro d set browser "yt-dlp -f 'bestvideo[height<=?720][fps<=?30]+bestaudio/best' %u"; open-in-browser; set browser "chromium %u"
Change the 720 to the maximum vertical resolution of the video that your computer can handle (in my case, 720p) and the fps of 30 to the maximum fps (30 should be fine in most cases of older hardware - if your device can handle higher, just leave off the whole fps part).
This will work just fine, but I mentioned earlier you might want to just download a single video from YouTube without going through the whole process of subscribing to the RSS feed, and in that case you would have to type out that whole -f format for the video on the command line every time. Technically, we can put this in our config file from earlier, but in case we wanted to download a video in archive quality instead, we wouldn't be capable of doing that easily anymore if we simply used the config file. So instead, we can make a simple script for it.
Create a file in some folder that you'd like to put scripts that you're going to run a lot. I chose ~/scripts/ and I named the file ytdlp-watch.sh, to differentiate it from another script I have in the same location called ytdlp-archive.sh that downloads in maximum quality for archive purposes.
Now, add the following to the file:
#!/bin/sh yt-dlp --no-playlist -f 'bestvideo[height<=?720][fps<=?30]+bestaudio/best' $1
Make sure to make the script executable by running chmod, e.g. chmod +x ~/scripts/ytdlp-watch.sh
The --no-playlist part is helpful when using the script outside of the context of newsboat, because yt-dlp defaults to trying to download an entire playlist instead of a single video in that playlist that you're linking it to, which can be annoying and potentially disastrous.
Next we're going to add the scripts folder to our PATH (if you know how to do this or are already using a folder in your PATH, skip this part). Go to the file ~/.bashrc and add the following new line at the end:
export PATH=~/scripts:$PATH
Now, exit and reload your terminal. You should now be able to call ytdlp-watch.sh as a script in your terminal just the same way as you could call yt-dlp before, and just copy/paste your link into the terminal as an argument to the script to download videos in a lower quality for your machine.
To set up newsboat to use this script instead of the manually typed version from before, change your ~/.newsboat/config file to the following:
#set browser browser "chromium %u" #map ,d to download video macro d set browser "ytdlp-watch.sh %u"; open-in-browser; set browser "chromium %u"
Congratulations, now you have the same setup as I do for newsboat on my ThinkPads. I hope this was helpful to somebody, and if not, please let me know at xerxes@persrex.xyz. Thank you!
May 17, 2022