Signals from... not so far away

Since we last spoke to you properly, the internet has proceeded to - as Cory Doctorow would put it - enshittify. The goals of Riff.CC are more relevant today than they've ever been. The internet has become a less and less permanent place as discussion forums die and fall into the void. Streaming services, once thought to be the end of piracy and an answer to many questions, have shattered into dozens of individual sites, with subscriptions required for each.

We do not believe culture should be out of reach for anyone, and yet we're hurtling towards a future where that is increasingly the case for everyone.

The Riff.CC Project has existed in some form since 2010. This is a very long time in internet terms. It's seen the rise and fall of many things, and our policy for most of that time has been that "our time will come" - that brute force effort to try and make Riff happen regardless of the viability is not wise, that we should wait for the right technologies to come, that we should wait for "that day" - and in little bursts like a spark catching flame, we've seen that day come and pass a few times.

With Riff.CC Alpha, we launched a tiny community of nearly 1000 people who believed in this mission. They put up with downtime, jankiness, unclear direction, and the overall immaturity of Alpha, and while I can't speak for any of them except for myself, I feel that they did so for a simple reason - they saw in Riff the same potential we - I - saw in it way back when I made the first upload to Mininova and saw hundreds of people download something I'd shared. Something I cared about.

On the 25th of June 2021, I gave a talk about Riff as it existed back then. I was passionate and energised, despite being very sick at the time and despite the turmoil that I had no idea would follow - and the end of Riff.CC Alpha which would come soon thereafter.

One thing stands out to me today from that talk in particular -

"This needs to exist, there is no alternative, it simply has to. [...] This is just the road."

This post is an open letter, then, to the world and to the people who have come and gone, to lay out the past and future of Riff and explain what we want to do.

The past

Riff started when I downloaded an album I liked and wanted to share it with the world. At the time, very few if any good options existed for sharing legally free music (such as Creative Commons and public domain music) in high quality. Platforms like Jamendo would mangle tags and filenames, and the user experience was generally pretty terrible. Even purchasing commercial music legally was often a similar story.

And yet, when you wanted to "acquire" music instead, you found a totally different story. The pirates put actual love into distributing content, effort way beyond that of even large music warehouses like iTunes. Sites like What.CD - and especially What.CD - built the most accurate music library ever assembled. With hundreds of thousands of records, and a clever "trumping" system that allowed higher quality versions of releases to replace lower quality records, almost the entire catalog of music on What.CD was carefully curated, with track names and filenames that were clean and reflected the artist's actual intentions. Typos were fixed. They had not only just about everything you could think of in literal bit-perfect quality, they had just about every version of each piece of content as well.

It was a Library.

Inspired by What.CD and the general higher curation quality that was possible on the piracy sites, I started to wonder - how can sites like this exist for free (and in What.CD's case, with zero advertising), purely running on user donations? Clearly there was an efficiency here. Well, it's not a secret that the core technology at play, BitTorrent, allowed for very efficient distribution - it still does to this day. But ultimately my question was something different - why do I get a better experience pirating music than I do downloading legally free music? Why is there not a place for a content creator to upload their music for free, in high quality, for the world to listen to? I started researching and found that as an artist you actually had to pay to put your music on high quality platforms, and basically nobody was really solving this problem.

The beginning

This was around the time I was discovering music that inspired me, music that was Creative Commons licenced. The idea was new and rebellious. How would it work? One album stood out to me - Adam McHeffey's "Let's Kick Fire". I downloaded it off FrostClick, a blog that to this day distributes legally free music using BitTorrent and other technologies, and had a listen. And I fell in love.

This was one of the most important albums I ever listened to. On a personal level, it spoke to me - themes of love, of longing, of distance - as a kid who'd been dragged from one hemisphere to the other, and no longer had access to any of my old friends or social circles, away from most of my family until further notice - and dealing with the day to day of growing up and experiencing love and loss and so on. Enough about that, though - the point is it spoke to me. And I wanted to share it.

Unfortunately, because it had been mangled by the platform it was uploaded to, it had poor sound quality and bad filenames and bad tags - the usual story - so I cleaned them as best I could and re-uploaded it to Mininova. Now you could grab the album in one go, with a simple BitTorrent client, and import it straight into your favourite music player and off you went. I emailed the artist who made it and asked if he could send me a higher quality copy of the album so that I could upload it properly - ideally in lossless quality - and he responded that he didn't exactly know how to do that, but very generously offered to send me a copy of the album. He sent me multiple pressed copies of it, and I eagerly ripped one of them and uploaded it.

Over the next few days and weeks hundreds of people downloaded the album. And I noticed something - Mininova had a program called Mininova CD - "Content Distribution" - where you could upload whatever you wanted as long as it was legally free, and Mininova would host it on their own servers and "seedboxes" and provide it to the world. So I applied to that program, under the name ZORLiN-CC - a sort of pseudo-record label, which would seek out high quality legally free content, clean up the tags and generally repackage it in higher quality, and upload it to Mininova CD. The first time I uploaded something, I realised my uploads were hitting the front page of one of the largest, most heavily visited sites on the entire internet.

Whoa.

I was extremely excited, and over the next few months I uploaded 137 releases. Everything from music to photo collections to comics and even movies that were produced in 2 weeks from start to finish - I shared culture, and I began to wonder how I could help bring that capability to more people.

The Fall of the Library

Then, one day, Mininova died when a Dutch court ordered that it begin implementing ineffective-by-definition filtering to block copyrighted content from being uploaded to their site. With literally no plausible way of actually adhering to that order, they instead chose to delete everything from their site that was not part of the Mininova CD program.

In the wake of Mininova essentially being wiped from the internet, ZORLiN-CC saw download numbers spike, and over the next couple of years would see over 650,000 downloads of the content I'd uploaded. Dozens of terabytes worth of downloads, at the time a huge amount of traffic, were fed to a community of users who were somewhat confused as to where the rest of Mininova had went and took a chance on this weird Creative Commons stuff.

I knew it wouldn't last, though. With all of the content Mininova had been known for gone, eventually the site would not be able to sustain the hosting and storage costs of all of that media - a much more expensive affair in those days.

Meanwhile, thinking back to my earlier dreams of making Mininova CD's opportunities available to more people, I envisioned a sort of hybrid of Reddit (at the time an open source project) and What.CD, and even tried mashing them together myself naïvely.

I found it more difficult than I expected, having never properly dabbled with programming, and gave up. But I'd purchased the domain - riff.cc - and the idea stuck firmly in my brain. I'd already experimented with a copy of the What.CD software and built a rudimentary version of Riff that had Creative Commons only material on it, invited a few friends, and played with the idea. It was a genuinely good experience - fast download speeds, a clean interface, simple to use and even somewhat usable on mobile, and all of this as far back as December 2010.

One day, Mininova fell for good. Some of the content uploaded to Mininova CD (and the wider Mininova, since some users uploaded their own content to it outside of Mininova CD) has never been seen again - culture was lost that day.

And I was angry.

The Second Fall

Riff went into hibernation at that point, and not a lot happened for a while. I worked on it passively, researching decentralised technologies and working out what could be done to eventually solve for what I wanted to create. I decided that trying to build it with the existing technologies that were out there at the time would simply mean creating another centralised site with high costs - or another glorified BitTorrent tracker - and moved on with my life, but kept Riff close to my heart the whole time.

Then What.CD's servers were seized by the French police, and a second Library burned to the ground. Once again, a cultural library assembled by thousands of volunteers was ripped from existence - its community and content scattered to the wind - and given that What.CD was a place where a lot of artists directly uploaded their own material, things were lost that the world has not heard since. It became clear to me that any model where the shutdown, whether by legal means or by resource exhaustion or simple lack of interest, of a site could take down all of its content would simply not work. In What.CD's case, eventually the users congregated on replacement sites and reuploaded some 70% of the catalog - but rebuilding the library of content took years, and some works were inevitably lost in the shuffle.

Once again, I found myself horrified. Whether you thought copyright infringement was okay or not, a second collection of media had been destroyed - some of it to never be seen again - and the world was worse off for it.

I started exploring every possibility I could, without any real resources to build anything meaningful in the space. Smart contracts, cryptocurrencies for borderless payments, decentralised technologies like IPFS - all of these things began appearing around this time.

Riff.CC Alpha

The first version of Riff.CC Alpha was born when I was checking out repositories on GitHub, specifically searching for a simple media player. I reasoned that I could build a storage system, bear the costs of hosting a lot of the content for it myself, and just start experimenting with decentralising the storage system and sticking a simple frontend media player in front. I would use things like Anycast and cheap storage boxes to create a storage system that spanned the world, and maybe just make something.

I didn't find anything suitable for that, but I did find Unit3D - an open source BitTorrent tracker and frontend suite that had just about everything I needed. It had moderation, modularity, a permissive licence that allowed for easy modification and sites run for any purpose - so we launched the first real version of Riff using it and we started to grow. We spawned a team of content moderators, a marketing lead, a small but passionate group of developers, and paid contractors to do everything from upload and tag content to modifying the core of Unit3D to achieve interesting things. We integrated IPFS for some releases, with a hack that allowed you to start streaming content instead of simply downloading it.

It was during the days of Riff.CC Alpha that I gave my talk - we created RIFF.CC FOUNDATION LIMITED as a company intended to become a nonprofit to run Riff, and really began scaling our efforts. At one point we had a petabyte of storage, and enough compute for encoding and other uses to make a medium sized business blush - I poured pretty much everything I had into making it work.

Riff.CC Alpha did not last, though.

The Second Great Hibernation

Due to health issues and mounting financial woes, Alpha became unsustainable and much of the hardware we'd stockpiled had to be sold off. The company was disassembled, the small team we'd assembled was sadly let go, and the dream was dead. I was sure of it.

Then one day, I thought back to what we'd come up with right near the end of Riff's life - a transpiler that would transform the website from a database of content, which required live servers, to a simple tree of files and metadata that would not technically require any infrastructure at all beyond creation.

The Revival

I began looking at IPFS and other technologies again with interest, and had a thought - what if I tried to do Riff again, but this time with a much lower, tighter budget? What if I finally hosted its finances on OpenCollective like we had been planning to, allowing for full transparency about where money came from and went, and more importantly allowing me to actually understand how much it was costing and control costs properly?

I thought about the core issues with something like Riff, and what I was actually trying to solve. Essentially, distributing the actual content was an easy, solved problem - BitTorrent could do it very efficiently, IPFS could do it reasonably well too, and a CDN combined with IPFS provided very good speeds and reliability - experiments that we'd done during Alpha showed seeking and buffering times that were competitive with and even beat Netflix at the time - no small feat.

One idea about IPFS stood out to me in particular - remember that the key weakness of BitTorrent trackers was the centralisation. One could make a very efficient site that brought together a huge community of people around the world, and run it on almost no hardware - something The Pirate Bay has managed to this day - but ultimately if the site and servers were to vanish, the content library would vanish until reuploaded to a different tracker. But IPFS was different. Once a piece of content is uploaded and becomes a CID, anyone in the world who knows the CID can retrieve the content as long as that content is reasonably available from the network of peers who are seeding (pinning, in IPFS terms) it - IPFS, unlike BitTorrent, is global by default with all content available to anyone on the network.

The main issue we needed to think about was actually not the content - although incentivising people to keep content alive and well seeded on a decentralised network was not an entirely solved problem, we could easily cover enough of the costs of keeping at least a few copies online, up to a point - it was the metadata and the distribution of the Content Identifiers or Infohashes (for IPFS and BitTorrent respectively) and the core idea that a single site can ever go down and take down an entire library in the process. But with IPFS being global, if you could somehow build a site that didn't individually matter at all, a simple node in a sea of nodes - distribute the actual site and database itself instead of just the content it pointed at - you could make a Library that would never fall as long as there were volunteers willing to keep it alive.

“What if you had the power to affect monumental change? Would you let fear consume, or would you overcome?”
STARSET - First Light

I researched and discovered distributed and decentralised database technologies like OrbitDB and Gun.eco, and I put a call out for developers, and a developer named Lexi came out of the woodworks. Together we built an early prototype for Riff called Omega.

Later, I brought on Julien Malard-Adam, who worked on a new version of Riff called Orbiter, and later still, Lucas Jovanovich, who made a version called CeramicRiff (based on ComposeDB by Ceramic).

Slowly, and then all at once, all three prototypes clicked into place.

On January 15th, 2023, we achieved a full working demo of federation for the first time - two completely independent Riff.CC instances, running on different domains - different servers on different continents, with different Ceramic private keys and different admin accounts, publishing and subscribing to each other.

It was a fairly basic demo, but we were able to:

• upload a pin (a package of IPFS content) on Riff.CC
• upload a different pin on FTWC.xyz (a demo domain purchased for testing)
• on FTWC.xyz, subscribe to Riff.CC and see its content appear

We then made Riff.CC subscribe to FTWC.xyz and saw the content from FTWC propagate to Riff, like magic.

The future

Over time, we chose to sunset some of the prototypes we'd created. We eventually settled on OrbitDB and our Orbiter prototype as the future of Riff, and as of today Riff is still in stealth mode - quietly in development, and waiting to one day step back into the light.

We think that being able to create relationships not just between users or abstract objects but between platforms and websites themselves is extremely powerful. Combine it with the power of global-by-default content distribution, and you have the ability for platforms to collaboratively share resources and users in a positive sum game.

We've invested back into hardware to run Riff.CC and related efforts, and spent around $75,000 AUD since Riff's last revival to bring us to where we are today. We have 1.6 pebibytes of our own storage, over 864 CPU cores worth of compute, and a dedicated computing facility with staffing. This will one day host Riff.CC for public use, and hopefully be the beginning of a network of sites distributing Creative Commons content and public domain media for anyone who wants to consume it.

The Pioneer Ecosystem

Our software and tools will form the Pioneer Ecosystem, allowing people to take back control of their media and build and remix culture. 

We're building Lis, a distributed filesystem which will cooperate with existing DSNs and allow for reliable and incentivised seeding of content across IPFS and other networks.

We're building Orbiter, the first tool to support Defederation, which will allow you to participate in a permissionless network of sites distributing content.

We're building Defederation, a protocol which allows for sites to follow each other a la Twitter of old, instead of users - flipping the Web 2.0 model on its head for the future. When sites follow each other mutually, they effectively merge as one - sites follow other sites that fit their moderation and ethical policies, and instead of the isolated pockets of the web of today, communities will emerge that harmonise and agree. Platforms like Mastodon, Bluesky and Lemmy have proven that such a model works for text - we believe a similar model will work for general media.

We're looking into support for ActivityPub, so that sites running Orbiter can publish their content to the Fediverse and cooperate with the new decentralised paradigm.

We can take the internet back, with these efforts and the efforts of others building on both the future and the past.


The future is coming, whether we like it or not. Let's make it a future we love.

Let's kick fire.