OGG Vorbis community servers

Hi all,

Today, ogg files take about :
- 800Mb for the streaming files
- 1,2Tb for the archives files (album downloads)

The software that we use to distributes thoses files in written in python, so we need a machine with python 2.5 preferably under Ubuntu (Hardy) or Debian, but we can still try if you have other Linux distros.
The software runs with a special user that need some privilegies, but no root access is really needed.

Concerning bandwith, anything should be better that what we have now, but as least 10mb/s should be great.

If you need more information feel free to ask

Amandine

Hello amandine, what level of I/O should we expect for the album downloads? Would a raid1 of 2TB 7200rpm disks be enough?

I think that would be more than sufficient. Even a "slow" harddisk should manage it. RAID1 is the redundancy one, right?

I offer my manpower and semi-decent Linux knowledge. I would also give 10-15€ per month to a community server project.

Yann2 wrote: Would a raid1 of 2TB 7200rpm disks be enough?

Yeah, sure ! As "The Chilling Spirit" said, it's more than sufficient ! You should more be worried about bandwith than I/O, even if ogg downloads are not as popular as mp3s. Raid 1 with 7200rpm disks is what we have on our regular download servers and it has never been a problem.

http://www.kimsufi.com/fs/

So a cheap server like that would cost around 700€ a year.. It doesn't sound *that* expensive - is there really no chance Jamendo would sponsor it -or at least part of it? It doesn't look that expensive... right, it's OVH, and a slow CPU, but it's cheap, and if it does the trick...

Bonjour, ok - oublions ma dernière remarque. Imaginons que la communauté s'occupe des charges liées à l'hébergement du serveur, serait-il envisageable:

* Que les fichiers ogg vorbis soient générés dans un bitrate plus élevé que les MP3s (Après tout, si on héberge le miroir...) ?
* Qu'il y ait une page sur Jamendo expliquant la différence entre le Vorbis et le MP3, expliquant clairement pourquoi Vorbis c'est le bien, tout en mettant en garde contre la non prise en charge par de nombreux matériels?
* Que le lien vers le fichier vorbis de telechargement soit mis en avant, avec une lien vers cette dernière page?

Enfin, ceci n'est pas nécessaire mais ca me filerait un sacré coup de main pour trouver un hébergeur sympa - est ce que Jamendo accepterait qu'il soit fait mention dans un fichier txt des archives ZIP quel provider fournit le serveur? Genre un "The download for the Vorbis version of this album has been provided by xxx" quelque part?

Enfin, je serais intéressé par des stats du serveur qui fournit les telechargements mp3s actuels - si on fournit un serv et que le CPU / les i/o suivent pas, ca serait bete :)

Merci!

Sorry this was meant to be in english, good for translating now... So let's forget about my last post. Imagine the community would sponsor the OGG server, would it be possible:

* That the vorbis files get encoded at a higher bitrate than the mp3 ones?
* To provide a page where Jamendo would tell its view on the subject (vorbis is the greater good) albeit warning that it might not play well on some devices, and explain the choices you made regarding supporting mp3 support
* That the link to the vorbis download is clearly visible on the download popup, with a link to the explanation page?

Finally, although not necessary, would it be possible to add a text file to the download archives that would state something like "The vorbis download of this file is provided by xxxx" - it would definitely help to find a friendly provider.

Thanks

Ogg Vorbis is already encoded at a higher bitrate than MP3. If I recall correctly MP3 is V2 while Vorbis is Q7. I never liked that because Q5 is more than enough for me and transparent for most people, but hey. :)

I too would like to see Vorbis promoted and it must be as easy to download as MP3.

Having a link next to each download link seems a nicer idea. Also that could mean a little pagerank increase for the linked provider which everyone likes. Good idea!

Hi Yann2,

These kimsufi servers are great you're right, they're new from this month so I'll choose them next time I need ! The problem with OVH is the bandwidth : you have 100mbps per account... so 10 servers leaves only 10mbps per server, and adding more servers just makes the problem bigger.

I'll let sylvinus answer your few questions, but I can give you the mp3 server stats :
As you can imagine, there is 3 different server types : streaming, album download (zipped), and track download.
There is 2 redundant servers for streaming, 2 servers for album download (used to be redundant too, but not anymore), and 2 servers for track download.
They're (almost) all dedibox pro servers (http://www.dedibox.fr/dedibox-pro/serveur_dedibox_pro.html).
Here are the graphs :
- for album download : http://imgjam.com/servstats/albumdl.png
- for track download : http://imgjam.com/servstats/trackdl.png
- for streaming : http://imgjam.com/servstats/streaming.png

I guess the one you're interested in is the album download one. Anyway, cpu and memory is not really the problem here, almost anything should do the trick.

Hi Fab, indeed, but one single server should be enough for a while... When 100mbits become not enough, we can move to the 150€ ones that have gbit, which should be enough for a while ;)

How needed is redundancy - can we have a 0 redundancy setup and just copy the music over again if it dies? (read: do a raid 0) ? If CPU and I/O not an issue, a raid 0 on a 50€ server should provide high bitrates downloads for a while. Fairly certain we could raise such a small amount of money too... Or get a deal with some hosting provider.

Oh and getting the "ogg.jamendo.com" domain to point to a separate, community-run website, would be awesome - or a jamendo page where some community members would have write permissions (thanks for making it point here already!)

yes, one single server (the basic one) for ogg should me more than enough. Raid0 is ok too, we still have the files if it crashes, you just risk some days of downtime.

Amandine

Ok, great, let me know for my other questions, having Jamendo help promote Ogg is pretty much the condition sine qua none for all this to happen

But if we agree on that, I'd be happy to lead some sorf of effort to get it all running. Can't promise anything, but I'll try hard :)

Thanks Sylvinus, I'll work on it and get back to you. This is good news though :)

Is the python software a must or would rsync work too?

I cannot find it anymore but didn't someone say that we should not worry about redundancy, because if "our" files died there would still be the files at Jamendo? That would suggest that we could simply rsync that regularly. I might forget about something important with this... Hm.

Anyways, singpolyma forwarded the reply he got from archive.org to me and I informed them about these new developments. With luck it will work out nicely. :-)

Hosting the streaming files could be independent from this. It would be much cheaper and should be no problem to get donations for. Something for 10€ per month would probably be more than enough. Is it really just 800Megabytes?

I am not interested in the single tracks myself. Not sure what the others think. I doubt public interest in that is significant compared to ZIPs and streaming.

I believe the streaming will need storing all the files, so probably another 1.5TB :)

To people reading: The chilling spirit and myself are coordinating efforts and contacting companies and non-profits that might be willing to sponsor us (servers, bandwidth or cash). Hopefully this will work out, else we will probably start a fund raising.

Sylvinus > Sounds good, but I've got doubts it would allow streaming of songs, at least not without a significant overhead? We're also looking at very cheap servers (celerons) which might be happier with another disk than with having to extract zip files :)

Can we have a look at this python script? I'd be interested in having a read through it, like the chilling spirit for the moment I dont understand why rsync wouldnt work - what exactly is it doing? Ogg conversion? Clever-er mirroring?

I have a small server in my basement I host my website on. It runs ubuntu (8.10 iirc? or was it 8.04). I think it's got python, but if not I can install it.

It's not much, but it's got a cable connection and 100gb of spare storage. Is there a way to throttle the bandwidth so that my family can still use the internet?

I should be able to help providing a 60€/mo kimsufi.com/fs/ with a friend, plus some sysadmin time.
More info in a bit.

Sorry for my English :)
I have a home server with big hard drives. It permanently accessible to Internet. I would happy to provide virtual machine for Jamendo Ogg mirror after two to three weeks.

@fab4am , @sylvinus : we have a first box to start the service!
We setup a VM with 1.5TB which is currently reachable at ziclibre.beuc.net .
Contact me at beuc@beuc.net with SSH keys so we can setup your access.

We still need more space to get redundancy and other kinds of services (streaming...), so post here if you can share boxes/bandwith!

Hi sylvinus,

I mailed you this afternoon about a proposition.
We discussed about it with Beuc, I'm waiting for your answer.

Hi,

I've summed up the setup requirements for community ogg servers : http://developer.jamendo.com/fr/wiki/OggServerSetup

Feel free to ask if something is not clear !

Amandine

The setup of the first ogg system is in progress :)

Also you can subscribe to open-music@listes.oxyradio.net if you're interested in providing resources.
http://listes.oxyradio.net/wws/info/open-music

Can anyone update us on this thread ? Have plans progressed ?

Wow, yes. Thanks so much!

I've talkted to a few people and what I may come up with is a donated dual opteron, 6GiB RAM dedicated server, a place with enough bandwidth to host it (for free) and an admin who'd be willing to put Gentoo on it and maintain it. I'll probably get together an interest group to donate for the needed disks (an idea was 4x1TiB disks to have a 3GiB RAID-5).

Would that work and would it help?

I've got a few questions thought:
* would these servers include torrent trackers and seeds?
* could we use magnet links?
* would it be possible to offer a FTP download to serve as webseeds to extend torrent seeds?

"The setup of the first ogg system is in progress :)"

Ok - good news.

"Also you can subscribe to open-music@listes.oxyradio.net if you're interested in providing resources.
http://listes.oxyradio.net/wws/info/open-music"

Not sure how to navigate this page. Is English available ? Where to go to create a login/password?

Question: You guys know more about this than me but - does Amazon Web Services lend itself to something this ? Evidently they offer a cost effective hosting service. Could Amazon host a server instance for downloading Ogg Files ? Remember - there is no hardware cost.

You can subscribe to the list by sending a mail at openmusic-subscribe@listes.oxyradio.net

@sylvinus: Why's that? I'd rather think that outsourcing the bandwidth to users would be beneficial. Plus if both a direct HTTP/FTP download is available and P2P at the same time, it'd be very unlikely that the albums would be lost or unavailable.

Because there are 33000 albums x 2 (ogg & mp3). That's almost 70000 torrents you would have to seed. Most of them will have very few peers, probably single digit downloads per month. Maybe 5% would attract 2-3 seeders. So you would still be the sole provider for most of them but with a lot of overhead and hassle. Providing HTTP downloads "on demand" is much simpler, lightweight and convenient (for example if an album gets changed a torrent becomes invalid).

You guys rock! I support the idea of http as most of the albums I like are not pretty popular, and I end up with days and days of 0 activity in the torrents. Any estimation on when the option will be ready and operational? (Don't wanna sound pushy, just wondering, this is something I've been waiting for years)

@The Chilling Spirit:

Since it's about _community_ _ogg_ servers, that means that 1) it's up to the community to share, where P2P is quite a logical (part of the) solution 2) it's just Ogg, so we're talking about 33 K albums/torrents. But even if it were 100 K where's the problem?

Could you explain the overhead and hassle to me then? My logic being since the torrent files seem to be automagically created by Jamendo anyway and the content would be on the server anyway, I don't see a problem there. Making it available to the community in different ways at the same time should IMHO be encouraged — especially if this also encourages the community that doesn't have run its own mirror of Jamendo to contribute to sharing as well!

Concerning the overhead itself, do you think it would really be too huge for a dedicated server to seed a few thousand torrents next to a sync cronjob and HTTP/FTP downloads? If so, please let me know.

OK, I'll grant you the problem with torrent validity when an album changes. Is there any statistical data on how often that actually happens?

I ran seeding with a couple of thousand torrents once. Clients are a pain in the ass with that many torrents. Some will abuse your cpu, others will not work, other will be buggy. A very low percentage of the torrents had any traffic ever and an even lower percentage had seeders sticking with them.

http://www.jamendo.com/en/forums/discussion/4878/torrents-might-have-no-seeders/
http://www.jamendo.com/en/forums/discussion/6157/bug-report-torrents-are-not-automatically-seeded/#Item_11 (my reply below)

With some manual browsing I got some numbers (no download numbers from the API):
20000 albums have not been download more than 100 times according to the site. Less than 3000 have more than 1000 downloads. That is for both formats of course. Vorbis is a single digit amount or even less (I forgot the number sylvinus said).

Super tall to make the point: http://enjoys.it/jamendo/jamendo-totalnumberofdownloads20100415.png

Publishing and advertising torrents for popular albums (or special promotions) would be a great idea of course. Torrents live from a healthy swarm. They are useless if there is one leecher every 5 days.

I know that uses torrents gives a warm fuzzy feeling. I LOVE torrents and the whole idea behind them. I guess you could make it work better if you invested some serious effort in a torrent-centric frontend but that surely is not something Jamendo would ever do.

Seeding the top 100(0) albums would make sense. :)

I vote for torrents as well - but maybe not just yet. Torrents require a significant user base to seed and share files. Jamendo is still evolving and its membership growing. Hopefully Ogg fans can grow with it.

At a suitable time in the future we can come back to the issue of torrents. At the moment with all files residing on one server, they are vulnerable to hardware failure and threats from Governments and Business seeking to maintain a monopoly on the download of music files.

Can someone advise of the costs associated with setting up the Ogg Server and how we may contribute to this ?

There is not one server. For the moment, there is multiple servers in order to start. And in June, OxyRadio plans to buy multiple servers and set up them in our own housing space (no rent or something like that). No hardware failure because of redundancy (2 servers with replication (if one shutdown, the other takes the place with the same data), raid, etc).

And governments won't be able to do anything about. It's just pure paranoia lol.
As OxyRadio is a non-profit association, we're able to receive contributions/donations and stuff.

For the record, music is being transfered on both Beuc and OxyRadio servers. And the professionnal plateform will be up in June.

"And governments won't be able to do anything about. It's just pure paranoia lol."

Maybe - but we thought that about Pirate Bay.

The Pirate Bay is mostly used to illegally (depending on your local legislation) share works with highly restrictive copyrights.
Jamendo is a site dedicated to music that is free to share.

I think the main concern would be a new financial crisis, with investments and public subsidies suddenly cut. But with all the anti-sharing legal mess these days (dadvsi, hadopi...), we could well imagine the government voting that DRM be mandatory or something :/

@sylvinus: do you have this libtorrent code around? We're looking for mass-seeding solutions but it looks like nothing is available.

@fab4am: album uploads stopped on Friday - any clue?

When I heard about the community effort I got other Ideas than I have read about in this thread.
The background is that I am myself willing to contribute a relatively small amount of resources that is not enough to cover all albums but if let's say 20 other do the same it would be enough.

My understanding of the current discussion is about to set up one large central cluster to supply all services, downloads, streaming, torrents. Why not instead have smaller groups where one is focusing on getting streaming working, another one on downloads and a third on torrents?

Ogg downloads. I don't see why python would be a must for serving http downloads. Although I'm not aware of how archive.org works, I guess it could work?

When it comes to torrents I can see the need for one central point of operation for the tracker.

When it comes to seeders I can think of a system where a number of community contributed systems each share a slice of the entire catalog and if more start to join, redundancy could be achieved. So Instead of a mass seeding server there would be masses(~50) seeding.
It would then need a central organization to make sure that all albums are taken care of by someone and that new contributors get appointed the most visited albums.

In short I suggest to split up the ogg initiative into smaller chunks so that it gets easier to contribute.

If anyone has an update, I would love to see what's going on. Also a road map or guess at a timeline for the individual initiatives would be great.

Thank you to all involved!

http://kge.mandragor.org/munin/ziclibre/ziclibre-df.html is a good indicator as well :) I'll bump a message on the mailing list to see how the new coming servers are doing, this one is supposed to be just temporary.