| |
|
|
|
1
| 2
|
Next Page
|
| Audio Codec Comparison: Vorbis vs MP3 vs AC3 vs FLAC... others? |
Redhatter
12/3/06 11:55:29 PM
Hero
Guru
|
Hi All...
For a long time, I've been using MP3 to store my music... originally it was constant-bitrate 160kbit, then later VBR... but basically, it's been using the same codec. The source of my music is generally 44.1KHz 16-bit linear PCM audio, ripped from CDs or vinyl records.
Just the other day, I started experimenting with the Vorbis codec... and noticed I was getting files about 1MB smaller than MP3, and the resulting audio didn't sound that much different to the MP3 (or even the original PCM).
What I'd like to know though... has anyone done an actual study on how these different codecs compare at various bitrates?
I'm looking at conducting a few experiments myself... the idea was, using uncompressed PCM as a control, I'd compress a given audio file using a particular codec for a given bitrate, then decompress it back to PCM and compare the output against the input in a package like Matlab. By subtracting the output from the input, I should get just the error -- I'd be looking for the least average error.
The codecs I use would have to be available under Linux ... since my version of Matlab is the Unix/Linux version.
What I'd like to establish, is the approximate compression ratios achieved at various common bitrate/VBR settings, and see how each codec compares.
So... my queries:
- Does anyone know of a similar study? What was their verdict?
- What sort of source audio would be best? I figured a couple of sample files ... e.g. a purely sinusoidal one like http://dev.gentoo.org/~redhatter/misc/ip32-waveform-in.ogg and more complex ones such as sample sounds distributed with KDE, and maybe a few DVD-ripped songs.
- Are there flaws in my thinking? -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.hopto.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
Kung Fu Hung-Su
13/3/06 2:01:23 AM
Champion
|
I haven't, and I would certainly read such a study, though i've already kind've decided on OGG for my "mp3" player + FLAC for archive... -----
To those who reply to my posts: Call me Hung, Hung-Su or Kung Fu!
Miyamoto Musashi: Become aquainted with every art.
Bruce Lee: 'I' do not punch. 'It' punches all by itself.
|
THE DEVASTATOR
13/3/06 1:15:19 PM
Overlord
|
http://www.hydrogenaudio.org/forums/index.php?showtopic=21904&hl=
There is a newer test which I do not have the link to but the results are about the same, also the newer test has less formats.
A few other tests: http://www.hydrogenaudio.org/forums/index.php?showtopic=36465
http://www.hydrogenaudio.org/forums/index.php?showtopic=35438
Edited by THE DEVASTATOR: 13/3/2006 1:33:00 PM -----
Do you want to touch my Lightsaber.
The force is strong...... in my pants!
|
boomerang_beeby
13/3/06 2:27:40 PM
Overlord
|
Quote by Redhatter
I'm looking at conducting a few experiments myself... the idea was, using uncompressed PCM as a control, I'd compress a given audio file using a particular codec for a given bitrate, then decompress it back to PCM and compare the output against the input in a package like Matlab. By subtracting the output from the input, I should get just the error -- I'd be looking for the least average error.
Thats good thinking however, you will only be getting the physical differences between the waveforms, this is one part of how lame (mp3) works, by stripping out waves that you can't here. If you then compare it as a waveform you will see a large difference that you may not actually be able to hear.
You should really include an ABX test :
http://wiki.hydrogenaudio.org/index.php?title=ABX
The tester assigns one file to the "a" button, the other file to the "b" button, and then the abx program randomly assigns either the first or the second file to the "x" button. The listener can listen to a, b, or x, in any order, as many times as he wishes, then decides whether x is the same as a or the same as b.
You also mentioned FLAC in the title. For those that are living under a (sound proof) rock, FLAC is a lossless codec, a little bit like if you zip the wav file you rip off the CD. You are able to unzip (decompress) it and get back your orignal wav.
Your test should prove that the conrol is exactly the same as the waveform that came out of FLAC. If not, you did something wrong :p
(this is the same for ALL other lossless codecs)
When choosing a codec to use, you really need to take into acount the source, where it will be used, and the output media. OGG Vorbis is great but not all Digital Audio Plays will read it. MP3 would just be silly for a movie soundtrack when you want 6 channels. FLAC is not useful if you want to podcast. The list goes on.
There is also variations between defult and recommended settings of different encoders, not to mention the different versions and compiles of them. I would try and to a HEADtoHEAD (atomicmpc style :D) between just 2 codecs first, and then you con try some others.
I am happy to be an ABX'er for you. I consider my hearing up there for a youngin. -----
I was at the MILLIONTH post threat in General Chat
You?
The OC AND SG sucks v2.5
^Those were the days^
|
Redhatter
13/3/06 9:51:04 PM
Hero
Guru
|
Quote by boomerang_beeby
Quote by Redhatter
I'm looking at conducting a few experiments myself... the idea was, using uncompressed PCM as a control, I'd compress a given audio file using a particular codec for a given bitrate, then decompress it back to PCM and compare the output against the input in a package like Matlab. By subtracting the output from the input, I should get just the error -- I'd be looking for the least average error.
Thats good thinking however, you will only be getting the physical differences between the waveforms, this is one part of how lame (mp3) works, by stripping out waves that you can't here. If you then compare it as a waveform you will see a large difference that you may not actually be able to hear.
Agreed... I figured this would help narrow it down a bit, and my hearing would do the rest.
You should really include an ABX test :
http://wiki.hydrogenaudio.org/index.php?title=ABX
The tester assigns one file to the "a" button, the other file to the "b" button, and then the abx program randomly assigns either the first or the second file to the "x" button. The listener can listen to a, b, or x, in any order, as many times as he wishes, then decides whether x is the same as a or the same as b.
That sounds like a good idea. :-) I should be able to rig one up pretty easily here. This would be more practical at the latter phases of the tests though.
You also mentioned FLAC in the title. For those that are living under a (sound proof) rock, FLAC is a lossless codec, a little bit like if you zip the wav file you rip off the CD. You are able to unzip (decompress) it and get back your orignal wav.
Your test should prove that the conrol is exactly the same as the waveform that came out of FLAC. If not, you did something wrong :p
(this is the same for ALL other lossless codecs)
Yeah... I had heard that... I figured I'd throw it in for completeness more than anything, and perhaps as a second control. I'd also be looking at compression ratios, and so it'd be interesting to see how FLAC compares with its lossy cousins, and against uncompressed PCM.
When choosing a codec to use, you really need to take into acount the source, where it will be used, and the output media. OGG Vorbis is great but not all Digital Audio Plays will read it. MP3 would just be silly for a movie soundtrack when you want 6 channels. FLAC is not useful if you want to podcast. The list goes on.
Indeed... a horses for courses market. You wouldn't use Speex as your music archive codec either. ;-) (Speex is a cousin of the Vorbis codec, designed especially for voice recordings)
There is also variations between defult and recommended settings of different encoders, not to mention the different versions and compiles of them. I would try and to a HEADtoHEAD (atomicmpc style :D) between just 2 codecs first, and then you con try some others.
That's a possibility... but for a test to be truly scientific, you need a control. I realise the implementation is going to make a difference too (e.g. Xing's MP3 encoder, which I used when I first started messing with MP3, vs the LAME MP3 encoder, BladeENC ... etc). I'll probably stick to a well known implementation -- other people can do their own comparisons between the alternatives.
I am happy to be an ABX'er for you. I consider my hearing up there for a youngin.
Heh, mine's not too bad... but there's the issue of reference material. Copyright is not our friend here. -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.hopto.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
bum
13/3/06 9:58:47 PM
Overlord
|
Isnt a comparison between lossy and lossless codecs kinda meaningless? -----
Quote by Salty peanut
You fit in as much as the rest of us dont.
|
Redhatter
13/3/06 10:37:05 PM
Hero
Guru
|
Not quite... it is, in that the lossless codec (by definition) should be a perfect replica of the original... but what I'd then be comparing, is it's storage size, compared with the original PCM waveform. -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.hopto.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
boomerang_beeby
15/3/06 11:00:30 PM
Overlord
|
Quote by Redhatter
Not quite... it is, in that the lossless codec (by definition) should be a perfect replica of the original... but what I'd then be comparing, is it's storage size, compared with the original PCM waveform.
Lossless compression ratio hovers between 45-65% of the orignal PCM waveform.
You can really only have 3 (directly measurable) variables between lossless codecs: Compression Ratio, Encoding Speed and Decoding Speed.
Lucky for you, it has been done before. http://web.inter.nl.net/users/hvdh/lossless/All.htm
It would be nice to see it updated a bit. With some cleaner graphs. -----
I was at the MILLIONTH post threat in General Chat
You?
The OC AND SG sucks v2.5
^Those were the days^
|
Redhatter
9/8/06 12:08:38 AM
Hero
Guru
|
Okay... yeap... I'm bumping this thread...
For uni, as part of the statistics component of my Engineering Maths subject, I need to gather some statistical data up (as part of a group project). This idea was brought up... and it looks like we'll be doing an actual study into this.
I'll let you all know what the results are, and how it's all going.
Stay tuned. :-) -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.yi.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
The_Salty_Peanut
10/8/06 11:14:02 AM
Mod
Hero
Immortal
|
Keep us posted ! -----
The Dude abides.
|
The Fuzz damn you!
10/8/06 11:44:47 AM
Hero
Immortal
|
It could be worth taking a look at encoding/decoding times, as well as quality/size. This comes into play especially for lossless codecs, where size differences will be negligible, and quality difference (should be) non-existent. -----
You are about to begin reading The Fuzz's .sig. Relax. Concentrate. Dispel every other thought.
.....|...
..\...../
-- o o -- << Duly elected Mayor of Atomic >>
../.....\
.._|..|_
|
Redhatter
10/8/06 6:29:00 PM
Hero
Guru
|
Some non-conclusive test results...
MP3 (encoded using LAME) is a dismal failure against Vorbis and AAC (encoded using FAAC). 48Kbps Vorbis is passable, but isn't great... 56Kbps MP3 isn't good for anything except voice.
I experimented today with comparison of waveforms in the time domain, and I'm also looking into ways of conducting listening tests on-mass. Nothing much at the moment.
I haven't done any serious analysis, but I've coded some shell scripts up that take a bunch of segments of a song (I was using Fleetwood Mac's "Tusk" for this one), encodes each segment at a different rate, decodes them, and then concatenates the pieces together.
The result being a track that drops in quality as time progresses, starting at 320Kbps, dropping to 256Kbps after ~20sec, then 224Kbps, 192Kbps ... etc, down to 48Kbps. The idea is someone would listen to that, and pause when the distortion becomes apparent.
I also did some experiments in Matlab, although I've still got my L plates when it comes to doing FFTs with it. (Doing FFTs in Matlab is bloody awful, and I might end up biting the bullet and coding something myself in C). I might be able to get some sensible results subtracting the original signal from the encoded signal, and calculating the RMS value for this "differential" signal. I'll have to try this across a few bitrates, and see what happens.
As far as frequency domain stuff... just using the FFT analysis built into XINE (which I found more convenient than XMMS), I could definately see differences in the audio samples as the bitrate dropped -- the upper frequency response drops, as you'd expect.
I have been timing the encoding/decoding user time, system time and real time, as well as noting compression ratios... at this stage, anything I could measure, has been measured. So yeah, I'll definately be looking into these, as they've got great significance for portable music player companies who may decide to use this information. (which they'll be welcome to do so, I plan to make this publically accessible under a free license)
One question for the assembled masses though...
There is the possibility of me getting some hosting somewhere, and conducting a listening test online. In order to do this, we'd need a sample of music that would be a good test for the codecs, is in a lossless codec (or uncompressed), and can be freely distributed legally. This is important, as I do not wish to violate any broadcasting/copyright laws in the process of doing this assignment.
Does anyone know a site where one might find such music? -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.yi.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
dlite
10/8/06 9:59:38 PM
Serf
|
Redhatter,
The quality of hardware a person has to reproduce the music, especially speakers will have a big impact on where the music is heard to distort, in my experience the better the system the earlier the distortion is heard.
There are many codecs that are free to use even in commercial situations like FLAC. As for music, try www.besonic.com and links here http://www.vocalist.org.uk/artists_upload.html
Also whilst some codecs are lossless some players I believe especially software players do not replay them losslessly.
Feel free to PM me if you have questions I actually sell a few Network media players like the Squeezebox, Pixel Magic and Hifidelio etc
-----
My Webpage : http://www.audiosupermart.com
|
Redhatter
10/8/06 10:33:18 PM
Hero
Guru
|
dlite: Exactly. :-) Perceived audio quality depends on so many factors... which is why I'm still persisting with the analytical approach.
On those music sites... I haven't looked yet... Do they offer music in a lossless form? This is important... transcoding from MP3s (or any lossless format) will distort the audio -- skewing the results rather badly. -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.yi.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
zebra
11/8/06 4:06:22 AM
SuperHero
Titan
|
If I am using an encoded format, at all - it will be FLAC, else...all pure .wav/wave/AIFF. I can't stand to listen to anything else. -----
#SPLCrew on liteonchat.no-ip.com on IRC!
-------------
speedlabs.org; Burn like you mean it.
Vicariously I, live while the whole world dies. Much better you, than I.
|
dlite
11/8/06 9:12:44 AM
Serf
|
Quote by Redhatter
dlite: On those music sites... I haven't looked yet... Do they offer music in a lossless form? This is important... transcoding from MP3s (or any lossless format) will distort the audio -- skewing the results rather badly.
I beleieve the besonic site is user uploaded and normally in lossy format, however you should be able to approach some of the uploaders who would have the files in original un compressed formats. -----
My Webpage : http://www.audiosupermart.com
|
tycho102
15/8/06 1:45:04 AM
Banned
|
APE generates better compression than FLAC. FLAC routinely gets 50% compression for me, and APE will generate 60-65% compression. If you use APE for your lossless, I believe one of K-Lite's codec packs (probably the "Mega") come with the codec. If not, I am certain that Ace's Mega pack does, although you don't want to install anything else out of that pack unless you are prepared for it.
Cutoffs are very easy to see in any decent audio editor. The Vorbis and DDL/AC3 cutoffs, for a given bitrate, are always higher than MP3 or MP3PRO. If MP3 cuts-out at 15khz, the Vorbis will go to 16.5khz, and the AC3 to 17.5khz.
Other than that, the only comparison I've ever done was a long time ago. I converted Bach's "Toccata and Fuge (D-minor)", and looked at the same timecode in comparison to the original WAV. The audio editor will "decode" the compressed formats into PCM, so it ends up being a precise method of comparison.
If you graphed the signal strength vs time waveform of the various formats, whichever comes closest to the original PCM-WAV will be the clear winner. In some cases -- notably frequencies below 12khz -- it will probably take a statistical analysis to produce a winner. -----
-> Everybody was kung-fu fighting!
-> 2x2ghz Opteron,Tyan Thunder K8W,2GB PC3200,BFG 6800GT,Dell 2405FPW,X-Mystique DDL,2xBenq 1655 Lightscribe,3ware 9550SX-8,RaptorX raid0,UPS,Enermax EG851,MG Magnum 500
|
Redhatter
7/9/06 11:21:32 PM
Hero
Guru
|
Another bump... We'll soon be putting the files online, and letting you loose... however... in the meantime... there's a little technical hitch.
http://stuartl.longlandclan.hopto.org/blog/2006/09/07/audio-codec-comparison-survey-looking-for-a
Anyone know of a web-based FLAC player applet? -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.yi.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
King_TriTAN
8/9/06 5:53:24 AM
Overlord
|
AC3!AC3!AC3! -----
A good tagline is indicative to ones good nature... I obviouly have a crap one
|
Redhatter
9/9/06 12:44:33 AM
Hero
Guru
|
Quote by King_TriTAN
AC3!AC3!AC3!
.... Right... your post neither adds useful information to the thread, nor answers my above question.
Just what exactly is your point? -----
Stuart Longland (aka. Redhatter)
I haven't lost my mind it's backed up on a tape somewhere...
[ http://atomicl.berlios.de | http://atomicdoc.yi.org ]
International Asperger's Year http://dev.gentoo.org/~redhatter/iay
|
Forums | Audio
|
|
. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.