yogsogoth
22/5/05 1:51:15 PM
Champion
|
There is a feature of the NT file system that is some what documented but not highly promoted, it is called “alternate data streams” and they are worth knowing about from both an interest point of view and also from a security point of view.
Alternate Data streams are data that is associated with a file but is not part of the file, this data can only exist on the NT file system, if the file is ever moved to a FAT partition the alternate data is lost.
It has been a feature of the NT file system since NT 3.1.
Windows uses these streams to store information like Title, Subject, Keyword, Comments, etcetera for files.
If you right click on a file, select properties and then the summary tab you will see a bunch of fields that can be filled out, all the information stored in these fields is added to a data stream and not to the file it self. You can check this, simply note the length of the file before adding to the summary fields and you will see that it does not grow with the addition of more text.
ADS was apparently included in the NT file system for compatibility with macintosh HFS, which uses alternate streams that they call forks to store information about the application used to open the file.
That's all a bit boring until you discover that you can create your own ADS to hide any thing you can think of, i will demonstrate in a moment.
Firstly, things to be aware of;
*Once an ADS has been created it can not (to the best of my knowledge) be deleted, though it can be over written. The only way to remove the alternate stream is to delete the file (default stream).
*An ADS follows the same rules as any other file, that is it can be as large as what the file system will allow and it will use that amount of disk space.
*Windows does not take into account the disk space consumed by ADS, when reporting “available disk space”. Not sure what will happen if the physical capacity is exceed and windows believes other wise, but it can't be good, best avoided.
*Never attach an ADS to the root directory e.g. C: . as you can not delete the root directory you can not remove the ADS.
*You can have more then one ADS per file, to my knowledge there is no limit.
Creating Alternate Data Streams
Proceed at your own risk!
For these experiments I recommend you create a new directory that can later be deleted if things get out of hand :)
All these examples are using XP sp2, other versions may have different behaviour.
I am going to use D:\ADS-test\
First create a zero length text file called myfile.txt
D:\ADS-test>copy con myfile.txt
then press F6 then return/enter
Should look like this
D:\ADS-test>copy con myfile.txt
^Z
1 file(s) copied.
That was just for some thing different, you can simply create the file in explorer if you prefer.
Verify that the file is reported as zero bytes
D:\ADS-test>dir myfile.txt
Volume in drive D is My Documents
Volume Serial Number is 7C73-4E80
Directory of D:\ADS-test
22/05/2005 11:54 AM 0 myfile.txt
1 File(s) 0 bytes
0 Dir(s) 92,632,068,096 bytes free
Now lets create an ADS to go with this file. The ADS will be given the name “ADS”.
D:\ADS-test>echo this text is hidden > myfile.txt:ADS
notice the colon {:}, it is key to creating and accessing an ADS.
You can check your “myfile.txt” and see that it is still zero bytes.
To access the text that we just added to the Stream called ADS we need to use the “MORE” command.
D:\ADS-test>more < myfile.txt:ADS
this text is hidden
And -----
"I don't want the world;
I just want your half."
***************
"Save the earth.
It's the only planet with chocolate!"
|
bnew
22/5/05 3:15:41 PM
Champion
|
Great info mate, thanks for the heads up. I've only ever read the theory about ADS, haven't actually played around with it as yet -----
||AthlonXP 2800 @ 2.2ghz 1.7vcore||Tt Triblade HSF ||2 x 512 Kingmax DDR500 2.5-4-4-7||Abit NF7-S rev.2||Abit X800pro||2 x 160gb WD 8mb cache||Gigabyte DVD Burner||Audigy2zs||Sennheiser212pro||CM Centurion||Trueblu350w||
|
myne
22/5/05 7:18:54 PM
Disciple
|
Ive vaguely known about this for a while, but after reading this I thought of an actual use.
Checksums.
"md5.bat"
md5sum %1 > %1:md5
"md5c.bat"
md5sum -c %1:md5
You'll need this : http://www.etree.org/md5com.html
Unfortunately I cant figure out how to make wildcards work with it
What you can do is something like this :
c:\> md5 filename.ext
This will create a stream called :md5 and copy the md5 string into it.
Then, to ensure the integrity, you use md5c
c:\> md5c filename.ext
If it reports OK it worked.
Now I have to figure out how to make it do wildcards and it can actually be useful
-----
|
myne
22/5/05 11:14:31 PM
Disciple
|
Sometimes I scare myself.
"Md5s.bat" to be used in conjunction with above linked md5sum.exe
*****start*****
@Echo off
set a=0
set errors=0
if "%1"=="-c" goto %1
if "%1"=="-m" goto %1
echo.
Echo Usage is " md5s.bat -m filename.ext " to create an md5 checksum
Echo Or " md5s.bat -c filename.ext " to check the md5 checksum of a file
echo -c or -m MUST be present
echo.
goto end
:-m
for %%i in (%2) do (md5sum "%%i">"%%i":md5 || (set /a errors=errors+1 & echo. & Echo %%i ***********ERROR***********)
echo %%i done
set /a a=a+1
)
echo.
echo.
Echo ================================
echo %a% files checked
Echo %errors% failed
goto end
:-c
for %%i in (%2) do (
md5sum -c "%%i":md5|| (set /a errors=errors+1 & echo. & Echo ***********ERROR*********** & echo.)
set /a a=a+1
)
echo.
echo.
Echo ================================
echo %a% files checked
Echo %errors% Bad or missing Checksums
echo.
:end
*****end*****
copy that and the md5sum.exe into your windows dir (or anywhere in the path variable)
md5c -m *.* will calculate the md5 for all files and save it to the file in an alternate filesystem stream called :md5
md5c -c *.* will calculate the md5's and compare them with the previously caculated md5 for that file
Obviously *.* works, so any wild card should work fine.
(only for ntfs obviously)
Edited by myne: 22/5/2005 11:32:40 PM
Edited by myne: 22/5/2005 11:44:22 PM -----
|
yogsogoth
23/5/05 5:11:18 AM
Champion
|
Nice idea myne
you found a great use for ADS, I believe that some virus scanners do a similar thing, but to come up with it on your own is way cool.
I will give your script a go later today if I get the chance
Cheers
-----
"I don't want the world;
I just want your half."
***************
"Save the earth.
It's the only planet with chocolate!"
|
boomerang_beeby
24/5/05 5:24:07 PM
Overlord
|
myne, thats freakishly sounds like time was waisted on it. cool though.
I'll look into a few of those Stream Explorer, should be fun. Im surpried MS didn't provide one to use.
That would be a good way to do Intrusion Detection, run the check on the systme dirs every day. And compaire the "tree" output.
Is there a way (with out a remote desktop solution) to put files from an NTFS to NTFS filesystem over a network? -----
I was at the MILLIONTH post threat in General Chat
You?
The OC AND SG sucks v2.5
^Those were the days^
|
80proof
24/5/05 5:43:38 PM
Charge
|
*Once an ADS has been created it can not (to the best of my knowledge) be deleted, though it can be over written. The only way to remove the alternate stream is to delete the file (default stream).
Sure there is. Copy the file to a FAT(32) partition and then back again or to a CD and the back again will work just aswell. -----
You were my sun rays.
Without you girl there was no days.
Never dreamt I'd speak the phrase,
Now what the fuck just happened?
|
myne
24/5/05 7:24:03 PM
Disciple
|
Quote by boomerang_beeby
myne, thats freakishly sounds like time was waisted on it. cool though.
Way too much time. Though I have prettymuch perfected it and I am going to use it. I tend to archive large video files for long periods of time. The pain with videos is they can become corrupt and still work. Atleast now I can verify their integrity and/or source a replacement copys from an alternate backup.
It's way way way too big to post in plain text so here's a link to the latest scarily uber batch file.
http://users.on.net/jvizard/myne/md5s.zip
It now supports... uh lots. Run it and find out.
Quote by boomerang_beeby
I'll look into a few of those Stream Explorer, should be fun. Im surpried MS didn't provide one to use.
That would be a good way to do Intrusion Detection, run the check on the systme dirs every day. And compaire the "tree" output.
Is there a way (with out a remote desktop solution) to put files from an NTFS to NTFS filesystem over a network?
Actually comparing files to a known good checksum is an ancient method of intrusion detection. Nonetheless it's still as effective as it ever was.
And yes, Normal windows file sharing sends alternate streams with the file so you can easily copy hashed files to a backup server.
Zip doesnt, ftp is unlikely to.
Still, It's going to be handy - the thing that made me so reluctant to do checksums before is I wanted to hide them and attatch them to the main file some how. Amazingly it's a feature that's been there for over 10 years :)
-----
|
boomerang_beeby
25/5/05 9:32:26 AM
Overlord
|
I thought the Alternative Streams would be lost once they hit SMB, just like NTFS ownership permissions. Or atleast if they're on a domain.
I'll do some testing then.
EDIT: No way to delete streams my arse:
http://www.sysinternals.com/ntw2k/source/misc.shtml#streams
Usage: streams [-s] [-d] <file or directory>
-s Recurse subdirectories.
-d Delete streams.
Edited by boomerang_beeby: 25/5/2005 11:10:58 AM -----
I was at the MILLIONTH post threat in General Chat
You?
The OC AND SG sucks v2.5
^Those were the days^
|
myne
26/5/05 1:36:14 AM
Disciple
|
I already did test it. Uploaded a file to other machine, ran the md5 checker and voila it worked.
had a look at my err... insano batch file? -----
|
yogsogoth
29/5/05 7:57:18 PM
Champion
|
Hey myne do you think your idea (method) could be adapted to check for duplicate files?
Just a thought,
I know there are already plenty of Duplicate file finders out there, but it could be a fun academic exercise.
Would Md5 be infallible for this purpose? Or at least good enough with some sort of user verification before deleting?
boomerang_beeby point taken; but you don't need the attitude to make it!
Third party apps can add lots of functionality to windows, my first post did not cover there use, only the use of native windows apps to work with ADS. I see your point but I will stick with mine until such time that I am shown a way to delete ADS with windows native tools/commands.
Nice find though I hope it works for you.
-----
"I don't want the world;
I just want your half."
***************
"Save the earth.
It's the only planet with chocolate!"
|
myne
30/5/05 3:10:19 PM
Disciple
|
I'd probably check the size as well.
Md5 is like several billion to 1 of having the same hash, but the same hash from a file the same size would be like several squillion trillion to 1 chance of happening -----
|
myne
30/5/05 7:27:50 PM
Disciple
|
hehe I am insane.
Assuming you're using my insane md5s.bat, you can check for doubleups like so:
md5s -vsr *.* >c:\fdrive.txt
that will output all the saved md5s to c:\fdrive.txt
Then, a simple
c:
cd
and then an insanely long single string :
for /f "tokens=1,2 delims=*" %i in (fdrive.txt) do find /c "%i" c:\fdrive.txt | find "C:\FDRIVE.TXT: 1" || echo %i %j >>c:\doubles.txt
And that'll create c:\doubles.txt
Open that in notepad and you'll see what doubles you have
I found 8 doubles on my f:
I'm thinking about making that a feature of the batch file. Who needs sanity anyway?
-----
|
. This material may not be published, broadcast, rewritten or redistributed in any form without prior authorisation.