Mass storage and Efficiency

[Guest loarevalo]

Should we care about efficiency in storing data, when now days storage capacity problems have virtually disappeared?

 

I thought about this when noticing that my digital pictures aren't really 5MB resolution but about 500KB in the worst cases, and 3MB in the best - though they all took 5MB of hard drive. While it's a problem of my camera, I thought: let's write a program that converts any image to its real resolution (optimize pixel usage). But then I thought, why do I care? I have more than enough memory space, and can burn DVDs. So I decided to not mind the outrageous wastefulness.

 

Yet then I thought, such ineficiency might not matter in hard-drive terms, yet what about networking, cache, ram, searching and sorting, etc...

 

So, should we mind the outrageous wastefulness of memory in modern computers?

What do you all think?

[Buffy]

yes, because if your data is inefficiently laid out on the disk or bloated, it will take longer to get the data off the disk or write it to the disk and your apps performance will degrade...

 

Cheers,

Buffy

[alxian]

when now days storage capacity problems have virtually disappeared?

 

 

hahahaha, pathetic digital media mortals... but of course i mean that in the nicest possible way...

 

with several hdds and tons of video data i beg to differ.. hundreds of gigs and i still run out of room.

 

give a man an inch and he'll grow to fill it and keep demanding more.. or something like that...

 

terabyte drives can't come soon enough. and the demand will become even louder once HD programming HD gaming and HD movie disks come along.. as much as they are trying to copy-proof the new disk technologies more than likley someone (dvd john) will figure out how to make 45 gigabyte 1:1 copies for distribution over the internet, which will raise demand for ISPs to keep pace, rolling out 30-50 megabit fiber connections.

 

 

pictures? they are about the only real HD technology that normal people can play around with.

 

even so they could have a better resolution in most cases since some digital cameras compress their files to save on precious flash memory. those micro hdd cameras will bust open the market of HD SLR photography (already do but they aren't generally affordable to most consumers).

 

IMO monitors and televisions lack resolution... 640x480 or 1080i its all pedestrian compared to whats to come.. UHDTV with 4000+ horizontal scan lines of resolution.. people become ill because its so real. uhdtv is due to become the standard in 2025

 

once you know about uhdtv, regular hdtv sounds like training wheels, a silly standard which at its core will only replace archaic incompatible analogue standards of the world to lay the ground work for real standards like shdtv and uhdtv (but even basic hdtv will have different flavours, you gotta love politics).

 

 

in your example, 3-5 megapixel images hopefully stored without compression or lossless compression.. the resolutions at the best setting usually are pretty high.. more than your desktop, enough to make a decent print.. i seem to remember my rinkydink 6mpx pine makes files in 3200x1900 or so. very large, and impressive when the settings on the camera and planetary alignement allow for a good shot or two to come out of a batch of a few dozen.

 

what if, you had a monitor that could display them 1:1? meaning that the raw images would have a use at full size as soon as you downloaded them to your PC?

 

further.. since static images are boring.. and digital video is much more fun.. you could record video with that kind of resolution? 3840x2400/2160 to be precise...

 

hdtv is "nice" but it'll be used to broadcast syndicated television, not very impressive to the videophile.. we need something better than jo-every-american will have access to.

 

S-HDTV, like s-video could be a spec for people with a more discerning palette. (and bottomless wallets with a wall in their homes they can devote to a proper s-hdtv in home better-than-sticky-floored-crowded-smelly conventional movie theater we currently suffer ourselves to just for the opportunity to see a feature film, for 10$ a pop.

 

the numbers 3840x2400 for 4:3 and 3840x2160 for widescreen. [the similar width ratio would mean the same set could operate in both 4:3 and 16:9 aspect ratios depending on the content]

 

i find that a television at 50+ inches should have a resolution at least as good as a monitor or better. and annoyed the hell out of me when i think that most people who currently own such gargantuan sets are content with 640/720x480 resolution content! gag,

 

 

could you imagine though? a bigscreen with such a resolution running at 120hz? you could easily read 10p fonts, play games with unheard of realism to surpace anything hollywood could produce.. maybe something like home IMAX..

 

TTFT would allow for a TV set in the conventional sense to take over for bigscreen LCDs plasmas and DLPs but they are a ways off.. and i haven't seen a 1080i projector at retail in ages..

 

would there be a market for higher end customers for S-HDTV between now and 2025?

 

and more importantly will there be TB optical disks to support higher end hdtv and shdtv within the next few years?

[Greg_G47]

Aside from stealing things, it will probably take changes in the kinds of files we store.

 

Back in the days of text-only computing a gigabyte was an impossibly large space for the average person to fill. Very few people have 250-400000 pages of text stored on their computer I'm sure. Enter images, video and 3D gaming. Half-Life 2 (a game you should play if you haven't!) for example is well over a gig.

 

If we add an extra dimension to imaging (i.e, record an actual 3D image of an area) we'll see image and video size go up exponentially. Including color and x/y/z coordinates of every corner and every shape in a room will take up a lot more space than a 2D picture recording a few million points of light on a chip. You'll fill terrabyte drives pretty fast with that.

[alxian]

thats an unfair assumption

 

i know many people with row up row of dvds and vhs casettes.

 

 

what i like to do is store all the ones i watch most often on my computer, that way everything can be accessed immediately.

 

be it television, through cable, movies, by loading the relevant player and selecting the movie from a playlist, or a game (that i own), by ripping the disk and creating an iso image i can mount and play the game simply by pressing a desktop icon.

 

these are conveniences anyone can benefit from.

 

your assumption that when i said upload to the internet was condoning piracy, you misunderstand, the way the system is supposed to work you won't even be able to play hd movies in your PC at all.

 

so that creating a server for your own home, a digital movie jukebx (filled with movies you own) to serve multiple televisions in your own home will be illegal.

 

even ripping a dvd to create that iso to create a virtual drive is illegal.

 

adding that people will leave that collection open to the internet is just being realistic.

 

If we add an extra dimension to imaging (i.e, record an actual 3D image of an area) we'll see image and video size go up exponentially. Including color and x/y/z coordinates of every corner and every shape in a room will take up a lot more space than a 2D picture recording a few million points of light on a chip. You'll fill terrabyte drives pretty fast with that.

 

consumer entertainment class holographics that would rival uhdtv resolutions.. may be a few decades away.

 

it likely that interactive TV will hit first. then holographics will have to be interactive as well to compete/replace 2D television.. nevermind the logistics of holophotography, how can you stream TBs of data to make holographic TV interactive?

[Greg_G47]

Sorry for suggesting you were pirating movies. I don't watch many videos off of my PC so I wouldn't have even thought of storing ones I own there.

 

As for logistics of how to take holographic photos and stream terrabytes of data. ask the same question 15 years ago about logistics of digital SLRs that rival conventional film and streaming those little 320x240 WMV videos.

 

On the subject of logistics of holo-photography, http://www.trnmag.com posted an article a month or so back about a program which can process diffuse reflections from surroundings to see "around" objects. i.e from a front-on picture of you holding a book up to your face it could discern some of the text on the page you were looking at. Something like this isn't an entire answer, but is certainly a necessary step.

[alxian]

i didn't mean for you to apologize.. just don't generalize, you won't always be right.

 

 

I don't watch many videos off of my PC so I wouldn't have even thought of storing ones I own there

 

neither do i, i export them to my tv using component cables, the experience thus is exactly like having a dvd player and high end audio setup without the extra hardware and total control.

[Turtle]

I thought about this when noticing that my digital pictures aren't really 5MB resolution but about 500KB in the worst cases, and 3MB in the best - though they all took 5MB of hard drive.

I borrow a friends digital camera sometimes; when I put one of those photos here in the Science Gallery, I noticed when you click to view it (members only) & scroll down, there is a list of all kinds of info on/from the camera. Apparently the disk space used is not all image data.

http://hypography.com/gallery/showimage.php?i=407&c=3&userid=796

[Guest loarevalo]

So I take it from the responses that the assumption that we have "unlimited" storage space is short-sighted, like the classic question "why would anyone need 1GB of hard drive?" or so it goes. I have to check, I think Bill Gates said that.

 

So, we should care like always about storage efficiency. Yet, it seems to me that the average user is so caught up in the excitement (or awe) of 200GB HD, that they would care little - like I :lol: Therefore, software capitalists care little about efficiency - that is my perception. At least, Microsoft seems to have never cared for efficiency. Is that judgement accurate? What would it take to make people like I care again about storing efficiency? being more restricted as to HD space?

 

Would storage efficiency FOREVER be an issue? Could storage capacity grow so fast, so that processing capacity won't reach it, and come short of making efficient use of all the storage space?

[Tormod]

So I take it from the responses that the assumption that we have "unlimited" storage space is short-sighted, like the classic question "why would anyone need 1GB of hard drive?" or so it goes. I have to check, I think Bill Gates said that.

 

Actually, I think he said "Nobody will ever need more than 640K RAM".

 

But sadly it seems not to be true:

http://www.usnews.com/usnews/biztech/gatesivu.htm

[Guest loarevalo]

Thanks. I think I heard it from an unreliable source.

[alxian]

indeed, 640k of ram was too little

 

2gigs for a windows machine running xp is about as much as anyone needs for now

 

however 64 bit processing and dual core processors and higher resolutions will require more system and video memory. windows vista @64bit will need 2 gigs and a 512 meg video card to operate as well as a winxp machine with 1 gig of ram and 128/256 video ram.

[Guest loarevalo]

What is that equation, predicting the growth of memory, speed, etc. in computers? What does the formula predict will be the HD size in 2020?

 

Taking in consideration the physical constraints, like the speed-of-light, mass/space ratio, etc., what is the max. storage of a hard-drive of physical size 1"X5"X5"? When will the industry reach that limit?

[alxian]

many really, the most famous being moores law which IMO is going to fail very soon. not because we can't make cpus faster and faster, after vista there will be little reason to keep up the break neck pace since people are willing to pay more for less performance, intel at least is calling it increasing the potential pre chip with more l2 ram higher bus speed, upping the horse power rather than the top speed.

 

moores law: the doubling every 18 months of cpu processing power you can get per dollar

 

another informal growth rate is hardrive aureal denstiy, capacity per platter square inch, roughly 133 gigs per platter. the aureal density increases very quickly like cpu hertz growth but without a moores law of aureal density the growth rate is more dynamic. i.e. perhaps having moores law reduced the rate of growth of cpu performance to only doubling every 18 months where its could be faster.

 

http://news.zdnet.co.uk/story/0,,t269-s2129861,00.html

 

1000 times increase planned. impressive? sure, but what is the seek time? what is the read write speed?

 

hdd versus hvd for main storage, hvd technically has better performance, but hdd has and may always have much bigger capacity.

 

but like cpu growth even if we can increase the aureal density of an hdd the performance is paramount. meaning those massive increases in capacity seldom have the performance necessary to make the old technology obsolete.

 

not long ago the maximum aureal density was 1 megabyte per square inch, then came big blue with ruthenium "pixie dust" to quadruple hdd aureal density, we're currently at 133 per platter, and we're looking at 230-250 gigs per square inch very soon. the goal for the HD era and onward may be 1 TB per square inch, with 5 platters 50 TB on a drive may not be more than a few deacdes away.

 

500 megabytes [yesterday], 50-500 TB [tomorrow] versus 500 gigs [today] (which you can buy for 300-400$). so very informally you can think about it like 1000x increase per decade.

 

similarly optical media or should i say removeable storage since who knows if removeable storage in 5 years will include organic methods, DNA and proteins for storing data. hvd being holographic king [so far] at multiple gigabytes per square inch.

 

optical media is at 4-9-18 gigs now, for the last few years, will jump to 50, and efforts to create 500 gig disks are bearing fruit.. where 500 gigs is going to herald the HD era.. not sure what 45-50 gig hd dvd and blu-ray have to do with HD since they will require compression and will be obsolete in ~5 years anyway.

 

lastly and least easy to quantify is the growth of consumer grade videocard performance. lets say polygons per second. 3dfx voodoo 2 breaking a million polys per second [yesterday], xbox 360 a half billion polygons per second [today], and god knows what in 5-7 years [xbox 3 and ps4 10s of billions of polygons per second?!], and what about holographics [next week...] something like 120 hertz at 8000x8000x8000 holopixels or polyhedrons (~60 trillion points) where the number of viewers (viewing an 8000x8000 holograms) will determine the real horse power required. such that each viewer requires his own zbuffer, the system thus draws the scene but only renders what needs to be displayed in full detail in realtime based on the position of the eyes of the viewers.

[alexander]

I thought about this when noticing that my digital pictures aren't really 5MB resolution but about 500KB in the worst cases, and 3MB in the best

heh, when i take pictures with my digital camera, they tend to take 100MB at least, lol.

 

as to why the files are larger then they really are, well there is all kinds of data stored in the picture file, quite a lot of it is just info about the file, but that doesnt bring a 3 meg file to show to you that it is really a 5 meg one! there are a few more things that go into the consideration, namely:

The way that the size of the file is computed, windows does weird things to calculate the size of anything, i think that somehow they use the 10 based file size calculation instead of 2, kinda weird, i remember a few occasions that my files appeared way larger then they really were, i think the worst was a 44k file that appeared not to even fit on a floppy, but then at some point it was recognised as a 44k file and did, quite weird, but windows is like that. I guess that their file system contribute to that factor as well, but not exactly sure.

 

Nobody will ever need more than 640K RAM

should we open a thread for all the stupid things billy's said, we'd take up all of our space on the server with all the text we'll put in the database... :lol:

 

windows vista @64bit will need 2 gigs and a 512 meg video card to operate as well as a winxp machine with 1 gig of ram and 128/256 video ram.

wow, <claps in amazement>, Linux doesnt even require half the ram and even an unsapported 32 meg graphics card does just fine. But not surprised, thats what you get for writing a completely OO OS all based on the .net "technology"... need to go and throw up just from thinking about it...

You know whats funny? Windows works on what like 2 platforms (x86 32, 64 bit), so thats 2 for the ol' billy boy vs what x86 32 and 64 (any cisc architecture procs), amiga, mips, spark 32 and 64, power pc 32 and 64, alpha, cell, risc, parisk, m68, m68k, arm, arm26, cris, m68k, m32r, ia64, minimips, motorolla 680x0, mach, l4, ap1000+, ultraspark, s/390 32 and 64, superh, vax, v850, esa/390 (IBM mainframe) and elks (I'm sure there is more, i just dont have time to research...) for Linus....

[Buffy]

I've been wondering.... Developers tend to get the most powerful systems, and they don't notice their lack of attention to performance until some products get deployed. Don't be surprised if Vista slips more because they're having problems getting it to run on "mere mortal" machines...the early beta's aren't performing well, but with the usual excuses of "it hasn't been tuned yet." I had one machine I upgraded to XPPro, and it was slow as a pig until I put 756Mb on it. Vista has got to be worse, but I'm waiting until Beta 2 to even try it... If they blow this, Linux has got a once in a life-time shot...

 

In this week's eWeek there are a bunch of big IT shop directors openly saying "XP is more than good enough, there's going to have to be something really special to get me to go through the cost and pain of converting to Vista." :lol:

 

Cheers,

Buffy

[alexander]

funny perspective, buffy, just about nothing in this world will convert me to vista :lol: