What do you consider "large" amounts of textual data? (Edit - I see you answered that question. Yes, that is large. :) )
If you are going to be crunching through very large files, I would consider upgrading the hard drive. Unfortunately, the fastest hard drives currently available are still available only with a SCSI interface, and they are expensive. (They would probably still be expensive, even if they weren't SCSI. :) ) They would be the Seagate Cheetah 15K.3 series.
Another possibility would be a RAID-0 configuration using the fastest IDE or (perferably) SATA drives you can find. (If you REALLY need speed, go RAID 0 with the Cheetahs...) RAID 0 increases disk read/write speed by spreading your data across multiple drives and reading/writing multiple drives at the same time.
However, you have to be really careful that you have an effective backup scheme in place with RAID 0. Every drive you add to a RAID 0 array doubles the probability of a failure affecting your data. (Because the data is spread across multiple drives.) In most cases, it makes sense to put the OS and software on a non-RAID drive, and your data on the RAID array. Then make sure you back-up your data. It can be difficult to properly back-up the OS on a RAID array of any kind. You have a lot more flexibility backing up data.
(The problem is that you can't back-up open files, and you ALWAYS have open files associated with the OS and other running software. Drive cloning is one of the few effective ways to back-up the system. It works because you do it from a DOS program, and so the files aren't open. But RAID-0 arrays can be difficult to clone.)
Anyway, I think the hard drive is the most common bottleneck in today's systems. I certainly felt the difference moving from my (now dead) server to my notebook. (I am now building a new, dual-Xeon workstation.) I use my machine primarily for software development. Some of the things that are a real pain on my notebook with it's slower hard drive are a bit of a surprise - you wouldn't think they would be very demanding of disk speed. For example, getting help (say, looking-up documentation on an API call) in Microsoft Visual Studio takes FOREVER on my notebook! This really slows me down when I have to wait several seconds for a page to come up, and I am zipping around (or trying to) to different topics. |
