Things to do when growing your project from 1TB to 50TB, or 50TB to 500TB, or above.

* Establish governance across the IT staff supporting and maintaining the VLDW

* Get training on performance and tuning for the database you are using

* ask the vendor to show you live, working examples of their database/ETL, <insert your favorite vendor here>  of customers working with data sets in the size range you are considering

* Read VLDW reports from the web (some of which the vendor will happily provide to you), some cost money, and others are free

* Ensure your throughput rates for ETL/ELT + loaders are 80,000 rows per second on average with 1k row sizes (without partitioning/parallelism), anything slower is seriously detrimental to your ability to grow the environment

* Check the Data Model.  If you are coming from a column based database (because perhaps you outgrew it), then you must map, create, and manage the data model for relational database that you are moving to.   Even if you are using a column based appliance, you should have a solid data model foundation for logical data representation.  Governance...  the bigger the system gets, the harder it becomes to manage without strict rules and standards and good data models.

* Be flexible.  Learn to align IT, and to turn IT into a lean machine, that can execute rapidly and adapt to business needs.

* don't be afraid to scale out, or in some cases - scale up (mostly with Big-Iron, and no, Big-Iron is not dead, far from it).

* Learn the terminology MPP, SMP, NUMA, Clustered, Grid, Cloud

* Establish a mitigation plan, and a risk analysis plan for "what if this node fails?"

Technically what you can do:

* Test the limits of your machines, networks, disk, cpu, RAM, and so on- understand their maximum throughput, average throughput for multiple parallel processes.

* Test the database, how does it perform with queries running at the same time as a large load?  The bigger the system gets, the harder it will be to "manage this".  Testing with 100,000 rows of data won't cut it when one feed might deliver 1.5 Billion every night.

* Increase I/O disk speed, 300 to 400 mbits throughput per second from the server to the disk and back again.  Watch the BUFFERING that occurs on the disk, ensure that the test clears the buffer on the disk before you run it again, otherwise your results will be invalid

* Increase the I/O channels. The number of I/O channels can have a huge impact on performance of very large systems

Again, theses are some of the things you can do today.  Love to hear about your environment, along with your results.

Cheers,

Dan L  danL@GeneseeAcademy.com