Mining Data Streams

When a space shuttle takes off, tiny sensors measure thousands of data points every fraction of a second, pertaining to a variety of attributes like temperature, acceleration, pressure and velocity. A data gathering server at a networking company receives terabytes of data a day from various network elements like routers, reporting on traffic throughput, CPU usage, machine loads and performance. Each of these is an example of a data stream. Many applications of data streams arise naturally in industry (networking, e-commerce) and scientific fields (meteorology, rocketry). Data streams pose three unique challenges that make them interesting from a data mining perspective. 1. Size: The number of measurements as well as the number of attributes (variables) is very large. For instance, an IP network has thousands of elements each of which collects data every few seconds on multiple attributes like traffic, load, resource availability, topography, configuration and connections. 2. Rate of accumulation: The data arrives very rapidly, like “water from a fire hydrant”. Data storage and analysis techniques need to keep up with the data to avoid insurmountable backlogs. 3. Data transience: We get to see the raw data points at most once since the volumes of the raw data are too high to store or access.