Large-Scale Datastreams Surveillance via Pattern-Oriented-Sampling

Monitoring large-scale datastreams with limited resources has become increasingly important for real-time detection of abnormal activities in many applications. Despite the availability of large datasets, the challenges associated with designing an efficient change-detection when clustering or spatial pattern exists are not yet well addressed. In this article, a design-adaptive testing procedure is developed when only a limited number of streaming observations can be accessed at each time. We derive an optimal sampling strategy, the pattern-oriented-sampling, with which the proposed test possesses asymptotically and locally best power under alternatives. Then, a sequential change-detection procedure is proposed by integrating this test with generalized likelihood ratio approach. Benefiting from dynamically estimating the optimal sampling design, the proposed procedure is able to improve the sensitivity in detecting clustered changes compared with existing procedures. Its advantages are demonstrated in numerical simulations and a real data example. Ignoring the neighboring information of spatially structured data will tend to diminish the detection effectiveness of traditional detection procedures. Supplementary materials for this article are available online.



Work Title Large-Scale Datastreams Surveillance via Pattern-Oriented-Sampling
Open Access
  1. Haojie Ren
  2. Changliang Zou
  3. Nan Chen
  4. Runze Li
  1. Design-adaptive test
  2. Generalized likelihood ratio method
  3. Kernel smoothing
  4. Optimal sampling
  5. Sequential change detection
  6. Statistical monitoring
License In Copyright (Rights Reserved)
Work Type Article
  1. Journal of the American Statistical Association
Publication Date October 23, 2020
Publisher Identifier (DOI)
Deposited July 19, 2022




This resource is currently not in any collection.

Work History

Version 1

  • Created
  • Added POSJASAPublishedVersion.pdf
  • Added Creator Haojie Ren
  • Added Creator Changliang Zou
  • Added Creator Nan Chen
  • Added Creator Runze Li
  • Published
  • Updated Keyword, Publication Date Show Changes
    • Design-adaptive test, Generalized likelihood ratio method, Kernel smoothing, Optimal sampling, Sequential change detection, Statistical monitoring
    Publication Date
    • 2022-01-01
    • 2020-10-23
  • Updated