Big Data and Dynamic Data Driven AppsCourse DescriptionDynamic Data-Driven Application Systems (DDDAS) is a paradigm whereby applications and measurements become a symbiotic feedback control system with the ability to dynamically incorporate additional Big Data into executing applications and dynamically steer the measurement process, which provides more accurate analysis and prediction, more precise controls, and more reliable outcomes.PrerequisitesAn eclectic group of students with varied backgrounds so that computational science projects can be completed small teams. Some programming experience is helpful.Office227 Ross HallSuggested reading
- Anand Rajaraman, Jure Leskovec, and Jeffrey D. Ullman, Mining of Massive Datasets, Stanford, 2013 (self published on Ullman’s web site). See Amazon.com for the hardcopy edition published by Cambridge University Press in 2011.
- Derrick C. Cerwinsky, Craig C. Douglas, Mookwon Seo, and Xiaoban Wu, An Introduction to C and Parallel Programming with Applications, 2014.
- Peter Pacheco, An Introduction to Parallel Programming, Morgan Kaufmann; 1st edition (January 21, 2011), ISBN-10: 9780123742605, ISBN-13: 978-0123742605, Kindle edition.Longer Version of Course DescriptionDynamic Application Systems (DDDAS) is a paradigm whereby applications and measurements become a symbiotic feedback control system with the ability to dynamically incorporate additional data into an executing application and to dynamically steer the measurement process, which provides more accurate analysis and prediction, more precise controls, and more reliable outcomes.
Big Data is a paradigm for methods to handle nearly infinite amounts of data that is either streamed (the DDDAS preferred method) or historically stored (and potentially ever growing) datasets for data mining. Almost all interesting DDDAS cases overlap with Big Data. Solving one solves for the other one, so it makes sense to study both simultaneously.
The ability of an application to control and guide the measurement process and determine when, where, and how it is best to gather additional data has itself the potential of enabling more effective measurement methodologies. Furthermore, the incorporation of dynamic inputs into an executing application invokes new system modalities and helps create application software systems that can more accurately describe real world, complex systems. This enables the development of applications that intelligently adapt to evolving conditions and that infer new knowledge in ways that are not predetermined by the initialization parameters and initial static data.
The need for such dynamic applications is already emerging in business, engineering and scientific processes, analysis, and design. Manufacturing process controls, resource management, weather and climate prediction, traffic management, systems engineering, civil engineering, geological exploration, social and behavioral modeling, cognitive measurement, and bio-sensing are examples of areas likely to benefit from DDDAS.
In this course, we will study successful DDDAS applications that are extensively documented through the DDDAS community web site, http://www.dddas.or DDDAS is already in use in the Kingdom of Saudi Arabia: the entire oil/gas pipeline system is run using a DDDAS that has run continuously since 1978 (that the instructor helped create) and has been running continuously since then even with hardware and software upgrades and going from 2,000 pieces of telemetry per minute in 1978 to tens of millions per second in 2014. We will also study real-time data mining techniques where if the data is not processed almost instantly it is lost.
The class will work in small groups to produce working DDDAS, one per group. The final project will hopefully produce a conference and/or archival journal submission (that is successfully published after the class is over). Groups must know how to program in C, C+, FORTRAN, or Java (note groups, not necessarily individuals). Being able to translate data using some tool such as Python, sed, awk, or Matlab is also useful. This is a hands on project oriented class to produce a useful DDDAS with Big Data to more than just the class.