37 Processes and Analyzing Data
Smoothing Data to Remove Noise
Data from mechanical systems usually changes fairly slowly compared to electrical systems and we can often define a characteristic time or frequency and ignore changes in the data that happen any faster. Some sort of time averaging over multiple samples will smooth out the sample to sample variations, reducing the noise at frequencies near the sampling frequency and providing a better estimate of actual values. Often we must make immediate decisions based on the data we have available to date. The recent COVID-19 pandemic provides a stark example of the need to interpret limited data in as timely a fashion as possible. We have all been watching this data closely and need the skills to assess it critically. Different control choices made in the spring have so far led to significantly different outcomes.
Notice how important it is to use the right approach to smoothing when interpreting the data. Averaging over larger sample sizes also reduces the noise in the data.
An Exponential Smoothing example is provided in the RWS-Notes/Code-Arduino/Learning Sequence SAMD/ S1.3_New_Tab_and_Smoothing sketch (Arduino S1.3)
Post-Processing of Time Series Data
If you wait until after a full time series data set is collected, you can base your estimates for the actual value at a particular time on data from both before and after that time. Knowing something about the future allows better estimates of the present. Processing offline reduces the pressure to complete calculations quickly, allowing for more sophisticated techniques. However, it doesn’t allow you to take any action in the present.
Any technique that can be applied in real time can also be applied in post-processing. This can be really useful for testing different estimating approaches to processing the same data.
Real Time Processing of Time Series Data
Improving your estimates from time series data during acquisition is essential for acting immediately on those estimates to control the system being measured, however it comes with two key limitations:
- The current estimate for a measured quantity can only depend on the current measurement and other measurements made in the past.
- The current estimate must be calculated quickly enough to be useful in real time, usually before the next measurement is made.
Exponential Smoothing (Python 4.2) is easier to implement on a microcontroller (Arduino S1.3) than Moving Averages (Python 4.1). Smoothing provide some influence of the long time history of your data to make an estimate in the present, without needing to store all the past values in an array.
Taking Derivatives of Measured Data
Numerical differentiation is simple in concept, but those small differences in will amplify any noise that was there in either the time base, or in the measured values. This video (7:26) provides the basics of differentiating position to get velocity and acceleration estimates.
These two videos will help understan