Signals and systems using python

From theory to practice: here’s how to perform frequency analysis, noise filtering and amplitude spectrum extraction using Python

If you want to work with data one thing is for sure: specialize or die.
This idea of a data scientist which can work with textual data, signals, images, tabular data and legos is an old fashioned way of seeing this profession.

Nowadays, with the advent of Machine Learning and the huge improvements in terms of data analysis as well as the ones of the hardware of the computers, it is crucial to have a deep understanding of how to work with a specific class of data. This is because there is simply too much to know about a single branch of data science before jumping into another one.

In my case, I work everyday with signals. I understand them, as I have a master’s degree in Physics, and I can get useful information out of them, as I am a data scientist.

In this blog post, I will show you the basic operations of signal processing, namely the frequency analysis, the noise filtering and the amplitude spectrum extraction techniques.

We’ll start with a brief theoretical background about these techniques and we will show how they work when applied to a very simple (and public) dataset, which we will use as our “signal”.

1. Theoretical Background

1.1 Fourier transform

In our mind a (1D) signal is nothing but a time series. This means that we have an x axis, which is the time, and a y axis, which is the quantity we are considering (e.g. voltage).

From an intuitive point of view, doing a Fourier transform of a signal means to see this signal in another domain.

Let’s make it even more simple. Let’s say you are Superman. All your friends know you as Clark Kent. Then there is some crime, you do your transformation, and you become Superman! You are always you (even though they don’t know it), but they are seeing you in another domain, with another outfit, doing another kind of stuff.

Now we can do the same thing with the signals and their sinusoidal components.

The idea is to look the signal from the time domain to the frequency domain.

In the left side of this image we see that there is a single sine with frequency=1. In fact, on the right side, you only see a peak when frequency=1 and 0 everywhere else.

Imagine you have multiple sines and cosines interacting: you will see multiple amplitudes and multiple non zero frequencies as multiple modes are present in your signal.

Why is this useful? Well, multiple reasons, but let’s pick two:

1. You may filter your signal. You know that, generally, your signal will have a limited frequency band. If you see something outside of that band you can assume it is white noise (noise that lives at all frequencies) and you can filter it out. (Noise Filtering)
2. You may analyze the periodical pattern of your signal and find very useful insight about the seasonal part of your system. (Frequency Analysis)

1.2 Wavelet Transform

Let’s go back to the Fourier Transform for a second. Doing the transformation means to project all the components of your Fourier spectrum (all the sines with all the frequency) to your signal. In our example by doing this projection you get 0 everywhere except for frequency 1, when you get a value which is different from 0.

Now, let’s use something which is more complicated than a sine function, like the dyb2 shown below.

And we can play with this little guy, contracting it and expanding it and analyze the projections on the full signal.

The projection will give us a “filtered” signal, and the residual will be the difference between this projection and the original one.

It is a more various and general way of filtering your signal from the noise, and it has its great advantage as we will see later in the hands on part.

1.3 Hilbert Transform

Sometimes you see your signal full of up and downs, but you don’t really want it. You just want to get the envelope. Mathematically, this operation is done using a convolution. In particular, the kernel is 1/(pi*t). There is nothing much to talk about in here at this stage, but we will see the power of this algorithm in the hands-on part in a minute!

2. Hands on examples

The dataset we are going to use is a public one and can be found here.
It is basically a time-series of energy consumption, and it is perfect for our study as it can, by all means, considered as a signal. The dataset is a Public Domain one (CC0: Public Domain) and can be downloaded and used for free with no previous authorization.

Let’s start with the coding.

2.1 Importing the libraries

This is what we’ll need:

2.2 Data Pre Processing

Let’s plot our dataset

We can see that it has a kind of trend and it has not zero mean. These two features can disturb our analysis. Let’s correct it using the detrend option of scipy.

2.3 Frequency Analysis

Now it’s time to use the Fourier transform to do the frequency analysis of our signal. Let’s plot the Fourier Spectrum:

We can see very interesting peaks here. For example there is a periodicity around 24 hours (1 day) or 12 hours (0.5 day) or around 80 hours (three days).

In particular, we can sort the peaks and print the correspondent indexes:

Now the indexes are the ones in bold.

And we can print these index in “number of days” unity as well:

2.4 Noise Filtering (Fourier Transform)

Now we can take the highest peak as a reference and start to filter the ones that are lower than 0.1, 0.2, …0.9 than this highest peak. For example, we can set everything that is below 0.5 times the highest peak amplitude to zero.

We need to be careful. If we set this threshold too high we don’t filter the noise only, but we filter important features of the signal as well. If the threshold is too low, we basically don’t filter anything and we still keep all the noise.

Let’s make some tries:

These are some results:

0.30 seems to be, in this case, a good surrogate. :)

P.S. For more information about this specific application, please relate to this article.

2.5 Noise Filtering (Wavelet Transform)

We can do noise filtering using the Wavelet Transform as well. In this case we have multiple levels of filtering. The first levels will filter the pure noise only, but they will be conservative. The deeper you go, the less noise you will have, but you will loose features of the signal as well (the residuals will be similar to the original signal).

P.S. For more information about this specific application, please relate to this article.

2.6 Amplitude Extraction

For this part, I just made up an up-and-downy signal. Imagine we want to extract only its envelope (i.e. the so called amplitude). You just need to apply the scipy Hilbert Transform and compute the absolute value:

Super easy, yet super useful in the everyday life of a data scientist that works with signals.

3. Conclusions

If you liked the article and you want to know more about Machine Learning, or you just want to ask me something you can:

A. Follow me on Linkedin, where I publish all my stories
B. Subscribe to my newsletter. It will keep you updated about new stories and give you the chance to text me to receive all the corrections or doubts you may have.
C. Become a referred member, so you won’t have any “maximum number of stories for the month” and you can read whatever I (and thousands of other Machine Learning and Data Science top writer) write about the newest technology available.

Can Python be used for signal processing?

How do you use signals in Python?

What is SIGUSR1 in Python?

What is Scipy signal?