What is the difference b/n Depth wise and standard convolution?

In this blog, I want to show you about the difference b/n depth wise and standard convolution. You will learn how much difference is present on the number of total scalar multiplication (output parameters). The model computes the total parameter to learn from the input data.

Standard convolution

I want to start this blog by giving examples. Let us take an input image with size of 12x12x3 and apply 5×5 convolution on the image with no padding and a stride of 1. If we only consider the width and height of the image, the convolution process is 12×12 — (5×5) — >8×8. The 5×5 kernel perform scalar multiplication with every 25 pixels. It gives out 1 number every time. We can get 8×8 pixel image. Because there is no padding (12–5+1 = 8). However, the Colored image has 3 channels, the convolutional kernel needs to have 3 channels. So, instead of doing 5×5=25, we compute 5x5x3=75 multiplications every time when the filter moves and compute with the image pixel. After going through a 5x5x3 kernel, the 12x12x3 image will change to 8x8x1 image.

If we want an output feature with of size 8x8x256, we should use 256 filters to get 256 8x8x1 image features. The feature stack them up together to get a 8x8x256 image features.

So fare, we see how standard convolution is working. 12x12x3 — (5x5x3x256) — > 12x12x256. Where 5x5x3x256 represents the height, width, number of input channels and number of output channels of the filter.

Depth wise separable convolution

The standard convolution process transforms a 12x12x3 image to a 8x8x256 image features. But the depth wise convolution has transformed the 12x12x3 image to 8x8x3 image features. Now, we need to increase the number of channels of each image. The point wise convolution uses a 1×1 kernel, or a kernel that iterates through every single point. This kernel has a depth of many channels. The colored image has three color channels. Accordingly, the point wise convolution has 3 channels. Therefore, we iterate a 1x1x3 kernel through our 8x8x3 image to create 8x8x1 image.

Each 5x5x1 kernel iterates 1 channel of the image, getting the scalar products of every 25-pixel group, giving out 8x8x1 image. Stacking these images together creates 8x8x3 image.

We can create 256 1x1x3 kernels that create 8x8x1 image each to get a final image of shape 8x8x256.

Comparison of Depth wise and standard convolution

Based on the above explanation let us make comparisons between standard convolution and depth wise separable convolution.

• First, we calculate the number of multiplications the computer has to do in the standard convolution. There are 256 5x5x3 kernels that move 8×8 times. That’s 256x3x5x5x8x8=1,228,800 parameters.
• In the depth wise convolution, we have 3 5x5x1 kernels that move 8×8 times. That is 3x5x5x8x8 = 4,800 parameters. In the point wise convolution, we have 256 1x1x3 kernels that move 8×8 times. That is 256x1x1x3x8x8=49,152 parameters. Adding them together, we can get 53,952 total parameters.
• 52,952 is a smaller than 1,228,800. 53,952 parameters. It is not comparable with 1,228,800 parameters
• i.e. Depth wise separable convolution is 1/20 of standard convolution. Hence, this huge difference makes the model lean fast and perform accurate classification.

Deep Learning Model based on depth wise separable convolution

The Mobile Net model is operates based on depth wise separable convolutions. It separates a standard convolution into a depth wise convolution and a point wise convolution. The depth wise convolution applies a single filter to each input channel. The point wise convolution then applies a 1×1 convolution to combine the outputs of the depth wise convolution.

Many pre trained models are used a standard convolution that produces huge parameters. Because it does both filters and inputs combinations into a new set of outputs in a single step. Huge training parameter consumes computational resources and needs long time to complete the training process. In Mobile Net the depth wise separable convolution splits this into two layers. A separate layer for filtering and a separate layer for combining.