An Introduction to Convolutional Neural Networks: Overview, Implementation, and Example Shuo Yu and Hsinchun Chen, AI Lab University of Arizona April 2019 01/26/2020 1 Acknowledgments Many of the images, results, and other materials are from: Deep Learning (2016), Ian Goodfellow, Yoshua Bengio, and Aaron Courville Lee Giles and Alex Ororbia, Penn State University Yann LeCun, New York University

01/26/2020 2 Outline Introduction Neuroscientific Basis Building Blocks Convolutional Layer Non-linear Layer Pooling Layer Implementation Build a CNN with Keras in Python Research Example: 2D-hetero CNN for Mobile Health Analytics Introduction Research Design Evaluation and Results

01/26/2020 3 Introduction Convolutional Neural Networks 01/26/2020 4 Convolutional Neural Network (CNN) Convolutional Neural Networks, or Convolutional Networks, or CNNs For processing data with a grid-like topology 1-D grid: time-series data, sensor signal data 2-D grid: image data CNNs are neural networks with convolution operations. The most well used deep learning networks

Image recognition, AlphaGo 01/26/2020 5 Convolutional Neural Network (CNN) Convolutional Neural Networks are inspired by mammalian visual cortex. The visual cortex contains a complex arrangement of cells, which are sensitive to small sub-regions of the visual field, called a receptive field. These cells act as local filters over the input space and are well-suited to exploit the strong spatially local correlation present in natural images. Two basic cell types: Simple cells respond maximally to specific edge-like patterns within their receptive field. Complex cells have larger receptive fields and are locally invariant to the exact position of the pattern. 6

The Mammalian Visual Cortex Inspires CNN Convolutional Neural Net output input 7 A Deep Classification Network 01/26/2020 8 CNN Architecture Intuition: Neural network with specialized connectivity structure, Stacking multiple layers of feature extractors

Low-level layers extract local features. High-level layers extract learn global patterns. A CNN is a list of layers that transform the input data into an output class/prediction. There are a few distinct types of layers: Convolutional layer Non-linear layer Pooling layer 9 Building-blocks for CNNs Feature maps of a larger region are combined. Feature maps are trained with neurons. Shared weights

Each sub-region yields a feature map, representing its feature. Images are segmented into sub-regions. 10 CNN Architecture: Convolutional Layer The core layer of CNNs The convolutional layer consists of a set of filters. Each filter covers a spatially small portion of the input data. Each filter is convolved across the dimensions of the input data, producing a multidimensional feature map. As we convolve the filter, we are computing the dot product between the parameters of the filter and the input.

Intuition: the network will learn filters that activate when they see some specific type of feature at some spatial position in the input. The key architectural characteristics of the convolutional layer is local connectivity and shared weights. 11 CNN Convolutional Layer: Local Connectivity Neurons in layer m are only connected to 3 adjacent neurons in the m-1 layer. Neurons in layer m+1 have a similar connectivity with the layer below. Each neuron is unresponsive to variations outside of its receptive field with respect to the input. Receptive field: small neuron collections which process portions of the input data The architecture thus ensures that the learnt feature

extractors produce the strongest response to a spatially local input pattern. 12 CNN Convolutional Layer: Shared Weights We show 3 hidden neurons belonging to the same feature map (the layer right above the input layer). Weights of the same color are sharedconstrained to be identical. Replicating neurons in this way allows for features to be detected regardless of their position in the input. Additionally, weight sharing increases learning efficiency by greatly reducing the number of free parameters being learnt. 13 CNN Architecture: Non-linear Layer

Intuition: Increase the nonlinearity of the entire architecture without affecting the receptive fields of the convolution layer A layer of neurons that applies the non-linear activation function, such as, - Rectified Linear Unit (ReLU) 14 CNN Architecture: Pooling Layer Intuition: to progressively reduce the spatial size of the representation to reduce the amount of parameters and computation in the network, and hence to also control overfitting Pooling partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum value of the features in that region. Input 15

Building-blocks for CNNs Feature maps of a larger region are combined. Feature maps are trained with neurons. Shared weights Each sub-region yields a feature map, representing its feature. Images are segmented into sub-regions. 16 Full CNN pooling

pooling 17 Other Layers The convolution, non-linear, and pooling layers are typically used as a set. Multiple sets of the above three layers can appear in a CNN design. Input -> Conv. -> Non-linear -> Pooling -> Conv. -> Non-linear -> Pooling -> After a few sets, the output is typically sent to one or two fully connected layers. A fully connected layer is a ordinary neural network layer as in other neural networks. Typical activation function is the sigmoid function. 01/26/2020 18

Other Layers The final layer of a CNN is determined by the research task. Classification: Softmax Layer The outputs are the probabilities of belonging to each class. Regression: Linear Layer The output is a real number. 01/26/2020 19 Implementation Python, TensorFlow, Keras 01/26/2020 20

Python CNN Implementation Prerequisites: Python 3.5+ (https://www.python.org/) TensorFlow (https://www.tensorflow.org/) Keras (https://keras.io/) Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. Recommended: NumPy Scikit-Learn NLTK SciPy

01/26/2020 21 Build a CNN in Keras The Sequential model is used to build a linear stack of layers. Building a CNN with the Sequential model is straightforward. The following code shows how a typical CNN is built in Keras. import numpy as np import keras from keras.models import Sequential from keras.layers import Dense, Flatten from keras.layers import Conv2D, MaxPooling2D from keras.optimizers import SGD 01/26/2020 Note: Dense is the fully connected layer;

Flatten is used after all CNN layers and before a fully connected layer; Conv2D is the 2D convolution layer; MaxPooling2D is the 2D max pooling layer; SGD is stochastic gradient descent algorithm. 22 Build a CNN in Keras (continued) model = Sequential() # We create an empty Sequential model and add layers onto it. model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(100, 100))) # We add a Conv2D layer with 32 filters, 3x3 each, followed by a detector layer ReLU. # This is the first layer we add to the model, so we need to specify the shape of the input. In this case we assume our input is a 100x100 matrix. model.add(MaxPooling2D(pool_size=(2, 2))) # We add a MaxPooling2D layer with a 2x2 pooling size.

01/26/2020 23 Build a CNN in Keras (continued) model.add(Conv2D(32, (3, 3), activation='relu')) model.add(MaxPooling2D(pool_size=(2, 2))) # We can add more Conv2D and MaxPooling2D layers onto the model. model.add(Flatten()) # After all the desired CNN layers are added, add a Flatten layer. model.add(Dense(256, activation='sigmoid')) # Add a fully connected layer followed by a detector layer with the sigmoid function. model.add(Dense(10, activation='softmax') # A softmax layer is added to achieve multiclass classification. In this example we have 10 classes. 01/26/2020 24

Build a CNN in Keras (continued) sgd = SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True) # Default SGD training parameters model.compile(loss='categorical_crossentropy', optimizer=sgd) # Compile the model and use categorical crossentropy as the loss function, sgd as the optimizer model.fit(x_train, y_train, batch_size=32, epochs=10) # Fit the model with x_train and y_train, batch_size and epochs can be set to other values score = model.evaluate(x_test, y_test, batch_size=32) # Evaluate model performance using x_test and y_test 01/26/2020 25 Research Example Two-Dimensional Heterogeneous Convolutional Neural Network (2D-hetero CNN) for Mobile Health Analytics

01/26/2020 26 Introduction We developed two-dimensional heterogeneous convolutional neural networks (2D-hetero CNN), a motion sensor-based system for fall risk assessment using convolutional neural networks (CNN). Five sensor system (chest, left/right thigh, left/right foot) for clinical tests Comprehensive assessment for gait and balance features CNNs are powerful in extracting low-level local features as well as integrating them into high-level global features. Feature-less; avoid feature engineering that is labor intensive, ad hoc, and inconclusive. Main novelty of this work: We proposed a novel CNN architecture to extract gait and balance features for fall risk assessment. Two-dimensional convolution: temporal convolution + cross-axial and cross-locational convolution

To the best of our knowledge, we are the first to apply CNNs for motion sensor-based fall risk assessment. 01/26/2020 27 Research Design Data Collection Data Preprocessing Model Design Sensor Attachment Signal Segmentation 2D-hetero CNN

Walking Test Data Augmentation Evaluation Fig. 2. Research Design 01/26/2020 28 Research Design Data Collection Twenty-two (22) subjects were recruited at a neurology clinic. 12 with high fall risks, 10 with low fall risks 5 tri-axial accelerometers attached to each subject Fig. 3. Shape and Size of SilverLink Sensors

Sampling rate: 25 Hz 25 sampling points per second; sufficient for capturing gait cycles Chest, left/right thigh, left/right foot (as shown in Fig. 4) To capture body and lower extremity movement (left/right) 10-meter ground walking tests were conducted to collect data for gait and balance. Subjects are instructed to walk in their comfortable paces for 10 meters in the clinic hallway. 01/26/2020 Fig. 4. Sensor Locations 29 Research Design 2D-hetero CNN

Stage 2: Cross-Locational Convolution Stage 1: Cross-Axial Convolution 3 x 5 conv. stride (3, 1) Left/right thigh 6 x 100 1 x 4 pool. 10 @ 2 x 96 1 x 4 pool. 2 x 5 conv. 10 @ 2 x 24

Stage 3: Integration 20 @ 1 x 5 20 @ 1 x 20 Fully connected Flatten 3 x 5 conv. 1 x 4 pool. Softmax classifier 1 x 4 pool.

1 x 5 conv. 2 Chest 3 x 100 10 @ 1 x 96 3 x 5 conv. stride (3, 1) 1 x 4 pool. 20 @ 1 x 5 20 @ 1 x 20 10 @ 1 x 24

1 x 4 pool. 2 x 5 conv. 10 @ 2 x 96 10 @ 2 x 24 Left/right foot 6 x 100 Note: The notation x @ y x z denotes x feature maps with height y and width z. 20 @ 1 x 20 20 @ 3 x 5 300 20 @ 1 x 5

Fig. 8. 2D-hetero CNN Architecture 01/26/2020 30 Research Design 2D-hetero CNN We partitioned the data into three parts based on sensor locations. Chest, left/right thigh, left/right foot Aim to capture balance features between left/right thighs and feet Stage 1: Cross-Axial Convolution Convolve among the three axes of a single sensor Extract features among axes within a sensor Stage 2: Cross-Locational Convolution Convolve between sensors on left/right thighs and left/right feet Extract balance features between the left and the right Stage 3: Integration

Integrate extracted features to provide final inference on fall risk assessment Main novelty compared to traditional 2D CNNs: Convolutions along the non-temporal dimension with explicit semantics to handle dimension heterogeneity Cross-axial and cross-locational convolutions 01/26/2020 31 Research Design 2D-hetero CNN Technical details: A rectified linear unit (ReLU) layer is added after each convolutional layer for model nonlinearity. Most widely used non-linear function for CNNs The maximum is used as the pooling layer. Common settings for CNNs A dropping layer is added after each pooling layer and the densely connected layer to avoid over-fitting.

Dataset split: Training (60%), validation (20%), test (20%) The validation set is used for model selection. The test set is used for reporting performance. As the model training process can get into local maxima, we train the model for five times and report the average performance. 01/26/2020 32 Evaluation Benchmark 1: Feature-based fall risk assessment Most widely used approach for fall risk assessment Stride variability (SVAR), acceleration root mean square (ARMS), walking speed (SPD) Benchmark 2: CNN models with alternative architectures 2D homogeneous CNN (2D-homo CNN) as applied in image recognition tasks 1D CNN (1D-CNN) as applied in activity recognition and ECG classification tasks

Benchmark 3: Ablation analysis 2D heterogeneous CNN with cross-axial convolutions only (2D-axis CNN) 2D heterogeneous CNN with cross-locational convolutions only (2D-loc CNN) 01/26/2020 33 Results 1.00 1.00 1.00 0.80 0.80

0.80 0.60 0.60 0.60 0.40 0.40 0.40 0.20 0.20

0.20 0.00 Precision Recall Specificity F-measure 2D-hetero CNN ARMS SVAR SPD 0.00 Precision

Recall 2D-hetero CNN 1D CNN Specificity F-measure 2D-homo CNN 0.00 Precision Recall 2D-hetero CNN 2D-loc CNN Specificity F-measure

2D-axial CNN Our proposed 2D-hetero CNN significantly outperformed all three sets of benchmark systems. Showing the advantage of 2D-hetero CNN over traditional feature-based and 1D/2D CNN methods. 01/26/2020 34 Conclusions In this work, we developed 2D-hetero CNN to provide fall risk assessment based on motion sensor data. A novel CNN architecture with cross-axial and cross-locational convolutions was proposed to optimize in our application context of fall risk assessment. Considered as a general approach for gait/balance assessment 10-meter ground walking test data from patients with Parkinson's disease were

collected at a clinic to evaluate our model. Our model achieved F-measure of 0.962, significantly outperforming the benchmarks. 01/26/2020 35