/
Neural Network Fundamentals
Save to my account
Sign up
Neural Network Fundamentals
Neural Network Fundamentals
Study
1
Question
What is the aim of the lecture on neural networks basics?
Answer
To give a very brief overview of neural networks, how they operate, and their key ingredients.
2
Question
When did the idea of mimicking the human brain computationally originate?
Answer
During World War II in the 1940s, specifically 1943-1944.
3
Question
Who is considered the proposer of the perceptron, the first realistic neural network?
Answer
Frank Rosenbelt in 1957-1958.
4
Question
What limitation did Minsky and Papert identify in the perceptron?
Answer
It cannot model the XOR function.
5
Question
What training method was introduced around 1986?
Answer
Backpropagation, the building block of neural network training.
6
Question
What theorem was proposed in 1989-1990 for multilayer perceptrons?
Answer
Universal Approximation Theorem (UAT): A single hidden layer MLP can approximate any continuous function to desired precision.
7
Question
What are the main parts of a biological neuron?
Answer
Dendrites (receive signals), soma (processes information), axon (transmits output), synapse (connection points).
8
Question
How does a neuron process inputs biologically?
Answer
Collects signals via dendrites, processes in soma, outputs via axon to next neurons.
9
Question
What are the inputs and output in an MPNet?
Answer
Binary inputs (0 or 1), binary output y: sum of inputs >= theta produces 1, else 0.
10
Question
What threshold theta mimics the AND operation in MPNet with two inputs?
Answer
Theta = 2, since only 1+1 >=2 produces 1.
11
Question
What threshold theta mimics the OR operation in MPNet with two inputs?
Answer
Theta = 1, since any 1 makes sum >=1.
12
Question
Why can't MPNet model XOR?
Answer
XOR data points are not linearly separable; no single line separates 1s from 0s.
13
Question
What key difference does perceptron introduce over MPNet?
Answer
Real-valued inputs and learnable weights w_i for each input, plus bias.
14
Question
What is the activation function g in the original perceptron?
Answer
Signum: g(x) = -1 if x<0, 0 if x=0, +1 if x>0.
15
Question
How does perceptron mimic AND with weights w1=1, w2=1, w0=-1.5?
Answer
x1 + x2 -1.5 <0 for (0,0),(0,1),(1,0) ->0; =0.5>0 for (1,1)->1.
16
Question
Why can't perceptron handle XOR?
Answer
Linear decision boundary (hyperplane); XOR requires non-linearity.
17
Question
What structure defines a multilayer perceptron (MLP)?
Answer
Input layer, one or more hidden layers, output layer; fully connected feedforward.
18
Question
What notation is used for weights in MLP?
Answer
w_{ji}^l where i indexes from/to neuron, l layer superscript.
19
Question
How does MLP solve XOR using two hidden neurons?
Answer
h1 acts as OR (w=1,1 bias=-0.5), h2 as NAND (w=1,1 bias=-1.5), y = h1 - h2 -0.5 with signum.
20
Question
What are the two operations in each MLP neuron?
Answer
Aggregation (weighted sum + bias = a_j^l), activation h(a_j^l) = z_j^l.
21
Question
Why introduce non-linear activation functions in neural networks?
Answer
Linear activations lead to constant derivatives (0 or 1), no effective non-linearity; needed for complex functions like XOR.
22
Question
What is the sigmoid activation function?
Answer
\( \sigma(x) = rac{1}{1 + e^{-x}} \), squashes to (0,1).
23
Question
What problem does sigmoid have besides vanishing gradients?
Answer
Not zero-symmetric; outputs 0-1 instead of -1 to 1.
24
Question
What is the tanh activation function?
Answer
\( anh(x) \), squashes to (-1,1), zero-symmetric; derivative 1 - \( anh^2(x) \).
25
Question
What is softmax activation?
Answer
\( rac{e^{x_i}}{\sum e^{x_j}} \), turns vector into probabilities summing to 1.
26
Question
What is ReLU activation?
Answer
\( f(x) = \max(0, x) \), linear for x>=0, 0 otherwise.
27
Question
What is the dead neuron problem in ReLU?
Answer
If pre-activation a<0 always, output z=0 forever; neuron never activates.
28
Question
What is Leaky ReLU?
Answer
\( f(x) = x \) if x>=0, else \( 0.01 x \); small slope for negatives.
29
Question
What is Parametric ReLU (PReLU)?
Answer
Like Leaky ReLU but leak alpha is learnable parameter.
30
Question
What is Concatenated ReLU (CReLU)?
Answer
Vector [ReLU(x), ReLU(-x)]; captures both positive and negative parts.