What is a Neural Network?

Prologue

I have been interested in artificial intelligence since I watched the movie Transcendence. I kept wondering what shape AI could take in the real world. My dream was to have an AI like Marvel's Jarvis. Who did not have this dream?

Currently, I am learning more about the technical details, models, and architectures around neural networks. Neural networks are one of the underlying building blocks of many modern AI systems, including LLMs and a lot of non-LLM systems.

What is a Neural Network?

A neural network is a function that turns inputs into outputs by passing numbers through layers of small calculation units called neurons.

That sounds abstract, so imagine a simple decision system:

0

inputs -> hidden calculations -> outputs

The inputs are numbers that describe the current situation. The outputs are numbers that describe what the network thinks should happen next.

For example, a small neural network could receive these inputs:

0
1
2
3

energy = 0.8
hydration = 0.4
food_is_left = 1.0
water_is_near = 0.0

And it could produce these outputs:

0
1
2
3

move_left = 0.72
eat = 0.18
drink = 0.05
rest = 0.31

If the system says that every output above 0.6 should trigger an action, this network would choose move_left.

The Basic Parts

A neural network usually has four important parts:

• Inputs
• Weights
• Biases
• Activation functions

Inputs are the values you give to the network.

Weights decide how important one value is for the next neuron. A large positive weight means the input strongly increases the next value. A negative weight means the input pushes the next value down.

Biases are extra values added to a neuron. They let a neuron become active even when the weighted inputs alone are not enough.

Activation functions shape the result. Without activation functions, the whole network would mostly behave like one large linear formula. Activation functions make it possible to model more complex behavior.

A single neuron can be described like this:

0

output = activation(input_1 * weight_1 + input_2 * weight_2 + bias)

Real networks repeat this many times across multiple layers.

Layers

Most neural networks are organized into layers.

The input layer receives the original data. The hidden layers transform that data step by step. The output layer returns the final result.

0

input layer -> hidden layer -> hidden layer -> output layer

The hidden layers are where the network builds internal representations. In image recognition, early layers might react to edges and colors, while deeper layers might react to shapes or objects. In a simulation, early layers might combine hunger, thirst, and nearby resources, while deeper layers might turn that into an action preference.

Feed-Forward Networks

A feed-forward neural network is one of the simplest network types. Data moves in one direction:

0

inputs -> layer 1 -> layer 2 -> outputs

There are no loops inside the network itself. You give it inputs, it calculates the layer values, and then it returns outputs.

This is the type of network I used in my Rust project zerya.

My Project: zerya

zerya is a small Rust life simulation. It simulates simple single-cell organisms on a 2D grid with soil, water, rock, food, moss, sickness, reproduction, energy, hydration, sleepiness, inventory, and relationships.

Each cell has a neural-network brain. Every simulation tick, the cell receives information about itself and the world around it. The brain then decides which actions should fire.

Some inputs describe the cell's internal state:

• Energy
• Hunger
• Hydration
• Thirst
• Age
• Fitness
• Sickness
• Sleepiness
• Happiness
• Previous action
• Memory values from the previous tick

Other inputs describe the world:

• Direction to nearest food
• Direction to nearest water
• Global food abundance
• A centered 5x5 vision grid around the cell

The vision grid is encoded as numbers. For example, rock and out-of-bounds tiles are negative, water is positive, food is more positive, and another cell has the strongest positive value.

The network outputs action scores. In zerya, the outputs include actions like:

• Reproduce
• Move up, down, left, or right
• Eat
• Drink
• Push food
• Nurture moss
• Attack
• Rest
• Pick up food, moss, or seeds
• Plant seed
• Feed a nearby cell
• Store memory values for the next tick

The simulation uses a threshold. If an action output is high enough, the cell tries to perform that action.

A Brain as a Decision Function

In zerya, the neural network is not magic. It is just a function that maps a list of numbers to another list of numbers:

0

brain(inputs) = outputs

The interesting part is that the inputs are chosen to describe survival-relevant information, and the outputs are connected to actions in the world.

If the brain receives low hydration and sees water nearby, a useful network should produce a high drink or move_toward_water output. If the brain receives low energy and sees food nearby, a useful network should produce a high eat or move_toward_food output.

At the start, the brain does not know this. Its weights and biases are random. Some cells make bad decisions and die quickly. Some survive longer by chance. The surviving brains become more likely to continue through reproduction.

Mutation and Learning

There are two important ways the brains in zerya can change.

The first one is mutation. When a cell reproduces, the child receives a cloned copy of the parent's brain with a chance of mutation. Mutations can slightly change weights or biases. Rarely, they can add a hidden neuron or insert a new hidden layer.

This creates an evolutionary pressure:

0

better behavior -> longer survival -> more reproduction -> more copied brains

The second one is reinforcement. The brain can adjust selected action outputs based on rewards. Positive reward pushes an output higher for a similar input situation. Negative reward pushes it lower.

This does not make the simulation instantly intelligent. But it gives the cells a mechanism for gradually changing behavior based on what worked or failed.

Why Activation Functions Matter

zerya uses ReLU for hidden layers and sigmoid for output values.

ReLU is simple:

0

relu(x) = max(0, x)

If the value is negative, it becomes 0. If it is positive, it stays positive. This gives hidden neurons a simple on/off style behavior.

Sigmoid turns a value into a number between 0 and 1:

0
1
2

very negative -> close to 0
0             -> 0.5
very positive -> close to 1

That works well for action outputs because an output can be interpreted as a score or probability-like value. For example, 0.9 means the action is strongly activated, while 0.1 means it is not.

What Can You Do With Neural Networks?

Neural networks are useful whenever you need a system to learn a mapping from inputs to outputs.

Common examples are:

• Image classification
• Speech recognition
• Text generation
• Recommendation systems
• Anomaly detection
• Game agents
• Robotics control
• Simulation agents

The important part is not that the network is called neural. The important part is that it can adjust many small parameters until the final behavior gets better for a given goal.

The Key Idea

A neural network is not a brain in the human sense. It does not automatically understand the world.

It is a parameterized function.

You choose what the inputs mean. You choose what the outputs control. Then training, reinforcement, or evolution changes the internal parameters so the function becomes more useful.

In zerya, this means a cell can start with random behavior and, over many generations, produce brains that are better at surviving in a small artificial world.

While the simulation was able to survive for 200 ticks in the beginning (1 tick is one timeframe in which a action can be taken), it was able to survive 15.000 ticks after a long training session. This means that the improvement was definetly achievable with a neural network and it was significant.