‘We’re still at the beginning of the AI journey’

Interest in neural networks resurged in the early 2000s, driven by advances in hardware and access to massive datasets [File]
| Photo Credit: REUTERS

Artificial Intelligence (AI) has made remarkable strides in recent years, yet according to machine learning expert Shreyas Subramanian, there is still much to uncover.

To understand AI’s evolution and where it’s headed, I connected with Subramanian, a Principal Data Scientist at AWS, on The Interface podcast, to gather his insights on the development of deep neural networks, the rise of transformers, and the broader trajectory of AI.

Tracing the roots of neural networks to the early conceptualisations of artificial neurons, Subramanian explained the work of Frank Rosenblatt in the 1950s with the Perceptron model.

“These perceptrons are simple pattern recognition compute units,” he explained. “Surprisingly, they are still used today in dense layers and adapters to make large language models more efficient.”

This foundational technology laid the groundwork for neural networks, which aimed to become universal function approximators that could map any input to any output.

The Rise of Neural Networks

Interest in neural networks resurged in the early 2000s, driven by advances in hardware and access to massive datasets. Subramanian highlighted the pivotal role of AlexNet, which popularised deep convolutional neural networks (CNNs) by winning the ImageNet competition. “AlexNet set the baseline by stacking perceptrons, introducing deep convolution neural networks, and leveraging GPU advancements,” he noted.

CNNs transformed image recognition by using convolution operations, akin to sliding a coloured plastic filter over an image to highlight specific features, dramatically improving classification tasks and paving the way for modern AI applications.

While CNNs revolutionised image processing, they lacked memory capabilities, which Recurrent Neural Networks (RNNs) addressed. “The missing component from early CNN literature was the ability to form memory,” Subramanian said. RNNs introduced mechanisms to retain information over time, though they struggled with issues like the vanishing gradient problem.

To overcome these limitations, architectures like Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) enabled specialised memory modules, allowing AI to handle complex temporal dependencies more effectively.

Inspired by the Human Brain

AI development has often drawn inspiration from neuroscience, though Subramanian acknowledged the gap between artificial and biological intelligence. “We still don’t understand enough about the brain to replicate its efficiency,” he admitted. The human brain’s ability to process sensory inputs in parallel, adapt through neuroplasticity, and perform energy-efficient computations remains unmatched.

Despite AI models boasting billions of parameters, they still fall short of the brain’s capabilities. “Today’s large language models exceed the number of neurons in the human brain, yet they’re limited in comparison to the brain’s efficiency and flexibility,” Subramanian observed.

One milestone in AI’s journey was the development of AlphaGo by DeepMind, which demonstrated the power of reinforcement learning and deep neural networks. “It surpassed human performance in the game of Go, relying on deep learning and reinforcement strategies to handle complex decision-making,” Subramanian reflected. AlphaGo’s success highlighted AI’s potential in strategic reasoning and long-term planning, influencing advancements beyond gaming.

The Transformer Revolution

In 2017, AI took a leap forward with the introduction of the Transformer architecture, particularly in natural language processing. The seminal paper, ‘Attention Is All You Need’, introduced self-attention mechanisms that outperformed previous RNN-based models. “Transformers eliminated sequential bottlenecks and handled long sequences efficiently,” Subramanian explained.

This architecture led to models like BERT and GPT, which fueled the rise of large language models and generative AI. The adaptability of Transformers made them foundational for diverse applications, from text generation to image and audio processing.

The AI Industry and Competitive Landscape

Discussing the competitive landscape, Subramanian noted how OpenAI’s strategic focus on scaling language models gave it an edge over tech giants like Google. “OpenAI invested heavily in compute resources and data curation at a time when others were hesitant,” he said. This early commitment paid off with the success of GPT-2 and GPT-3, leading to widespread adoption and commercial success.

Meanwhile, companies like Google and Amazon pursued different strategies, balancing research with business priorities. “Google’s focus was more on moonshot projects, while Amazon emphasised providing diverse AI solutions to customers,” Subramanian observed.

A Journey Still Unfolding

AI’s evolution from simple perceptrons to transformative language models reflects decades of innovation, interdisciplinary research, and technological advancements. Yet, as Subramanian emphasised, “We’re still at the beginning of this journey. The more we learn about AI and neuroscience, the more we realize how much there is yet to uncover.”

The future of AI promises exciting developments, driven by continued exploration of both artificial and biological intelligence. As the field advances, understanding AI’s history and foundational principles will be crucial in shaping its potential and addressing the challenges ahead.

(Listen to the full discussion with Shreyas Subramanian on the Interface podcast or watch the YouTube video for more insights.)