The Biology of LLMs

Large language models are popping up in the digital world, changing how people and businesses interact with technology. With every new iteration, these AI systems can manage everything from customer service to content creation.

Large language models (LLMs) drive human productivity, product and capability gains in customer service, healthcare and education industries.

While we have made progress, LLMs are not accurate. They do not understand causality, and they still struggle with cultural nuance. If organizations want to incorporate these LLMs into their processes, they need to learn the biology of these tools.

How Do LLMs Work?

Modern language models are, at their core, neural networks that form massive webs of mathematical relationships that transform and generate human language. In contrast to the usual software, they learn patterns from data rather than executing explicit programming instructions.

LLMs were revolutionized by Google researchers in 2017 with the transformer architecture, which allows models to put different weights on the words according to their mutual relations. This breakthrough meant that LLMs could begin to capture longer-range dependencies and contextual relationships that previous systems missed.

Over time, networks have evolved more sophisticated language representations from naive word prediction to more nuanced concepts of context, tone, and cultural references. Training these networks is computationally expensive, and leading labs run thousands of specialized chips for weeks or even months.

Today’s largest current models have trillions of connections of such complexity that we can only compare them with biological organisms, but on an entirely different set of principles.

How LLMs Develop Their "Genetic Code"

The training data for LLMs is massive collections of text, which are the formative experiences for these systems. The model is built on this digital corpus. This can consist of hundreds of billions of words and dictate what it can understand and what biases it may have.

Most major AI labs are now keeping their exact training methodologies more and more secret while still using sophisticated filtering to remove harmful content and keep diverse knowledge samples. The data is directly connected to performance. Researchers have found that high-quality, well-curated datasets typically exceed the performance of larger but less refined collections.

A key issue is the cultural representation in training data, since models trained primarily on English or Western sources suffer from other cultural contexts. Newer training approaches provide human feedback and can teach models to follow human values with billions of corrections. Authors and publishers keep challenging the use of copyrighted materials in training, and legal battles continue.

Limitations of LLMs

LLMs suffer from limitations, such as pattern recognition and the difference between pattern recognition and comprehension itself. Some of the limitations include:

Factual Inaccuracy

Models often hallucinate, confidently asserting things without any basis in their training data.

Neven Dujmovic, privacy engineering director at Arthrex, explained that "AI models such as GPT-4 process extensive text data, relying on patterns and statistical associations rather than genuine comprehension, which can result in misinterpretations and subsequently lead to the generation of coherent yet factually inaccurate or non-existent information."

Moreover, formidable challenges persist for even advanced models to make elementary arithmetic errors regarding mathematical reasoning. The second is temporal awareness. This means that the models lack insight into the progression of time and cannot update their knowledge beyond the cutoff date of training.

Poor Reasoning

Some models have fragile reasoning capabilities that can sometimes produce illogical analysis. Others sometimes make egregious errors on very similar problems. The LLM systems may not understand causality — that is, why things happen the way they do — and cannot distinguish between correlation and causation. Cultural nuance tests these systems, and minority languages and references are the hardest to get right.

Building Ethical LLMS

As LLMs continue to advance rapidly, learning institutions and the government have been compelled to introduce safeguards to mitigate the possibility of the harmful uses of this technology. One of the most immediate threats is misinformation, i.e., models that can generate sufficiently convincing fake news or deceptive content at unprecedented scale and speed.

"As AI technology continues its relentless march forward, the malevolent actors behind disinformation schemes quickly adapt," warned Leesa S, managing director at R3i Capital.

Models have begun to reproduce sensitive personal data present in the training data. This has led to legal liabilities for developers, and privacy concerns have escalated.

As tasks that previously were carried out by knowledge workers are automated, employment disruption has sped up. Goldman Sachs estimates that up to 300 million jobs worldwide will be affected by generative AI. Mitigation efforts are still insufficient to eradicate bias and discrimination, as models can heighten the societal prejudices inherent in training data.

As creative professionals object to the legality of training on copyrighted works, copyright infringement claims have flourished. While regulation remains very uneven in different regions, industry leaders and ethicists increasingly recommend further regulation.

"AI guidelines should provide the skills required to spot AI-generated trickery," Leesa said. "The algorithms can inherit biases from the training data, leading to discriminatory outcomes.”

The Sustainability Challenges Facing LLMs

Despite its benefits, AI has some sustainability challenges, including:

High-Energy Costs

The seemingly effortless intelligence of LLMs is produced at a high energy cost, which poses a serious threat to AI’s environmental sustainability. Training today's largest models require enough electricity to power hundreds of American homes for a year. The energy costs of a single training run can reach millions of dollars.

Excessive Water Requirements

Although this massive computing comes at a considerable environmental cost, the water consumption necessary to cool these big computing clusters has become an environmental concern.

Data centers in drought‐prone areas have come under fire for their large water footprint. The environmental costs are real and growing. Developers are creating LLMs that do not consume a lot of water.

Requires Rare Materials

Environmental impact is also felt in hardware manufacturing, where specialized chips require rare minerals like cobalt and indium and energy-intensive production processes. Google and Microsoft have committed to carbon neutrality for their AI operations through renewable energy purchases or offset programs, but verification is difficult.

While computational giants continue to emerge, smaller and more efficient models are gaining traction thanks to knowledge distillation, which makes compact models capable of reaching the capabilities of larger counterparts. The industry needs fundamental breakthroughs in computing efficiency, not just incremental improvements.

Applications of LLMs Across Industries

"We're seeing that the early adopters who overcame barriers to deploy AI are making further investments, proving to me that they are already experiencing the benefits from AI," said Rob Thomas, Senior Vice President of IBM Software. Various industries utilize these models, including:

• Healthcare: Healthcare organizations use LLMs for medical documentation, literature reviews and patient communication. However, clinical applications have been slowed by regulatory concerns.
• Financial Services: Financial institutions have adopted LLMs for market analysis, report generation and customer service, driven by productivity gains anticipated by early adopters such as Goldman Sachs.
• Education: Adoptions range from personalized tutoring systems to automated grading to helping educators overcome their struggle with academic integrity concerns.
• Legal: In the legal sector, LLMs are used in document review, contract analysis and case research. The American Bar Association has reported that 32 percent of attorneys now use AI tools to some extent.
• Customer Service: LLMs have been used to create chatbots that significantly improve customer service.
• Scientific Research: Progress in scientific fields, from materials science to genomics, is being accelerated by LLMs that are increasingly used by research laboratories to generate hypotheses and synthesize results.

The Next Generation of Language Models

Language models are the next evolutionary leap from statistical pattern recognition to true understanding. The next frontier is multimodal models, incorporating text, vision, audio and other sensory inputs. Companies like OpenAI and Anthropic are leading with newfound advances in these directions.

The future isn't just about better text prediction but AI systems that can reason across different types of information. Memory architectures are being developed to provide models with a longer-term recall to maintain context across longer conversations.

Models are also increasingly capable of using tools. They are starting to interact with external systems, databases, and applications, effectively extending their capabilities beyond language generation. Specialized training of models through reasoning enhancements has enabled these models to become more logical, but true causal reasoning is yet to be achieved.

There’s substantial progress in areas that were considered AI-hard problems just two years ago. These developments hint at the prospect of artificial general intelligence, whereby some researchers believe that sufficiently advanced language models can match human intelligence. Around the world, regulatory frameworks are still evolving, with the EU's AI Act setting boundaries that will inform development for years to come. The pace of innovation suggests we're at an inflection point in technological history.

The distinction between biological and artificial intelligence may blur in ways we're only beginning to comprehend.