Key Insights from a Year of Working with LLMs (1/4)

Introduction

The past year has seen significant developments in the field of Large Language Models (LLMs). From the release of GPT-4 to the emergence of open-source alternatives, the landscape has changed considerably, presenting new opportunities and challenges. As a patent attorney working at the intersection of law and technology, I’ve had the opportunity to observe and work with these advancements firsthand.

Over the last twelve months, I’ve explored the practical applications of LLMs, examining their potential to improve legal processes, support decision-making, and change how we interact with complex information. This experience has been a mix of insights, setbacks, and unexpected discoveries. What I’ve learned goes beyond technical knowledge; it’s a different approach to problem-solving, creativity, and human-machine collaboration.

This matters, particularly for those of us in legal technology. As LLMs become more capable, they’re not just enhancing our abilities; they’re prompting us to reconsider our approach to legal research, document analysis, and legal writing. There’s potential to automate routine tasks, find new insights in large document sets, and generate more nuanced legal arguments. However, like any tool, the value lies in understanding its capabilities, limitations, and how to use it effectively.

In this series of four posts, I’ll share key insights from a year of working intensively with LLMs. This first post focuses on general process insights, exploring how we can effectively integrate LLMs into our workflows and thinking processes. Subsequent posts will cover model-specific observations, technical implementation lessons, and persistent challenges and issues.

Whether you’re experienced with AI or just starting to explore LLMs in your practice, I hope you’ll find something of value in these observations.

(PS: Bear with me – some of these posts were written with help from Claude as I can then write 4 posts in the normal time it takes me to write 1.)

Series Overview

Over the next four weeks, I’ll be sharing insights gleaned from a year of intensive work with Large Language Models. Each post will focus on a different aspect of my experience:

General Process Insights: This post explores how working with LLMs has changed our approach to problem-solving, creativity, and information processing. We’ll look at strategies for effective LLM integration into existing workflows and thought processes.
Model-Specific Observations: The second post will look into the characteristics of different LLM models, including GPT-4, Claude 3.5 Sonnet, and various “mini” models. We’ll discuss their strengths, limitations, and optimal use cases.
Technical Implementation Lessons: In the third post, we’ll get into the nuts and bolts of working with LLMs. This will cover topics like parallelization, asynchronous programming, and building robust tech stacks for LLM-powered applications.
Persistent Challenges and Issues: The final post will address the ongoing challenges in working with LLMs, including issues of variability, evaluation difficulties, and limitations in certain types of tasks.

In this first post, we’ll focus on the general process insights. These are the broad lessons that apply across different models and applications, forming the foundation for effective work with LLMs. Let’s dive in.

General Process Insights

The crucial role of context

It’s easy to underestimate just how much implicit information we, as humans, bring to even the simplest tasks. When using LLMs, we need to be acutely aware of this context and find ways to make it explicit.

Here are some key observations about context:

Implicit knowledge is vast: We often don’t realize how much background knowledge we’re using when we perform tasks. This can range from common-sense understanding to domain-specific expertise that we’ve internalized over years of practice.
Context shapes interpretation: The same input can be interpreted differently depending on the context. For example, legal terms often have specific meanings that differ from their everyday usage.
LLMs need explicit context: While LLMs have broad knowledge, they don’t have the same intuitive understanding of context that humans do. We need to provide this context explicitly.

To effectively provide context to LLMs, consider:

Defining the scope: Clearly outline the domain and specific area of focus for the task.
Providing background information: Include relevant facts, definitions, or principles that a human expert would typically keep in mind.
Specifying the desired outcome: Be explicit about what you want the LLM to produce or achieve.

For instance, when using an LLM for patent analysis, you might need to specify:

The specific area of technology
Relevant sections of patent law
Any particular conventions or standards in the field
The specific type of analysis you’re looking for (e.g., novelty, non-obviousness)

By making context explicit, we not only improve the performance of LLMs but also often gain a deeper understanding of our own thought processes and the knowledge we bring to bear on complex tasks.

It’s worth noting that determining what context is necessary can be an iterative process. As you work with LLMs, you may discover gaps in the provided context that need to be filled to achieve optimal results.

Leveraging LLMs for information condensation

A particularly useful capability I’ve observed in LLMs over the past year is their proficiency in condensing large volumes of information into concise, high-quality outputs. This ability, coupled with the relatively low cost of LLM operations, opens up new possibilities for information processing and synthesis.

Key observations on LLM-based information condensation:

Efficient summarisation: LLMs excel at distilling the essence of lengthy documents or multiple sources into clear, concise summaries.
Cost-effective processing: The computational cost of running LLMs for condensation tasks is often lower than one might expect, especially when compared to the time and resources required for human-led summarisation.
Scalability: This approach allows for the processing of much larger volumes of information than would be feasible with manual methods.

To leverage this capability effectively, consider the following approaches:

Iterative condensation: For very large datasets, use a multi-step process where initial summaries are further condensed in subsequent passes.
Guided focus: Provide specific prompts or questions to direct the LLM’s attention to particular aspects of the information.
Cross-referencing: Use LLMs to identify and synthesise common themes or contradictions across multiple documents.

In legal practice, this capability can be particularly valuable. For instance:

Case law analysis: Condensing multiple relevant cases into a brief overview of key principles.
Contract review: Summarising lengthy contracts to highlight key terms, obligations, and potential issues.
Patent landscaping: Distilling large patent databases into concise overviews of technological trends or competitive positions.

It’s worth noting that while LLMs are powerful tools for information condensation, the output should always be verified by a knowledgeable human. LLMs can occasionally miss nuances or make errors, especially in complex or specialised fields.

By incorporating LLM-based condensation into our workflows, we can process and synthesise far more information than previously possible, potentially uncovering insights that might otherwise remain hidden in the sheer volume of available data.

Navigating document and thought hierarchies

Another insight I’ve gained over the past year is the importance of navigating hierarchies of documents and thoughts when working with LLMs. This is particularly relevant in legal contexts, where documents often have complex, nested structures.

Key observations on working with hierarchies:

Structure mirrors thought: Document hierarchies often reflect the structures our brains use to organise complex information. For instance, the layout of legislation or legal arguments typically mirrors the logical progression of legal reasoning.
Thought process hierarchies: By mapping out the hierarchy of thoughts or decision-making processes, we can guide LLMs to produce more coherent and logical outputs. This is essentially teaching the LLM to “think” in a structured manner similar to a legal professional. This is what OpenAI is doing at run-time with their o1 series of models.
LLMs and structure: While LLMs can work with hierarchies, they are not hard-coded within their architectures – they are embedded in the millions of weight values over multiple transformer layers. We need to provide this structure explicitly, which often reveals insights into our own cognitive processes.
Hard-coding helps: For specific tasks, hard-coding the document or thought structure into your prompts or preprocessing steps can significantly improve LLM performance.

Practical approaches to leverage hierarchies:

Document mapping: Create clear maps of document structures, especially for complex legal documents like contracts or legislation.
Hierarchical prompting: Structure your prompts to reflect the hierarchy of the document or thought process you’re working with.
Recursive analysis: Use LLMs to analyze documents at different levels of the hierarchy, then synthesize the results.

In legal applications, this approach can be particularly valuable:

Legislation analysis: Navigate through acts, sections, subsections, and clauses in a structured manner that mirrors legal interpretation processes.
Contract review: Analyze contracts by clearly delineating main sections, clauses, and subclauses, reflecting how legal professionals mentally organize contract information.
Legal reasoning: Structure legal arguments by explicitly laying out primary issues, sub-issues, and supporting points, mirroring the thought process of building a legal case.

By leveraging these hierarchies, we not only improve LLM performance but often gain deeper insights into the structure of our own knowledge and cognitive processes. This approach can lead to more precise, contextually aware analyses and outputs, particularly valuable in fields like law where structure and logical progression are crucial.

Starting small: The power of mini models in complex tasks

When working on complex projects with LLMs, such as content generation or intricate legal analysis, it’s often necessary to use more capable models like GPT-4 or Claude 3.5 Sonnet. However, an important insight I’ve gained is the value of identifying and isolating simpler subtasks within these complex projects where mini models can be effectively employed.

Mini models, including GPT-4o Mini, Claude 3.5 Haiku, and Gemini Flash, have proven surprisingly capable for well-defined, narrower tasks. These might include:

Classification of document types or legal concepts
Extracting specific information (dates, case numbers, party names)
Formatting or standardising text
Simple translations or paraphrasing

By breaking down larger, complex tasks into smaller, clearly defined subtasks, it becomes possible to leverage these mini models effectively. This approach offers several advantages:

Efficiency: Mini models often process requests faster and at a lower cost.
Focused Performance: They can be highly accurate within their specific, limited domains.
Scalability: Easier to deploy and manage for high-volume, repetitive tasks.

For example, in a complex legal document analysis project, while the overarching analysis might require a more sophisticated model, tasks like initial document classification, key information extraction, or standardising citation formats could be handled efficiently by mini models.

This strategy of task decomposition not only optimises resource usage but also allows for more modular, maintainable systems. It’s a reminder that in the world of LLMs, bigger isn’t always better – the key is matching the right tool to the right task.

The iterative path to quality

Working with LLMs over the past year has reinforced the importance of iteration in producing high-quality outputs. The process is remarkably similar to making a croissant: add information, fold, refine, add more information, fold, refine. This iterative approach allows for a level of refinement that would be difficult to achieve in a single pass.

Key observations on the iterative process:

Layered refinement: Each iteration adds a layer of nuance or precision to the output, much like the layers in a croissant.
Rapid cycles: LLMs can perform these iterations much faster than humans, allowing for numerous refinement cycles in a short time.
Evolving context: Each iteration can incorporate new context or feedback, leading to progressively more accurate or tailored results.

Practical approaches to leverage iteration:

Staged prompts: Break down complex tasks into a series of prompts, each building on the output of the previous one.
Feedback loops: Incorporate human feedback or automated evaluation metrics between iterations to guide the refinement process.
Parallel iterations: Run multiple iterative processes simultaneously with slightly different parameters, then synthesize the results.

In legal applications, this iterative approach can be particularly valuable:

Contract drafting: Progressively refine contract language through multiple passes, each focusing on different aspects (e.g., clarity, legal compliance, risk mitigation).
Legal research: Iteratively refine search queries and synthesize findings, gradually building a comprehensive understanding of a legal issue.
Argument construction: Build legal arguments layer by layer, starting with main points and progressively adding supporting evidence, counterarguments, and rebuttals.

It’s worth noting that while LLMs can perform iterations rapidly, the process still benefits from human oversight. The role of the legal professional shifts from performing each iteration manually to guiding the overall process and providing critical evaluation.

This iterative approach allows us to harness the speed and processing power of LLMs while still maintaining the nuanced understanding and quality control that are crucial in legal work. By embracing this process, we can produce higher quality outputs more efficiently than ever before.

Embracing a new creative process

Working with LLMs over the past year has led to some interesting shifts in how we approach creative and business processes. These changes can seem quite unusual at first, but they offer new possibilities for efficiency and quality in our work.

Key observations on the new creative process:

Emphasis on refinement: The creative act is no longer just about the initial draft from mind to keyboard. A significant part of the process now involves editing and refining LLM-generated content.
Perfection isn’t the starting point: With LLMs, it’s often more efficient to generate a rough output quickly and then refine it, rather than striving for perfection in the first draft.
Iterative improvement: The process becomes more about guiding and shaping the output through multiple iterations, rather than creating everything from scratch.
Hidden steps become visible: Steps in the creative process that were previously internal and hidden now become explicit when working with LLMs.

Practical approaches to this new process:

Rapid prototyping: Use LLMs to quickly generate multiple versions or ideas, then refine the most promising ones.
Structured refinement: Develop systematic approaches to reviewing and improving LLM outputs.
Collaborative creativity: View the process as a collaboration between human expertise and LLM capabilities, each contributing their strengths.

In legal and business contexts, this approach can be valuable for:

Document drafting: Quickly generate initial drafts of contracts, memos, or reports, then focus human expertise on refining and customizing.
Brainstorming: Use LLMs to generate a wide range of ideas or solutions, then apply human judgment to select and develop the best ones.
Analysis and research: Generate initial analyses or research summaries, then iteratively refine and expand based on human insight and additional prompts.

It’s important to note that this shift doesn’t diminish the value of human creativity and expertise. Instead, it allows us to focus our efforts on higher-level thinking, critical analysis, and nuanced refinement.

By embracing this new creative process, we can often produce higher quality work more efficiently. However, it does require a mindset shift, moving away from the idea of the solitary creator and towards a more collaborative, iterative approach to creativity and problem-solving.

Conclusion

As we’ve explored in this post, working with LLMs over the past year has led to several important insights about how we approach complex tasks and creative processes. We’ve seen the value of starting small with mini models for well-defined subtasks, the importance of externalising our thought processes, and the crucial role of context in guiding LLM outputs.

We’ve also discovered the power of LLMs in condensing large amounts of information and navigating complex document hierarchies, mirroring our own thought processes. The iterative nature of working with LLMs has emerged as a key factor in producing high-quality outputs, leading us to embrace a new creative process that emphasises refinement and collaboration between human and machine.

These insights have implications for how we approach legal and business tasks. By leveraging LLMs effectively, we can often work more efficiently, process larger volumes of information, and potentially uncover insights that might otherwise remain hidden.

However, it’s important to remember that these tools are just that – tools. They require thoughtful application, guided by human expertise and judgment. As we continue to integrate LLMs into our workflows, we’ll need to remain adaptable, continuously refining our processes to make the most of these powerful capabilities.

In the next post in this series, we’ll jump into specific observations about different LLM models, exploring their strengths, limitations, and optimal use cases. Until then, I encourage you to consider how these general insights might apply to your own work with LLMs, whether you’re just starting out or already deeply engaged with these technologies.

Key Insights from a Year of Working with LLMs (1/4)

Introduction

Series Overview