How to Preprocess Unstructured Data for LLM Applications?

Hello Learners…

Welcome to the blog…

Table Of Contents

• Introduction
• What is an Unstructured Data?
• How to Preprocess Unstructured Data for LLM Applications?
• Summary
• References

Introduction

In this post we get an overview of How to Preprocess Unstructured Data for LLM Applications?

What is an Unstructured Data?

Unstructured data refers to data that does not have a predefined data model or is not organized in a predefined manner.

What is Data Preprocess?

Preprocessing refers to the steps taken to prepare and clean raw data before it is used for analysis or modeling.

In this we learn to extract and normalize content from a wide variety of document types, such as PDFs, PowerPoints, Word, and HTML files, tables, and images to expand the information accessible to your LLM.

How to Preprocess Unstructured Data for LLM Applications?

Retrieval Augmented Generation or RAG has been widely adopted in may enterprises. The typical RAG pipeline has key components like data loading,chunking,embedding, storing in the vector database, and then retrieval.

Here, we learn the techniques of representing all sorts of unstructured data like text, images, and tables from many different sources like PDF and PowerPoint and Word in a way that lets your LLM RAG pipeline access all of this information.

A particularly challenging task in RAG is data loading and chunking due to data being stored in many different file types and data formats.

For example, we have numeric data in Excel spreadsheets or text reports in PDF or Markdown, or presentations in PowerPoint or Slides Or Keynotes or communications in Outlook or Slack or Teams and so on.

Each of these file types also in turn might support data stored inside them in different formats.

A PDF or PowerPoint file, for example may itself contain tables, images or bulleted lists.

So a data loaders must first able to parse many different file formats. But once it’s parsed that data then what?

It turns out that it’s very useful to normalize the data from these different sources. So when we normalize tables from a PDF or a PowerPoint or other data format, it can all be represented in a similar way or may be bulleted list.

Summary

Also you can refer this for more learning about llms,

• https://galaxyofai.com/tag/llms/

References

• https://www.deeplearning.ai/short-courses/preprocessing-unstructured-data-for-llm-applications/