Skip to main content
Open In ColabOpen on GitHub

Source Code

This notebook covers how to load source code files using a special approach with language parsing: each top-level function and class in the code is loaded into separate documents. Any remaining code top-level code outside the already loaded functions and classes will be loaded into a separate document.

This approach can potentially improve the accuracy of QA models over source code.

The supported languages for code parsing are:

  • C (*)
  • C++ (*)
  • C# (*)
  • COBOL
  • Elixir
  • Go (*)
  • Java (*)
  • JavaScript (requires package esprima)
  • Kotlin (*)
  • Lua (*)
  • Perl (*)
  • Python
  • Ruby (*)
  • Rust (*)
  • Scala (*)
  • TypeScript (*)

Items marked with (*) require the packages tree_sitter and tree_sitter_languages. It is straightforward to add support for additional languages using tree_sitter, although this currently requires modifying LangChain.

The language used for parsing can be configured, along with the minimum number of lines required to activate the splitting based on syntax.

If a language is not explicitly specified, LanguageParser will infer one from filename extensions, if present.

%pip install -qU esprima esprima tree_sitter tree_sitter_languages
import warnings

warnings.filterwarnings("ignore")
from pprint import pprint

from langchain_community.document_loaders.generic import GenericLoader
from langchain_community.document_loaders.parsers import LanguageParser
from langchain_text_splitters import Language
loader = GenericLoader.from_filesystem(
    "./example_data/source_code",
    glob="*",
    suffixes=[".py", ".js"],
    parser=LanguageParser(),
)
docs = loader.load()
len(docs)
6
for document in docs:
    pprint(document.metadata)
{'content_type': 'functions_classes',
 'language': <Language.PYTHON: 'python'>,
 'source': 'example_data/source_code/example.py'}
{'content_type': 'functions_classes',
 'language': <Language.PYTHON: 'python'>,
 'source': 'example_data/source_code/example.py'}
{'content_type': 'simplified_code',
 'language': <Language.PYTHON: 'python'>,
 'source': 'example_data/source_code/example.py'}
{'content_type': 'functions_classes',
 'language': <Language.JS: 'js'>,
 'source': 'example_data/source_code/example.js'}
{'content_type': 'functions_classes',
 'language': <Language.JS: 'js'>,
 'source': 'example_data/source_code/example.js'}
{'content_type': 'simplified_code',
 'language': <Language.JS: 'js'>,
 'source': 'example_data/source_code/example.js'}
print("\n\n--8<--\n\n".join([document.page_content for document in docs]))
class MyClass:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, {self.name}!")

--8<--

def main():
    name = input("Enter your name: ")
    obj = MyClass(name)
    obj.greet()

--8<--

# Code for: class MyClass:


# Code for: def main():


if __name__ == "__main__":
    main()

--8<--

class MyClass {
  constructor(name) {
    this.name = name;
  }

  greet() {
    console.log(`Hello, ${this.name}!`);
  }
}

--8<--

function main() {
  const name = prompt("Enter your name:");
  const obj = new MyClass(name);
  obj.greet();
}

--8<--

// Code for: class MyClass {

// Code for: function main() {

main();

The parser can be disabled for small files.

The parameter parser_threshold indicates the minimum number of lines that the source code file must have to be segmented using the parser.

loader = GenericLoader.from_filesystem(
    "./example_data/source_code",
    glob="*",
    suffixes=[".py"],
    parser=LanguageParser(language=Language.PYTHON, parser_threshold=1000),
)
docs = loader.load()
len(docs)
1
print(docs[0].page_content)
class MyClass:
    def __init__(self, name):
        self.name = name

    def greet(self):
        print(f"Hello, {self.name}!")


def main():
    name = input("Enter your name: ")
    obj = MyClass(name)
    obj.greet()


if __name__ == "__main__":
    main()

Splitting​

Additional splitting could be needed for those functions, classes, or scripts that are too big.

loader = GenericLoader.from_filesystem(
    "./example_data/source_code",
    glob="*",
    suffixes=[".js"],
    parser=LanguageParser(language=Language.JS),
)
docs = loader.load()
from langchain_text_splitters import (
    Language,
    RecursiveCharacterTextSplitter,
)
js_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.JS, chunk_size=60, chunk_overlap=0
)
result = js_splitter.split_documents(docs)
len(result)
7
print("\n\n--8<--\n\n".join([document.page_content for document in result]))
class MyClass {
  constructor(name) {
    this.name = name;

--8<--

}

--8<--

greet() {
    console.log(`Hello, ${this.name}!`);
  }
}

--8<--

function main() {
  const name = prompt("Enter your name:");

--8<--

const obj = new MyClass(name);
  obj.greet();
}

--8<--

// Code for: class MyClass {

// Code for: function main() {

--8<--

main();

Adding Languages using Tree-sitter Template​

Expanding language support using the Tree-Sitter template involves a few essential steps:

  1. Creating a New Language File:
    • Begin by creating a new file in the designated directory (langchain/libs/community/langchain_community/document_loaders/parsers/language).
    • Model this file based on the structure and parsing logic of existing language files like cpp.py.
    • You will also need to create a file in the langchain directory (langchain/libs/langchain/langchain/document_loaders/parsers/language).
  2. Parsing Language Specifics:
    • Mimic the structure used in the cpp.py file, adapting it to suit the language you are incorporating.
    • The primary alteration involves adjusting the chunk query array to suit the syntax and structure of the language you are parsing.
  3. Testing the Language Parser:
    • For thorough validation, generate a test file specific to the new language. Create test_language.py in the designated directory(langchain/libs/community/tests/unit_tests/document_loaders/parsers/language).
    • Follow the example set by test_cpp.py to establish fundamental tests for the parsed elements in the new language.
  4. Integration into the Parser and Text Splitter:
    • Incorporate your new language within the language_parser.py file. Ensure to update LANGUAGE_EXTENSIONS and LANGUAGE_SEGMENTERS along with the docstring for LanguageParser to recognize and handle the added language.
    • Also, confirm that your language is included in text_splitter.py in class Language for proper parsing.

By following these steps and ensuring comprehensive testing and integration, you'll successfully extend language support using the Tree-Sitter template.

Best of luck!