Data formats for AI work: JSON, JSONL, Parquet and embeddings on disk

Why formats matter in AI projects

Every AI project moves data between places: from documents into chunks, from chunks into embeddings, from your code to an API and back, from your logs into an analysis, from a training run into a saved model. Each of these steps writes something to disk or sends it over the network, and the format you pick decides how easy it is to read, append to, compress, share and trust that data later.

Most beginners save everything as one big JSON file or one big CSV, and it works until the file has a million rows, or a field contains a comma, or two processes try to write to it at the same time.

JSON

JSON is the format of the web. API requests and responses, tool arguments, model outputs and configuration files are usually JSON, and in Python a JSON object becomes a dictionary.

import json

config = {"model": "gemini-2.5-flash-lite", "temperature": 0.2, "max_output_tokens": 512}

with open("config.json", "w") as f:
    json.dump(config, f, indent=2)

with open("config.json") as f:
    loaded = json.load(f)

JSON is good for one object, like a configuration or one API payload. It is bad for large or growing collections, because the whole file is one object, so to add one record you have to read the whole file, change it and write it back, and if the program crashes in the middle, the file can be left broken.

JSONL

JSONL, also called JSON Lines, puts one JSON object on each line.

{"question": "How many leave days do I get?", "answer": "24 days", "source": "hr_policy.pdf"}
{"question": "Who approves travel?", "answer": "Your manager", "source": "travel_policy.pdf"}

That small change solves the problems JSON has with collections:

• You can append a new record by adding a line, without reading the rest of the file.
• You can read a huge file one line at a time, without loading it all into memory.
• If one line is broken, the rest of the file is still readable.
• Tools like head, tail, wc -l and grep work on it directly from the terminal.

That is why JSONL is the default for LLM work. Fine-tuning datasets, batch API requests, evaluation sets and call logs are usually JSONL, and several providers expect it for fine-tuning and batch jobs.

import json

records = [
    {"question": "How many leave days do I get?", "answer": "24 days"},
    {"question": "Who approves travel?", "answer": "Your manager"},
]

with open("eval_set.jsonl", "w") as f:
    for record in records:
        f.write(json.dumps(record) + "\n")

with open("eval_set.jsonl") as f:
    for line in f:
        item = json.loads(line)
        print(item["question"])

CSV

CSV is a plain text table, with a header row and one row per record. It is the format people open in Excel or Google Sheets, so it is useful for small datasets you want a non-technical person to review, like a list of test questions and expected answers.

Its weaknesses show up quickly in AI work. CSV has no types, so everything is text until you convert it. Long text with commas, quotes and line breaks needs careful escaping, and LLM outputs are full of all three. And nested data, like a list of sources for each answer, doesn't fit a flat table.

Use pandas or Python's csv module to read and write it, never string splitting, because they handle the escaping for you.

Parquet

Parquet is a columnar file format. Instead of storing data row by row, it stores each column together, compressed, with its data type.

That makes it much smaller than CSV for the same data, much faster when you only need some columns, and it keeps types like integers, dates and lists intact. It is the common format for large datasets, data lakes and many datasets on Hugging Face.

import pandas as pd

df = pd.DataFrame({
    "doc_id": [1, 2],
    "text": ["Refunds take 5 to 7 days.", "Travel needs manager approval."],
    "tokens": [8, 6],
})

df.to_parquet("chunks.parquet")
print(pd.read_parquet("chunks.parquet", columns=["doc_id", "tokens"]))

You need pyarrow installed for pandas to read and write Parquet. You can't open a Parquet file in a text editor, so for small data that people need to read, JSONL or CSV is still easier.

Embeddings on disk

An embedding is a list of numbers, often hundreds or thousands of them, for every chunk of text. For 100000 chunks with 768 dimensions, that is 76.8 million numbers, so the format matters a lot.

Don't store embeddings as JSON text. It is several times larger than the binary version and slow to parse.

For learning and small projects, keep the vectors in a NumPy array saved with np.save, and keep the text and metadata separately in JSONL or Parquet with the same order or an ID column.

import numpy as np

vectors = np.random.rand(1000, 384).astype("float32")
np.save("embeddings.npy", vectors)

loaded = np.load("embeddings.npy")
print(loaded.shape, loaded.dtype)

float32 uses half the space of NumPy's default float64, and it is the precision most embedding models and vector databases use.

For anything beyond a few hundred thousand vectors, or anything that needs fast search, filtering or updates, use a vector database, which stores and indexes the vectors for you.

Saving models: serialization

When you train a model, it lives in memory, in RAM. When the program ends, it is gone, and you would have to train it again from the beginning.

Serialization is converting a Python object, like a trained model, into bytes that can be saved to disk, and deserialization is loading those bytes back into a Python object. What you are really saving are the model's learned parameters, which in a neural network are its weights and biases. Once they are saved, you can load the model for predictions without training it again, and when new data arrives, you can continue training from the saved parameters instead of starting from zero.

For classical machine learning models, like a scikit-learn spam classifier, the common libraries are pickle and joblib.

import joblib
from sklearn.linear_model import LogisticRegression

model = LogisticRegression().fit([[0], [1], [2], [3]], [0, 0, 1, 1])
joblib.dump(model, "spam_model.joblib")

loaded = joblib.load("spam_model.joblib")
print(loaded.predict([[2.5]]))

joblib is usually better than pickle for models with large NumPy arrays inside them.

Why pickle can be dangerous

Pickle, and joblib which uses it underneath, can run code while loading a file. That is how they rebuild complex Python objects, and it also means that a malicious pickle file can run any code on your machine the moment you load it.

So only load pickle or joblib files that you created yourself or that come from a source you trust completely. For sharing model weights, use safetensors, a format created by Hugging Face that stores only the tensors and can't run code. It is now the default format for most models on Hugging Face.

For PyTorch models, save the state_dict (the dictionary of weights) rather than the whole model object, and load it into a model class you define in your code.

Model registries

When you start training or fine-tuning models regularly, saving files with names like model_final_v2_new.joblib stops working. A model registry, like the one in MLflow or Hugging Face Hub, stores each version of a model with its metrics, the data it was trained on and the code version, so you always know which model is in production and can roll back to an earlier one.

Other formats you will meet

• YAML: configuration files that people edit by hand, like Docker Compose files and CI pipelines. Easier to read than JSON, but sensitive to indentation.
• Markdown: READMEs, documentation, and often prompts and the documents you feed to RAG, because LLMs handle it well and it keeps structure like headings and lists.
• .env: environment variables for keys and settings, covered on the secrets page.
• PDF, DOCX and HTML: the formats most company documents arrive in, which need parsing before you can use them, covered in document parsing.

Which format to use

Data	Format
App configuration	JSON, YAML or .env
One API request or response	JSON
Fine-tuning data, eval sets, batch requests, call logs	JSONL
Small table for people to review	CSV
Large tables, processed chunks, analytics	Parquet
Embeddings in a small project	NumPy .npy plus metadata in JSONL or Parquet
Embeddings at scale	A vector database
Classical ML models you made yourself	joblib
Neural network weights to share	safetensors
Docs, prompts, RAG source text	Markdown

From the floor

In the 75 day challenge, day 5 was serialization. I explained it with a spam classifier: you train it once, and if you don't save it, the knowledge it learned disappears with the program, so you save the parameters and reuse them, and when new data comes in, you continue from what is saved instead of training the whole model again.

On Build30 day 1, while building the token tracker, I changed the log file from JSON to JSONL, where every call is a separate object on its own line, instead of everything being wrapped inside one big object.

Related

• Data for AI
• Building a fine-tuning dataset
• Embeddings
• Pydantic and structured data

Sources

• JSON Lines
• Apache Parquet documentation
• pickle, Python documentation (security warning)
• safetensors, Hugging Face
• Saving and loading models, PyTorch