fenic: semantic DataFrames for humans and agents

fenic turns AI-assisted exploration of structured and unstructured data into reusable, inspectable DataFrame pipelines.

It's a DataFrame query engine for semantic data processing, with AI operators — extract, classify, summarize, embed, semantic join, and more — built into the query model. Use it to turn documents, transcripts, logs, eval traces, tickets, tables, and APIs into typed rows and repeatable workflows.

The point is a shift in what your data work produces. Humans and agents work on the same pipelines — both can author, inspect, and reuse them. The result isn't a one-off prompt or a brittle regex script that has to be reverse-engineered later — it's a durable artifact: typed, inspectable, rerunnable, and callable.

From exploration to artifact.

pip install fenic

Writing fenic with an AI coding agent? Run fenic skill install so Claude Code / Cursor / Codex write it correctly, and fenic check to lint it.

What is fenic?

fenic is a semantic DataFrame engine. You write the PySpark/SQL-style operations you already know — select, filter, join, group_by, agg — alongside semantic operators that call language models as a first-class part of the query. You configure models once on a Session, build a pipeline lazily, and fenic compiles and runs it on a query engine built for inference: automatic batching, rate limiting, retries, token/cost accounting, and response caching.

Two ideas make it different from gluing an LLM onto pandas:

• Inference lives inside the query model. Extraction, classification, summarization, and embeddings are operators with schemas and types — not side calls you orchestrate by hand.
• The pipeline is the artifact. Because the work is expressed as typed operators, it's already inspectable (row-level lineage, explain, per-query metrics), rerunnable (lazy plans + caching), and promotable into a named table, view, or MCP tool an agent can call.

60 seconds: messy text → typed rows

Replace brittle parsing and one-off prompts with a typed, schema-bound operator. Define the shape you want as a Pydantic model; fenic returns structured columns you can query.

import fenic as fc
from pydantic import BaseModel, Field

class Ticket(BaseModel):
    product: str = Field(description="The product the user is asking about")
    sentiment: str = Field(description="positive, neutral, or negative")
    issue: str = Field(description="One-line summary of the user's problem")

session = fc.Session.get_or_create(
    fc.SessionConfig(
        app_name="quickstart",
        semantic=fc.SemanticConfig(
            language_models={
                "mini": fc.OpenAILanguageModel(model_name="gpt-4o-mini", rpm=500, tpm=200_000)
            },
        ),
    )
)

df = session.create_dataframe([
    {"id": 1, "text": "The CSV export in Reports keeps timing out since the last update."},
    {"id": 2, "text": "Love the new dashboard, but SSO login is broken on mobile."},
])

# Free text -> typed, queryable rows
tickets = (
    df.select("id", fc.semantic.extract("text", Ticket).alias("t"))
      .unnest("t")
)
tickets.show()

Why fenic?

Unstructured data is everywhere, and working with it is brittle. Teams reach for regex, one-off scripts, notebooks, and prompt chains to pull meaning out of documents, logs, tickets, transcripts, and traces. The results are hard to reproduce and hard to inspect.

Agents made exploration easy and introduced a new problem. An agent can dig through messy data and find something useful — but unless that discovery becomes code, data, or a pipeline, it dies as a chat transcript. The next person has to reverse-engineer what happened.

fenic gives semantic data work a DataFrame abstraction. Express the exploration as fenic operators and it's already the artifact.

Featured workflow: from eval exploration to durable eval intelligence

import fenic as fc
from typing import Literal
from pydantic import BaseModel, Field

class FailureMode(BaseModel):
    failed: bool = Field(description="Whether the agent failed the task")
    category: Literal["tool_error", "instruction_following", "retrieval", "reasoning", "none"] = Field(
        description="Primary failure category, or 'none' if the run succeeded"
    )
    evidence: str = Field(description="Short quote or summary justifying the classification")

session = fc.Session.get_or_create(
    fc.SessionConfig(
        app_name="eval_triage",
        semantic=fc.SemanticConfig(
            language_models={
                "mini": fc.OpenAILanguageModel(model_name="gpt-4o-mini", rpm=500, tpm=200_000)
            },
        ),
    )
)

traces = session.read.docs("eval_runs/**/*.json", content_type="json", recursive=True)

failures = (
    traces
    .with_column("analysis", fc.semantic.extract(fc.col("content").cast(fc.StringType), FailureMode))
    .unnest("analysis")
    .filter(fc.col("failed"))
)

failure_modes = failures.group_by("category").agg(
    fc.count("*").alias("n"),
    fc.semantic.reduce(
        "Summarize the common root cause across these failures",
        column=fc.col("evidence"),
    ).alias("pattern"),
)

failure_modes.write.save_as_table("failure_modes", mode="overwrite")

Query meaning and metadata together

# Match on meaning, not exact values
matches = candidates.semantic.join(
    roles,
    predicate=(
        "Candidate background: {{ left_on }}\n"
        "Role requirements: {{ right_on }}\n"
        "The candidate is a strong fit for the role."
    ),
    left_on=fc.col("resume"),
    right_on=fc.col("job_description"),
)

# ...then group, aggregate, and rank with ordinary DataFrame ops
ranked = (
    matches.group_by("role_id")
    .agg(fc.count("*").alias("n_candidates"))
    .order_by(fc.desc("n_candidates"))
)

Make it an artifact your agents reuse

from fenic import SystemToolConfig

session.catalog.set_table_description(
    "failure_modes", "Recurring agent failure modes with counts and root-cause summaries"
)

server = fc.create_mcp_server(
    session,
    "Eval Intelligence",
    system_tools=SystemToolConfig(
        table_names=["failure_modes"],
        tool_namespace="evals",
        max_result_rows=100,
    ),
)

fc.run_mcp_server_sync(server, transport="http", port=8000)

Semantic operators

Inspect and operate

• df.explain() — the logical/physical plan
• df.lineage() — trace specific rows forwards and backwards
• Per-query metrics — tokens and cost
• Caching — an LLM response cache plus .cache()