On 18 November 2025, our paper **“FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance”** will be presented at the 6th ACM International Conference on AI in Finance (ICAIF ’25) in Singapore. FAITH is our attempt to answer a simple but uncomfortable question:

> When an AI reads your financial statements, how often is it *quietly wrong*?

In this blog, we share the motivation behind FAITH, how the framework works, and what we learned when we put today’s leading large language models (LLMs) under stress.

## Why hallucinations are especially dangerous in finance

LLMs are already being used to read annual reports, draft research notes, answer client questions, and help analysts navigate complex disclosures. They are remarkably good at turning long documents into concise explanations.

But finance is unforgiving:

- A misplaced decimal or wrong unit (million vs billion) can completely change the story.
- Many important numbers are not written in plain text – they sit inside dense tables.
- Decisions must be traceable: “where did this number come from?” is not a rhetorical question for regulators and risk managers.

When an LLM produces an answer that **sounds** plausible but does not actually match the underlying table or text, we call this an **intrinsic hallucination** – the model contradicts the context it was given. FAITH focuses precisely on this type of failure.

## What FAITH actually measures

Most existing hallucination benchmarks test models on open‑domain facts (often built from Wikipedia) or simple question‑answering over short passages. Financial work is different: we care about **numbers grounded in real filings**, often involving multi‑step calculations over tables.

FAITH is built around three principles:

1. **Real documents, not toy examples**  
   We build the benchmark directly from **2024 10‑K annual reports of S&P 500 companies**. These filings are long, table‑heavy, and subject to regulatory scrutiny – exactly the kind of documents financial institutions use every day.

2. **Masked‑span prediction instead of Q&A**  
   Rather than asking open questions, FAITH turns the problem into a *fill‑in‑the‑blank* task:

   - We find sentences in the “Management’s Discussion and Analysis” section that mention financial quantities (e.g. “Revenue increased by **$11.30 billion** or **17%** in 2023…”).
   - We then **mask** one of the numeric spans:  
     “Revenue increased by **[MASK]** or 17% in 2023…”
   - The model receives:
     - the corrupted sentence,
     - nearby text,
     - and the relevant financial tables.
   - Its job is to recover the missing value based *only* on this context.

   If the model fills in the blank with the wrong value or wrong unit, that counts as a hallucination with respect to the document.

3. **Automatic, scalable dataset creation**  
   Manually labelling thousands of such examples would be painfully slow. Instead, FAITH uses a carefully designed pipeline:

   - Detect numeric spans and their associated units (%, $, million, per share, etc.).
   - Filter out spans that are ambiguous, inconsistent, or impossible to recover from the tables.
   - Use a committee of strong LLMs to help decide whether a span is *answerable* from the given context, and keep only those with strong agreement.
   - Normalize numbers to a shared scale (e.g. “1.23 billion” vs “1,230 million”) so equivalent answers are not unfairly penalized.

   The result is a **large‑scale, realistic benchmark** that can be recreated on proprietary documents inside an institution’s own perimeter.

<img src="/images/blog/pipeline.png" alt="Pipeline for constructing FAITH masked-span tasks from 10-K filings" data-caption="*High-level FAITH pipeline: starting from 10-K reports, we extract sentences and numerical spans, have LLMs judge which spans are answerable, format masked-span inputs, pull supporting tables as context, and evaluate model predictions against ground-truth values.*" />

## Four types of financial reasoning we test

Not all numeric questions are created equal. To understand *where* models struggle, we categorize each masked span into one of four reasoning scenarios:

1. **Direct Lookup (A)**  
   The answer is simply copying a value from a single cell in a table.  
   *Example:* “Total revenue in 2024 was [MASK].”

2. **Comparative Calculation (B)**  
   The answer requires a simple comparison over time or category.  
   *Example:* “Revenue increased by [MASK]% year‑over‑year,” when revenues for 2023 and 2024 are both present in the table.

3. **Bivariate Calculation (C)**  
   The answer is computed from **two** different metrics.  
   *Example:* “Gross margin was [MASK]%,” combining revenue and cost of goods sold.

4. **Multivariate Calculation (D)**  
   The most demanding case: multi‑step formulas involving several variables and latent quantities.  
   *Example:* computing the equity portion of an investment when only the total purchase price, leverage, and ownership share are given across text and tables.
   
<img src="/images/blog/scenarios.png" alt="Toy income statement illustrating the four FAITH reasoning scenarios" data-caption="*In this toy income statement, **A** is a direct lookup ($x_1$), **B** a comparative calculation over time ($y_1$ vs $y_2$), **C** a bivariate calculation combining taxable income and the effective tax rate to recover $k_2$, and **D** a multivariate calculation chaining several quantities across the table.*" />

By tracking accuracy across these scenarios, FAITH can tell you not only *how often* a model hallucinates, but *how quickly* things break down as the reasoning chain becomes more complex.

## What we found when we tested today’s LLMs

We evaluated a mix of proprietary and open‑source models, from lightweight 8B‑parameter models to frontier‑scale systems.

### 1. A clear hierarchy of reliability

- **Top‑tier proprietary models** (e.g. Claude‑Sonnet‑4, Gemini‑2.5‑Pro) reach **around 92–96% accuracy** on our main benchmark split.
- **Strong general models** like GPT‑4.1 and GPT‑4.1‑mini follow closely behind.
- **Many smaller or mid‑sized open‑source models** fall far short, often below 50% accuracy overall and near zero on the hardest reasoning cases.

In other words: even among very capable models, there is a measurable and meaningful gap in how faithfully they handle financial tables.

<img src="/images/blog/overall_accuracy.png" alt="Overall model accuracy on FAITH pilot and main splits" data-caption="*Overall accuracy by model on the pilot and main FAITH splits: a handful of frontier systems cluster in the 90%+ range, while many smaller open-source models lag far behind, especially on the main split.*" />

### 2. Complexity is the main failure driver

For almost every model, performance follows the same pattern:

> Direct lookups are relatively easy; multi‑step quantitative reasoning is where hallucinations explode.

Models do quite well when they only need to copy a number from a table. But as soon as we ask them to combine multiple figures, translate between percentages and absolute values, or reason across both text and tables, error rates rise sharply.

On the **Multivariate Calculation** scenario, many models completely collapse, mis‑computing key quantities or ignoring parts of the context altogether.

<img src="/images/blog/accuracy_by_scenario.png" alt="Accuracy by reasoning scenario from A (direct lookup) to D (multivariate)" data-caption="*Accuracy by reasoning scenario: performance is highest on direct lookups (A) and declines steadily through comparative (B) and bivariate (C) cases, before dropping sharply on multivariate reasoning (D).*" />

### 3. Scale errors are surprisingly common

One of the most striking error types we observed is the **scale mismatch**:

- The model identifies the *right* number, but attaches the *wrong magnitude* – for example, answering “$150” instead of “$150 million”.
- These mistakes are easy to miss in a quick skim, yet devastating in practice.

When we hypothetically “fixed” only scale errors in some models, their accuracy jumped by 15–20 percentage points. This suggests that explicit handling of units and scales remains a major open problem in LLM design and prompting.

### 4. A case study: who really understands the capital structure?

For one challenging example, the model had to infer the equity investment in a shopping centre from:

- an ownership percentage,
- a pro‑rata share of mortgage debt in a table, and
- the total purchase price described in the text.

Only one frontier model successfully reconstructed the full chain:

1. Backed out the total debt from the pro‑rata amount and ownership share.
2. Subtracted this from the purchase price to get total equity.
3. Applied the fund’s ownership percentage to arrive at the correct equity investment.

Most other models simply multiplied the purchase price by the ownership share, ignoring leverage entirely – a plausible‑sounding but financially wrong answer. This illustrates how **intrinsic hallucinations can stem from shallow “pattern matching” rather than true financial reasoning.**

## Why FAITH matters for financial institutions

For banks, asset managers, insurers and regulators, FAITH offers two things:

1. **A realistic benchmark**  
   Because the dataset is built from real S&P 500 10‑Ks, performance on FAITH is much more indicative of how a model will behave when exposed to actual reporting pipelines, not just curated academic examples.

2. **A reusable methodology**  
   The same pipeline we used to build FAITH can be applied to *your* own documents:

   - ingest internal credit memos, risk reports, or client decks,
   - generate masked‑span tasks automatically,
   - and measure how faithfully candidate models recover numbers and units.

This makes FAITH not just a static dataset, but a **general recipe for in‑house hallucination testing** under your own definitions of “what matters”.

## Try FAITH yourself

We have open‑sourced:

- the **benchmark dataset**,  
- the **data construction pipeline**, and  
- example **evaluation scripts** for popular LLMs.

If you work with financial AI and care about model risk, we invite you to:

- run FAITH on your own models,
- adapt the pipeline to your proprietary documents,
- and share feedback or extensions with us.

You can find more details and links to the data and code here:

- **Project page:** <https://zhang-mengao.github.io/FAITH/>
- **GitHub repo:** <https://github.com/zhang-mengao/FAITH>


Together, we hope FAITH can help move the conversation about “AI in finance” from *demo‑level excitement* to **measured, evidence‑based trust**.


FAITH: A Novel and Scalable Way to Measure Hallucinations in Financial AI



## **Anatomy of a REITs Research Report**

Real Estate Investment Trusts (REITs) own or finance income-producing real estate and distribute most of their taxable income as dividends. Because they are inherently yield vehicles, investors assess them through cash-flow stability, balance-sheet resilience, and valuation versus peers and bonds. A good REIT report therefore combines property fundamentals, financing structure, and pricing logic to judge whether the income is sustainable and fairly valued. The goal is not just to list metrics, but to guide a reader from a stated call to the evidence that supports it.

A clear, logical flow helps turn data into conviction. A professional REIT report should be structured around how investors actually evaluate income-producing assets — focusing on **cash-flow stability**, **financial resilience**, and **valuation discipline**. Think of the report as a courtroom brief: start with the verdict, then present the facts in an order that makes the conclusion feel inevitable.

### **1. Executive Summary & Thesis**

Start with a concise investment call **(Buy / Hold / Sell)** and a one-paragraph rationale that foreshadows the evidence to come. Mention the price target, implied upside/downside, and how the stated catalysts or risks will move the distribution profile.

Highlight key drivers such as yield outlook, asset quality, leverage trend, and macro sensitivity. In two to three sentences, preview what will be proven later: “We rate Buy because improving logistics occupancy offsets higher funding costs, keeping the 6.2% yield secure while P/NAV trades at a 10% discount.”


### **2. Revenue & Cash Flow Analysis**

Focus on how the portfolio generates stable and visible distributable income, linking operational trends back to DPU visibility. Move from top-down (asset mix, geography) to bottom-up (tenant quality, lease terms), then tie these to the trajectory of cash flows after capex and interest.

**Key analytical angles:**

- **Portfolio Mix:** geography, sector exposure, and asset quality
- **Tenant Profile:** concentration risk and WALE (Weighted Average Lease Terms)
- **Occupancy & Rent Reversion:** forward growth visibility
- **DPU Trend:** stability and payout consistency

Use visuals such as *Revenue by Segment* or *Top Tenants by Contribution* to illustrate diversification and risk exposure. Clarify whether cash flows are contracted (long WALE, built-in escalations) or volatile (short leases, index-linked rents), and quantify sensitivity to vacancy or rent reversions.

***Example:  Portfolio Breakdown (Source: DBS)***

![image.png](/images/blog/image.png)

### **3. Liquidity & Financial Position**

Assess the REIT’s financial position and leverage headroom, linking the balance sheet to distribution sustainability. Explain whether today’s leverage supports growth (acquisitions, AEI) or forces deleveraging (asset sales, dividend cuts).

**Core indicators:**

- **Gearing Ratio:** debt headroom for acquisitions
- **Interest Coverage:** resilience to higher borrowing costs
- **Debt Maturity Profile:** refinancing timing risk
- **Fixed vs Floating Debt:** sensitivity to rate movements

Supplement data with **management commentary** — e.g., refinancing intentions or capital recycling plans — to determine the future leverage roadmap. Spell out rate sensitivities (“A 50 bp hike cuts DPU by ~2% given 60% floating-rate exposure”) and any covenant buffers.


### **4. Valuation Framework**

Combine **absolute** and **relative** methods to form a rounded view. Present valuation as the bridge between fundamentals and the call: how the quality of cash flow and balance-sheet risk translates into yield, P/NAV, or target price. Show why your assumptions are credible versus consensus and history.

### (a) Absolute Valuation — Dividend Discount Model (DDM)

For a **multi-stage DDM**, the price today is the sum of discounted dividends over the explicit forecast period plus the discounted terminal value:

$$
P_0 = \sum_{t=1}^{N} \frac{D_t}{(1 + r)^t} + \frac{\frac{D_{N+1}}{r - g}}{(1 + r)^N}

$$

Where: $P_0$ = intrinsic value today; $D_t$ = dividend (or DPU) in year $t$; $r$ is the cost of equity (often estimated via CAPM); $g$ is perpetual growth rate after year $N$.

The terminal value after the forecast horizon is based on a perpetual growth assumption:

$$
TV = \frac{D_{N+1}}{r - g}
$$

**Key inputs:**

- Forecast horizon: 5–10 years
- Cost of equity ($r$): often via CAPM
- Perpetual growth ($g$): typically 1–2 %

Run sensitivity tests on $r$ and $g$ to show valuation range.

### (b) Relative Valuation — Peer Comparison

Benchmark against peers using:

| Metric | Formula | Interpretation |
| --- | --- | --- |
| **P/NAV** | Price ÷ Net Asset Value | Premium/discount to book |
| **Dividend Yield** | DPU ÷ Market Price | Income return vs peers |
| **Cap Rate** | NOI ÷ Asset Value | Asset-level yield |

### **5. Investment Thesis & Risks**

Conclude by integrating financial, operational, and qualitative insights so the recommendation feels earned. Link the drivers back to the target price (“higher occupancy supports mid-cycle rent reversions, justifying a 6.0% cost of equity and 1–2% terminal growth”). Make explicit what could break the thesis and how fast you would change the call.

**Typical risks:** interest-rate hikes, tenant defaults, FX exposure, or sector oversupply.

**Typical mitigants:** long WALE, fixed-rate debt, high-quality sponsors.

## **Writing Tips for Professional REIT Reports**

### **1. Focus on Cash-Flow Sustainability over Growth**

The key to REIT analysis is **income sustainability**, not revenue expansion. Because most REITs operate mature, yield-generating assets, investors care less about top-line growth and more about **whether distributions are sustainable** under different conditions.

***Example:*** Instead of saying “Rental income grew 5% YoY,” explain *why* — e.g., “Rental income rose due to higher occupancy in logistics assets and positive rent reversions, supporting stable DPU despite higher financing costs.”

### **2. Integrate Quantitative and Qualitative Insights**

A professional report blends metrics with management guidance. Numbers alone don’t tell the whole story — interpret what management actions imply for future cash flow or risk exposure.

***Example:  Combine Management’s Expectation with Financial Data (Source: UBS)***

![image.png](/images/blog/image%201.png)

![image.png](/images/blog/image%202.png)

### **3. Use Peer Comparisons Effectively**

Benchmarking provides valuation context. Compare the REIT’s **P/NAV**, **Dividend Yield**, and **Gearing** against sector peers to highlight mispricing or relative resilience.

***Example:  Company Data vs Market (Source: UBS)***

![image.png](/images/blog/image%203.png)

### **4. Maintain Analytical Neutrality**

Institutional readers expect objective tone. Avoid promotional or emotional language; instead, anchor conclusions in verifiable data and market consensus.

***Example:*** Replace “The REIT has an excellent growth outlook” with “Portfolio rental reversions turned positive after three quarters of decline, indicating early signs of recovery.”

### **5. Craft a Cohesive Narrative**

Each analytical section — from revenue to valuation — should reinforce the same central argument: *Is the REIT’s income sustainable, and is the yield justified by risk?*

***Example:*** Link the pieces together: “Stable occupancy and a longer debt maturity profile underpin DPU visibility, while a 50bp yield spread over peers offers moderate upside potential.”

## **Final Thoughts**

A professional REIT report is more than a compilation of ratios — it is a **structured narrative of income sustainability**. The best reports connect operational metrics, management actions, and market valuations into a single story that helps investors understand *how distributions today can be preserved tomorrow*.

> In essence, clarity, consistency, and conviction define the quality of REIT research. Numbers inform the view, but structure and reasoning make it credible.
>


How to Write a Professional REITs Research Report: From Structure to Valuation Logic



<div style="display:flex;justify-content:center;align-items:center;gap:clamp(12px,5vw,28px);flex-wrap:nowrap;margin:2rem auto;background:#f5f7fb;border-radius:16px;padding:clamp(12px,4vw,24px);max-width:min(100%,860px);box-sizing:border-box;">
  <img
    data-inline="true"
    src="/images/blog/NUS%20AIDF%20logo.png"
    alt="NUS AIDF"
    style="height:clamp(40px,12vw,80px);width:auto;flex:0 1 auto;min-width:0;display:inline-block;object-fit:contain"
  />
  <img
    data-inline="true"
    src="/images/blog/ACM%20ICAIF%20logo.png"
    alt="ACM ICAIF 2025"
    style="height:clamp(40px,12vw,80px);width:auto;flex:0 1 auto;min-width:0;display:inline-block;object-fit:contain"
  />
</div>

I'm happy to share a good news: our paper **"FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance"** has been accepted to the **ACM International Conference on AI in Finance (ICAIF 2025)** for oral presentation, which will be held in Singapore.

This work is done by **CAESARS** team at **Asian Institute of Digital Finance, National University of Singapore**, and it sits at the intersection of two things we care about a lot: **trustworthy AI** and **reliable financial information**.

## Why this matters

Today, large language models (LLMs) are increasingly used to read financial reports, summarise company performance, and answer questions about earnings, risk, and growth. They can be very helpful—but they also sometimes **“hallucinate”**: confidently giving answers that sound right but are actually wrong.

In finance, even a small mistake in a percentage or a billion vs million can matter a lot. Our work looks at this problem from a simple angle:

> When an AI model is given a real company’s financial tables, can it stick to the facts that are **already there**, instead of making numbers up?

FAITH is our attempt to systematically test this behaviour on real-world financial filings. Just as importantly, the way we build the dataset is scalable: with relatively little manual work, anyone can plug in their own reports and generate a similar hallucination-assessment dataset. For enterprises, this makes it much easier to quickly construct in-house test sets and gauge the risk of adopting AI models in their day-to-day workflows.

## An open dataset for financial AI

To make this useful beyond our own group, we have **open-sourced the FAITH dataset**.

In plain terms, FAITH is a curated collection of:

- Real **S&P 500 companies’ annual reports (10‑K filings)**
- The financial tables inside those reports
- Short snippets of text that refer to numbers in those tables
- The **“correct” numbers** that the model is supposed to recover

If you are building or evaluating AI models for finance, FAITH gives you a way to check how often a model stays true to the original tables vs. drifting into made‑up numbers.

You can find more details and links to the data and code here:

- **Project page:** <https://zhang-mengao.github.io/FAITH/>
- **GitHub repo:** <https://github.com/zhang-mengao/FAITH>


## A quick look at the numbers

<figure style="max-width:60%;margin:0 auto;">
  <img src="/images/blog/statistics.png" alt="…" style="width:100%;height:auto;" />
</figure>

Here are a few headline stats about the FAITH dataset:

- It covers **hundreds of S&P 500 companies** from their latest 10‑K filings.
- It includes **over two thousand short text snippets** where the discussion in the report refers to specific numbers in the tables.
- All in, there are **a few thousand individual data points** (number‑and‑unit pairs) that can be used to test whether an AI model reads and uses the tables correctly.

In other words, FAITH is large enough to stress‑test modern AI models, but focused enough to stay close to real, everyday financial analysis tasks.

## Want to dive deeper?
This post is just a high-level overview. For a more detailed walkthrough of FAITH—the research questions behind it, how we build the dataset step by step, and what we learn from benchmarking different models—please see [this in‑depth blog post](https://report.nuscri.org/blog/FAITH:%20A%20Novel%20and%20Scalable%20Way%20to%20Measure%20Hallucinations%20in%20Financial%20AI)

---

Thanks to everyone who contributed to this project. If you’re interested in trustworthy AI in finance, feel free to explore the dataset, run your own experiments, or reach out—we’d love to hear what you discover.


CAESARS Blog

FAITH: A Novel and Scalable Way to Measure Hallucinations in Financial AI

How to Write a Professional REITs Research Report: From Structure to Valuation Logic

Our FAITH paper is going to ACM ICAIF 2025 🎉