Twenty questions to ask a data catalog vendor about AI agents

Vendor demos in this category look alike: a search box, a lineage graph, a chat window. The differences show up in specific questions. These twenty follow the nine criteria in our scoring model, so you can compare answers with our scores and with each other.

1. Which tools does your MCP server expose, by name, and which agent clients have you tested with?

2. Is the MCP server generally available, or in preview or early access?

3. Besides MCP, which APIs or SDKs can our agents call, and are there call limits by plan?

4. Show column-level lineage for one of our metrics, from source columns through dbt to the BI field. Where does it stop?

5. Do you capture logic inside BI calculated fields, LookML or workbook formulas?

6. What is mapped automatically on day one, and what needs someone to write or approve it?

7. How long after we rename a column does the change reach what an agent receives?

8. Can we express rules such as certified sources only or no personal data as automatic labels, and do they update when lineage changes?

9. How is context scoped to the person or task an agent is serving?

10. Can you show which data an agent touched to produce a given answer?

11. Do you hold metric definitions, extract them, or read them from a semantic layer we already have?

12. When two definitions conflict, how does an agent know which one is certified?

13. Do you provide a way to run benchmark questions and score agent answers over time?

14. How do you detect context drift, and who is notified?

15. Which of our systems are covered by maintained connectors, and are integrations capped by plan?

16. What does a custom connector cost in time and money?

17. What is the pricing unit, and what is the price at our size today and at three times that size?

18. What happens at a tier boundary: a step change, or a conversation?

19. Can we self-host, run in our cloud account or use a marketplace listing?

20. Exactly which metadata do you store, where, for how long, and do you ever read rows?

Some answers deserve a follow-up. A vendor that cannot name the tools on its MCP server may not have one in general availability. A lineage demo that only uses the vendor's sample data tells you nothing about your BI tool. Governance that requires someone to apply every label by hand will drift. A price that is only available after the proof of concept leaves you negotiating after you have invested time. None of these is disqualifying on its own, but each should be written down and resolved before a decision.

Score each answer on the same 0 to 10 scale we use, with a one-line reason, and keep the reasons. When a vendor says a feature exists, ask for the documentation page; when it says a feature is coming, record it as not available. Our own scores come only from public pages, so a vendor may answer better in a call than its pages suggest. That is useful to know, and it should still be written down.

Several of these questions came straight from gaps in our research. For example, column-level lineage was not stated on Atlan's connectors page, deployment options were not stated on the Alation and Collibra pages we read, and Euno caps integrations by plan. A good vendor will answer these directly.

Source: Atlan connectors

Source: Alation homepage

Source: Collibra MCP Server

Source: Euno pricing

Send the questions in writing a week before the first technical call, and ask for written answers with links. It saves a meeting, and it gives you a record to compare against the proof of concept. Vendors that answer in writing tend to answer more precisely, because the answer can be checked.