Data Validation
Develop comprehensive data validation rules and tests for data pipelines
Prompt template
You are an experienced data quality engineer. Develop a complete data validation framework for the following use case. Dataset and context: [DATASET – e.g., daily product catalog import from ERP system, ~10,000 records] Schema and columns: [SCHEMA – Column names, data types, required fields, e.g., product_id (VARCHAR, required), price (DECIMAL, >0), category (ENUM)] Critical business rules: [BUSINESS_RULES – e.g., price cannot be negative, product_id must be unique, stock_quantity ≥ 0] Validation framework: [FRAMEWORK – e.g., Great Expectations, dbt tests, Pydantic, SQL Constraints, Python unittest] Please provide: 1. Complete validation rules for schema, format, and value checks 2. Referential integrity checks for relationships between tables 3. Statistical plausibility checks (e.g., volume deviations greater than 20% vs. prior day trigger an alert) 4. Executable code for all validations in the chosen framework 5. A reporting format for validation results with clear error messages and recommended actions
How to use this template
Copy the prompt, replace the placeholders in square brackets with your details and paste it into the AI assistant of your choice. The more precise your details, the better the result.
Tips for better results
- Distinguish between critical errors (stop pipeline) and warnings (log but continue processing).
- Define thresholds for statistical deviations based on historical data.
- Plan validations both at data ingestion and before loading into the target system.
Suitable open models
The selection names current open models as a starting point for your own tests. It is not a measurement for this template.
Keywords: #validation #data-quality #great-expectations #testing