Time Series Forecasting for Karnataka Mandi Prices
Time Series
Deep Learning
Agriculture
Forecasting
Transformer
PatchTST
This project develops a deep learning-based forecasting system for agricultural commodity prices across Karnataka mandis (markets). Using historical price data spanning 2015-2025, the system employs PatchTST, a state-of-the-art transformer architecture, to predict 30-day forward prices for 1,065 unique market-commodity-variety combinations.
Dataset Size
3.14M
Time Range
2015-2025
Model Name
PatchTST
Forcast Horizon
30 Days
Dataset Size
3.14M
Time Range
2015-2025
Model Name
PatchTST
Forcast Horizon
30 Days
Dataset Size
3.14M
Time Range
2015-2025
Model Name
PatchTST
Forcast Horizon
30 Days
Project Framing
Agricultural commodity pricing in India is characterized by high volatility, information asymmetry, and limited price transparency. Farmers, traders, and policymakers lack reliable tools to anticipate market movements, leading to suboptimal decision-making.
This project aims to democratize price intelligence by building an accurate, scalable forecasting system using publicly available mandi (market) data from Karnataka state. The system targets multiple stakeholders:
Farmers
Planning harvest timing and crop selection
Traders
Inventory management and procurement decisions
Policymakers
Market intervention and price stabilization
Researchers
Understanding agricultural market dynamics
Farmers
Planning harvest timing and crop selection
Traders
Inventory management and procurement decisions
Policymakers
Market intervention and price stabilization
Researchers
Understanding agricultural market dynamics
Farmers
Planning harvest timing and crop selection
Traders
Inventory management and procurement decisions
Policymakers
Market intervention and price stabilization
Researchers
Understanding agricultural market dynamics
Problem Statement
Primary Challenge: Develop a time series forecasting model that can accurately predict modal (most common) prices for agricultural commodities 30 days into the future, across diverse market-commodity combinations with varying data quality and trading patterns.
Key Challenges:
Serve data missingness (up to 80% in some series)
Irregular trading patterns (markets don't operate every day)
Heterogeneous time series (different commodities, markets, seasonality)
Need for scalable solution across 1,065+ series
Maintaining temporal consistency during imputation
Objective & Success Criteria
Primary Objective
Build a production-ready forecasting system capable of generating 30-day price predictions for all active market-commodity combinations with acceptable accuracy and computational efficiency.
Success Criteria
Accuracy
Mean Absolute Percentage Error (MAPE) < 15%
Data Quality
Successfully impute 80%+ of missing values
Scalability
Train on 1,000+ series within 24 hours
Robustness
Handle series with varying lengths and patterns
Accuracy
Mean Absolute Percentage Error (MAPE) < 15%
Data Quality
Successfully impute 80%+ of missing values
Scalability
Train on 1,000+ series within 24 hours
Robustness
Handle series with varying lengths and patterns
Accuracy
Mean Absolute Percentage Error (MAPE) < 15%
Data Quality
Successfully impute 80%+ of missing values
Scalability
Train on 1,000+ series within 24 hours
Robustness
Handle series with varying lengths and patterns
Literature Review
Time series forecasting for agricultural prices has evolved significantly with the advent of deep learning. Traditional approaches relied on statistical methods like ARIMA and seasonal decomposition, which struggle with non-linear patterns and multivariate dependencies.
Deep Learning for Time Series
Architectures such as LSTM networks, Temporal Convolutional Networks (TCN), and Transformer models have demonstrated superior capability in capturing long-range dependencies using self-attention mechanisms.
Patch-based Transformers
PatchTST (Nie et al., 2023) introduced patch-based segmentation of time series, treating subsequences as tokens similar to Vision Transformers.
Reduced computational complexity
Improved local feature learning
Channel independence for multivariate series
State-of-the-art benchmark performance
Agricultural Market Intelligence
Domain research emphasizes seasonality, weather integration, and regional market variability. However, scalability remains limited, as most studies focus on single commodities or isolated markets.
Literature Review
Time series forecasting for agricultural prices has evolved significantly with the advent of deep learning. Traditional approaches relied on statistical methods like ARIMA and seasonal decomposition, which struggle with non-linear patterns and multivariate dependencies.
Deep Learning for Time Series
Architectures such as LSTM networks, Temporal Convolutional Networks (TCN), and Transformer models have demonstrated superior capability in capturing long-range dependencies using self-attention mechanisms.
Patch-based Transformers
PatchTST (Nie et al., 2023) introduced patch-based segmentation of time series, treating subsequences as tokens similar to Vision Transformers.
Reduced computational complexity
Improved local feature learning
Channel independence for multivariate series
State-of-the-art benchmark performance
Agricultural Market Intelligence
Domain research emphasizes seasonality, weather integration, and regional market variability. However, scalability remains limited, as most studies focus on single commodities or isolated markets.
Shortfalls in Existing Approaches
Limitation
Impact
Statistical models (ARIMA, ETS)
Cannot capture non-linear patterns; poor with missing data
Historical mandi (agricultural market) price data from Karnataka state, sourced from government repositories. The dataset covers daily modal prices reported by market committees across the state.
Data Schema
Field
Type
Description
Arrival_Date
DateTime
Trading date
State
String
Karnataka
District
String
Administrative district
Market
String
Market/mandi name
Commodity
String
Crop/product name
Variety
String
Commodity variety/cultivar
Grade
String
Quality grade (FAQ, Medium, etc.)
Modal_Price
Float
Most common price in rupees/quintal
Latitude/Longitude
Float
Geographic coordinates
Data Processing Pipeline
The data undergoes a multi-stage processing pipeline to prepare it for modeling:
01
Data Cleaning
Objective: Filter out low-quality time series that would compromise model training.
Process: 1. Group data by (State, District, Market, Commodity, Variety, Grade) 2. Calculate missing data percentage for each series 3. Filter series where: • Missing data ≤ 80% • Total days ≥ 1004. 4. Result: Retained 1,065 series (representing high-quality data)
02
Date Expansion
Objective: Create continuous daily timeline for each series to enable uniform forecasting.
Process:
• For each series, create date range from min to max date
• Fill categorical features (State, Market, etc.) using forward/backward fill
• Leave Modal_Price as NaN for missing dates (to be imputed)
• Result: Uniform daily grid for all series
03
Data Imputation
Objective: Fill missing price values intelligently while respecting temporal patterns.
Algorithm: Gap-Aware Linear Interpolation
• For gaps ≤ 7 days: Linear interpolation between known values
• For gaps > 7 days: Forward fill from last known value
• Rationale: Short gaps likely represent continuous market conditions; long gaps may indicate structural changes
• Add 'is_traded' flag to distinguish actual vs imputed values
04
Smoothing
Objective: Reduce noise in imputed values to prevent unrealistic price jumps.
Process:
• Apply 7-day rolling median filter to imputed sequences
• unique_id: Concatenation of Commodity_Variety_Grade_State_District_Market
• dayofweek: Day of week (0=Monday, 6=Sunday)
• month: Month of year (1-12)
• year: Calendar year
• is_traded: Binary flag (True=actual price, False=imputed)
01
Data Cleaning
Objective: Filter out low-quality time series that would compromise model training.
Process: 1. Group data by (State, District, Market, Commodity, Variety, Grade) 2. Calculate missing data percentage for each series 3. Filter series where: • Missing data ≤ 80% • Total days ≥ 1004. 4. Result: Retained 1,065 series (representing high-quality data)
02
Date Expansion
Objective: Create continuous daily timeline for each series to enable uniform forecasting.
Process:
• For each series, create date range from min to max date
• Fill categorical features (State, Market, etc.) using forward/backward fill
• Leave Modal_Price as NaN for missing dates (to be imputed)
• Result: Uniform daily grid for all series
03
Data Imputation
Objective: Fill missing price values intelligently while respecting temporal patterns.
Algorithm: Gap-Aware Linear Interpolation
• For gaps ≤ 7 days: Linear interpolation between known values
• For gaps > 7 days: Forward fill from last known value
• Rationale: Short gaps likely represent continuous market conditions; long gaps may indicate structural changes
• Add 'is_traded' flag to distinguish actual vs imputed values
04
Smoothing
Objective: Reduce noise in imputed values to prevent unrealistic price jumps.
Process:
• Apply 7-day rolling median filter to imputed sequences
• unique_id: Concatenation of Commodity_Variety_Grade_State_District_Market
• dayofweek: Day of week (0=Monday, 6=Sunday)
• month: Month of year (1-12)
• year: Calendar year
• is_traded: Binary flag (True=actual price, False=imputed)
Exploratory Data Analysis
Key Findings
Metric
Value
Total Records
3,141,486
Unique Series
1,065
Date Range
2015-01-01 to 2025-09-09
Avg. Missing %
~54% (pre-imputation)
Price Range
₹100 - ₹30,000 per quintal
Insights
Strong seasonality in vegetables (tomato, onion) with 3-4 month cycles
Staple grains (rice, wheat) show longer-term trends with less volatility
Cash crops (cotton, areca nut) exhibit high variance
Weekend and festival effects visible in trading patterns
Geographic clustering: nearby markets show correlated price movements
Modeling Strategy
Selected Model
PatchTST
Patch Time Series Transformer uses patch-based segmentation of time series, treating subsequences as 'tokens' similar to image patches in Vision Transformers. This approach reduces computational complexity, improves local pattern recognition, and achieves state-of-the-art results on benchmark datasets.
Input
180 days
Patching
Convert to patches
Transformer
Self-attention
Output
30-day forecast
Selected Model
PatchTST
Patch Time Series Transformer uses patch-based segmentation of time series, treating subsequences as 'tokens' similar to image patches in Vision Transformers. This approach reduces computational complexity, improves local pattern recognition, and achieves state-of-the-art results on benchmark datasets.
Input
180 days
Patching
Convert to patches
Transformer
Self-attention
Output
30-day forecast
Train-Test Split
Temporal split approach:
• Training set: All data up to 1 year before the most recent date • Test set: Final 1 year of data • Rationale: Simulates real-world forecasting scenario; prevents data leakage
Training Dynamics
Training Observations
Loss converged smoothly after ~5-6 epochs
No significant overfitting observed (train/val loss tracked closely)
GPU acceleration provided ~8x speedup vs CPU
Total training time: ~2-3 hours on Apple M1 MPS
Model Comparison
PatchTST was selected after comparison with baseline approaches:
Model
Strengths
Weaknesses
Status
ARIMA
Interpretable
Poor with non-stationarity
Rejected
LSTM/RNN
Sequential modeling
Vanishing gradients, slow
Rejected
Standard Transformer
Long-range dependencies
Quadratic complexity
Considered
PatchTST
Efficient, patch-based
Requires more data
Selected
Results & Performance
MAE
₹285
MAPE
12.3%
RMSE
₹412
Coverage
100%
MAE
₹285
MAPE
12.3%
RMSE
₹412
Coverage
100%
MAE
₹285
MAPE
12.3%
RMSE
₹412
Coverage
100%
30-Day Price Forecast
Historical
Forecast
30-Day Price Forecast
Historical
Forecast
Strong Predictive Performance
The model demonstrates strong predictive performance across diverse commodity types and markets. With a MAPE of 12.3% (below the 15% target), the system successfully forecasts prices for all 1,065 market-commodity combinations, providing actionable insights for stakeholders.
Deployment Architecture
The forecasting system follows a batch prediction architecture suitable for production deployment:
Layer 1
Data Ingestion Layer
• Scheduled ETL pipeline to fetch latest mandi prices
• Data validation and quality checks
• Storage in PostgreSQL/MongoDB
Layer 1
Data Ingestion Layer
• Scheduled ETL pipeline to fetch latest mandi prices
• Data validation and quality checks
• Storage in PostgreSQL/MongoDB
Layer 2
Processing Pipeline
• Data cleaning and imputation (as described in pipeline)
• Feature engineering
• Preprocessing for model input
Layer 2
Processing Pipeline
• Data cleaning and imputation (as described in pipeline)
• Feature engineering
• Preprocessing for model input
Layer 3
Inference Service
• Model loaded into memory (PyTorch/ONNX runtime)
• Batch prediction for all active series
• GPU acceleration for faster inference
Layer 3
Inference Service
• Model loaded into memory (PyTorch/ONNX runtime)
• Batch prediction for all active series
• GPU acceleration for faster inference
Layer 4
API Layer
• REST API (FastAPI/Flask) for forecast retrieval
• Query by commodity, market, date range
• Response caching for frequently accessed forecasts
Layer 4
API Layer
• REST API (FastAPI/Flask) for forecast retrieval
• Query by commodity, market, date range
• Response caching for frequently accessed forecasts
Layer 5
Monitoring & Updates
• Performance tracking (MAPE drift detection)
• Scheduled model retraining (monthly/quarterly)
• Alerting for data quality issues
Layer 5
Monitoring & Updates
• Performance tracking (MAPE drift detection)
• Scheduled model retraining (monthly/quarterly)
• Alerting for data quality issues
Learnings & Tradeoffs
Key Learnings
Imputation Strategy Matters
Gap-aware imputation (different strategies for short vs long gaps) prevented unrealistic price jumps and improved forecast stability.
Single Model for All Series
Training one PatchTST model on all 1,065 series outperformed individual models, likely due to transfer learning across similar commodities.
Temporal Features Are Critical
Day-of-week and month features significantly improved accuracy by capturing weekly market cycles and seasonal patterns.
Data Quality > Model Complexity
Investing time in rigorous data cleaning and imputation yielded better results than experimenting with more complex architectures.
Tradeoffs & Limitations
Computational Cost
PatchTST requires GPU for practical training times. This limits deployment to environments with GPU access or requires ONNX conversion for CPU inference.
Interpretability
Deep learning models are less interpretable than ARIMA or regression. Stakeholders may require additional explanation layers (SHAP, attention visualization).
Extreme Events
Model struggles with unprecedented shocks (e.g., COVID-19 lockdowns, sudden policy changes). Ensemble with rule-based adjustments could help.
Data Freshness
Forecasts depend on recent data availability. Delayed reporting from mandis reduces forecast accuracy for the first few days.
Key Learnings
Imputation Strategy Matters
Gap-aware imputation (different strategies for short vs long gaps) prevented unrealistic price jumps and improved forecast stability.
Single Model for All Series
Training one PatchTST model on all 1,065 series outperformed individual models, likely due to transfer learning across similar commodities.
Temporal Features Are Critical
Day-of-week and month features significantly improved accuracy by capturing weekly market cycles and seasonal patterns.
Data Quality > Model Complexity
Investing time in rigorous data cleaning and imputation yielded better results than experimenting with more complex architectures.
Tradeoffs & Limitations
Computational Cost
PatchTST requires GPU for practical training times. This limits deployment to environments with GPU access or requires ONNX conversion for CPU inference.
Interpretability
Deep learning models are less interpretable than ARIMA or regression. Stakeholders may require additional explanation layers (SHAP, attention visualization).
Extreme Events
Model struggles with unprecedented shocks (e.g., COVID-19 lockdowns, sudden policy changes). Ensemble with rule-based adjustments could help.
Data Freshness
Forecasts depend on recent data availability. Delayed reporting from mandis reduces forecast accuracy for the first few days.
Future Work
Short-Term Enhancements
• Add weather data (rainfall, temperature) as exogenous features
• Incorporate festival calendars for demand spikes
