Kimi K3 for Data Science: I Replaced My Entire Pandas Pipeline — Here's What Broke

The Original Pipeline: 2,000 Lines of Pandas

Every data scientist has that one pipeline. The one that started as a quick analysis script and grew, over months, into a 2,000-line monolith of data cleaning, feature engineering, model training, and visualization. Mine processes customer behavior data for a mid-size e-commerce company — about 5 million rows across 47 features — and produces churn predictions, customer segmentation, and executive dashboards.

The pipeline worked. It was tested, documented (mostly), and had been running in production for six months. But it was also brittle — any change to the data schema required updates across multiple files, and the 47 features were engineered through trial and error with limited documentation of why each transformation existed.

So when I started evaluating K3's capabilities, I decided to use this pipeline as a stress test. Could K3 replace the entire thing — not just generate individual functions, but understand the complete data science workflow and produce a coherent, working pipeline from scratch?

The short answer: it replaced 4 out of 5 stages successfully, and the one failure was genuinely instructive. Here's the complete breakdown, stage by stage.

Stage 1: Data Cleaning — K3 Nailed It (9/10)

Data cleaning is where K3 shines. I gave it a description of my raw data schema — 47 columns with their types, common missing value patterns, and known quality issues — and asked it to produce a cleaning pipeline.

What K3 generated in 45 seconds was genuinely impressive:

• Missing value handling: It correctly identified that 'last_purchase_date' should be forward-filled within customer groups rather than globally, that 'customer_segment' missing values should be labeled 'unknown' rather than imputed, and that numerical features should use median imputation grouped by 'account_age_bucket'.
• Outlier detection: It implemented IQR-based outlier detection for transaction amounts, with separate thresholds for B2B vs B2C customers — a nuance that had taken me two days to discover manually.
• Type corrections: It converted date strings to datetime objects, created proper categorical types for ordinal features (reducing memory usage by 35%), and handled timezone normalization for timestamps across 12 time zones.
• Data validation: It added Great Expectations-style validation checks at the end of the cleaning pipeline — verifying row counts, null percentages, value ranges, and referential integrity between customer and transaction tables.

The cleaning stage required only two modifications: K3 initially used a deprecated Pandas method (DataFrame.append instead of pd.concat) and missed one business rule (negative refund amounts should be flagged, not just zero amounts). Both were quick fixes. Overall, K3 reduced my cleaning code from 450 lines to 280 lines while actually improving quality. This stage alone justified the experiment.

Stage 2: Exploratory Data Analysis — Surprisingly Insightful (8/10)

I asked K3 to perform EDA on the cleaned dataset, looking for patterns, correlations, and anomalies that might inform feature engineering and modeling. This is where I expected K3 to struggle — EDA requires intuition about what's interesting, not just mechanical computation.

K3 surprised me. It generated a comprehensive EDA notebook with 34 cells covering:

• Univariate distributions for all numerical features with skewness and kurtosis statistics
• Correlation heatmap with hierarchical clustering, correctly identifying three feature clusters
• Time-series decomposition of purchase frequency, revealing weekly and monthly seasonality
• Cohort analysis by customer acquisition channel, showing retention curves diverging at month 3
• Interaction effects between top-5 correlated feature pairs

The most impressive insight: K3 identified that 'days_since_last_login' had a non-linear relationship with churn — customers who logged in very frequently (daily) AND very infrequently (monthly+) both had higher churn than the moderate group (weekly). It suggested creating a 'login_frequency_category' feature with three bins, which turned out to be one of the top-10 predictive features in the final model.

Where K3 fell short: it missed a critical data leakage issue. The feature 'customer_support_tickets_last_30d' included tickets filed after the churn decision point, which would not be available at prediction time in production. I caught this during review, but it's the kind of subtle error that could have led to an overoptimistically accurate model. The Excel data analysis article covers similar patterns with structured data.

Stage 3: Feature Engineering — Good but Not Great (7/10)

Feature engineering is where domain expertise matters most, and this is where K3's limitations became more apparent. I asked it to generate features based on the EDA insights and the business context I provided.

What worked: K3 generated 28 new features, of which 18 were meaningful additions. RFM features (Recency, Frequency, Monetary) were correctly implemented. Rolling window aggregations (7d, 30d, 90d averages) were properly computed with appropriate handling of sparse data. Interaction features between correlated pairs were sensible. Text-based features from customer review text (sentiment score, review length, keyword extraction) were well-implemented using a lightweight NLP pipeline.

What didn't work: K3 generated 10 features that were either redundant or problematic. Three features were mathematically identical to existing features under different names (it "rediscovered" customer_lifetime_value using a slightly different formula). Four features had look-ahead bias — they used data that wouldn't be available at prediction time. Three features were based on spurious correlations that wouldn't generalize (it found that customers with email addresses starting with 'A' had slightly higher churn in this dataset — obviously noise, not signal).

After manual review and curation, I kept 18 of K3's 28 features. Combined with the original 47, the final feature set had 65 features. The model's AUC improved from 0.78 (original features only) to 0.84 (original + K3 features). A meaningful improvement, but the 35% "bad feature" rate means you can't skip the review step.

Stage 4: Modeling — The Stage That Broke (5/10)

This is where my experiment hit a wall. I asked K3 to build a complete modeling pipeline: train/test split, model selection, hyperparameter tuning, cross-validation, and evaluation metrics.

K3 generated a reasonable modeling scaffold: stratified train/test split, implementations of Logistic Regression, Random Forest, XGBoost, and LightGBM, Optuna-based hyperparameter tuning, and 5-fold cross-validation with proper stratification. The code was syntactically correct and would run without errors.

But the decisions were wrong in ways that a data scientist would catch but a novice might not:

• Imbalanced classes: The churn rate was 12%, making this a moderately imbalanced problem. K3 didn't apply SMOTE, class weighting, or threshold optimization. The default 0.5 threshold produced a model that predicted "no churn" for 97% of customers — high accuracy, useless for the actual business need.
• Feature scaling: K3 applied StandardScaler to all features, including tree-based model inputs where scaling is unnecessary and potentially harmful for interpretability.
• Hyperparameter ranges: The Optuna search space was too narrow for XGBoost (max_depth limited to 3-6 when the optimal value was 9) and too wide for Logistic Regression (C from 1e-5 to 1e5, wasting search budget on absurd values).
• Evaluation metrics: K3 reported accuracy and AUC but didn't include precision, recall, F1, or business-relevant metrics like the cost of false positives vs false negatives. For a churn prediction model, recall at a fixed precision threshold is usually the right metric.

I spent 3 hours fixing the modeling stage — more time than I would have spent writing it from scratch. The scaffold was helpful as a starting point, but the statistical decision-making was clearly K3's weakest area. The benchmark comparison shows K3 excels at code generation, but data science modeling requires statistical judgment that current AI models haven't mastered.

Stage 5: Visualization — Excellent (8.5/10)

The final stage was another pleasant surprise. K3 generated publication-quality visualizations using Plotly and Matplotlib:

• Executive dashboard: A 4-panel Plotly dashboard with churn probability distribution, feature importance bar chart, ROC curve with confidence intervals, and monthly churn trend line. Interactive, responsive, and export-ready.
• Customer segmentation scatter plot: A 2D t-SNE visualization colored by predicted churn probability, with hover tooltips showing customer details. Visually clear and technically correct.
• Feature interaction network: A network graph showing the top-20 feature correlations as edges, with node size representing feature importance. Used Plotly's network graph capabilities effectively.
• Model comparison radar chart: A radar chart comparing all four models across accuracy, AUC, recall, precision, and training time. Clean and informative.

The only modification I made was adjusting color palettes for accessibility (K3's default colors weren't colorblind-friendly) and adding proper axis labels and titles to two charts. Otherwise, the visualizations were ready to present to stakeholders.