What this PR adds

bigdata-analysis — an AI coding skill that prevents silent data bugs in Hive/Impala/Spark ETL pipelines.

What problem it solves

When AI assistants generate Spark/Hive code, they frequently produce code with invisible bugs — row counts look correct, no errors thrown, but data is silently wrong. This skill encodes 10 battle-tested rules from real production incidents:

• Control characters in GROUP BY causing silent row explosion
• SELECT * in INSERT causing column position shifts
• foreachPartition as a no-op for cache materialization
• Broadcast JOIN threshold causing 26k+ task explosion
• And 6 more rules covering OOM, JOIN safety, UDF type safety, metadata refresh, etc.

Who uses this workflow

Data engineers writing Hive/Impala SQL or Spark Scala ETL jobs on HDFS/YARN, especially those using AI coding assistants.

Real-world impact

One rule alone (Rule 4: Spark SQL vs DataFrame API) reduced a production job from 367 lines / 45 Jobs / 3 hours to 50 lines / 1 Job / 15 minutes.

Category

Added to Data & Analysis (alphabetical order).

Install

npx skills add Oak-B/bigdata-analysis-skill@bigdata-analysis