dhiraj kumar
chaudhary

senior software engineer · new york

i build ai systems and the data platforms they run on. at kkr (global atlantic), my work supports 10,000+ production jobs across shared infrastructure, developer tools, and applied ai.

outside kkr (global atlantic), i design and ship independent software products.

Dhiraj Kumar Chaudhary, smiling in a navy jacket.

education.

st. joseph’s university

new york city

bsc honors · mathematics & computer science · dual major

gpa 3.85 · presidential scholarship · dean’s list · peer tutor

experience.

kkr (global atlantic)

2022 – present

senior software engineer, data platforms

grew core data platforms from a single-engineer function into a team of five, supporting the insurance data engineering organization. the team owns the platforms behind 10,000+ production jobs and the ai workflows that automate their operations.

Reusable Spark components connect source data to shared production pipelines.

spark etl framework

own and maintain the insurance data engineering organization’s spark/scala etl framework: 50+ reusable components powering 10,000+ production jobs. performance tuning reduced key runtimes by roughly 80%, while shared operational controls propagate improvements across dependent pipelines.

components cover database reads and writes, csv generation, s3 delivery, and email notifications. centralized logging, incident creation, and configuration let teams assemble pipelines with minimal custom code and apply shared fixes across every dependent job.

Many small JSON files with inconsistent structures pass through normalization into one consistent dataframe.

json ingestion

rebuilt a pipeline processing 300–400k json files per load (now supports 1M files). on the original workload, runtime fell from 17–18 hours to 10–15 minutes for 95% of runs. distributed file movement reduced a five-hour operation to two minutes.

used spark rdds as a distributed work queue for file movement and normalized inconsistent json structures before dataframe creation. handling list-versus-dictionary schema drift at the rdd layer removed a major ingestion bottleneck.

Logs, code, database evidence, and incident history inform an investigation and drafted next steps for human review.

automated incident resolution

designed an agentic system for diagnosing production data-job failures and drafting remediation for engineer review. evaluated against 200 real failures, the proof of concept classified root causes correctly about 95% of the time. the system is moving toward production.

each investigation runs in a managed claude code session grounded in the job’s code, logs, database evidence, and incident history. the system drafts a github pull request for code defects or a routed email for data and infrastructure issues; engineers review every proposed action.

proof of concept · moving toward production

serverless platform

one of the lead architects of the migration from always-on ec2 to a centralized serverless platform on aws glue, lambda, and emr. proved feasibility, built the initial version, and supported team migrations with custom trigger handling, dependency tracking, and operational alerts. automation saved 8,000+ engineering hours; alerting reduced error-detection time by roughly 40%.

Legacy procedures and dashboards pass through rewriting and validation into migrated versions.

llm migrations

built llm-assisted pipelines to migrate 2,000+ sybase stored procedures, 500 sas jobs, and roughly 1,200 tableau dashboards. combined parsers, schema checks, scoped model rewrites, and rendered dashboard comparisons for validation. shipped in 2024, reducing months of migration work to days and saving roughly $1m in labor.

Changed partner-file columns become a proposed mapping correction for approval.

partner schema drift resolver

built a production system that detects schema drift in insurance partner files and proposes mapping corrections as reviewable diffs. approved files ingest without code changes or partner resends. shipped in 2024 and now handles roughly 120 files per month across dozens of deals.

data quality agent

built a two-layer data-quality system: an agent profiles tables and drafts 20–30 rules per table, testing each against real data before human review. a daily llm pass analyzes the results for anomalies and relationships that individual rules miss. the proof of concept covers roughly 15 tables and produces readable reports and structured summaries.

poc

document translation

built an internal pipeline that translates pdf and powerpoint documents while reconstructing their original layout. open-source models run inside the vpc, with round-trip evaluation to verify meaning. the production service supports on-demand english–japanese translation without sending data outside the environment.

Separate source databases feed shared tables while owners retain control over access.

data fabric

architected a central data fabric on apache iceberg, consolidating oracle, snowflake, db2, and redshift data into a shared queryable store. more than 500 tables across four teams are unified under centralized controls and self-service access, while dataset owners retain access authority.

variance tracker

built and deployed a full-stack variance tracking and sign-off platform end to end using flask and aws. shipped through enterprise architecture review, with a shared approval workflow used by 50+ stakeholders. reduced variance approval cycles by roughly 70%.

deal code generation

built an llm pipeline generator grounded in previous insurance deals’ code, with human review before integration. accelerated development of 15–20 pipeline modules per deal, reducing integration time by roughly 60%. later retired the pipeline and preserved its grounding knowledge in claude code skills.

retired (2024) · lives on as claude code skills

A plain-language question and database schema inform a SQL query and result table.

natural language to sql

built the organization’s first natural-language-to-sql system in 2024, grounding generated queries in real schemas, column context, and curated examples. the implementation was incorporated into the production tool whose descendant now runs across the organization.

merged into the org's production tool · 2024

tarifica

2021

software engineer intern

rebuilt and maintained 100+ python scrapers for telecom data, improving data quality by about 25%. worked on etl pipelines and flask apis for faster reporting.

projects.

A resume and job description become a tailored resume while preserving existing experience.

get new resume

[getnewresume.com ]

an ai resume-tailoring platform that produces a targeted resume, ats match score, and cover letter from a resume and job description. a four-stage, multi-model llm pipeline enforces typed, validated outputs and a zero-fabrication constraint, preserving the candidate’s actual experience.

A personalized news briefing delivered with timezone-aware scheduling.

timebrew

[timebrew.news ]

personalized ai news briefings powered by a three-stage pipeline for curation, editing, and delivery. aws step functions coordinates 14 lambdas with timezone-aware scheduling, at an operating cost of about $0.02 per briefing.

Goals shape an email, with feedback informing the next message.

inspireinbox

[inspireinbox.com ]

an llm-powered personal-growth companion that generates and schedules emails around individual goals. feedback from each delivery informs subsequent content, connecting generation, scheduling, and personalization in a continuous feedback loop. built with fastapi and react.

Embeddable clock, timer, and counter widgets for Notion pages.

notion crafts

[notioncrafts.com ]

a collection of embeddable clocks, timers, counters, and icon packs for notion pages. built with a python/flask backend on ec2, served through nginx and cloudflare.

research work.

A colored Flood-It grid and successive moves that spread one color across the board.

flood-it solver[read ]

designed and implemented a python algorithm that predicts the minimum number of moves needed to flood the grid in the flood-it game.

presented at the mathematical association of america

bio-robotics[read ]

combined artificial and biological structures into a hybrid robotic design that outperforms conventional single-structure robots.

presented at the sjc undergraduate research symposium