PySpark Big Data Developer

Key Responsibilities

• Design & Development: Create and optimize scalable ETL (Extraction, Transformation, Loading) pipelines using PySpark for massive datasets.
• Coding & Engineering: Write clean, efficient, well-documented code primarily in Python (PySpark) often leveraging frameworks/tools.
• Collaboration: Work with cross-functional teams (senior developers, data engineers, analysts, business partners) to understand data requirements and ensure seamless solution integration.
• Troubleshooting & Optimization: Debug and resolve data processing issues and performance bottlenecks in Spark applications and other big data technologies.
• Full SDLC Involvement: Participate in the entire software development lifecycle, from requirements analysis and design to testing, deployment, and operations.
• Data Integrity: Ensure high data quality and integrity throughout the data lifecycle.
• This candidate possesses 4-8 years of experience in developing and managing Enterprise Applications, demonstrating a robust foundation in Big Data technologies and a strong grasp of software development principles.
• Key Experience & Expertise: Enterprise Application Development: 4-8 years in developing and managing enterprise-grade applications.
• Object-Oriented Programming (OOP): Solid foundation in OOP concepts.
• Big Data Development: Expertise in PySpark, HDFS, Hive, Sqoop, and Hadoop for Big Data environments.
• Database Technologies: Good exposure to SQL Server and ORACLE databases.
• Experience with query writing for data validation/manipulation Scripting & Automation: Proficient in Shell Scripting and experience with job scheduling tools like Autosys .
• BI Reporting Tools: Some exposure to BI tools, specifically Tableau .
• Tools & Practices: Proficient with Git; experience with JIRA, Confluence.
• Familiarity with DevOps and CI/CD pipelines. ------------------------------------------------------ Job Family Group: Technology ------------------------------------------------------ Job Family: Applications Development ------------------------------------------------------ Time Type: Full time ------------------------------------------------------ Most Relevant Skills Please see the requirements listed above. ------------------------------------------------------ Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------ Citi is an equal opportunity employer
• qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran
• any other characteristic protected by law.
• If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi .
• View Citi’s EEO Policy Statement and the Know Your Rights poster.