PySpark Big Data Developer

Responsibilities

• Conduct tasks related to feasibility studies, time and cost estimates, IT planning, risk technology, applications development, model development
• establish and implement new or revised applications systems and programs to meet specific business needs or user areas Monitor and control all phases of development process and analysis, design, construction, testing
• implementation as well as provide user and operational support on applications to business users Utilize in-depth specialty knowledge of applications development to analyze complex problems/issues, provide evaluation of business process, system process
• make evaluative judgement Recommend and develop security measures in post implementation analysis of business usage to ensure successful system design and functionality Consult with users/clients and other technology groups on issues, recommend advanced programming solutions
• install and assist customer exposure systems Ensure essential procedures are followed and help define operating standards and processes Serve as advisor or coach to new or lower level analysts Has the ability to operate with a limited level of direct supervision.
• Can exercise independence of judgement and autonomy.
• Acts as SME to senior stakeholders and /or other team members.
• Appropriately assess risk when business decisions are made, demonstrating particular consideration for the firm's reputation and safeguarding Citigroup, its clients and assets, by driving compliance with applicable laws, rules and regulations, adhering to Policy, applying sound ethical judgment regarding personal behavior, conduct and business practices
• escalating, managing and reporting control issues with transparency.
• Recommended Qualifications: 5-8 years of relevant experience Experience in systems analysis and programming of software applications Experience in managing and implementing successful projects Working knowledge of consulting/project management techniques/methods Ability to work under pressure and manage deadlines or unexpected changes in expectations or requirements Education: Bachelor’s degree/University degree or equivalent experience This job description provides a high-level review of the types of work performed.
• Other job-related duties may be assigned as required.
• Design & Development: Create and optimize scalable ETL (Extraction, Transformation, Loading) pipelines using PySpark for massive datasets.
• Coding & Engineering: Write clean, efficient, well-documented code primarily in Python (PySpark) often leveraging frameworks/tools.
• Collaboration: Work with cross-functional teams (senior developers, data engineers, analysts, business partners) to understand data requirements and ensure seamless solution integration.
• Troubleshooting & Optimization: Debug and resolve data processing issues and performance bottlenecks in Spark applications and other big data technologies.
• Full SDLC Involvement: Participate in the entire software development lifecycle, from requirements analysis and design to testing, deployment, and operations.
• Data Integrity: Ensure high data quality and integrity throughout the data lifecycle.
• This candidate possesses 4-8 years of experience in developing and managing Enterprise Applications, demonstrating a robust foundation in Big Data technologies and a strong grasp of software development principles.
• Key Experience & Expertise: Enterprise Application Development: 4-8 years in developing and managing enterprise-grade applications.
• Object-Oriented Programming (OOP): Solid foundation in OOP concepts.
• Big Data Development: Expertise in PySpark, HDFS, Hive, Sqoop, and Hadoop for Big Data environments.
• Database Technologies: Good exposure to SQL Server and ORACLE databases.
• Experience with query writing for data validation/manipulation Scripting & Automation: Proficient in Shell Scripting and experience with job scheduling tools like Autosys.
• BI Reporting Tools: Some exposure to BI tools, specifically Tableau.
• Tools & Practices: Proficient with Git; experience with JIRA, Confluence.
• Familiarity with DevOps and CI/CD pipelines. ------------------------------------------------------ Job Family Group: Technology ------------------------------------------------------ Job Family: Applications Development ------------------------------------------------------ Time Type: Full time ------------------------------------------------------ Most Relevant Skills Please see the requirements listed above. ------------------------------------------------------ Other Relevant Skills For complementary skills, please see above and/or contact the recruiter. ------------------------------------------------------ Citi is an equal opportunity employer
• qualified candidates will receive consideration without regard to their race, color, religion, sex, sexual orientation, gender identity, national origin, disability, status as a protected veteran
• any other characteristic protected by law.
• If you are a person with a disability and need a reasonable accommodation to use our search tools and/or apply for a career opportunity review Accessibility at Citi .
• View Citi’s EEO Policy Statement and the Know Your Rights poster.