Overview: Management Science Associates, Inc., (MSA) is a diversified information management company that for over half a century has given market leaders the competitive edge in data management, analytics and technology. We are seeking an experienced Senior Software Engineer/Hadoop Engineer to join the MSA Information Management Solutions (IMS) division, which has been offering data and insight solutions to the Consumer Package Goods industry since the early 1980s and now includes the Hadoop technology stack in our solution offerings.
As a Senior Software Engineer/Hadoop Engineer you will provide onsite and cloud support for the MapR Hadoop clusters and related software stack. This position is responsible for administering key Big Data systems and applications that handle data ingestion, processing, analytics and presentation layers, and scaling the system to handle billions of events in near real-time.
The engineer will work with exciting technologies such as Hadoop, Yarn, Spark, Kafka, Drill, ElasticSearch, HBase, Hive and similar components.
You will be responsible for installs, upgrades, administering the Big Data systems, and evaluating new software, and for configuring, performance tuning, troubleshooting, and security. You will ensure proper monitoring, alerting, capacity planning and reporting in the production environment. You will contribute to the evolving design and architecture of reliable and scalable infrastructure.
Responsibilities: * Implementation and ongoing administration of Hadoop infrastructure * Propose and deploy new hardware and software environments required for Hadoop services and expand existing environments * Maintaining the Hadoop platform and its ecosystem * Performance tuning * Capacity and software upgrade planning * Monitoring Hadoop cluster connectivity and security * Managing and reviewing Hadoop and System log files * File system management and monitoring * HDFS support and maintenance * Teaming with the infrastructure, network, database, application and business intelligence teams to guarantee high data quality and availability * Collaborating with application teams to install operating system and Hadoop and Hadoop ecosystem updates, patches, version upgrades when required * Point of Contact for Vendor escalation
Required Skills: * Requires a four-year degree in Computer Science/Software Engineering or Information Systems. A Master’s degree in Computer Science/Software Engineering or related degree program is preferred. * Minimum eight years of experience in systems analysis and programming, addressing issues of architecture and data management * Minimum 3 years Hadoop related work experience * Solid experience with Hadoop administration best practices * Solid experience with most common Hadoop ecosystem applications such as Spark, HBase, Kafka, etc. * Familiar with Hadoop architecture and principles such as parallel distributed processing, data locality, multi-tenant security, etc. * Experienced with Linux system administration best practices * General operational expertise such as good troubleshooting skills, understanding of system’s capacity, bottlenecks, memory, CPU, OS, storage, and networks * Monitor the cluster, configure high availability, backups * Familiarity with open source configuration management and deployment tools such as Puppet or Chef and Linux scripting * Knowledge of Troubleshooting Core Java Applications is a plus * Good communication and documentation skills * Ability to work independently on multiple tasks * Experience working in integrated product development team environment