Engineer/Senior Engineer – Observability

Job description:

The Engineer/Senior Engineer – Observability Engineering is key member of Service Reliability Engineering. He/she will be ultimately responsible for system Observability, reliability Monitoring and reducing time to detect by continuously finetuning the monitoring infrastructure of the services our SRE team supports.

As a Reliability engineering team member- With proactive and predictive monitoring our Production & development team can continue to innovate by spotting small bugs and big disasters before they actually happen. That’s your main mission as an Monitoring & Observability Engineer. Next to our Elastic community, you’ll be part of our multidisciplinary Innovative Tech team, where DevOps, Agile, Cloud & Software Engineering experts all work together to create remarkable solutions based on cutting-edge technology.

Location: Chennai (Preferred) /Mumbai

What will you be doing?

· Implement, maintain, and consult on the observability and monitoring framework that supports the needs of multiple internal stakeholders.

· Manage Opera/Prometheus/Grafana/Splunk to support custom metric delivery dashboards.

· Design and build an observability infrastructure for all engineering teams to consume

· Design and develop tools for metric collection, analysis, and reporting

· Educate and lead efforts to improve observability among all engineering teams

· Responsible for the availability, performance, scaling, monitoring and incident response of FSS technology platform and services.

· Ensure the site and services are up 24*7 with no unplanned downtimes. Participate in a rotating on-call schedule to troubleshoot and resolve production escalations from our 24x7x365 NOC & Customer Success teams;

· Debugging of the code issues based on web service and API responses, errors, events, logs, etc. Monitor and optimize application performance within the deployment architecture;

· Identify and collect the appropriate measurements, and synthesize the correct queries, to show intuitive and insightful visualizations which characterize the behaviour of complex systems

· Continue evolving monitoring tooling toward a standards-based self-service automated platform and come up with creative solutions to solve problems

· Ensure proper reviews are built to minimise the Mean Time to Recover (MTTR) and Mean Time to Failure (MTTF).

· Implementation of ITIL processes like Incident management, problem management and change management.

· You will add, tune and maintain alert configurations and documentation as needed;

What you will bring along

· BS/MS/MCA Degree in Computer Science, Electrical & Computer Engineering or Mathematics or equivalent experience;

· 3-8 years of relevant reliability engineering work experience in any of the Online technology companies.

· Ability to understand the business services and map it to the reliability engineering design and review

· Excellent analytical, problem-solving and communication skills

· Driven and self-motivated, work creatively to solve challenging problems.

· Experience with design and implementation of Continuous Delivery and/or DevOps solutions or architecture patterns.

· Experience with code repository management, code merge and quality checks, continuous integration, and automated deployment & management using tools like Jenkins, Git, Ansible, Artifactory, Jira, Sonar

· Abreast of industry standards and trends related to telemetry and software pipelines

· Experience rationalizing and implementing monitoring and observability toolchain at enterprise scale

· Previous experience of public clouds (AWS and Terraform)

· Knowledge and experience of containers and Kubernetes cluster

· Hands on experience consolidating application and system logs at enterprise scale

· Experience with automation tools (Chef, Ansible)

· Experience with metrics exporters and integrations

· Experience with metrics collection and storage (Prometheus, InfluxDB)

· Experience with log collection and storage (ELK, Splunk,logstash)

· Experience with metric and log query languages (PromQL, LogQL, Sumo Logic)

· Experience with alert and notification management (Alertmanager, PagerDuty, Teams integrations)

· Experience with building dashboards (Grafana, Loki, Sumologic, Tenable)

· Proven development background with Go, Python, Shell or Java

· Security awareness, with an emphasis on designing for security best practices