Job Description

Lendable works with fintechs across over 25 countries, processing data in the tens of billions of rows to provide real-time insights into fintech performance and economic trends. They have built a robust data engineering infrastructure to enable integrations with all of their fintech partners and the execution of data science and econometrics research using these data sets. This data is the core of their DNA. It is what allows them to understand the lending and payments fintechs that they support.

 

About the role:

You will be responsible for building the systems that they need to continue to scale Lendable.

In this role, you will collaborate with Lendable’s Director of Engineering to optimize Lendable’s Spark ETL systems and ensure that they can handle tens of billions of rows of data daily. You will design and build the tools to manage, verify, and test all of these data flows in real time. You will work with Lendable’s integrations team to implement generalized data models to represent the heterogeneous data that they collect around the world.

Key responsibilities:

• Implementing and optimizing Spark ETL systems for processing data sets in the tens of billions of rows.


• Designing and building data versioning, testing, and promotion systems for complex data flows.


• Building CLI and/or web app tools to manage ETL pipelines, data test suites, and data promotion.


• Porting existing data workflows from single-machine Pandas implementations to distributed Spark workflows.


• Maintaining production infrastructure and debugging issues quickly to ensure compliance with SLAs.


• Designing and implementing data models to enable analysis of fintech portfolios across sectors and continents.

Who are you?

You are a computer scientist at heart. You care about doing things the right way and understanding the why of systems. You take pride in putting in the extra effort the first time you do something so that it takes a tenth as long the next time through. You build systems robust to human error and data scale. You want to build systems that will last for decades to come.

Does this sound like you?

Must have:

• Strong computer science fundamentals, including things like algorithmic complexity and database design.


• Strong skills debugging distributed systems.


• 5+ years of experience working as a back-end engineer or data engineer.


• Experience deploying, monitoring, and debugging production systems.


• Expert Python skills.


• Strong UNIX and SQL experience.

Good to have:

• Experience working on data infrastructure and/or ETL pipelines.


• Big data processing and storage experience with tools like Spark, Athena, Redshift, or Parquet files.


• Skills in CI/CD systems and test suite design.


• Experience provisioning and deploying with AWS.


• Experience orchestrating workloads with Airflow.


• Experience packaging applications and development environments in Docker.