Data Pipeline Engineer
Job Type : Permanent, Full Time
Salary : £80000 – £100000/annum
My client has multiple product lines covering Art and NFT marketplace, Data Analytics for the art world including launching a number of Apps, Publishing and Printing and their own version of TED talks.
They are looking for Date Pipeline Engineers to join their wider engineering team who will be responsible for the management of large amounts of data produced on the web, social media and blockchains. The successful applicant would be joining a strong team of 30+ people (data scientists, engineers, and analysts), in a fast-paced and collaborative team environment.
There is no Tech challenge required, and this role can be done completely remote!
Responsibilities
Writing data pipelines in Python or R to extract data from blockchain & service APIs into relational databases.
Spark and Pyspark experience required.
AWS experience required.
Developing tools to monitor existing data pipelines and web scrapers.
Developing tools to monitor data quality in various databases.
Improving the data quality of the databases by working on hotfixes with the help of QA & fellow data engineers/data scientists.
Deploying machine learning models on a secured public facing API via Kubernetes/Kubeflow/Ray Serve.
Writing and securing public facing APIs on Kubernetes to allow external partners to access our services.
Work with data scientists in deploying the following on Kubernetes/AWS cloud
More Information
- Address City of London, London
