Job Description:
Data & AI Engineer – Customized LLMs & Databricks
We are seeking a highly skilled AI Engineer with strong expertise in customized Large Language Models (LLMs) and hands-on experience with the Databricks platform. The ideal candidate will design, build, fine-tune, and deploy enterprise-grade generative AI solutions , modern AI frameworks, and cloud technologies.
Key Responsibilities:
- Design, develop, and deploy customized LLM-based applications
- Build scalable Generative AI and RAG (Retrieval-Augmented Generation) solutions on Databricks Lakehouse architecture.
- Fine-tune and optimize open-source and proprietary LLMs using enterprise datasets.
- Develop prompt engineering frameworks and AI orchestration workflows.
- Work with Databricks Mosaic AI, MLflow, Vector Search, and Unity Catalog.
- Build and manage vector databases, embeddings pipelines, and semantic search solutions.
- Integrate AI solutions with enterprise applications, APIs, and cloud platforms.
- Optimize model performance, scalability, inference latency, and cost efficiency.
- Implement AI governance, monitoring, security, and responsible AI practices.
- Collaborate with business stakeholders, data engineers, and product teams to deliver AI solutions.
- Support AI model deployment, MLOps pipelines, and production monitoring.
Required Skills & Qualifications:
- Strong hands-on experience with Databricks and Lakehouse architecture.
- Experience with Databricks Mosaic AI, MLflow, Delta Lake, and Unity Catalog.
- Strong programming skills in Python and SQL.
- Hands-on experience with LLMs such as GPT, Llama, Mistral, Claude, or similar models.
- Experience with RAG architectures, embeddings, and vector search implementations.
- Knowledge of LangChain, LangGraph, Semantic Kernel, or similar AI orchestration frameworks.
- Experience with cloud platforms such as AWS, Azure, or GCP.
- Familiarity with APIs, Docker, Kubernetes, and microservices architecture.
- Understanding of AI governance, model evaluation, and monitoring.
Preferred Qualifications:
- Experience deploying GenAI applications in enterprise environments.
- Knowledge of distributed computing and Spark optimization.
- Experience with Databricks Model Serving and AI Gateway.
- Familiarity with CI/CD and MLOps practices.
- Experience with multimodal AI models and AI agents.
- Technologies & Tools
- Databricks, Mosaic AI, MLflow, Delta Lake
- Python, SQL, PyTorch, TensorFlow
- OpenAI API, Hugging Face
- LangChain, LangGraph
- Vector databases: Pinecone, FAISS, ChromaDB
- AWS, Azure, GCP
- Docker, Kubernetes, GitHub Actions
