The Future of Data Engineering: AI, LLMs, and Automation – YouTube Dictation Transcript & Vocabulary

1.[Music] hello and welcome to the data engineering podcast the show about modern data management data migrations are brutal they drag on for months sometimes years burning through resources and crushing team morale data folds AI powered migration agent changes all that they unique combination of AI code translation and a at data validation has helped companies complete migrations up to 10 times faster than manual approaches and they're so confident in their solution they'll actually guarantee your timeline in writing ready to turn your year-long migration into weeks visit dataengineering podcast.com folds today for the details your host is Tobias Macy and today I'd like to welcome back G maansi where we're going to talk about the work of data engineering to build AI to build better data engineering and all of the things that out of that idea so glib for folks who haven't heard any of your past appearances if you could just give a quick introduction yeah thanks for having me again to it's always fun to be at the podcast I'm glad I Am Co and co-founder of data fold we work on automating data engineering workflows now also with AI prior to starting data fold I was a data engineer data scientist data product manager and I got a chance to build three data platforms pretty much from scratch and at three very different companies including Autodesk and Lyft where I was one of the first founding data engineers and got to build a lot of pipelines and infrastructure and also break a lot of pipelines and infrastructure and I've always been fascinated by how important is data engineering to the business in that it unlocks the delivery of the actual applications that are data driven be that dashboards or machine learning models or now increasingly also bi applications and at the same time as a data engineer I've always been very frustrated with how manual airr prone TVs and toysome my personal workflow was and pretty much started data fold to solve that problem and remove all the manual work from the data engineering workflow said that we can ship high quality data faster and help all the wonderful businesses that are trying to leverage data actually do it so excited to chat in the context of data engineering AI obviously there's a lot of hype that's being thrown around about oh you just rub some AI on it and it'll be magical and your problems are solved you don't need to work anymore it's going to replace all of your Junior Engineers or whatever the current marketing spin is for it and it's undeniable that large language models generative AI the current ERA that we're in has a lot of potential there are a lot of useful applications of it but the work to actually realize those capabilities is often a little bit opaque or misunderstood or confusing and so there are definitely a lot of opportunities for being able to bring large language models or other generative AI Technologies into the context of data engineering work or development environments but the work of actually getting it to the point where it is more help than hindrance is often where things start to fall apart and I'm wondering if you can just start from the work that you're doing and the experience you've had of actually incorporating llms into some of your product some of the lessons learned about what are some of those impedance mismatches what are some of those stumbling blocks that you're going to run into on the path of saying I've got a model I've got a problem let's put them together yeah absolutely and I think that's SP on obs to bias in terms of there's a lot of noise and hype around AI everywhere but yet we don't have a really clear idea and consensus how actually it impacts State Engineering and maybe before we dive into like okay what is actually working it's worth kind of disambiguating and cutting through the noise a little bit and I've been thinking about this recently but I think there is probably two main things that everyone gets a bit confused about one is the confusion of software engineering and data engineering software engineering and data engineering are very related and in many ways there are similar in data engineering we ultimately also write code that produces some outcome but unlike software engineering typically we're not really building a deterministic application that performs a certain function we write in code that processes large amounts of data and usually that data is highly imperfect and so so we're dealing not just with uh code we're dealing also with extremely complex extremely noisy inputs and a lot of the times also unpredictable outputs and that makes the workflow quite different and I think one important distinction is when we see lots of different tools and advancements in tools that are affecting software engineers and impacting their work CLS for it better like one example is I think over the past year we've seen amazing amazing Improvement of the kind of co-pilot type of support within the software engineering workflow through various tools we at beta fold for example use cursor ID a lot and we really like how it seamlessly plugs in and enables our Engineers working on the application code just be more productive spend less time on uh a lot of like boiler plate toil tasks and those tools are really it's really exciting how it affects the software engineering workflow there's also a huge part in the software engineering space right now that is devoted to the agents so for example with cursor the idea is that you plug it in the IDE in a few touch points for developer like code completion and then kind of an assistant that helps you mck up and refactor the code and it's very seamless but it's still kind of part of the core workflow for human and then there's a second school of thought where there's an agent that takes a tasks that can be very Loosely defined and then basically Builds an app from scratch or takes a gr linear ticket and does the work from scratch and it's also very exciting I would say in our experience testing multiple tools the results there are far less impressive and actual impact on the business for us in terms of software engineering has been far less impressive than with more like a ID native enhancement but all of is to say that while those tools are really impactful for software engineers and there's a lot happening also in other parts of the workflow we've seen very limited impact of those particular tools on the data Engineers workflow and the primary reason is that although we also write in code as data Engineers the tools that are built for software Engineers they lack very important context about the data and that is kind of a simple idea and a simple statement but what's underneath is actually quite a bit of complexity because if you think about what data engineer needs to do in order to do their job they have to understand not just the codebase but they also have to have a really good grasp on the underlying data that their codebase is processing which is actually a very hard task by itself starting from understanding what data you have on the first place how is the data computed where it's coming who is consuming it what are relationships between all the data sets and absent of that context the tools that you may have supporting your workflow yes it can help you generate the code but the impact of that would be quite limited relative to um how complex your workflow is and I think that means that for data Engineers we need to see a specialized class of tools that would be dedicated at improving data Engineers workflow and would excel at doing that by having that context that is critical for data engineer to do their job that's kind of I think one aspect of the confusion sort of like all the advancements and software engineering tools are exciting and inspiring it doesn't mean that now data Engineers workflow is impacted as as significantly as a software engineer's workflow I think the other type of confusion that I'm seeing is there's a lot of talk about AI in the data space and all the vendors you see out there are I think smartly positioning them themselves as really relevant and essential to the fundamental tectonic ship we now seeing technology meaning they trying to position themselves as relevant in the in the world where llms are really providing big opportunity for businesses to uh to improve and grow and automate a lot of business processes but if you double click into what is exactly everyone is saying is it's pretty much we going to help you the data team the data engineer ship AI to your business and to your stakeholders like we are the best you know workflow engine so that you can get data delivered for AI or we are the best data quality vendor that will help you ensure the quality of the data that goes into AI or we have the most Integrations with all the vector databases that are important fori and kind of the the the message that you're getting from all of this and by no means this is not import this is definitely important and relevant but what's interesting about this is we're saying essentially data engineer you have so many things to do and now you also have to ship AI we're going to help you ship AI it's so important that you ship data for AI applications we are the best tool to help you ship AI but it's almost sounds like this is data engineers in the service Sur Ai and I think what's really interesting to explore and to unpack and what I would personally love for myself as a data engineer is kind of reversing that question and asking the question of okay so we have now this fundamental shift in technology amazing capabilities by llms how does it actually help me in my workflow so what does the AI for data engineer look like and I think we need much more of that discussion because I think that if we make people who are actually working on all these important problems more productive with the help of AI then they will for sure do amazing things with data and I think that's a really exciting opportunity to explore one of the first and most vocal applications of AI in that context of helping the data Engineers by maybe taking some of the burden off them that I've seen is the idea of talk to your data warehouse in English or text a SQL or whatever formulation it ends up taking where rather than saying oh now you need to build your complicated star or snowflake schema and then build all of the different dashboards and visualizations for your business intelligence you just put an AI on top of it and then your data consumers just talk to the AI and say hey what was my net promoter score last quarter or what's my year-over-year Revenue growth or how much growth can I expect in the next quarter based on current sales and it's going to just automatically generate the the relevant queries it's going to generate the visualizations for them and you as a data engineer or as an analytics engineer don't need to worry about it anymore and from the description it sounds amazing it's like great okay job done I don't need to worry about that toysome work I do all of the interesting work of getting the data to where it needs to be and then the AI does the rest but then you still have to deal with issues of making sure that you have the appropriate semantics mapped so that the AI understands what the question actually means in the context of the data that you have which that's the hardest problem in data anyway no matter what so the AI doesn't actually solve anything for you it just maybe exacerbates the problem because somebody asks the AI the question the AI gives an answer but it's answering it based on a misunderstanding of the data that you have and so you still have those issues of hallucination incorrect data or variance in the way that the data is being interpreted and I'm wondering what you have seen as far as the the actual practical applications of the AI being that simplifying interface versus the amount of effort that's needed to be able to actually make that useful yeah I think this is tax to SQL is the Holly Grail of the data space I would say for as long as I've worked in the space for over a decade that you know people really try to solve this problem multiple times and obviously now in hindsight it's obvious that pre- LM all of those um approaches using traditional NLP were doomed and now that we we have llms it seems like okay finally we can actually solve this problem and I'm very optimistic that it indeed will help make data way more accessible and I think it eventually will have tremendous impact on how humans interact with data and how that is leveraged but I think that the how and how it happens and how applied is also very important because I don't think that the fundamental problem is that people cannot write SQL SQL is actually not that hard to to write and to master I think the fundamental issue is that if we think about the life cycle of data in the organization it's very important to understand that the raw data that it gets collected from you know all the business systems and all the events and logs and everything we have in a data Lake it is pretty much on usable and it's unusable both by machines and AI or and and people if we just try to you know throw a bunch of queries that it and ask you know try to answer really key business questions and in order for the data to become use usable we need what is currently is the job of a data engineer of structuring filtering merging aggregating this data curating it and creating a really structured representation of what is our business and what are all the entities in the business that we care about like customers products orders so that then this data can be fed into all the applications right business intelligence machine learning Ai and I don't think that tax to SQL replaces that because if we just do that on top of the raw data we basically get garbage in garbage out I do think that in certain applications in certain applications of that we can actually get very good results even today if we put that level of a system on top of Highly curated semantically structured data sets right so if we have a number of tables that are well defined that describe how our business Works having a text tosql interface could be actually extremely powerful because we know that the questions that are asked and will be translated into code will be answered with the data which has been already prepared and structured and so it's actually quite easy for the system to be able to make sense about it but I don't think we are there where just like you don't need the data team let's just ask a question almost guaranteed that the answer will be wrong so data engineer in that regard data engineering and data Engineers uh are definitely not going to lose their jobs because now it's easy to generate SQL from text and in the context even of that text tosql use case what I've been hearing a lot is that it's not even very good at that one because llms are bad at math and SQL is just a manifestation of relational algebra thereby math but that if you bring a Knowledge Graph into the system where the AI is using the knowledge graph to understand what are the relations between all the different entities from which it then generates the queries and actually does a much better job but again you have to build the knowledge graph first and I think maybe that's one of the places where bringing AI earlier in the cycle is actually potentially useful where you can use the AI to do some of that wrote work of saying here are all the different representations that I have of this entity or this concept across my different data sources give me a first pass of what a unified model looks like to be able to represent that given all of the data that I have about it and all the ways that it's being represented and I'm wondering what you've seen in that context of bringing the AI into that data modeling data curation workflow of it's not the end user interacting with it it's the data engineer using the AI as their co-pilot if you will or as their assistant to be able to do some of that tedious work that would otherwise be okay well I've got 15 different spreadsheets I need to visually look across them and try and figure out the similarities and differences Etc yeah that's a great point I would say that there are I have two thoughts there on how the EI plugs in to actually make text tosql work yes you absolutely need that kind of semantic graph of what what data sets you have how are they related what are all the metrics how those metrics are computed and in that regard what's really interesting is that the metrix layer that was at some point a really hot idea in the modern Data stock probably about for you know 3 to five years ago and then everyone was really disappointed with how little impa it actually made on on a data team's productivity and just overall a data stack it almost like now now it's the metric layers time because if you take the metric layer and um which gives you a really structured representation of the core entities and the metrics putting the text to SQL is almost like the most impactful thing that you can do because then you have a structured representation of your data model which allows AI to be very very effective at being able to answer questions while being while while operating on a structured graph and so I think we'll see really exciting applications coming out of the hybrid of that kind of fundamental metric layer semantic graph and text to SQL in you know we already seeing that the early impacts of that but I think over the next two years it probably will become the a really popular way to open up data for ultimate stakeholders instead of classical B of like drag and drop uh interfaces and kind of passively consumed dashboards but then the second point which you made is basically cani actually help us get to that structured representation and I think absolutely um for the data Engineers workflow so not for a I would say business stakeholder or someone who is Data consumer but for data producer I think that leveraging llms to help you build data models and especially build them faster build FAS in the sense of understanding all the semantic relationships not just writing code is a very promising area and that comes back to the my point about how software tools are limited in their help of you know for data Engineers right I can write SQL but if I if my tool does not understand what are the relationships between the data sets then it can't even help me write joints properly and one of the interesting things we've done at data fold was actually build A system that essentially infers a entity relationship diagram from the raw data that you have combined with all the ad hoc SQL queries that have been written by people so previously that would be a very hard problem to solve but with the help of our lamps we can actually have a really good shot at understanding what is the what are all the entities that your business have in your data Lake how are they related and that's almost like a probabilistic graph because people can be writing joints correctly or incorrectly and you have noisy data and sometimes keys that you think are like primary keys and foreign keys are not perfect but if you have a large enough data set of queries that were ran against your Warehouse you can actually have a really good shot at understanding what's the semantic graph looks like and the context on which we actually did this was to help data teams build testing environments for their data but the the implications of having that knowledge is actually very powerful right so to your point we can use that to also help right SQL so I'm very bullish on the ability to help engine data engineers build pipelines by creating a semantic graph without the need for curation because previously that problem was almost pushed to people with all the kind of data governance tools the idea was let's have data stewards Define all the canonical data sets and all the relationships and obviously we just discovered this completely non-scalable so now we finally at the point where we can automate that kind of semantic data mining uh with llms that brings us back around to another point that I wanted to dig into further in the context of how to actually integrate the llms into these different use cases and workflows you brought up the example of cursor as an IDE that was built specifically with llm use cases in mind supposed with something like a vs code or Vim or emac where the llm is a bolt-on and something that you're trying to retrofit into the experience and it can be useful but it requires a lot more effort to be able to actually set it up configure it make it aware of the codebase that you're trying to operate on Etc versus the prepackaged product and we're seeing that same type of thing in the context of data where you mentioned there are all these different vendors of oh hey we're going to make it super easy for you to make your data ready for AI or use this AI on your data but most teams already have some sort of system in place and they just want to be able to retrofit the llm into it to be able to start getting some of those gains with the eventual goal of having the llm maybe be a core portion of their data system their data product and I'm wondering in that process of bringing an llm retrofitting it onto an existing system whether that be your code editor your deployment Environ enironment your data warehouse what have you what are some of those impedance mismatches or some of the issues in conceptual understanding about how to bring the appropriate I'm going to use the word knowledge even though it's a bit of a misnomer into the operating memory of the llm so that it can actually do the thing that you're trying to tell it to do yeah that's a great question Tobias I think that to answer this we kind of need to go back to what are the jobs to be done for a data engineer and how does the data engineer workflow actually look like and if we were to visualize it it actually looks quite similar to the software engineering workflow in just the types of tasks that a data engineer does day-to-day to do their work and by the way we're saying data engineer as sort of like a blank label but I don't necessarily mean just people who have data engineer in their title because all roles that are working with data including data scientists analysts analytics engineers and BM and M cases and software Engineers a lot of them actually do data Engineering in terms of building pipelines and developing pipelines as part of their job it's just data Engineers probably do this you know 100% a day time and if I'm a data analyst or data scientist that would be doing this maybe 30 40% of the time of my week and so if we think about what do I need to do to let's say ship a new data model like a table or extend an existing data model you know refactor definitions or add new types of information into an existing model it starts with planning right so I'm doing planning I'm trying to find the data that I need for for my work and a lot of the times a lot of information can be sourced from documentation from data catalog I think right now the data cataloging in the sense of like what data sets I have and what's the profile of those data sets has been largely solve there are great tools you know some are open source some are vendors but overall understanding what data sets you have now is way easier than it was 5 years ago you also probably are Consulting your tribal knowledge and you go to slack and you do like search for certain definitions and that's also now is largely solved with a lot of the Enterprise search tools and then you go into writing code and writing code I think this is also an important misconception like if you are not really you know doing this for for living you think that people spend most of their time actually writing SQL and in terms of like writing SQL to for production and in my experience actual writing of this SQL or other types of code is maybe like 10 to 15% of my time whereas all the operational tasks around testing it talking to people to get context doing code reviews shipping it to production monitoring it remitting ating issues talking to more people is where the bulk of the work is happening and if that's true then that means that probably as we talk about automation these operational workflows are where the bulk of the lift coming from LMS can actually happen and so for actual writing code as a data engineer I would still recommend probably using the best-in-class software tools these days like cursor it will even though it's not aware of the data it will probably still help you write a lot of boil boiler plate and will speed up your workflow somewhat and or you can use other IDs with Co know like V code plus copilot I think those tools will just help you speed up the writing of the code itself but back to the operational workflows that I think take the majority of the of the time within any kind of cycle of shipping some of shipping something when it comes to what happens after you wrote the code right typically if you have people who care about the quality of the data means that you have to do a fair amount of testing of your work and testing is both helping making sure that my code is correct right does it conform to the expectations uh does it produce the data that I expect but it's also about understanding potential breakages Data Systems are historically fragile in the sense that you have layers and layers of dependencies that are often AA because um I can be changing some definition of what active user is somewhere in the pipeline but then I can be completely oblivious of the fact that 10 jobs down the road someone builds a machine learning model that consumes that definition and tries to automate certain decisions for like for example spend and manipulating that metric and so if I'm not aware of those Downstream dependencies I could be actually causing a massive business disruption just by the sheer fact of changing it and so the testing that involves not just understanding how the data behaves but also how the data is consumed and what are the larger business implications for making any kind of modification to the code is where a ton of time is spent in the data engineering and so what's interesting is that is the use case where historically we had data fault spent a lot of time thinking about even prei and before LMS were thing what we did there was came up with a concept of data diffing and the idea is everyone can see code diff right my code looked like this before I made it change now it's it's a different you know it's a different set of characters that uh the code looks like and uh def the code is something that is like EMB bettered in GitHub right you can see that but the very hard question is understanding how does the data change based on the change in the code because that is not obvious that happens like once you actually run the goat against the database and so data diff allows you to see the impact of a code change on the data and that by itself was quite impactful and we've seen a lot of teams adopt that you know large Enterprise teams uh fast moving software you know startup teams but we were not fully satisfied with the degree of automation that feature alone produced because people are still required to like sip through all the data diffs and explore them for multiple tables and see how the downstream impacts manifest themselves through lineage and it felt like okay now at least we can give people all the information but they still have to sift for a a lot of it and some of the important details can be missed and the big unlock that llms bring this particular workflow is once llms became pretty good in comprehending the code and actually semantically understanding the code which pretty much happened over 2024 with the latest generation of fundamental large you know large large language models we were able to do two things one take a lot of information and condens it into like three bullet points kind of like an executive summary and those bullet points are essentially helping the data engineer understand on the high level what are the most important impacts that I need to worry about for any given change and for a code reviewer to understand the same and that just helps people to get on the same page very quickly and save everyone a lot of time that otherwise could spend be spent in meetings or back and forth you know putting comments on a code change and the second unlock that we've seen is the opportunity to drill down and explore all the impacts and do the testing by essentially chatting with your P request chatting with your code and that comes in the form of a chat interface where you're basically speaking to an agent that has a full context of your code full context of the data change data diff and also full context St for your lineage so that I can actually understand how every line of code was modified affecting the data and what does that mean for the business and you can ask questions and it produces the answer is way faster than you would by essentially looking at all the different you know code changes and and data diffs and that ended up saving a lot of times a lot of time for data teams and now that I'm describing this you kind of feel that it sounds like almost having a buddy that just like helps you think for the code almost like having a code reviewer except for with thei with llm this is a buddy that's always available to you 24/7 and probably makes youro mistake because it has all the context and can set through a lot of information so really quickly so that's an example of how an could be applied into an operational use case that historically has been really timec consuming and take a lot of manual work out of that context and I really want to dig into that one word that you said probably at least a half dozen times if not maybe a couple of dozen was that context where that I think is the key piece that is so critical and also probably the most difficult portion of making AI useful is context what context does it need how do you get that context to it how do you model that context how do you keep it up to date and so I think that really is where the difference comes in between the cursor example that we touched on earlier versus the retrofitting onto emac or whatever your tool or workflow of choice is is how do you actually get the context to the place that it needs to be and so you just discussed the use case that you have of being able to use the llm in that use case of interpreting the various data diffs understanding what is the actual ramifications of this change and I'm wondering if you can just talk through some of the lessons learned about how you actually populate and maintain that context and how you're able to instruct the llm how to take advantage of the context that you've given it that's a great question Tobi and I think what's interesting is that at face value it seems like you want to throw all the information you have at llm right just like tell it everything and then let it figure out things and in fact it is obviously not as easy as that and in fact it's actually counterproductive to over Supply the llm with context in part because the context window of large language models is limited and the trade-off there is one you just like can't physically fit everything and two even if you're dealing with a model that actually is designed to have a very large Contex window if if you overuse it and Supply too much information LM just get gets lost it's also a starts being far far less effective in understanding what's actually important versus not and the overall effectiveness of your system goes down so back to your question of like what is the actual information that is important to provide as context into llm it really depends on what is the workflow that we're talking about in the context of a code review and testing where we we are trying to fundamentally answer the question of a if we changed the code was a change correct relative to what we tried to do what was what the task was or did we not conform to the business requirement the second question is did we follow the best practices such as you know code guidelines and performance guidelines or not and the third question is okay let's say we conformed to the business requirements we did the good job at following our coding best practices but we may still cause a business disruption just by making a change that can be a surprise either for a human consumer of data Downstream or could throw off a machine learning model that was trained based on a different distribution of data right and so these are fundamental three questions that we try to answer and by the way even without AI That's what a good code review would ultimately accomplish done by humans so so what is the context that it's important for LM to have here first obviously it is the code difference right so we already know what the original code was what the new code is and feeding that inm is really important so that I can understand okay what are the actual changes in the code itself in the logic and I'm I won't go into the details here because obviously the code base can be very large sometimes your PR can touch a lot of code so you have to be quite strategic in terms of how do you feed that on a technical side but conceptually that's what we have to provide as an input number one the second important input is the data diff right it's understanding if I have a kind of main branch version of the code understanding what data it produces and what are the metrics showing right and then if I have a new version of the code let's call it a developer Branch what data it produces and what is the difference in the output let's say with my main branch code I see that I have 37 orders on Monday but with the new version of the code I see that I have 39 and so that already tells me that okay so this is the important impact on the output data and on the metric and that can and that's important both on the value levels understanding how the individual cells rows and columns are changing but it's also important to do rollups and understand what is the impact on metrics and coupling that context with the code diff allows us to understand how changes in the code affect the actual data output and the third really important aspect is the lineage so lineage is fundamentally understanding how the data flows throughout your system how it's computed how it's aggregated and how it's consumed and the lineage is a graph and there are kind of two directions of exploration one of them is Upstream which helps us understand how how did the geta data get to the point where you're looking at it right so for example if I'm looking at number of orders and I'm changing a formula where does the information about orders come from in the first place and that is important because that can tell us a lot about how a given metric is is computed and what is the source of Truth are we getting it from Salesforce are we getting it from our internal system and then the downstream lineage is also important because it tells us how the data gets consumed and that is absolutely essential information that can help us understand what Downstream systems and metrics will be affected and lineage graph in itself can be very complex and building it actually is a tough problem because you have to essentially scrape all of your data platform information all the queries all the bi tools to understand how data flows how it's consumed and produced but let's say you have this lineage graph it's actually also a lot of information by itself and so to properly Supply that lineage information into an LM context you actually kind of need uh your system to be able to explore lineage graph on its own to see like okay if I am if theel make made a change here what are the important Downstream implications of that so now we're talking about kind of the system to be able to kind of Traverse that and do analysis on its own for for the context I would say these are the three most important types of context and then the fourth one is kind of optional if you're again if your team has any kind of best practices SQL linting rules documentation rules you can also provide them as context and then your kind of AI code reviewer assistant can help you reason about well did you conform or not and if not making suggestions about what to correct eventually probably going in and correcting your code itself I think that's ultimately where this is going but again it's pretty much would be operating on the same set of input context another interesting element of bringing llms into the context of the data engineering workflow and use case one is the Privacy aspect which is a whole other conversation I don't want to get too deep into that Quagmire but also when you're working as a data engineer one of the things you need to be thinking about is what is my data platform what are the tools that I rely on what are the ways they link together and if you're going to rely on an llm or generative AI as part of that tool chain how does that fit into that platform what is some of the scaffolding what are some of the workflows what are some of the custom development that you need to do where a lot of the first pass and naive use cases for generative Ai and L m is oh well just go and open up the chat GPT UI or just go run LM Studio or use CLA or what have you but if you want to get into anything sophisticated where you're actually relying on this as a component of your workflow you want to make sure that it's customized that you own it in some fashion and so that is likely going to require doing some custom development using something like a lan chain or a lane graph or uh crew AI or whatever where you're actually building additional scaffolding logic around just that kernel of the llm and I'm curious how you're seeing some of the needs and use cases of incorporating the llm more closely into that actual core capabilities of the data platform through that effort of customization and uh software engineering that's a great point I think that the models themselves are getting rapidly commoditized in the sense that their capabilities the fun you know the foundational large language models their interfaces are very similar their capabilities are similar we're seeing a lot of race between the companies training those models in terms of beating each other in benchmarks looks like the whole industry is conver converging on adding more reasoning and then the ways that this is happening is also converging on the same experience and the matter and the difference is like who is doing this better right who is beating the metrix who provides the best the cheaper inference the faster inference uh more intelligence for for the same price and to that end I don't think that differentiation or the effectiveness of whatever is the automation that you're trying to bring really depends on the choice of a model maybe for certain narrow applications actually maybe choosing a more specialized model and or fine-tuning model would be more applicable but still I don't think the model is where you really where the magic happens these days model is important for magic but it's not something that actually allows you to build a really fact application by just you know choosing something better than what's available to everyone else the actual magic and the value ad and the automation happens in how you leverage that model in your workflow so all the orchestration in terms of how do you prompt the model what kind of context you provide how do you tune The Prompt how do you tune the inputs how do you evaluate the performance of the model in production how do you make various LM based actors that may be playing different roles interact with each other that is where the hard work is happening and that is where I think the actual value and impact is created and that's where all the complexity is so I think you don't have to be you know PhD and really understand how the models are trained although I would say just like in computer science is obviously very helpful to understand how these models are trained in the architectures and their trade-offs but you don't have to be good at um you know training those models in order to effectively leverage them but to leverage them you have to do a lot of work to effectively plug them in the workflows and I think that the applications and companies and teams that are thinking about what is the workflow what is the ideal user interface what is all the information that we can gather to make LM do the better job and then are able to rapidly iterate will ultimately create the most impact with alls and so on that note in your experience of working with the llms working with other data teams and keeping apprised of the evolution of the space What are some of the most interesting or Innovative or unexpected ways that you've seen teams bring llms into that inner loop of building and maintaining and evolving their Data Systems I think the most in hindsight obvious but not necessarily obvious when you just in realization is that no one really knows how to ship llm AI based applications they obviously you know guides and tutorials and still like there's a lot you can learn from looking at what people are doing but the field is evolving so fast that nothing replaces fast experimentation and just building things it's not that you can just hire someone who worked on building an llm based application like six months ago a year ago and all of a sudden you you know gain a lot of Advantage as you would with many other Technologies like you know if we were I guess working in a space of video streaming it would be very beneficial to have extensive experience with working with video streaming and codex and with llms one no one really knows exactly how they work even the company in terms of like how they behave right in terms even the companies that are shipping them are discovering more and more novel ways of leveraging them more effectively every week and from for the teams that are using leveraging LMS like like data folds the thing that we found matter the most is the ability to a just stay on top of the field and understanding what's the what's the like most exciting thing that people are doing how they relate to our field how can we borrow some of those ideas but most importantly is is rapid experimentation with some sort of methodology that allows you to try new things measure results quickly and then being able to scrap your approach that you thought was great and just go with a different one because a lot of times when a new model is released you have to kind of adjust a lot of things you have to adjust the prompts you have to even rearchitecturing and that is both difficult but also incredibly exciting because the pace of innovation and what is possible to solve is evolving extremely fast I would say the fastest of any previous technological wave of disruption that we've seen in your experience and in your work of investing in this space figuring out how best to apply llms to the problems facing data engineers and how to incorporate at that into your products what are some of the most interesting or unexpected or challenging lessons that you've learned personally yeah I I think that the the interesting realization was that specifically for data engineering domain again if you just take the problem at face value you think well let's just build a co-pilot or an agent that would kind of try to automate data engineer way and I don't think we have the tech ready for an agent to just like really take a task and run with it yet I don't think it's been solved in software space I think it's in some ways even harder to solve in data space we'll eventually get there I don't think we are there yet I don't think that the biggest feedback you can make in bing workflow again is like having a co-pilot because that's not where the engineers spend most of their time in terms of like writing production code it's all operational tasks and there are certain kinds of problems in the data engineering space where it's not even a dayto day you know you help you save like an hour two hours 3 hours but there are certain types of workflows where to complete a task a team needs to spend like 10,000 hours and a good example of such a project would be a data platform migration where for example you have millions of lines of code on Legacy database you have to move them over to a new modern data warehouse you have to refactor them optimize them repackage them into a new kind of framework right you may be moving from like store procedures on Oracle to DBT Plus data bricks and doing that requires a certain number of hours for every object and because you're dealing with a large database that at Enterprise level sums up to enormous amount of work and historically these projects would last years and be done by a lot of times outsourced Talent from you know Consultants or or Si and for data engineer that's like probably one of the most miserable projects to do I've done I've glad such a project at lift and it's been an absolute grind where you you're not shipping new things you're not shipping AI you're not shipping even data pipelines you're just like solving technical debt for years and what's interesting is that those types of projects and workflows are actually I would say where Ai and LMS can make today the most impact because we can take a task we can reverse engineer it we know exactly what is the target of you know you move the code you do all these things with the code and ultimately the data has to be the same right you're moving you're going through multiple complex steps but what's important for the business is once you move from let's say you know ter data to snowflake your output is the same because otherwise these wouldn't accept it and that allows us to a lever LS for a lot of the tasks that historically manual but also have a really clear objective function for LMS like diffing the output on a legacy system to a modern system and using it as a constraint and if you put those two things together you have a very powerful system that is a extremely flexible and scalable thanks to all lamps but also can be constrained to a very objective definition of what's good you know unlike a lot of this text tosql generation that cannot be constrained to the definition of what's good because like how do you know migration you do know and that allows AI to make tremendous impact on the productivity of a data team by essentially taking a project that would last for years cost millions of dollars and go budget and constrain that into weeks and you know just a fraction of the price I think that is where we can see real impact of AI That's like useful it's working and we also see the powerless in software space as well there also a lot of the like really impactful Enterprise applications of AI is actually taking these Legacy code bases and you know helping teams maintain them and or migrate them and I think that there are more opportunities like that in the data engineering space where we'll see AI make tremendous impacts and as you continue to keep in touch with the evolution in the space work with data teams evaluate what are the cases where llms are beneficial versus you're better off going with good old human Ingenuity what are some of the things you're keeping a particularly close eye on or any projects or context you're excited to explore in terms of where you where I think that llms would really make a huge impact on the workflow uh just llms in general how to apply them to data engineering problems how to incorporate them more closely and with less leg work into the actual problem solving apparatus of an organization yeah so I think that on multiple levels there's a lot of exciting things like for example being able to prompt an llm from SQL as a function call that's available these days in modern data platforms is incredibly impactful right because instead of trying to in many instances we're dealing with extremely massive data and instead of having to write like complex case when statements and rexes and like udfs to be able to clean the data to classify things and to just tangle the mass we can now apply llms from within SQL from within the query to solve that problem and that is incredibly impactful for a whole variety of different applications so I'm very excited about all these capabilities that are now you know brought by the major data platforms like you know snowflake data breaks uh B query I think that the if we go into the workflow itself like what does data engineer do and how to make that work better I think there's a ton of opportunity to further automate a lot of tasks I think a big big one is data observability in monitoring I honestly think that data observability in its current state is a dead end in terms of like let's cover all the data with alerts and monitors and then be the first to know about any anomalies it's useful but then it quickly leads to a lot of noise alert fatigue and ultimately kind of could be even net negative on the workflow of a data engineer I think that this is a type of workflow where putting an a to investigate those alerts do the root cause analysis and potentially remediation is where I see a lot of opportunity for saving a ton of time for a data team while also improving the slas and the overall quality of the output of the data engineering team and that's something that we are really excited about something we're working on data full and we're excited about coming later this year are there any other aspects of this overall space of using llms to improve the lives of data engineers and the work that data Engineers can do to improve the effectiveness of those llms that we didn't discuss yet that you'd like to cover before we close out the show I think that you know we talked a lot about kind of the the workflow Improvement I think that overall my recommendation to dat Engineers today would be to learn how to ship LM applications it's not that hard Frameworks like Leng chain make it very easy to compose multiple blocks together and ship something that works whether or not you end up losing using L chain or other frameworking production and whether your you know team allows that doesn't really matter but it's really really really useful to try and build and learn all the components and by it's just like software engineering you know learning how to code opens up so many opportunities for you to solve problems right you see a problem and you're like I can write a bon crep for that and I think that with LMS it's almost like a new skill that both software engineers and data Engineers need to learn where you see a problem and you think that okay I actually think I can scill the problem into three tasks that I can give to an llm like one would be extraction web could be like reasoning and classification and now it just solves the problem and so but but really learning how to build and trying helps you build that intuition and so my recommendation would be for all dat Engineers for listening to this is try to build your own application that solves either a business problem or helps you in your own workflow because knowing how to build with LMS just gives you tremendous superpowers and will definitely be helpful in your career in the coming years I definitely would like to reinforce that statement because despite the AI maximalists the AI Skeptics no matter what you think about it llms aren't going anywhere they're going to continue to grow in their usage and their capabilities so it's worth understanding how to use them and investing in that skill because it is going to be one of those core Tools in your toolbox for many years to come and So for anybody who wants to get in touch with you and follow along with the work that you are doing I'll have you add your preferred contact information to the show notes and as the final question I'd like to get what your current perspective is on the biggest gap on the tooling or technology for data management today I think that there's a lot of kind of skepticism and some bitterness around kind of modern data stack failed Us in the sense that we were so excited that more days stack will make things so great five years ago and we're kind of disappointed and I think that I'm an optimist here I think that modern data stack in the sense of infrastructure and getting a lot of the fundamental challenges out of the way like running queries and getting data in and out of different databases and visualizing the query outputs and having amazing notebooks all of that that we now take for granted is actually so great relative to where we were you know five seven eight 10 years ago I don't think it's enough so I think that uh I am with the data practitioners for like well it's 2025 we have all these amazing models why is it still so hard to ship data absolutely with you and I think what I'm excited about is now that we have this really great foundation with modern data stack in the sense of infrastructure I'm excited about one getting everyone on Modern Data stock to the point of migrations right let's get everyone on more infrastructure so that they can ship faster obviously a problem that I'm really passionate about in solving and working second once you are on the modern data infrastructure how to keep modernizing your team's workflow so that the engineers are spending more and more time on solving hard problems and thinking and planning on the Val activities that are really worth their time and less and less on operational toil that just is burnout inducing and keeps everyone back so I'm excited about the modern data stack Renaissance thanks to the fundamental capabilities of large language models absolutely well thank you very much for taking the time today to join me and sharing your thoughts and experiences around building with llms to improve the capabilities of data Engineers it's definitely an area that we all need to be keeping track of and investing some time into so I appreciate the insights that you've been able to share and I hope you enjoy the rest of your day thank you so much Tobias [Music] thank you for listening and don't forget to check out our other shows podcast onit covers the Python language its community and the innovative ways it is being used and the AI engineering podcast is your Guide to the fast moving world of building AI systems visit the site to subscribe to the show sign up for the mailing list and read the show notes and if you've learned something or tried out a project from the show then tell us about it email hosts at dataengineering podcast.com with your story and to help other people find the show please leave a review on Apple podcasts and to tell your friends and co-workers [Music]