In Spring 2025, I began serving as an Expert in Residence for BU Spark!, a program that matches Boston University students with non-profit and local government clients to work on data-for-good projects each semester. As an EiR, it’s also my role to serve as a mentor to these students - and I’ve been giving some thought to the career advice I’d like to give to the cohort after a busy spring semester, many of whom are now (or soon to be) on the job hunt. This is advice I’ve given many times in the past to students, recent graduates, colleagues, friends transitioning careers, and even strangers who have reached out via mutual connection (not strangers anymore!). And as they say, if you answer the same question or give the same advice over and over again, you should probably turn it into a blog post. So this is me formalizing my perspective and advice into a 12-step program for how to build a career.

But first, a spoiler: This is not, in fact, a blog post about how to get a job. This is a blog post about how to find and contribute to your professional community.

This particular post is also only part 1 of a series, which means we’ll only be covering the first 3 steps out of the 12. However, I think you’ll find there are plenty of actionable steps to keep you occupied until the next part is published. Especially step 3, "build a personal website", which could keep going for as long as you want... I say as someone who just recently updated her own website theme as a side-quest to publishing this blog post.

The “why”

Traditionally, we call the process of finding connections in your professional community “networking”. However, I have found that most people have an extremely negative association with “networking”. We think of networking as transactional, full of tit-for-tat favors and unethical biases towards the already privileged. Students shy away from networking because they feel they have nothing to offer, and asking for something (like a job) when you have nothing to give (except your time and labor) feels gross. I say this because this is exactly what I thought when I was in undergrad being told to network in order to find a job. I hated it, and so naturally I was awful at it. When I hear the old adage “it’s not what you know, it’s who you know,” I think of old boys’ clubs and a world where your privilege predetermines your success. But most of us are not starting with a ready-made network to lean on, and I would like to convince you that it’s just as important to grow your network as it is to grow your skills. More than that, I’d like to flip the whole idea of networking on its head.

I didn’t learn how to “network” until I went to grad school for library & information science, and I was surrounded by librarians. Here’s the thing about librarians: we love to be helpful. We want to share that interesting book, informative link, or knowledgeable person. At library school, everyone wanted to help connect me to resources that I might find interesting or useful, simply because making a great connection gave them joy. They were not asking for anything from me, and I barely had to ask for anything from them - simply expressing interest was enough to kick-start a series of introductions. And sometimes I also knew something or someone that could help someone else - and I found joy in that. I didn’t realize that library school taught me how to network until later, but by then it had been completely reframed for me. Networking was not about exchanging favors, rather it was about being part of a professional community. And in a community, you help each other – not because you will get something in return – because it is good for the community as a whole and contributing to your community makes you feel good. It turns out it’s good for you too; a sense of belonging and inclusion can do wonders for your mental health.

I cannot completely transmit my changed perspective and lived experience to you via this blog post, but I hope I can at least partially start to reframe the act of getting to know your professional peers. So this is the last time you will see the word “networking” in this post, and I hope the next time you see that word in the wild you can mentally replace it with “community building”.

So how can you start building your professional community?

1. Find where the existing communities live
2. Introduce yourself
3. Give to the community (time, knowledge, connections)
4. Make friends
5. Build new communities

That’s it!

I know, I know, it is much easier said than done. You may have already figured it out, but I have a very important caveat to this approach: it’s slow. It’s organic. It’s not going to follow your timeline. It won’t be predictable, or guarantee desired results. And it’s going to require an investment of your time and attention over the span of years, not weeks or months. This is a strategy for your career as a whole, not for finding a job tomorrow. But if you start investing the time now, it will pay off years down the road (when you may need to find a new job rather urgently).

And as with any good investment strategy, you should start seeing some returns on your investment in the short-term, with greater gains over the long-term the sooner you start. That is to say - you don’t need to reach the end of the 12-step program before you reap any benefits. You should be gaining additional benefits with each additional step… and furthermore, they should be compounding. After all, there is a reason I’m breaking this series up into multiple parts! Me being a slow writer is… yes, definitely one of them, I fully admit it. But also because I want you to start doing steps 1-3 now!

One more caveat: not all communities are created equal. There are some really toxic communities out there, and it’s important to remember that you get to choose which to be a part of. Here’s something else nobody tells you when you are choosing between potential career paths: when you choose a career, you are in many ways choosing your professional community. And while jobs will come and go, your professional community will stay… and follow you from job to job. But every career path has potential communities based on tools, domain knowledge, skills, conferences, etc. A word to the wise: explore widely, but choose carefully, and choose how you show up even more carefully. This is how reputations are built – or ruined.

I hope I’ve convinced you to give this approach a try. And if not, I hope you find some of the more concrete suggestions I will be offering next useful. And maybe, in a few years (after some experience in the workplace) you’ll revisit this post with a new perspective.

The “how”

Now that we’ve established goals (building professional community) and set expectations (this is not a blog post about how to find a job in 10 days), let’s get down to the nitty-gritty: how to build your career and grow your professional communities in 12 steps. The first 3 steps (the focus of this blog post) are all about laying the groundwork: this is the prep work you need to put in so that you are ready to introduce yourself to communities. The actual act of introducing yourself is relatively short and easy. But introducing yourself well – and having people discover you and reach out to you – requires a decent amount of upfront effort and thought.

The next three steps will focus on finding communities, both online and IRL. Guess what you should be doing once you find and join these communities? Introducing yourself! I know, once again, easier said than done. In a future blog post, I’ll talk through some of the strategies that have helped me, a die-hard introvert, with being more social. Don’t worry, though - after following the first three steps, you’ll be very well prepared for these self-intros!

Next, it will be time to start participating in your communities. Give your time, knowledge, resources, and support - first figure out in what capacity you are able to contribute, then figure out what gives you joy, and lean into that. You will probably stay in this phase for a long time, experimenting and iterating. You may find new communities, and start the whole process over. You may change mediums - maybe you decide to create video content instead of writing blog posts. It’s your journey, and you get to decide what direction to explore.

As you continue to explore, many years into your journey, you may find some uncharted lands. You may identify some gaps in the communities you belong to, and you may identify some opportunities for new communities. Maybe you’ve identified a unique group of people who could do amazing things together if they just had a place to connect. Finally, it’s time for you to build your own community. Even if it’s for a small friend group, bringing people together is a magical thing.

Here is your 12-step program for building a career (and community):

1. Write your resume (and CV)
2. Establish your social media accounts
3. Build a personal website
4. Cultivate your social feeds
5. Join communities (on socials, Slack, Discord, etc)
6. Attend community events (in person and/or virtual)
7. Start posting (on socials, on your blog)
8. Schedule coffee chats (in person and/or virtual)
9. Speak at community events
10. Promote good work (yours and others)
11. Identify gaps - and fill them
12. Build bridges (that grow into communities)

Write your resume (and CV)

This is the most obvious, boring, and also necessary first step: if you’re going to apply for jobs, you need to have a resume. A slightly less common recommendation: you should also have a CV (Curriculum Vitae).

A resume is a highly curated document - it should be 1 page (maximum 2 pages for those further along in their career) that is formatted according to good design principles. Hiring managers spend 6-8 seconds on average looking at a given resume - and that means that all relevant information needs to jump out at them in those 6-8 seconds. Use of whitespace, fonts, font size and styling, color, order and placement of elements all matter for a resume. If everything needs to fit on one page, you have to be selective in what you include. What you choose not to include is just as important as what you do include. Your resume will change depending on what types of jobs you are applying for, perhaps even depending on individual jobs, and you should also have a standard resume that gets periodically updated - even if you’re not actively job searching. Struggling to fit everything in 1 page? Here are some tricks that may help.

In contrast, a CV is where you list everything. Every degree, certification, paper, presentation, award, professional organization, internship, job… a CV is a document you always only add to. Formatting is also less important for your CV. Your resume, therefore, is a carefully selected, curated, and formatted version of your CV. While CVs are only required when applying to certain types of jobs (government, academic), you should still always maintain your CV. You never know when having that full documented history can come in handy!

So if you have neither, start with the CV. List out everything, and leave formatting to the bare minimummaybe even just write it in markdown!. Then, consider the type of role you would like to apply to. For every piece of information on your CV, consider whether including it will help you get an interview. Pick a design for your resume first - this can be from a template or something you design yourself. Different industries may have different expectations when it comes to resume design… and if you are in an industry that values design skills, you better bring your A-game.

Quick tips

Here are some of my tips for resumes, presented in no particular order:

• Pick two complementary fonts. One will be your header font, one will be your body font.
  • Here are some resources on font pairing: https://www.fontpair.co/, https://www.canva.com/learn/the-ultimate-guide-to-font-pairing/
• Choose a single accent color. You can use this color to draw attention to particular elements, as well as to visually distinguish sections on the page. Otherwise, stick to black text on white background.
• Your name is the headline. Make it large, at the top of the page, and centered.
• Contact information belongs right under your name. Make sure to include a phone number and an email address. Optionally, also include your LinkedIn username and (if applicable, and if you want hiring managers looking at it) your GitHub username.
  • Be professional here. Make sure your voicemail recording is professional, and your email is professionalno bball_dude99@hotmail.com please - just your name is best. Make sure this is an email you actually check often, and I would lean against an organization specific email (e.g. your university email).
  • Do not include your address. While this used to be common, it is outdated now. If you know you are only looking for in person positions in a specific area, then maybe include the city and state. Otherwise, leave it off.
• Optionally, you can have a short blurb introducing yourself and the type of role you are applying for. I generally recommend against this, but it can be helpful for recent graduates or those transitioning careers, to provide context for the rest of the resume. Make this short, sweet, and easy to read.
• You should have 4 sections: education, work experience, skills, and portfolio. Education should include both formal education, bootcamps/independent coursework, certifications, etc. Work experience can include volunteer activities as well. Skills can include both specific tools as well as more general/conceptual skills. The portfolio section can include personal projects, papers, presentations, and other publications that demonstrate your knowledge and skills.
• For students and recent graduates, put the education section at the top. Everyone else, put education at the bottom. Then the skills section should be at the top. Then work experience, then portfolio.
• For each work experience item, you should have a bullet points highlighting what you accomplished, quantified if possible. Avoid regurgitating the job description. Employers want to know what you actually did, not what you were hired to do. And they would really like to know what value you added to your team and organization. Each bullet should start with a verb - so instead of “I built…” simply say “Built…”.
• For each portfolio item, include enough information that someone could find it online if they went looking. Even better - include a direct link.
• Your resume should be designed for both paper and digital viewing. Someone should be able to print the resume out and review it, but most likely they will be viewing it from a computer. This means you can and should use hyperlinks - every item in your contact info section should have a hyperlink, and every portfolio item should have a hyperlink. Make it as easy as possible for an interested hiring manager to dig deeper once you have caught their interest.
• Your submitted resume should be a PDF file (even if you created the resume in another filetype).

Establish your social media accounts

Remember how the whole point of this post is to help you find and join your professional community? Well in this day and age, that mostly happens online, on a variety of social media platforms. And that means that you need to create accounts on said social media platforms. Our first stop: LinkedIn. But before we get there, we need to cover a few necessary preliminary steps.

The first pre-req: a professional-sounding personal email. At every university and job, you will get an email address. And when you inevitably leave that organization, you will also lose access to that email. Also, your organization has access to everything in that email account. You need an email that you will use throughout your career, and one that is private to just you. This email should be as close as you can make it to your name. Every social account is going to require an email to sign up, so if you don’t have one yet, do this first.

The next pre-req: a professional headshot. This should be a high quality photo of only you, shoulders and up, in at least semi-professional attire. Most universities offer the opportunity to get a professional headshot taken against a standard backdrop, but if you can I recommend hiring a professional photographer (or a very skilled friend) to take photos in a meaningful location and to try to inject some personality into your headshot. This headshot may be the only visual your online connections have of you, so make it good. You will use the same headshot across all of your social accountsat least, those you want to be connected to you in a professional capacity. I’m not talking about your private Instagram account here. And for the love of all that is holy, do not use an AI generated headshot. These are so easy to spot, scream fake, and go completely against the whole point of a headshot: showing that you are a real person, not a bot.

Final (optional but highly recommended) pre-req: a password manager. Ideally, you should choose a platform-agnostic password manager that you can use across any web browser or device. Personally, I use and like Bitwarden (it’s free). The most popular (paid) option is 1Password. These happen to be the two recommended options from Wirecutter. Other options include Keeper, Dashlane, and LastPass. Anyone who has accounts online (so, everyone) should be using a password manager, with generated passwords unique to each account. Using the same password for everything is a security hazard, plus more and more organizations are transitioning to passkeys. If you don’t already use a password manager, do yourself a favor and make that transition now.

Ok, time to walk through the process of creating your social media accounts! This section is going to be a bit hand-hold-y, so feel free to skim or skip as needed. But if LinkedIn or GitHub is a mystery for you - don’t worry, we got this.

LinkedIn

If you do not already have a LinkedIn account, go create one. Once that is done, head to your profile page - we’re going to walk through every element you need to fill out.

Let’s start at the top: the most important elements here are your name and your profile photo. Next is your headline (aka tagline): this is the text that will appear next to or under your name and profile photo when you post. This should be short and to the point - your current/aspiring title/role/profession. Everything else in the intro is optional, and you can fill it out later.

Next: your cover image. If you belong to an organization (university or employer), they probably have a standard set of cover images you can choose from in their Brand Guidelines. This is the safest option - just remember to keep it up to date with your current organization. Alternatively, you can design a cover image for yourself. Think about what you want to communicate with it. When someone visits your profile page, this will be one of the first things they see. Some people will use another photo of themselves speaking at an event or engaged in some professional activity. If none of these appeal to you, another safe option is to use a landscape photo. If you like to travel, use a favorite scenic photo you took (this can be a great conversation starter too!).

Final bit of the intro: the about section. Many people skip over this section because writing bios is hard, or they don’t keep it up to date. This is a mistake. The about section is the first and best opportunity you have to tell someone about yourself. Consider this an elevator pitch for any future employers. You should introduce yourself, highlight key accomplishments, and state what you are currently looking for. While you can include more details in a longer bio, your first paragraph should be 2-4 sentences and include all of the most necessary information you want to convey. Edit this section carefully and get friends/family to review it for you as well.

Okay, introductions complete! Are we done? Definitely not. Remember the previous step, creating a resume and CV? We’re now going to transfer all of that information over to your LinkedIn profile. Your LinkedIn can be functionally identical to your CV, or you can exclude any irrelevant experiences. Take the time to go through and fill out all of the sections in your profile - and don’t forget to add skills!

Finally, one bonus step: trade recommendations with a peer or ask for one from a professor/supervisor. Are you working on a significant group project, for a class or a club? Identify someone who you worked with closely, who can speak to your contributions (and vice versa), and offer to write them a recommendation if they would be interested in writing one for you. If your professor recently wrote positive feedback for you on a major project, ask if they could put that same feedback in a LinkedIn recommendation. For strangers on the internet, knowing that someone else vouched for you can help a lot with establishing initial trust.

Your profile page is now complete: make sure to add a link to your LinkedIn profile on your resume. You are also now ready to start adding connections. We’ll cover more of this in a future step (Cultivating your social feeds), but you should go ahead and start making connection requests to anyone you personally know. Family, friends, professors, students from your classes or clubs, current and past employers - if you know them and they have a LinkedIn account, add them.

GitHub

If you are reading this blog post, I’m assuming you probably write code. If you stumbled on this and you never write code and do not plan to, feel free to skip this section. But I’m guessing that this will be relevant for most people reading this.

If you do not have one already, go ahead and create a GitHub account. Fun fact: you can add as many email addresses as you want to your GitHub account. My advice is to only ever have 1 GitHub account. While some organizations may require you to create a specific GitHub account just for them, this is honestly rare. Instead, you should add your organization-specific email addresses, and route all notifications to the appropriate email based on the GitHub organization.

A GitHub account serves many purposes. Its primary function is for version control - you should be keeping all of your code in a git-tracked repository, and GitHub is (currently) the de-facto remote location. If you write code, and you have a GitHub account, I’m assuming that you know git (or are about to learn git, maybe even via this workshop I developed/adapted). If you don’t know git, this is something you need to learn pronto.

A secondary function of your GitHub account is to serve as a portfolio of your (coding) work. This means that you should have some public repositories with your work - personal projects, school projects, or work projects that can be safely made public. And guess what is just as important as having these projects in public repos? Documenting them. Every repo should have a well organized and thorough project README that introduces strangers on the internet to your work. (Note: READMEs should be markdown files. If you don’t know markdown syntax, invest 30 minutes in learning it). The audience may be other coders who want to build off your work, or prospective employers checking to see if your portfolio matches your resume. They are not just interested in if you can write good code - they want to know if you can write clean, documented, well-organized code. In other words - are you going to write PRs that they love or hate reviewing? So don’t just use a repo as a place to stash code that is functional enough to work - put effort into curating your repos and making them a joy to read.

However, we all have junk repos from that barely started personal project or that one-off workshop. I’m not saying you need to put a high level of care and attention into every repo - just those you want potential employers and community members to look at. And there are a few tricks you can use to direct their attention to the right repos.

First, put some effort into your GitHub profile. Go to your GitHub profile page, and edit your profile. Add your headshot, your (current or aspirational) role title, your LinkedIn and other socials, etc. Next, create a README for your profile. Remember your About section for LinkedIn? You can recycle and modify that bio for your GitHub profile. You can (and should) put links in your personal README - to your socials or other contact info, and to specific repositories that you want people to look at.

Right below your profile README will be your pinned repositories. You can choose up to 6 repos that you want to direct people to by pinning the repository. And if people go straight to your repositories (though it is far more likely they will look at your profile first if you’ve put any effort into it), they will see the repos in order of most recent activity by default. That means that if you put some extra effort into updating (documenting!) the repos you most want people to see, those will be the repos they see first.

Finally, consider making any repos you don’t want people to see private. While GitHub used to limit the number of private repositories for personal accounts, they no longer do, and you can freely switch between public and private in your repo settings.

One final tip: if you are a student, you should absolutely sign up for the GitHub Student Developer Pack. Among a myriad of other benefits, you can sign up for a .me domain name on Namecheap and get 1 year free. This is optional but highly recommended for the next step - building a personal website. Seriously, if you are a student, go sign up for the Student Pack right now - there are so many freebies and learning resources that are absolutely worth it.

Bluesky, Mastodon, etc

While LinkedIn and GitHub are the only two I’m going to require (well, theoretically require, I can’t actually force you to do anything), I am going to highly suggest you create (professional) accounts on other popular social media sites as well. My top recommendation (at the moment) is Bluesky - think of it as the successor to Twitter. While X may still exist atop Twitter’s ashes, I’m not going to refer anyone to that raging dumpster fire of a site. There are also many thriving communities on Mastodon, especially among academia. Mastodon is a bit more complicated and less user-friendly than Bluesky, so you may want to research it a bit first. Speaking of academia - if you have ever published a paper, you should sign up for ORCiD. If you make (or want to make) professional video content, then create a professional YouTube (and Instagram, and TikTok, etc). Do not conflate these professional accounts with your personal accounts, though - that way danger lies. Reddit accounts are usually anonymous, and I’ll stick with that trend - create a dedicated professional reddit account (that can remain anonymous) and follow relevant subreddits. This can be a great way to learn new things and figure out what your communities are talking about right now.

There are so many social media sites out there… it is really up to you to figure out which sites and communities you want to invest time in. Typically on these sites you can be more of a whole person - keep it kind, but be a real personvs on LinkedIn, where you are very specifically the most professional and filtered version of yourself. My top suggestions (for sites like Bluesky/Mastadon): use a standard username (if available) that is as close as you can make it to your name, and add your headshot and a short bio as soon as you create your account (before you start following people!).

Build a personal website

Creating accounts on LinkedIn, GitHub, and other socials gives you an online presence where you can connect with people and present some information about yourself. It is also relatively easy to establish a presence on these sites. So you may be strongly inclined to skip this next step - after all, building a personal website is more difficult, time-intensive, and requires some technical skills. I’m going to first attempt to make the case for why you should absolutely build a personal website, and then I’m going to walk you through the process.

Why you should have a homepage on the internet

There are both ideological and practical reasons why you should have a personal website. Let’s get the manifesto out of the way, then we’ll discuss practicalities. A personal website is your home on the internet - your personal space that you have complete control over. You get to choose the design, the content, when and how you update it, and how people can engage with it. You own it - not some big corporation. As anyone who has been around on the internet for long enough can tell you, you don’t own anything on the social media sites. And eventually, they will die, or become so enshittified that they might as well be dead. The content that you put on social media sites is not really yours - it’s provided by you, to the site, just with your name attached. You are subject to their algorithm… for better or for worse (and believe me, it can get pretty bad).

Think of social media sites as places in your neighborhood that you go visit - the local coffee shop, the bar, the book shop, the cafe. But you don’t live there - you live in your house. Your house is the space that belongs just to you, and you get to furnish and decorate it as you choose. You get to decide who can visit and when. And when your favorite local coffee shop closes down, that’s very sad, but you aren’t suddenly homeless. You can convince your friends to meet up with you at the book shop instead.

If you aren’t convinced by the ideology of having a personal website as your home on the internet (and trust me, I’m far from the only one with this perspective), then let’s talk about the practical reasons. Your personal website is a one-stop-shop for someone to learn everything (you want them to) about you. Your website has your bio, resume, cv, portfolio, writings (blog posts!), socials, publications, projects, learnings, hobbies…. Everything and anything you want to put on it. It is a place for you to curate all of the information about you on the internet - and that content can either live on your website or you can point to it. With a personal website, you are making a potential employer’s job much easier - they don’t have to go look you up on all these different sites to try to see if you do good work and are a potential fit. On your personal website, it’s all there in one place, and you control how it is presented.

Convinced yet? Hopefully you are… and if not maybe you’ll trust me enough to follow along and reap the potential benefits later. Let’s move on to talking about how to actually accomplish this fantastic internet feat.

Getting started

There are many, many ways to build a personal website. If you are in web development, you do not need my help or my convincing - your personal website is your portfolio, and it’s probably custom. If you have zero technical skills and do not aspire to gain them, you’re probably going with one of the classics like WordPress.

But if you know basic git, have a GitHub account, and are comfortable writing markdown and editing YAML files, then I’m going to suggest my personal favorite method: GitHub Pages and Jekyll. This website is built on GitHub Pages, uses a pre-made Jekyll theme (al-folio), and lives at a custom domain name (which I obtained via the GitHub student pack and have been paying NameCheap approx. $15/year for after the free first year ran out). You don’t need a custom domain name (though I personally think it is worth it), and you get to choose whatever theme or technology you want to build on.

I’m going to continue the “how-to” portion of this section with the assumption that you are following this recommendation, though if you prefer a different method you may still find some of the following content helpful.

Step 1: Brainstorming

What do you want out of your personal website? Before you start building, you need to plan, design, and prioritize. Come up with a list of features that are important to you. This is going to help enormously with the next step (picking a theme).

Here are some examples of features that you might want for your personal website:

• Resume/CV page
• Bio/About page
• Blog
• Social media links
• Social media feed (latest posts)
• Bookshelf (books you’ve read/recommended)
• News/recent updates
• Github portfolio/repos
• Academic papers published
• Portfolio/project pages
• Hype/Recommendations page (for example)
• Media pages (photography, videos)
• Customizable color themes
• Light/Dark themes
• Language options

What features do you absolutely want, and what features are nice-to-have?

Next, consider what you want your homepage to look like. Do you want your bio front and center? Or do you want people to see your blog posts first? Maybe you want a portfolio carousel to be near the top. Or maybe you just want visitors to see your resume. Even if your website has all of these capabilities, only one primary feature can be the first thing visitors see when they visit your website - everything else will require clicks (or scrolling) to navigate to the right spot. What you choose to feature on your homepage communicates something about your priorities. While many themes are flexible enough that you can alter what shows up as your homepage, it is usually easiest to go with the default.

By choosing what features you need, what features you want, and what you want your homepage to focus on, you are now ready to filter through the multitude of themes and select one to build your website with.

Step 2: Choose a Theme

Even if we restrict the options to Jekyll themes, there are many, many options out there. Browse through the many options at jekyllthemes.io, though you probably want to restrict the options to only the free ones. You can also find curated lists, like this one of awesome jekyll themes. I personally have used (and can highly recommend) two themes: minimal-mistakes and al-folio (I started out using minimal-mistakes, then a few years later switched to al-folio. This is part of the beauty of having a personal website - you can change the theme/design any time!)

If none of the Jekyll theme options are speaking to you, then go ahead and look at the Hugo options as well. Hugo is becoming more popular, and many folks are switching from Jekyll to Hugo. Check out some Hugo themes here. I’ve noticed that the blowfish theme is pretty popular, as well as the LoveIt theme. Or maybe you want to build your website with Quarto!

Based purely on available features and appearance (explore the examples!), pick out ~3 themes that you like. But before making your final choice, you should read the documentation. You should know how much work it is to first set up the website, and to maintain and extend the website. Different themes may have different setup steps, and before committing to a theme, you need to take this into consideration.

Step 3: Create & clone the repo

Before you go creating a repo called my-website, slow down and read some documentation first. Start with the Quickstart for Github Pages. You should notice something very important here - your repo name needs to follow the formula of <username>.github.io. You also need to read the setup documentation of your chosen theme. For example, al-folio recommends creating the repo by using the theme as a template. Any theme you choose should have well-documented setup instructions. Read the instructions first, then follow them - exactly.

After creating your repo on Github, you will need to clone the new repo to your local computer, and then set up your local development environment. Why do you need a local development environment? Because you are going to want to be able to preview changes to your website before publishing them. Again, this should be documented as part of the theme’s setup instructions. There are also generic Jekyll instructions in the Github pages documentation.

Step 4: Make your first updates

One of the most important files in your new website repo will be the _config.yml file. This is where all of the core settings for your site will live, including key information like your site’s name. Go through this file and make all the updates you want. After you are finished, save the file. Then, open a local preview of your website. If you are using Jekyll, you will want to type this in the command line: bundle exec jekyll serve. Either a localhost webpage will automatically open, or you should see a line like this in the terminal: Server address: http://127.0.0.1:4000. This is your localhost URL, and if you copy and paste http://127.0.0.1:4000 into your browser, you should see your website.

At this point, the version of your website you see on localhost should be different from the published version, because you haven’t yet pushed your changes. Let’s compare by looking at the currently published version at <username>.github.io. See steps 20-23 at this documentation page for further help.

After making the most important updates to your site (updating _config.yml and perhaps removing some of the example material), and previewing the changes on localhost, go ahead and push your changes to Github. This will kick off a Github Action to update your website (see step 24 of the aforementioned docs). Within a few minutes, you should see these updated changes on your published site (at <username>.github.io).

It is during steps 3 and 4 that you are most likely to encounter confusing issues. Whether you missed a step, something wasn’t documented, or something has changed since the documentation was updated, it is totally normal for you to encounter some bumps along the road. I highly encourage you to persist, and reach out for help if you need it.

Step 5: Use a custom URL (optional)

While you are welcome to use the github.io URL for free in perpetuity, there are some advantages to using a custom domain name. For one, custom domain names are pretty useful. They can help you establish your identity and brand online, and they also help communicate authenticity and credibility. It is also one further step in you owning your home on the internet (and again, don’t just listen to me!). You can do cool things like set up your own email hosting or switch your bluesky handle to your domain. In the long run, you are likely to find many uses for owning a domain name. If you have the Github student pack, this should be a very easy decision - register a .me domain name for free, and hopefully your name (or some variation of it) will be available.

After registering your domain name (and you may need to give it up to 24 hours after actually buying the domain name), you will then want to configure Github pages to use your new custom domain. As always, read the documentation, and follow the steps exactly. You will also need to update your _config.yml file with your new domain name.

This is the last tricky setup step. After you have completed all of the setup (following the documentation!), visit your new custom URL and verify that you can see your Github Pages website (debug as needed). You are now ready to start curating the content on your website!

Step 6: Update the bare bones basics

There is a lot of content that you can put on your website, and fully building out your website will take time. Blog posts, projects or portfolio items, and other such content are important, but they are not essential right now. Your first goal is to get your website into a shareable state - that is, you can put a link to it on your resume and not be embarrassed. So let’s go through the key items in roughly their order of importance.

Your home page. This is the first thing a visitor sees when they visit your website URL. Some websites only have a homepage, but most separate out the pages. If you choose a single-page theme, then consider your homepage the portion you view without scrolling. Look at all of the elements on your home page, and update them. Your website will have a title - I highly recommend using your name as the title. After all, this is your personal website, all about you. You’ll also likely want to include a profile photo, and some kind of “about me” blurb and/or tagline. Sound familiar? These are the same key elements you should have updated on your social media profiles. Be consistent, and re-use the same content for your personal website.

Your bio. The theme you chose may have a separate, longer “about me” page, or the about me may be simply part of the home page. Re-use the same “about me” content you wrote for your LinkedIn profile - you can tweak, add to, and improve it later!

Your socials. Most themes have a specific way to display your social media profiles - figure out which profiles you want to link to from your website, and make sure you have added the correct URLs in the correct place. At a minimum, include your LinkedIn and GitHub profiles. You should also include your professional email address.

Your CV. Most personal website themes include a dedicated page for displaying your CV. If not, there should at least be some way to link to your resume PDF file. When filling out your CV page, again, re-use content - copy exactly from the resume/CV you’ve already written, which you also should have already re-used on your LinkedIn profile.

The menu bar. Likely at the top of your website, there are a number of pages and menu options. There may be some pages you plan on filling out later, while others may not be relevant to you. You don’t need to delete the pages, necessarily - just remove the links to them in the menu. When you visit your website, you should only be able to navigate to pages that have actual content on them.

Optionally, a placemark blog post. You don’t need to actually write a blog post yet, but if your website has a blog section (which it should), make sure to remove any posts from the template and write a single blog post with the title of “Coming Soon!” or something similar. Include a brief blurb that this is a new website and you’ll be updating with your first blog post soon.

Throughout this process of updating the bare bones basics of your site, you will also be learning how to use your website. This is a learning process! So take it slow, forgive yourself for mistakes, and treat this as an exploratory exercise. You’ll need to learn how to add and reference images and files appropriately. You’ll need to learn how to update yaml files, and where the files that control certain elements of your website live. Throughout this entire process, you should have a local preview of the website in your browser, and you should preview each change after each file save. After you are finished, make sure you commit and push all of these changes, and then go check it over on the actual live website. You may find things not displaying as intended, and you will need to further learn how to debug these issues. While the destination (of having a personal website) is the goal, the (learning, skill-building) journey is just as important. Not only are you learning new technical skills, but you are also learning how to write about and market yourself. You are telling the world a story about who you are - exactly what you will need to do in interviews.

Step 7: Request a review

You now have a shareable website - congratulations! But before you go publicizing this fact to the world, ask a few trusted people to review it first. Whether it be family members, friends, colleagues, or other advisors, invite feedback from people who you know can give you an honest critique. At the end of the day, it is your choice on whether to take any of that feedback, but the process of getting an outside perspective is invaluable. Whether it be typos, phrasing, framing, or design, there is something that can be improved, and I promise you will benefit from a second (and third!) perspective. All you have to do is share the URL, what you are trying to accomplish with the website, and what type of feedback you are open to. Then, make sure to update your website with any of the feedback you agree with!

Step 8: Share & Publicize

Now, you are finally ready to share your website with the world. First, distribute the URL for your website - put the URL on your resume and in the bio of all of your socials (whether it be in an official spot for your personal website or just at the top of your bio). Then, share the news by posting about your new website on all of your socials. It’s official: you now have a personal website! The hard upfront work is done, and you can start to reap the rewards. But remember: what you just created was the foundation. It is now your responsibility to build on that foundation over time. After all, your website just has the bare bones basics right now. You have some content to add!

Step 9: Your first portfolio entry

Your portfolio is something that you will build over time. You may already have a number of projects you can write about for your portfolio, in which case you have a backlog to slowly work through. Or you may not have a lot of projects right now - but you will in the future, and you need a place to write about those projects. For right now, focus on getting just one entry into your portfolio. If you want to keep going and add more, go for it!

For your first portfolio entry, pick the project you’ve completed that you are most proud of (and also can be fully shared on the internet). Include all important materials about this project - all of the relevant files and links (including to the code!). If possible, include as many images as you can. Write a basic description of the project, and then write a longer narrative - what you learned, what the outcomes were, etc.

Step 10: Your first blog post

Most people are not quite sure what to write about in a blog. You will find that you probably have quite a bit to write about over time, but we will cover that later. I am a strong believer that your first blog post should essentially be a longer “about me” page. Your bio should be short and sweet - only the headlines, not the full story. This first blog post is your chance to tell your story in your real voice. You can talk about your accomplishments, important and formative experiences, your hobbies and interests, and even your ambitions and goals in life. Don’t worry as much about sounding “professional” - you want to let your voice and your personality shine through. After reading this post, I want to feel like I know you better - that you’re someone I could comfortably strike up a conversation with.

Again, go through a review process before you publish this post. And again, officially “publish” your first blog post by posting about it on your socials. These are good practices to follow with every new blog post or addition of content to your website - ask someone to review it first, and then share it with the world via your social media (this goes for new portfolio entries as well!)

Over time, keep adding your website. Continue to flesh out your portfolio, write new blog posts, and keep your website up to date. We’ll cover more about writing content in Step 6, “Start posting” (in part 2 of this blog post series). But you now have a solid foundation and are ready to introduce yourself to the world.

Introduction Interregnum

Thus concludes my advice on how to prepare to introduce yourself, and part 1 of this blog post series. There is a reason we are taking a pause here - completing these first 3 steps takes a lot of time and effort! I’d rather get this section into your hands sooner, and hope that some of you can get started and benefit from my advice now, than wait to publish until all 12 steps are written out. I hope you can also see how these 3 steps build on each other, and you can and should re-use the work you put into one step into the nextthere is no need to re-write your job descriptions 3 times!.

The next steps rely on you having followed these 3 steps first. For example, the very next step is “Cultivate your social feeds”, which boils down to following and connecting with people. Every time you click follow, there is a chance they will look at your profile and decide whether they want to follow you back. Your chances of being followed back are much greater if you are clearly a real person who is doing interesting work - as demonstrated in your bio and on your website!

There is one final piece of advice that I want to leave you with as you embark on this self-introduction journey. Please, please, do not use AI* (*LLMs, a.k.a ChatGPT, Claude, Gemini, etc) to write any of your content. Do not use AI to write your bio. Do not use AI to write your intro blog post. Do not use AI to edit/create your profile photo. Talking and writing about yourself is hard, but it is a muscle you must start exercising. Here’s another reason why: by having a social media presence and a personal website, you are giving potential employers a big signal that you are a real person. Right now, recruiters and hiring managers are drowning in a sea of AI-generated resumes and cover letters. They are painfully easy to spot, and while some may be attached to real people, it simply isn’t worth the effort to sort “real” AI-generated resumes from fake - so they all go in the trash. If your social media profile or personal website smells AI-generated, you end up accomplishing exactly the opposite of what you are intending - people may assume that you are not real. So, don’t. It doesn’t matter if your bio is a bit awkward, if you have the occasional typo, if your profile photo isn’t studio perfect. What matters is that you are genuine, and that your authenticity shines through.

I hope I have convinced you that establishing a presence online - through social media platforms and through a personal website - is worth the investment. And hopefully, I have also demystified what it takes to get started. Still, learning and doing new things can be scary - and it’s not going to stop being scary until you actually try. So if you’ve made it to the end of this blog post, I want you to make a deal with you: I’ll keep writing the next parts of this series, and you follow the 3 steps I’ve already laid out that is, those that you haven’t done already!. And if you have any questions or run into any obstacles, you can message me and I will do my best to help. Deal?

The Coalesce session recording is now available! See the Appendix

During my time as a consultant with Analytics8, I worked with two different companies who were both implementing dbt Mesh. I learned so much from both of those engagements, both from my very smart colleagues at both Analytics8 and the clients, and from the problems I had to solve while implementing dbt Mesh. I also started learning more about data mesh, and added the Data Mesh bible to my bookshelf. I saw the chaos that could result from implementing dbt Mesh simply as a convenient higher level of organization, and I became convinced that one of the best ways to implement dbt Mesh was with data mesh as a grounding, opinionated philosophy.

Data Mesh exploded into the data scene in 2022 with Zhamak Dehghani’s book “Data Mesh: Delivering Data-Driven Value at Scale”.Every quote throughout this blog post, unless otherwise noted, is from this book. Zhamak characterized data mesh as a “decentralized sociotechnical approach to share, access, and manage analytical data in complex and large-scale environments – within or across organizations.”

She clarifies that data mesh is about analytical data, not operational data, and that it is an approach to data management, not an architecture. It belongs properly as part of an organization’s data strategy, and provides a target state to iterate towards for both the organizational operating model (socio) and enterprise architecture (technical).

Data mesh is defined by its 4 core principles, which when applied can shift how data is managed organizationally, architecturally, technically, operationally, principally, and infrastructurally. And perhaps most importantly, data mesh is not for every organization - it is only intended for organizations that have reached a certain scale and maturity (and therefore complexity) in their data practices.

Zhamak is purposefully tech & tool agnostic in her book, though she went on to found her company Nextdata to create the tech to best implement her vision of a data mesh. This means that as long as you are following the 4 principles of data mesh, you can use any set of tools you want to build your own data mesh - including dbt.

We are going to explore these 4 principles through the lens of dbt, with the goal of outlining an opinionated approach to how you should architect your dbt Mesh of project(s). While dbt is extremely flexible in how it can be implemented, often a strong structure and grounding philosophy can be more helpful than endless possibilities.

Much like data mesh, dbt Mesh should really only be considered if your organization has reached a certain scale and maturity in your data practices. Like data mesh, dbt Mesh was designed to solve problems encountered by organizations maintaining very large and very mature dbt projects. If this is not you, congratulations: you don’t have to deal with all of the additional complications data/dbt mesh introduces for a while yet. Enjoy it while you can!

A brief history of data mesh (and further resources)

Zhamak first started writing about Data Mesh in 2019, when she published “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” on her ThoughtWorks colleague Martin Fowler’s blog. She followed this up with “Data Mesh Principles and Logical Architecture” in 2020, and then fully articulated the data mesh approach in her 2022 book “Data Mesh: Delivering Data-Driven Value at Scale”.

Data mesh made it onto Gartner’s 2022 Hype Cycle for Data Management in the “Innovation Trigger” phase (the first phase), but was predicted to become obsolete before reaching the final phase, the “Plateau of Productivity”.

Data mesh quickly gained a following among data practitioners who believed this approach could solve many of the pain points they had been experiencing in managing data at very large organizations. The Data Mesh Learning Community was founded in 2021 to help practitioners connect and learn about data mesh from each other.

After Zhamak’s book was released in 2022, she quit her role at ThoughtWorks to found her own company, Nextdata, to create her ideal implementation of a data mesh architecture. For 3 years they were quietly building, and meanwhile many in the data community declared that data mesh was dead (as they are often wont to do) - that while it sounds great in theory, it is impossible to actually implement, and data teams should instead adopt pieces of the data mesh strategy and then move on. But in April 2025 Nextdata publicly launched their product Nextdata OS, an operating platform purpose-built for autonomous data products. Data mesh is once again in the public dialogue - it seems to have survived the “trough of disillusionment” and is now climbing the “slope of enlightenment” (though I suppose we’ll have to wait for next year’s hype cycle report to see if Gartner agrees).

Data mesh as a concept has also evolved as the thought leadership on data mesh has expanded beyond Zhamak to include others like Jean-Georges Perrin (JGP), who co-authored “Implementing Data Mesh” with Eric Broda in 2024 and authored the soon-to-be-published “Building Data Products”. JGP’s focus is primarily on the principle of data as a product, and you can read more about his approach in the blog post “The Next Generation of Data Platforms is Data Mesh”. JGP is also the chair of the Linux Foundation’s Bitol project, which has published the Open Data Product Standard (ODPS) and Open Data Contract Standard (ODCS).

Why data mesh and dbt

Data mesh and dbt both share a single fundamental mission: bringing software engineering best practices to data engineering. With dbt, analytics engineers got version control, CI/CD, isolated development and production environments, testing, and documentation out of the box. Similarly, Zhamak positions data mesh as bringing the decentralization revolution in software engineering (microservices architecture) to the data world. Notably, both data mesh & microservices architecture were deeply influenced by Eric Evans’ book Domain-Driven Design.

While the product name choice of “dbt Mesh” may make you think that it is dbt’s solution to data mesh, there have actually only been a few explicit lines drawn between dbt Mesh and data mesh. For the most part, dbt Labs maintains that dbt Mesh is a natural evolution of dbt as enterprises scaled up from a single dbt project into needing multiple interconnected projects. However, Hope Watson (a dbt Labs Resident Architect) made the connection between data mesh and dbt Mesh explicit in her training course “Data Mesh with dbt Cloud for dbt Practitioners.”

dbt Mesh’s product history reflects this natural evolution, and I could not have always recommended dbt as a way to implement data mesh. There are still several serious limitations that will influence your dbt Mesh architecture, but I also hope that the perspective in this blog post - dbt Mesh as a way to implement a version of data mesh - can influence the dbt Mesh product roadmap enough that these shortcomings will eventually be resolved (read: dbt product and developer experience teams, this is for you too!).

dbt Lab’s product strategy has also shifted to better enable a data mesh implementation - beyond support for Iceberg and Cross-Platform Mesh, the metadata capabilities unlocked by dbt Fusion are hugely important for enabling key features needed in a data mesh. We’ll cover the most important feature, state-aware orchestration, in the section on the principle of domain ownership.

One final note: for the most part, folks implementing data mesh using dbt will be dbt Labs Enterprise customers. Most dbt Mesh features require an Enterprise or Enterprise+ plan. However, if you are really set on a data mesh with dbt core, it is not an impossibility - you will just need to work a bit harder and figure out your own setup. I’ll point you to the dbt-loom plugin for cross-project dependency management and Dagster for multi-project orchestration (see a demo of how it works!).

The 4 principles of data mesh

Data mesh is defined by 4 principles that work in concert:

1. Domain Ownership
2. Data as a Product
3. Self-serve Data Platform
4. Federated Computational Governance

Each of these 4 principles can guide how you should implement dbt Mesh for distributed data management at scale. Let’s take them one at a time.

Domain Ownership

The principle of Domain Ownership is the first stumbling block for organizations trying to implement data mesh, because it requires organizational change. While other principles can be (mostly) accomplished by technologists, this principle requires buy-in from leadership and long-term investment in data mesh as the data strategy.

Rather than having a central data team that serves all departments, organizations should decentralize the ownership of analytical data to the business domains, and ensure that team structure is aligned by domain. Each business domain (already its own organizational unit) should be responsible for curating, maintaining, and sharing the data that it is most familiar with.

The implication here is: if your organization currently has a single centralized data team, and cannot or does not wish to functionally have many small domain-oriented data teams, then your organization is not ready for data mesh (and that’s okay!). But if your organization is already organized by domain with data practitioners embedded, they are probably itching for ownership over their own data and the support to do so successfully.

Applied to dbt

Simply put, the recommendation here is simple: align your dbt projects to your domains. Each business domain, with its own set of self-sufficient data practitioners, should have ownership over their own dbt project.

Of course, nothing can be simple when you are playing in the waters of distributed systems. Up until very recently, it was not possible (and then not recommended) to completely align your dbt projects to your domain teams due to complications around bidirectional project dependencies.

When dbt Mesh was first announced at Coalesce in 2023, dbt Labs advocated a hub-and-spoke model, where a central data team still maintained the “hub” project, but now downstream domain teams could use models from the “hub” while developing their own custom models in isolated “spoke” projects. This architecture was necessitated by the fact that project dependencies could only go one way - the “hub” project could never depend on models from one of the “spoke” projects. Not very meshy, was it? In a data mesh, domain teams should be able to serve and consume data products to and from each other.

This meant that, functionally, all dbt projects had to be a DAG (directed acyclic graph) as well. As a result, organizations were either forced into a hub-and-spoke strategy (though this likely was not an issue for organizations wanting to continue with having a single central data team), or domain-oriented projects would get split up into “producer” and “consumer” projects.

Fortunately, after product feedback from customers (including yours truly), dbt Labs made bidirectional project dependencies possible in 2024, with the feature generally available in October along with a suite of other mesh-related product announcements at Coalesce (notably cross-platform Mesh). While all of the models across all of the projects still had to adhere to the “acyclic” rule of DAGs, the projects themselves no longer had to form a DAG.

But - and this is a very big but - there was still no elegant way to orchestrate for these bidirectional dependencies. dbt Labs knew that they would need to figure this out, and pre-emptively announced adaptive jobs as a feature that would be coming soon, but until then it would be inadvisable to architect a dbt Mesh with bidirectional project dependencies.

We finally got resolution for this story in May at dbt’s Launch Showcase with “State Aware Orchestration”, a feature of the new dbt Fusion engine… which as of this week at Coalesce 2025 is now in public preview! This means that there is now a way to orchestrate for dbt Meshes with bidirectional project dependencies, but only if you are also able to transition over to the dbt Fusion engine. dbt Fusion and state-aware orchestration are also not yet generally available, so implement with caution (and report bugs).

Now that we’ve covered some of the major dbt Mesh product developments that have resulted in the fact that I can finally now recommend organizing your dbt projects according to the first data mesh principle of domain ownership, let’s talk about the other complicating factor: your organization. There are two different organizational profiles that are likely to implement dbt mesh:

1. The organization that has been using dbt for years, with model counts ballooning into the thousands and DAG that looks like the Cthulu version of a spaghetti monster, finally needing to split their single project up into many for the sake of everyone’s sanity
2. The organization that is new to adopting dbt but knows they either want a data mesh or are already operating with a data mesh strategy, and therefore want many domain-aligned dbt projects

Both types of organizations may read my recommendation of “align your dbt projects to your domains” and say, “yes, let’s do exactly that!” and immediately proceed to create many domain-aligned dbt projects. To which I must say… “wait! Not yet.”

Cue the disgruntled groans and disappointed sighs. To the organizations of the former type - those with old and overgrown dbt projects - you will only need to take a short pause. To the organizations of the latter type - those new to dbt - you will likely need a much longer pause (but only a pause! I promise you can get to multiple projects soon enough). Both are on the same journey, but one is much further along, while the other can get to the final goal (true mesh with many projects) much faster but still needs to go on the journey of adopting dbt in the first place.

I know that dbt Mesh sounds very exciting and promising (and really, it is), but it also introduces a lot of complications that you really should not want to deal with until you absolutely have to. As dbt Labs DevEx advocate Benoit so elegantly stated, “dbt mesh is about adding good friction to the data transformation process.” Adopting dbt introduces enough friction in the first place, and jumping straight to dbt Mesh may introduce too much friction - enough that the transition may fail entirely.

All of this to say: if your organization is new to dbt, start in a single project. You can just architect this single project with the goal in mind that you will eventually transition to multiple projects. The adage of “slow is smooth, and smooth is fast” comes to mind - I promise that you will have a smoother time adopting dbt if you start with one project, and you will get to a successful dbt Mesh implementation faster than if you had started with multiple projects.

So, let’s talk about how to design your single dbt project such that the transition to multiple dbt projects will be as painless as possible. Type 1 organizations (those with old & overgrown projects), this is what you want to transition your existing dbt project to look like. Type 2 organizations (those new to dbt), this is how you want to design your new dbt project from the ground up.

First, use groups to plan out your projects. Before creating one dbt project per domain, create a group per domain, and ensure every model belongs to a group. Planning is cheaper than doing (see: How Big Things Get Done), and groups provide a cheap (time is money!) way for you to plan out (and iterate!) your domain-oriented projects. Trust me - it is way easier to change what group a dbt model belongs to than the project it belongs to, and it is way way easier to change what groups exist than to change what projects exist. The group config can be assigned by folder in the dbt_project yaml file, and overridden on a model-by-model basis.

Next, align your folder structure to your groups. You may already have a well organized dbt project and find that all models in the same folder belong to the same group - in that case, assigning groups and consolidating the folder structure will be very easy! But if you find that you need to assign groups on a model-by-model basis, then you are going to need to spend some time reorganizing your models and folder structure.

The typical folder structure for a single dbt project is to organize by pipeline stage first - staging, intermediate, and marts as the top-level folders in the model folder. Then you should organize by source or domain inside those folders. As you get closer to migrating to multiple projects, however, I recommend flipping that structure: your domain (a.k.a all models in a single group, a.k.a all models that will be split out into their own project) should be your top level folders, with the pipeline stage folders (staging, intermediate, marts) replicated inside each top-level domain folder. If you are starting out in a new dbt project that you know will be split out later on, I recommend using this configuration to start with. This structure of top-level domain folders should be replicated across all of the top level dbt project folders (models, seeds, tests, macros, analyses).

Once your project is organized by domain folders, you can start enforcing domain ownership within your git workflow by implementing a codeowners file. Create teams in your git platform of choice that aligns with the domain-oriented team that will be owning the future domain-oriented dbt project (a.k.a the dbt groups). Any changes to folders belonging to that team must be approved in a pull/merge request by the team that owns it. Similarly, you can now start to slowly shift more and more dbt project ownership activities to these domain teams, like maintaining orchestration jobs, monitoring for test failures, etc.

It can be easy to underestimate just how much work it is to manage a dbt project. It requires more advanced skills in git, dbt, and cloud platforms, and by slowly transitioning these responsibilities one at a time, you can provide a safe space for domain teams to advance these skills and figure out for themselves how they want to distribute responsibilities. This is one of the primary reasons I strongly encourage organizations new to dbt to start with a single project: to centralize and thus reduce this burden until enough people have the skillset required.

Assigning your dbt models to groups provides another key advantage in drafting out your dbt projects: enforcing model access. By default, models have an access level of “protected” - they can be referenced by any other model in the same project. Once you have implemented dbt groups, you should change the default access level for models in your project to “private”, limiting only models within the same group to reference each other. Then, you will want to decide which models should be referenceable by models in other groups (and eventually other projects), and assign them to have an access level of “public”. These are the models that will turn into “data products” - more on that in the next section.

You may encounter some errors in this process, as you realize that models with access level “private” were actually being used by other groups. This is a good thing! You are finding these errors now, while everything is still in the same project and errors are easier to resolve through refactoring or configuration changes, versus after migrating models into many projects (after which it becomes much more difficult to refactor and resolve).

At some point, your groups will reach a point of stability, your domain teams will reach a point of self-sufficiency, and your public models will be mature enough that you can finally say: we are ready. We planned, we drafted, we prepared, and now we are ready to migrate into multiple projects. Even if domains do not reach this point on the same timeline, if you have followed my recommendations on your dbt project structure, pulling a single domain out into its own project one-at-a-time should be relatively easy. If you want to do it all at once, I recommend you check out dbt-meshify.

Once you reach the end of this process and all of your domain teams have moved into their own dbt projects, don’t be so fast to deprecate the original project. You may have new domains who need help getting started with their own project, and the original project can continue to serve as an incubator - a project that is already set up, fully functional, and just needs the model files created. Much like when your organization was in the mesh transition phase, the original project provides a safe place for domain teams new to dbt to gradually learn how to manage their own projects. Furthermore, there may be a need for models that don’t fit neatly into one of the domains. The original project is a convenient place to stash them until ownership can be decided.

Data as a Product

The principle of Data as a Product is the one that has received the most attention and may be considered the most valuable or pressing to adopt. And while there is a lot to say about this principle and how to implement it with dbt Mesh, I will remind everyone that the four principles are designed to work together: if you focus only on this principle and forget the others, you may be producing data products but you will not be following the data mesh approach.

According to this second principle, we should apply product thinking to domain-oriented data. Zhamak cites Marty Cagan’s book “Inspired”, which describes successful products being feasible, valuable, and useful. Data products should likewise have a common set of characteristics that make them usable, valuable, and feasible - and Zhamak outlines the eight characteristics she believes are fundamental to data products:

1. Discoverable
2. Addressable
3. Understandable
4. Trustworthy & Useful
5. Natively Accessible
6. Interoperable
7. Valuable on its own
8. Secure

These eight characteristics are inclusive of and expand beyond the older concept of FAIR data - Findable, Accessible, Interoperable, & Reusable - to include the traits necessary in a distributed system. (However, FAIR is a much easier acronym to remember than DAUTNIVS!).

Beyond these characteristics and the technical methods to implement them, Zhamak highlights the cultural shifts needed to truly treat data as a product, vs treating data as an asset. Success should not be measured simply by vanity metrics like how many “data products” exist or the volume of data (measures that assume data is valuable by virtue of its very existence), but rather in how many people are using the data products and their satisfaction with the experience of using the data. Quality data products need continuous care and curation, and furthermore, this care should be administered by the domain-oriented teams that are producing it in the first place, not by downstream centralized data teams that often lack needed context for the data.

In other words, you cannot simply declare a table in one of your mart schemas to be a data product by virtue of its existence. That is “data as an asset” thinking, not “data as a product” thinking. Instead, you should carefully consider how (and whether!) you can evolve this mart model to meet each of the eight characteristics of a data product.

Applied to dbt

Not every model in a dbt project should be considered a data product - nor do you want them to be. So, which models are our candidate data products? Most likely the models living in your marts (or equivalent) layer, which have an access level of “public” (assuming you followed the recommended steps from applying the first principle of Domain Ownership to dbt). These are the models that live at the intersection of projects, the models that other projects will use as cross-project references. These are the models that other teams are going to use - teams that lack the domain knowledge and development context that your team has. All of this means that these models should adhere to a stricter set of rules and documentation standards than models reserved for internal use. Your goal should be for another domain to successfully use your data product without ever needing to contact your team. So, let’s use the eight fundamental characteristics of data products to determine the rules mart models should follow in order to evolve into true data products.

Discoverable

When a user is browsing a catalog of data products and needs to find one that will suit their needs, a discoverable data product can be easily surfaced during this search. The data product itself should include the necessary and standardized metadata needed to enhance discoverability. In dbt, this metadata is provided in the model yaml file. While dbt does not require any metadata to be provided, there is a standard set of properties, as well as the extremely flexible meta config. Your team should decide on the minimum acceptable metadata needed for a mart model. Furthermore, the more detailed your descriptions, the more discoverable your model will be. Having a controlled vocabulary can also aid users in discovering the right data products. Try putting your librarian hat on when crafting the metadata for your models ;)

Addressable

Functionally, this means that a data product should have a permanent and unique address (and/or id). You get this by default with dbt, as all model names must be unique within a project. The model name also serves to group together all relevant information about the model in the catalog. One implication of this, however, is that once a mart model is considered a data product, its model name should not change. The relation name in the database can change via the alias config, but the model name should remain consistent. Model contracts can enforce this, among other things, with model versions providing a way to change the model over time without breaking references. The deprecation date should be set for individual versions and the root model prior to the model being removed (and at this point, changing its name is equivalent to removal).

Understandable

Once a data product is found, in order to use it, the user must be able to understand it - its context, its meaning, the entities and relationships involved, and how to relate it to other data products. In other words, it must be well documented! A few sentences in the description field are really not sufficient. This is where doc blocks can come in handy. Beyond enabling the DRY principle to apply to documentation, doc blocks are a great mechanism to isolate the description into a markdown file, which can be lengthier and include standard markdown formatting. You can even include images and diagrams! Your team should decide on a standard for documentation of data products that will allow users to independently and successfully use the data. Table and column names should follow clearly documented naming conventions. You may find that using doc blocks can help consolidate columns that mean the same thing but have slightly different names! I also highly recommend making writing documentation someone’s entire job - technical documentation is its own specialty, and it requires a different perspective to write documentation for the end users. Hire a documentation specialist!

Trustworthy & Truthful

Zhamak cites Rachel Botsman’s concept of trust, and states that “a data product should close the gap between what data users know confidently about the data, and what they don’t know but need to know to trust it.” One way to do this is to attach service-level objectives & agreements to the data product, where uncertainty is measured and some degree of certainty is guaranteed. Users want to know how often the data is updated (and the gap between the time the data is updated and when business fact happened in reality), how complete the data is (measuring nullness across key fields), basic descriptive statistics, and data lineage. Within dbt, data tests and freshness checks are a way to measure and enforce some of these SLAs. (In fact, there were some very exciting announcements at the Coalesce 2025 keynote on setting freshness SLAs and smart dbt test execution using column level lineage!).

Beyond the basic dbt tests you get out of the box, dbt packages like dbt-utils, dbt-expectations, and elementary allow analytics engineers to easily add sophisticated tests to models and columns. By carefully adding these tests prior to the mart, deciding when they should error vs warn, and materializing the marts as tables (not views), you can ensure that the data available in a mart meets data quality agreements. With robust observability using packages like elementary or other products like Metaplane (now a Datadog company) and established triage processes when issues inevitably arise, you can identify and resolve data quality issues before they ever reach the user. And of course, data lineage features are included out of the box in any dbt project.

Natively accessible

For a data product to be natively accessible, it should be available wherever and in whatever format the user needs it. Since dbt has traditionally operated within a single platform, your data product would only be available as a table/view within a database. However, there are two features that address this limitation: cross-platform Mesh via Iceberg support and the dbt Semantic Layer.

If you materialize your data product model using an Iceberg materialization - and assuming you are on a platform that supports Iceberg - users can now access that data from their own data platform (assuming their platform also supports Iceberg). Open table formats are changing the game around accessibility and interoperability - at least that is the vision that we seem to be on our way to realizing. Until support is more widespread and reliable, additional infrastructure may still be needed to move data between platforms.

If you create a semantic model for your data product model and define the metrics that can be calculated on it, your data product becomes accessible in many different BI tools (again, subject to the BI tool’s support of the semantic layer). This opens up the world of dashboards and even spreadsheets to analysts who want to use the data product.

Interoperable

While each data product should be self-sufficient and usable on its own, it should also be able to be used together with other data products. The closest that dbt gets to offering a feature to serve this characteristic is the semantic layer - you can identify your entities across your models and how they can be related together, with the semantic layer handling how to query the underlying tables in order to produce the desired metrics sliced and diced any which way you want (as long as these dimensions are defined in the semantic layer).

However, the bulk of the work in making data products interoperable relies on you, the analytics engineers, data modelers, and data architects. It is up to you to model your data in a sensical way, to have understandable and consistent naming conventions, and to document your data products well and focus on how they should be related to each other. You should go through the exercises of conceptual data modeling and even master data management to make sure that the entities defined in one domain (project) match up to the same entities defined in another domain (project). I do think that dbt has room to grow in better enabling interoperability: for example, while the data lineage graphs are extremely useful when it comes to tracing the provenance of a data product, they are not so useful when it comes to querying the final product. I’d love to be looking at the documentation page for a mart model and be able to see the star schema diagram as well!

Valuable on its own

This principle seems self-explanatory on the surface, but it has some interesting implications for data product developers. For a data product to be valuable on its own, all components of a data product must be packaged up and considered together: the data, the code, the documentation, and even the infrastructure that produces it all. Zhamak defines a new unit of logical architecture, the “data quantum”, which encapsulates all of the structural components needed to build, publish, and share the data product. Within dbt, this means that your “data product” is not simply your sql model file - it is the combination of the sql file, yaml file, markdown files with any associated doc blocks, the data in the database created by the dbt model, and even the entire upstream lineage of models that produce the final data product model. One interesting implication here is that in a data mesh world you may want to shift from orchestrating dbt projects to orchestrating individual data products. The updates that will be made to dbt core to support state-aware orchestration should be in line with this shift towards model-centric orchestration.

Zhamak also uses this principle to caution against lifting and shifting your dimensional data model into data products in a data mesh. While data products should be able to be joined to each other, they should not have to be joined prior to being useful. When you think about your traditional star schema, usually you must join dimensions to your fact before it can really be used for analysis. So, you should not consider your typical “fact” table to be a data product - rather, the entire “star” is the data product. There are some interesting implications here: you may want to have a standard 4th layer between your intermediate and mart layers for these facts and dimensions, leaving the mart layer to follow the “One Big Table” methodology and pre-joining needed dimensions to your facts to form the final data product. Or perhaps you leave your facts and dimensions in your mart layer and introduce a new “product” sub-layer to follow it. Regardless, if you choose the data mesh approach, this must influence your data modeling methodology.

Secure

Since we have already established that a pure dbt implementation of data mesh is limited to the medium of relations in a database (see principle 5, natively accessible), this principle is likewise limited to database security capabilities. Zhamak states that access control policies should be defined centrally but enforced at runtime by the data product, taking into account the level of access any individual may have. Fortunately, modern data warehouses can have quite complex and expressive access control policies. However, to truly take advantage of these, you will want to look outside of dbt, and use a tool like Terraform instead. There are some excellent resources on how to use terraform with dbt and data warehouse platforms, such as Katie Claiborne’s talk at Coalesce last year. The one dbt feature it is important to be aware of in respect to security is the grants config. However, at the scale of users participating in a data mesh, you’re likely to quickly outgrow the capabilities of the grants config alone, as you seek to centrally manage access control policies for your data platform in a way that can be “versioned, automatically tested, deployed and observed, and computationally evaluated and enforced” - for this, you need a tool like terraform (or OpenTofu).

One last side note: while it is not available yet, the future dbt Fusion capability of tracking PII via project-wide column lineage will be highly relevant to this principle (when it is available).

Score your data product models

So, let’s summarize the actions you should take in order to evolve your mart models into data product models:

1. Data product models should have model access set to “public”.
2. Data product models should have enforced contracts, which means any changes applied needs to go through the model versioning process, and before a model can be removed it must go through the deprecation process.
3. Data product models should be thoroughly documented, at both the model and column level, such that a new user could successfully use the data product as intended based on the available documentation alone. It’s highly recommended that doc blocks be utilized in the documentation process.
4. Data product models must be tested - and these tests should occur upstream of the final data product model, with all assertions that are tested documented with the data product model, and the results of these tests made available. (sidenote for the dbt product team: make it easy to see what upstream tests were run!)
5. Data product models should have a standard set of metadata fields that provide further context and enable discoverability (pro-tip: include data product owners in this metadata)
6. Data product models should be useful on their own, with all needed columns already included (OBT methodology over Kimball). Don’t just call your fact and dimension mart models data products - create new models that join them together.
7. Data product models should have explicit documentation about who has access to the data, and this should be enforced via grant configs.
8. If possible and needed at your organization, materialize the data product model as an iceberg table.

This is a lot! Data products require a lot of careful curation, and this requires time and effort. We want to evolve our mart models into data product models iteratively, not all at once (or it will never happen!). So let me make a further suggestion: develop a scoring system for your data products. Whether it be a 5-star rating, “medallion” score, or low-to-high maturity rank, have an agreed upon methodology for scoring your data product models, and surface this to users. You don’t want a user to expect the same thing from a 1-star model that meets the bare minimum standard of having access = public as they do from a 5-star model that is tested and documented up the wazoo. Utilize a meta config to document your model maturity. I’m not going to tell you what your scoring system should be, but I am going to tell you that you need to get together with all of your teams across the mesh and agree on one.

Self-serve Data Platform

The principle of Self-serve Data Platforms is not, as you may initially think, about “self-service analytics”, enabling end users to access and use the produced data products. Instead, it is all about enabling the domain teams to produce those data products in the first place. This third principle is essential to making the first two principles actually work in practice. Without a self-serve data infrastructure as a platform, decentralized is just a synonym for fragmented, and domain ownership over data starts to feel like an untenable burden. Organizations that focus on the first two principles of data mesh will soon find themselves falling back to previous patterns, and return the onus of data management to a centralized data team.

What does a self-serve data platform look like? Primarily, it should make it easy for a new domain team to go from 0, to 1, to many published data products. Publishing that first data product should not require going through a central team to set up infrastructure - teams should be able to accomplish this independently, without needing deep specialized technical expertise. There should be a low barrier to adoption, and integration with other systems (namely, operational systems) should be as seamless as possible. And while working with this platform should be as easy as possible for generalists (or rather, those with specialties besides data) to work with, it must not compromise on software engineering best practices of testing, versioning, and modularity.

However, the platform must enable all of this… while being decentralized. Zhamak posits that there are very few existing platforms/technologies that meet these requirements, as most platforms currently in vogue are highly centralized. She argues that centralization only introduces bottlenecks that slow down the speed of change & innovation. A self-serve data platform must strike the right balance of enabling decentralized data ownership while only centralizing the most essential tasks and resources so that domain teams can operate at a higher level of abstraction. Furthermore, the self-serve platform must centralize enough of the underlying services that data products developed by different domains remain interoperable.

Supporting all of this is a platform team. If you do not already have a platform team in your organization, this may be what your centralized data team evolves into. It is the responsiblity of this platform team to support all of the features and tools needed by all domain teams. It is the platform team who should support the dbt resources I’m about to recommend in this next section.

Applied to dbt

Before describing how you should apply this principle to dbt, I must first acknowledge something: I think it is highly unlikely that Zhamak would agree that it is even possible to create a self-serve data platform in accordance with her vision using dbt. Zhamak has spent a considerable amount of time and effort creating her vision of a true self-serve data platform with her company’s product, Nextdata OS.

So, I am not going to even try to recommend you implement this principle with dbt, especially with respect to the decentralization component. Instead, I am going to focus on a sub-principle: that it should be as easy as possible for new domain teams to get started producing their first data products.

First, I want to point out some of the excellent product developments out of dbt Labs that have made it easier than ever for non-data specialists to contribute to dbt projects - products that you should be making full use of if you are a dbt Labs customer. Then, I’ll outline some specific suggestions that you can implement in your organization to make it easier for other teams to manage their own dbt projects and start producing data products.

dbt Product shoutouts:

• The dbt Studio IDE and dbt VS Code extension (also, big shoutout to Altimate AI for their work in developing & maintaining the dbt Power User extension for VS Code). These tools make the experience of developing within a dbt project smooth and the Studio IDE in particular walks new developers through the necessary steps in a user-friendly way.
• dbt Canvas for opening up dbt projects to contributions from those who feel uncomfortable with code and prefer a GUI-centric drag & drop interface. Bonus shoutout to dbt Copilot for further enabling these users to still contribute to the underlying codebase.
• dbt Insights to more easily enable users to query and explore data - think ad-hoc analysis with superpowers from the additional context of your project’s documentation, metadata, and an assist from AI.
• dbt Semantic Layer for enabling analysts to define and correctly query metrics across different dimensions in their BI tool of choice (so long as it is a supported option)
• dbt Catalog for providing the best presentation layer of dbt projects, turning version-controlled code repositories into documentation that is easy for users to explore and discover data products they can use - as well as all of the context that goes with it.
• The dbt Learn Course Catalog for providing the best self-serve courses (all free!) to learn every aspect of dbt - both the open source dbt Core and paid products.

Now, I will acknowledge that it is very much in dbt Labs’ self-interest to lower barriers of entry and make it easy for a wider user base to adopt their products. However, I think they have done an excellent job at striking that balance of lowering barriers to entry while not compromising on the software engineering best practices that they brought to data management - a core part of their founding mission. On the other hand, these dbt products all belong to one centralized platform - they certainly do not embody the principle of decentralization that Zhamak emphasizes. Perhaps at some point there will be an integration between dbt and Nextdata OS, and we can see what it looks like to develop data products using dbt on a true decentralized self-serve data platform that fully embodies the original data mesh vision.

I’ll also mention that some of these tools are likely to have widely varying adoption across your teams, and can significantly impact the code that gets committed to your project. So you may want to keep this in mind when deciding what your domain projects look like - a team of downstream marketing analysts may want to make heavy use of dbt Canvas for developing their dbt models, and this will influence what the sql in their models looks like. This is not a bad thing - and this team should be able to operate with their tools of choice! The self-serve data platform should enable all of these types of workflows, while the principle of federated computational governance protects the independence of domain teams (within the bounds required for interoperability).

3 things your platform should build/support

Now, let’s move on to some suggested dbt Mesh best practices that will also help your organization strike the right balance between decentralized domain-oriented dbt projects and centralized support for those projects.

Pro-tip #1: create an internal dbt package. You have used public dbt packages before from the dbt Package Hub (I dare you to find someone who hasn’t at least used dbt-utils), but did you know that you can also create your own dbt packages purely for internal use at your organization? And it’s easier than you may think - after all, a dbt package is just a dbt project. There are number of great resources on how to develop a dbt package, including this official guide, this tutorial from dbt PM extraordinaire Amy Chen, this past Coalesce talk, and this blog post from phData.

What does an internal dbt package get you? Code sharing between all of your projects in the mesh. The top use case is macros: there are going to be certain macros that you want available to all projects - and you only want there to be one version of it (imagine the confusion if everyone developed their own version of a macro that performs a common transformation… in slightly different ways!). Macros can go through a promotion process - one domain project may develop a new macro, iterate on it, generalize it, and then add it to the internal dbt package when another team has a use for it as well. Custom generic tests are just macros - you can create standarized custom tests for the whole mesh to use as well. There may also be certain models you want made available to everyone - for example, a common reference table made queryable through a seed + model in the package. It is up to your organization to decide what code would be useful to be centralized and shared across all projects, and decide on a process for what code gets added to this centralized repository.

Pro-tip #2: create a dbt project template. There are a number of ways to do this, ranging in sophistication. It may be as simple as a repo with a standardized dbt project setup that can be forked, or as sophisticated as a terraform template that can set up the new dbt project & all related infrastructure. Katie Claiborne gave an excellent talk at Coalesce last year about exactly how to do this! Regardless, the goal is to make it fast, easy, and efficient to create new dbt projects that comply with all expectations your organization has of a dbt project in the mesh. With infrastructure-as-code tools like terraform, this setup is reproducible and version controlled.

Pro-tip #3: create a dbt project (or multiple projects) that are only for the purpose of learning dbt. While dbt Learn offers many amazing courses, learners are still meant to follow along by creating their own projects to practice on. By pre-creating these learning projects, you are reducing barriers for new dbt contributors. My recommendation is to do the set-up steps for the lessons, and then ask learners to create their own branches in that dbt project repo in order to complete the course and follow along. They may submit Pull/Merge Requests for their work to be evaluated, but learners should be blocked from merging any of these changes to main - the main branch should remain clean for any new learner to create a branch and progress through the course from the beginning on their own.

For the dbt Learn course developers/maintainers: it would be lovely if y’all could consolidate the various course-specific dbt projects in order to minimize the total number of “learning projects” that need to be set up. There are many different versions of the Jaffle Shop project out there!

Federated Computational Governance

Finally, we arrive at our fourth and final principle: Zhamak’s modern re-imagining of data governance for the data mesh, Federated Computational Governance. Zhamak’s approach uses systems thinking to govern the mesh, balancing the autonomy of decentralized domains with the harmony of global interoperability. The challenge is in finding equilibrium while the two opposing forces fight for optimization. Zhamak was inspired by Donella Meadows’ “Thinking in Systems”, and describes how to use feedback loops to adjust behavior and leverage points to adjust the strength of those feedback loops and significantly impact the overall system. In a large-scale distributed and dynamic system like a data mesh, it is essential that traditional data governance evolves to become federated and automated.

To become automated, policies that were previously enforced by manual intervention must now be enforced automatically by the platform (and this means these policies must be expressed in such a way that they can be automatically enforced - as code). For example, a standard might be that all models should have a primary key test. A policy might be that all PII data must be masked.

To become federated, domains must be allowed to create their own policies to some extent (primarily around how data products are created, maintained, and served), while also adhering to some minimum viable set of global policies & standards (primarily those that facilitate interoperability). As a reminder on what “federated” means, think of the governance model outlined by the US Constitution: the federal government sets certain policies/laws that apply across states, but state governments have the power to set policies/laws that apply only to people in their state (so long as compliance with federal laws is maintained. If not… that’s what the Supreme Court is for).

Standards and policies are code - administered, enforced, and customized locally after inheriting from global policies - and this code should be subject to the same agile principles as other software: easy to change, iterate, and evolve.

Applied to dbt

If there is one key leverage point to adjust people’s behavior when contributing to a dbt project, it is during a Pull/Merge Request. The CI/CD process as a whole is a feedback loop, and the event of a CI job passing or failing is a leverage point. It is an excellent demonstration on how policies can be defined in code and enforced automatically.

A typical setup for a dbt project looks something like this: when a Pull/Merge Request is opened, a CI job is automatically kicked off. This CI job should either build all new/updated models and their downstream models (slim CI) or the entire project (inefficient, but ok for smaller projects with no way to handle state or deferral yet), in an isolated environment (typically, a new schema is created for the PR, then dropped after the PR is merged). Note that if you have enforced model contracts in your project, any violation of the contract will cause the job to fail. If the dbt job is successful, you can safely merge the code to main (or a develop branch, depending on your git strategy) - after a code review by a human as well. CI jobs offload some of the burden from human reviewers, by letting computers do what computers are good at (making sure code runs without error).

On the surface, requiring a successful CI job prior to merging code to main is a way to protect against bugs that could cause production jobs to fail. However, you can test for more than just the fact that models run successfully - you can also test that the code follows certain agreed upon standards. This is where the dbt Project Evaluator package comes in handy.

The dbt Project Evaluator turns your dbt project(s) into data - and then builds dbt models on that data and tests for certain assertions. While dbt Project Evaluator comes with a standard set of policies it enforces out of the box, you are able to customize which assertions it tests for and even the parameters for some of those tests. For example, let’s say that your team has an expectation that every model should have a primary key test. Typically, it would be the human reviewer’s responsibility to ensure this policy is followed. However, with dbt Project Evaluator enabled and run during the CI job, the missing primary key test will fail if any model is missing a primary key, thus preventing a merge until the issue is resolved.

One useful side effect of using the dbt Project Evaluator package is that your dbt project - the actual code repository - is modeled out and materialized in your database. This means that you can build further models and tests on top of these Project Evaluator models, in order to enforce custom policies that are not included out of the box in Project Evaluator’s ruleset.

And if you have multiple dbt projects, all with dbt Project Evaluator enabled for staging (or production) jobs, you can put these project-specific tables together to form a cohesive dataset about the entire mesh. And then you can test new and interesting things in this comprehensive set of tables - in other words, you create mesh-wide tests. For example, let’s say that you only want one project in the mesh to use a given table as a source (in order to reduce duplicative and possibly diverging efforts). This mesh-wide test is now possible to run and enforce once you create the model that sources in and consolidates the Project Evaluator core models across all projects.

Once your dbt project - the code itself - is data that can be queried and tested, the possibilities are endless. And a great place to put these mesh-wide tests and models? In your internal dbt package, which should be imported by every domain project, enabling these mesh-wide tests to be run during the CI job as well. I would further suggest that you turn your internal dbt project into a full project, with its models materialized in the database in a single permanent location, as this will make it much easier to query and monitor the status of data products across the mesh. (Remember: CI schemas should be dropped after the PR is merged, so you can’t build permanent infrastructure on models built just during CI jobs).

Here comes another request for the dbt Labs team: it would be amazing if default project configurations (think: the configs that you set in your dbt_project.yml file) could be inherited from that internal dbt package (which evolves into a central project that underlies the entire mesh). While code can be imported from packages (models, macros), configurations cannot - but it would make setting up and governing the mesh easier if they could be!

People and Process

As I’m sure you’ve noticed while reading this blog post, there are a lot of decisions to be made about how, exactly, your organization will decide to implement dbt Mesh. And when you are operating at the scale of a mesh, with governance and ownership distributed across the domains, it is no longer feasible for a single leader or centralized governance team to make these decisions for everyone. While it may be tempting to put the responsibility of crafting data mesh governance policies on the existing (likely highly centralized) governance team (or, conversely, to leave them out completely!), please resist the temptation. Instead, you need a “federated team of domain product owners, subject matter experts, and central facilitators.” Identify a core group of decision makers that allows for every domain to be represented, and charge this group collectively with crafting the policies, making decisions, and steering the development of the mesh. Start to transition data stewards from the centralized governance team into the domains, and pull domain data product owners into the new (federated!) mesh governance team. (Note: this federated mesh governance team is an important component of the fourth principle).

In short - you need to convene your mesh! Let’s review some of the decisions this Mesh Governance Council will need to make, one principle at a time.

Decision time

Domain Ownership

• What qualifications must a domain team meet in order to manage their own dbt project?
• Who determines when they have met these qualifications?
• Who manages the creation of the new dbt project and the migration of models into it?

Data as a Product

• What is the scoring system that will be used to rate data product models?
• What qualifications does a model need to meet in order to reach the next level?
• Who is able to make this determination, and how is this score represented in the model metadata?
• What other metadata is expected for all data product models across all projects?

Self-serve Data Platform

• Who is responsible for maintaining the internal dbt package?
• Who can contribute to the internal dbt package?
• When should code be promoted from being managed in domain dbt projects to being generally available in the internal dbt package?
• Who is responsible for creating and maintaining a dbt project template?
• Who is responsible for making sure existing dbt projects get any needed updates after changes are made to the template?
• Who is responsible for supporting new dbt learners, and maintaining the learning projects?

Federated Computational Governance

• Who will be monitoring the health of the overall mesh, and making sure that individual domains and projects are adhering to the mesh-wide policies?
• How will issues be detected, and who is responsible for ensuring those issues get resolved?
• What will the standard CI job look like across dbt projects, and when can CI jobs diverge from this standard template?
• What is the standard suite of tests dbt Project Evaluator should run for all projects, and in what tests can individual projects have more flexibility?
• Who is responsible for setting, maintaining, and monitoring the mesh-wide tests?

Map out your process

Hopefully these questions clarified that the transition to a dbt-powered data mesh introduces a lot of new problems and complexities… ones that will take time, iteration, and input from every participating dbt project team to solve. I would highly recommend mapping out the process that the teams should follow in order to craft new policies and solve each problem. (Shameless plug: helping teams figure out what this decision-making process would look like for a dbt Mesh environment was the goal of my peer exchange at Coalesce in 2024!)
Here is an example of what this process may look like at one organization:

1. Someone (hereafter the Champion) brings a proposal to the Mesh Governance Council.
2. The Council can either reject the proposal immediately (process stops), or let it go forward.
3. If the process goes forward, the Council evaluates whether this is a major or minor proposal. In essence, a minor proposal can be approved immediately, while a major proposal requires a working group to flesh out the details.
4. If the proposal is minor, the proposal is either approved or rejected by the Council (by a vote or some other decision making function).
5. If the proposal is major, the Champion forms and leads a working group, recruiting interested parties. This working group crafts the proposal, and gathers feedback. When they are ready, they bring this detailed proposal to the Council.
6. Once again, the Council decides whether to reject (process stops), request changes (process loops), or approve the proposal (process moves forward).
7. The proposal may need to go through additional approval processes, depending on the organization’s requirements. The Council determines what further approval processes the proposed policy needs to go through. If the policy is rejected by any other approving body, the Champion and working group can decide whether to stop or loop back through the process again with revisions.
8. Once the policy has all needed approvals, the final version of the policy should be thoroughly documented and stored in a central and accessible location. This final policy should also include the planned (automated if possible) enforcement mechanism. The policy should be communicated out to all projects, along with an expected timeline for compliance. (Pro-tip: the internal dbt package - which may double as a central monitoring project - would be a great initial place to put these policies as markdown files!)
9. Individual domain teams can petition the Council for exceptions to the new policy. The Council decides whether to approve or reject these petitions.
10. New mesh-wide tests (or other automated enforcement mechanism) should be implemented according to the communicated timeline and any approved exceptions. I highly recommend a gradual approach - set tests to warn for a period first, before transitioning the tests to error on failure.

As a reminder, this heavier process should only be needed for policies that must apply across the mesh for the sake of interoperability. Individual domains should be able to set their own policies internal to their projects in a much faster and lighter process. Mesh-wide policies should lean towards being minimal, with individual domains empowered to enforce stricter policies for their own projects.

Wrapping Up

I hope that you walk away from this blog post (and accompanying talk) also convinced that dbt Mesh + data mesh is a strong approach for organizations using dbt to manage data at scale. I also hope that I’ve caught you at the right time in your dbt Mesh implementation journey, and you can put into practice some (or all!) of the suggestions I’ve made. Of course, while this is (naturally, for me) an absurdly long blog post, I can only cover so much, and I’m limited by my own experiences. I would love to hear from y’all if there is anything I should add - if you have any pro-tips for the others who have chosen to adopt data mesh by way of dbt - please reach out and let me know!

Appendix

Note: An abridged version of this article was published Oct 1, 2025, on The Data Intelligence Platform

Thank you Ole Olesen-Bagneux for featuring my writing as a Guest Expert!

If you’re an analyst on a data team, the following scenarios are probably very familiar to you:

1. A stakeholder you’ve worked with before comes to you with a new request. They saw what kinds of questions could be answered with a dashboard in the last project, and now they have some new questions to try and find answers for
2. You get an email from someone in the organization who wants a dashboard for their data. Right now they do a lot of manual work and one-off visualizations in a spreadsheet, and their friend in another department recently showed them a dashboard that the data team made
3. An executive asks your boss for a set of very specific metrics. Your boss tells you to write some SQL queries to generate numbers for those specific metrics, so they can then email those numbers to the executive

New project requests can come from many sources: an email to a shared inbox, a DM from a former stakeholder, a verbal request from your boss, or even just a jira ticket created by your project manager. No matter how you get the request, once you’ve received it, you will need to choose how to respond.Not responding at all is still a form of response

The data team dilemma (the ad-hoc deluge)

Perhaps your team is already over maximum capacity and you get requests like this every day. You could choose to ignore the request, or mentally bookmark it to respond to later, only to forget about ithello executive function overload!. Perhaps you have a project request form, so you redirect all inquiries to filling out a generic form. Maybe you don’t have capacity for this request, but your coworker just wrapped up a project, so you ask them to take it on. If interrupt tasks are common for your team and you are lucky enough to have time allocated to ad-hoc requests, you may try to quickly provide an answer before moving on to the next request or project. If it’s obvious this request is a huge ask, you may reply that you don’t have capacity right now, but they can put a meeting on your calendar for sometime next month. The request may get added to the project backlog, to be re-examined during the next sprint planning phase.

You’re on a data team, and in today’s world that meansyou've trauma-bonded you are understaffed, always working at maximum capacity with pre-planned projects, and also expected to drop everything to answer some higher-up’s one-off question. Half of your week is filled with meetings, and you know that if you just had some uninterrupted quiet time you’d be able to actually crank out some real work and make real progress on that dashboard that was supposed to be done last week. Fortune forbid that the data isn’t fresh today or the column you were counting on is a free-text field full of inconsistent garbage. Why did your stakeholder organize their spreadsheet like this, don’t they know that will break the import? Why does your boss think you can just select * from some_perfect_table_that_does_not_exist? You don’t have enough time to clean up the data going into the dashboard the executives supposedly look at every day (they don’t), you certainly don’t have time to model the data used in 1 project into a star schema!

Let’s all stop and take a deep breath. In… out… in… out. If you are very lucky, and you have always worked on a fully staffed, well-managed data team with a reasonable number of projects and minimal interrupts, perhaps the above situations have never happened to you. But for the rest of us (and, I bet, the vast majority of us), these situations are familiar (and perhaps slightly triggering). And I’m about to give you some advice that you really won’t want to hear: you need to talk to your stakeholders more.

“What? Didn’t you hear that half my day is already filled with meetings and I can’t find the time to do the work I’ve already been assigned? I already talk to my stakeholders every week and they never tell me anything helpful. They’re just not data peopleI'm going to need you to do the big airquotes for me here, they don’t get it. I can make the dashboard they need without them micromanaging every pixel of it. They’re always going off on tangents unrelated to the project, how is talking to them more going to help?”

I hear you. I understand your doubts. I know you don’t have enough time to dive into each project as deeply as you want to. I know how frustrating it can be to talk to “non-data” people who just don’t understand why their seemingly simple ask isn’t reasonable. Trust me - I’ve been there, and I get it. But I want to zoom out for a moment, and look at the bigger picture.

What is the role of a data professional? What is the purpose of a data team? What is the function of data in an organization? What even is data?

Pause. Take just a few minutes to really think about these questions. What are your answers? I really want to hear! Wherever you may have seen this posted, please let me know in the comments.

No really, come up with your own answers before you read what I think - then come back and hear how a librarian would approach this dilemma.

The DIKW pyramid

My background is in Library & Information Sciences, and I also have a deep fondness for philosophy, which informs my opinion. You may have a totally different answer informed by your own background and expertise, and that is awesome! If we put together all of our perspectives, we can arrive at a more complete picture.

What is data?

When I think of what data is, I think of it in the context of the DIKW (data-information-knowledge-wisdom) pyramid, which is a foundational concept in the field of Library & Information Sciences. Rather than try to explain the DIKW pyramid myself, I’m going to pull quotes from this excellent encyclopedia page, which itself pulls quotes from significant literature on the DIKW pyramid (mostly Ackoff’s “From Data to Wisdom”).

On data:

Data are symbols that represent properties of objects, events and their environments. They are products of observation. To observe is to sense. The technology of sensing, instrumentation, is, of course, highly developed (Ackoff 1989, 3).

On information:

Information is relevant, or usable, or significant, or meaningful, or processed, data (Rowley 2007, Section 5.3 “Defining Information”).

On data vs information:

The vision is that of a human asking a question beginning with, perhaps, “who”, “what”, “where”, “when”, or “how many” (Ackoff 1989, 3); and the data is processed into an answer to an enquiry. When this happens, the data becomes “information”. Data itself is of no value until it is transformed into a relevant form. In consequence, the difference between data and information is functional, not structural (Ackoff 1989, 3).

Information systems generate, store, retrieve, and process data. In many cases their processing is statistical or arithmetical. In either case, information is inferred from data. (Ackoff 1989, 3)

On knowledge:

Knowledge is know-how, for example, how a system works. It is what makes possible the transformation of information into instructions. It makes control of a system possible. To control a system is to make it work efficiently. (Ackoff 1989, 4)

On wisdom:

Wisdom adds value, which requires the mental function we call judgement. […] The value of an act is never independent of the actor [… ethical and aesthetic values] are unique and personal. […] wisdom-generating systems are ones that man will never be able to assign to automata. It may well be that wisdom, which is essential to the effective pursuit of ideals, and the pursuit of ideals itself, are the characteristics that differentiate man from machines. (Ackoff 1989, 9)

All organizations collect and store data as a by-product of doing business. However, this data cannot be used beyond its functional purpose of making the widget work until it is processed into information. That information can be used to help the people in the organization gain a better and more accurate understanding of how well the organization is functioning. Knowing how an organization is functioning helps the people serving the organization make decisions, which will then affect how the organization functions. Leaders making these decisions should have a vision of the organization’s ideal state, and we consider these leaders wise when they can effectively move the organization closer to that ideal state.

The purpose of a data team is to turn data into information, so that their stakeholders elsewhere in the organization might gain knowledge and make wise decisions. The role of a data professional is to navigate this process of turning data into information in the most efficient and useful way possible. An excellent data professional will partner with stakeholders to see through the process of generating knowledge from the processed information, and the most effective data leaders are those empowered to make decisions alongside other leaders.

In order to provide information that will increase the knowledge of your stakeholder and help them make wise decisions, you must talk to them in order to find out what the decisions they need to make are, what their current understanding of the knowledge landscape is, and what information they need in order to enhance their understanding. This conversation requires time, empathy, and trust. Most likely, your stakeholder has already thought about this… and then gone the extra step of trying to translate this into a request for a data team.

Their idea of a data professional is probably just that - someone who only operates at the level of data. But it is your job to demonstrate that while you have the technical know-how to work at the data level, you also have the business acumen to understand and converse at all levels of the DIKW pyramid. You can do your best work at the foundation when you have the entire context. The translation from a business-oriented request into a data-oriented request should be on the data professional, not the stakeholder.

This is where I’m going to introduce another concept from the field of Library & Information Sciences… and the primary topic of this blog post: the reference interview.

The reference interview is designed to solve a problem common to most information professions - one that you have undoubtedly encountered before and may have already developed techniques to address. My hope is that by learning how librarians approach a reference interview you can add another set of techniques to your toolbox, and perhaps even adopt a new perspective that will help you approach one of the most frustrating aspects of your job.

The reference interview

Before adapting the reference interview for data teams, let me first explain it in its original context: that of a librarian responding to patron inquiries. Have you ever walked into a library with a research question, but weren’t quite sure where to start? Soon after walking in, you probably saw a circular desk with signs above it proclaiming “Reference Desk”. Behind that desk are one or two librarians, next to computers with the catalog ready for searching, ready to help you with your research quest. You walk up to the desk, ask your first question - and what happens next is the reference interview.

My primary source for this section will be the Reference and User Services Association (RUSA) Guidelines for Behavioral Performance of Reference and Information Service Providers - as these guidelines are industry-standard and referenced by most institutional guides to the reference interview. I’ll also be drawing from the book “Conducting the Reference Interview: A How-To-Do-It Manual for Librarians”, which goes into much more depth than the short RUSA Guidelines.

The importance of the reference interview is founded on a simple assumption: that patrons rarely start out asking the librarian for the information they actually need. The goal of the reference interview is to identify what the patron really wants to know, so that the librarian can then more efficiently search the system for resources that will fulfill the patron’s true information need. Librarians should take the time to establish rapport with the patron, engage in active listening, and confirm that the information provided is what they actually needed - in the end, this will save the time of both patron and librarian (compared to jumping straight to the search and repeating through trial and error until an adequate resource is found).

While on the surface the reference interview is a structured format for going from initial question to identifying resources that can fulfill the true information need, the real magic is in the empathetic communication style and active listening that underlies these techniques. Excellent interpersonal skills and emotional intelligence are key for the librarian to build trust with the patron, cultivate an inclusive and collaborative learning environment, and work towards a mutual understanding of what the patron really wants to know.

Reference interviews generally follow these 6 steps:

1. Establish contact with the patron and listen to the initial request, without interruption or judgement
2. Paraphrase the initial request, to demonstrate you were listening and verify you correctly understood the question
3. Ask open-ended questions first that encourage the patron to elaborate on their initial request (focusing on the big picture), followed by clarifying close-ended questions to understand the specifics of certain parts of the request
4. Verify the patron’s request by restating the question and asking for confirmation, incorporating what you have learned during the question phase. If the patron verifies this rephrased request is accurate, start the search. If not, repeat the question phase until the patron verifies the restated question is accurate.
5. Conduct a search of the system collaboratively, sharing your screen/resource and explaining each step of the process. Check in with the patron to see if they want to do the search themselves at any point in the process, and if so, continue to offer assistance as needed. This is the phase where you can offer instructions or advice, guiding the patron with your special expertise and knowledge.
6. Follow up with the patron to verify that the patron’s information need was met. You may need to review the resources found and identify different/additional resources if needed, repeating the search phase. It may also be necessary to explain when a request is beyond the scope/capacity of the library, in which case the patron should be referred to an external resource/institution. This final phase is also the appropriate time to ask for feedback on the process.

A reference interview is most successful when the librarian and patron share a common purpose: finding materials that will fulfill the patron’s information need. It is the librarian’s responsibility to find out what the patron wants and guide the interview in order to achieve this goal. The modern reference interview comes from the perspective that libraries are primarily service-oriented, and should provide information in the manner that is most useful to its patrons.

Of course, there are plenty of opposing viewpoints in the LIS literature - that the reference interview is not always necessary, that it is only useful for certain types of inquiries, that libraries shouldn’t be service-oriented, that the reference interview can only be performed in person… and the dominant viewpoint in the literature has changed over time. If you’re curious, feel free to explore the literature, but I’m not going to dive any deeper in this post beyond noting the fact that the importance and role of the reference interview has been contested and evolved since it first became a thing over a century ago.

It should also be noted that even when librarians are trained to always conduct a reference interview, studies have shown that a majority of the time they do not conduct a full reference interview, or default to the “question-oriented model” (answering the patron’s original question) rather than the “needs-oriented model” (addressing the patron’s information need - the initiating problem). Being empathetic, curious, and patient all of the time for every patron query is hard work, and nobody is perfect - but the reference interview provides an ideal guideline for these engagements.

Roles of the reference librarian

Before we move on to how this translates to the work data teams do, I want to spend some time on the different roles that a librarian takes on during the reference interview. My primary source for this section is a chapter in the book “Leading the Reference Renaissance: Today’s Ideas for Tomorrow’s Cutting Edge Services”, called “Inventing the future by examining traditional and emerging roles for reference librarians”.

The most dominant role of the reference librarian is that of the “information provider”: the highest priority for a reference librarian is to provide answers to questions - the more complete the better, whether that be through directly telling the enquirer the answer to their question or helping them find the information resources they need (e.g. books, journal articles, news articles, magazines, etc). After all, the ultimate goal of the reference interview is to fulfill the patron’s information need. But while “information provider” is the most obvious role, it is far from the only (or even, arguably, the most important) role that a reference librarian fulfills.

The other major role for a reference librarian is that of the “instructor”. No reference librarian wants to be the sole provider of answers - think of all the questions that would never get answered simply due to time and resource constraints! Instead, reference librarians can instruct users in library and research skills, so that they can be self-sufficient and navigate the library to find the answers and resources they need on their own. While reference librarians can lead classroom-style teaching sessions, instruction can also occur during the reference interview.

Read the first sentence of step 5 again: “Conduct a search of the system collaboratively, sharing your screen/resource and explaining each step of the process.” By the end of the reference interview, the patron should feel more comfortable conducting a similar search on their own next time. While this may mean the current reference interview may take slightly longer, it will save time in the long run.

There are some further minor roles that are implied by, or overlay, these two major roles, such as:

• Communicator: librarian as the human connection between a user and the resources
• Relationship builder: librarian building a long-term relationship with the user
• Guide/advisor: librarian guiding the user in selecting good and appropriate resources
• Counselor: librarian mentoring users in the research process, and treating the user as a holistic person
• Partner: librarian and user act as a team, each bringing their own knowledge and skillsets

The final role I want to focus on is that of the librarian as an expert: this is the foundation for all of the previously outlined roles. A librarian is an expert at navigating (and cultivating!) the system that organizes the library’s resources. That expertise allows the librarian to quickly find needed information, and some portion of that expertise is what the librarian is attempting to transfer with instruction (though much of the expertise is likely to remain as tacit knowledge"What is Tacit Knowledge? Tacit knowledge is knowledge that cannot be captured through words alone." Sidenote: I'm very happy to be linking out to some Cedric Chin content here, his essays are gold!). This expertise is what the librarian brings to any research partnership, and what allows the librarian to be a guide for users.

Data professionals are Information professionals

The parallels between data professionals (especially analytics engineers) and librarians have been drawn before (many times), albeit usually by those on the data side of the divide. If you’ve read this far, I’m hoping that by now it is obvious that this is because both data professionals and librarians are really information professionals. And while data (engineer/analyst/scientist/etc) is a relatively new profession, librarianship is not, and there is a lot for the data professional to gain by considering themselves part of the wider information profession - and the larger body of knowledge and practice that comes with it.

One such practice? The reference interview.

Before I give my take on how data teams can learn from and adapt the reference interview, I once again want to ask you, dear reader, to pause and consider for yourself: based on everything you’ve learned so far, how can you borrow from the typical reference interview the next time a stakeholder comes to you with a new request for the data team?And if you shift how the data team engages with the rest of the organization, how can this change the role of the data team? Revisit the scenarios from the very start of this blog post: how would you have approached these scenarios before, and how would you approach them now from the perspective of a data-savvy reference librarian who starts every stakeholder engagement with a reference interview?

Remember, the difference between data and information is functional - it is only by processing data into information that we give it meaning, purpose, and value. Which level do you want to be operating at? Which level does the rest of your organization think you are operating at? What do you need to do to get to the next level?(A question for a future post… how can data professionals further elevate their work to be *knowledge* professionals?)

The reference interview for data teams

So how can data teams learn from library science, adapt the reference interview, and elevate their practice from being data practitioners to information practitioners? I want to approach this from two perspectives: that of an individual contributor on a data team who has the power to adjust how they interact with stakeholders, and that of a data team leader who has the power to make structural and process changes that affect the environment that the individual contributors on a data team work in.

For the individual contributor

Let’s start with the individual contributor perspective. Remember the pithy and unwanted advice I gave near the start of this post? You need to talk to your stakeholders more. Beyond that, the attitude and emotional presence you bring to those interactions matters. Your stakeholder has an information need. They know you are busy and have a dozen other projects that need your attention, and they likely have already formed some opinion about what you know and are capable of doing.

So in an effort to be helpful and efficient, they have likely already taken that information need and refined it, restricted it, reshaped it in a query or request that they think you can do/answer in order to fulfill their information need. So here is my first and most important request to you: don’t accept that query at face value. It is your job, not theirs, to dig deeper and discover their true information need before you try to fulfill it.

Here is an abbreviated and reframed version of the 6 steps to the reference interview, which you can use and adapt in order to uncover their true information need. Think of this as a formula for a structured or guided conversation:

1. Listen to the initial request without interruption or judgement
2. Paraphrase the request back to them
3. Ask open ended questions first, followed by clarifying close-ended questions
4. Restate the request yet again, now altered by their responses to your questions. Repeat until the rephrased request is confirmed as accurate
5. Collaborate to find an answer to the rephrased information request, using the interviewer’s expertise in navigating the information system and the interviewee’s guidance and background knowledge. Depending on the need/relationship, either can be the “driver”.
6. Verify that the true information need was met. If not, the process can be repeated, or the requestor may need to be directed to further resources if their information need is beyond the capacity of this process. Finally, you can ask for feedback on the process.

Conversations like this (in which you are trying to unearth their true information need) will generally be more successful when you approach them with empathy, non-judgement, and genuine curiosity. Take the time to establish a rapport first. Apply the techniques of active listening. Intentionally cultivate an inclusive and collaborative space. This means that your language matters. Erase words like “simply”, “just”, and “easy” from your vocabularyAny other words to add to the list?. When you ask questions, show that you are paying close attention to their response by rephrasing and restating what you heard. Ask yourself first what assumptions you are making, and at the very least verify/clarify those assumptions or don’t bake them in to your first round of (open-ended!) questions.

Remember: the interpersonal skills and empathetic mindset that a librarian brings to the reference interview is just as important (if not more!) to the ultimate success of uncovering the user’s true information need as the structured interview process that they follow. I want to acknowledge that this is hard, and it is a skill to develop just as vital as any technical skill, so give yourself grace and time to reflect as you start to adjust your approach. And if you feel like this is already how you engage with stakeholders, that’s fantastic! Make sure you are collecting feedback - from your stakeholders, your peers, and your managers/mentors - so you can continue to improve your approach (everyone has blindspots!).

If this sounds preachy, I apologize, but I genuinely think this is both the hardest and most impactful skill you can develop. It is a skill I know I will need to continue to practice and improve on throughout my entire career, and it is a skill that is almost never taught in professional contexts (unless, perhaps, you were a therapist in a previous life). If you are struggling to make the jump to senior, demonstrating this skill will be one of the most important things you can do to show your manager and your team that you are someone who can handle the more nebulous projects - the kinds of projects typically given to seniors.

Ok, so what do we have so far?

• Talk to your stakeholders more (especially at the start).
• Never assume that their initial request matches their true information need.
• Make it your goal to uncover their true information need.
• Cultivate an empathetic, non-judgemental, and curious mindset.
• Practice active listening techniques.

What else can we adapt from the reference interview?

Assuming the Instructor role… and a path to self-service?

I want to pull your attention to step 5 of the librarian’s reference interview:

Conduct a search of the system collaboratively, sharing your screen/resource and explaining each step of the process. Check in with the patron to see if they want to do the search themselves at any point in the process, and if so, continue to offer assistance as needed. This is the phase where you can offer instructions or advice, guiding the patron with your special expertise and knowledge.

I also want to remind you of one of the major roles the reference librarian plays: the instructor. Most likely you already assume the other major role of the “information provider” when you receive an ad-hoc request, but how often do you assume the “instructor” role? What would that look like for a data analyst?

Let’s revisit the first scenario:

A stakeholder you’ve worked with before comes to you with a new request. They saw what kinds of questions could be answered with a dashboard in the last project, and now they have some new questions to try and find answers for

After some investigation (read: asking a series of open-ended and then clarifying close-ended questions, rephrasing their ask, and then repeating until you understand their underlying information need), you realize that this stakeholder is trying to do some exploratory data analysis (EDA) for a new project, and they want to gut check whether the trends that they think are happening are actually happening. They don’t actually need a dashboard (or at least, not yet), but that’s how they’ve been doing EDA for the other project, and they figured another dashboard for this new dataset would let them see whether these numbers were actually going up over time.

Do you want this stakeholder coming to the data team every time they want to do some EDA? Go that route, and you will soon have a graveyard of unviewed dashboards, not to mention the opportunity cost of devoting valuable analyst time that could’ve been spent on higher value projects. Why is the stakeholder asking for a dashboard when the underlying information need could be fulfilled more efficiently with some SQL queries? Likely, because dashboards and interactive visualizations are familiar. For the sake of this example, let’s say that this stakeholder knows some basic SQL. How could the data analyst approach step 5 of an adapted reference interview?

The first step is to invite the stakeholder to be a partner in this EDA. Assuming that this stakeholder is not an executive, if they are asking for the data team to invest time in answering their question, they should be willing to put that time in themselves, as well. Who knows - they may even be excited at this opportunity to learn some analyst skills from an expert! Set up some pairing sessions, and use those sessions to time-box the project - you’ll only work on it during those pairing sessions. During those sessions, you will be sharing your screen and narrating your process.

Don’t try to prepare a cleaned up version of the process - let the stakeholder see and participate in every messy step (yes, that includes googling for how that window function works again, and running queries with typos). The ultimate goal here is for the stakeholder to feel comfortable going through this process by themselves, and that includes knowing what to do when something goes wrongMy opinions here have been heavily shaped by The Carpentries pedagogical model - see: https://carpentries.github.io/instructor-training. You want the stakeholder to lose the perception that the skilled technical data analyst is able to magically produce a dashboard with a few precise keystrokes!

Start from the very beginning - the first thing you’ll have to do is identify the right datasets to query. You are an expert at navigating the data warehouse - you know how it is organized, the naming conventions, and how to identify high quality ready-to-use datasets. You know how to search the data catalog (or whatever other documentation there may be), and you are all set up to query the database. Your stakeholder is also an expertLet's say it louder for the folks in the back! - they may or may not know what the data looks like, but they do have a lot of knowledge in this domain. You need to use your expertise to guide the stakeholder in the research process, and you can lean on your stakeholder to know what information they expect and want to find. You might find that by working together, you can fulfill your stakeholder’s information need better and faster than going off on your own… a 1 + 1 > 2 effect.

The next time this stakeholder has a question, hopefully they are more comfortable performing some of this process themselves - and maybe in some future request, they will ask to drive in the pairing sessions, with you providing support when needed. By inviting the stakeholder into your workflow and process, you are enabling and empowering them to use the data resources available to them more independently, freeing up your time to focus on more complex analysis.

This is one of the foundational building blocks of true self-service - not only making the data and tools available, with the data documented and findable in a curated data catalog, but also providing the training, support, and guidance for users so that they are empowered to navigate the system by themselves or in partnership with a data expert if they so choose. Sorry, but no magic BI tool is going to be able to provide self-service if those core fundamentals aren’t in place.

Gathering requirements with the reference interview

Generally, the first step in a new project is to gather requirements. This phase is often responsible for most of the frustration and angst you experience while working on a project. How many times have you thought you reached the end of a project, only to be told that the report or dashboard you delivered does not actually meet the stakeholder’s expectations? This needs to be tweaked, that needs to be reworked… but sometimes that little tweak is a change to a fundamental assumption that requires overhauling the entire product!

How does your team gather requirements? When does the requirements gathering start? (Likely during the initial intake phase!) Who typically performs the initial requirements analysis? Does this person have the necessary context and technical skills to gather all requirements? Is there a handoff? What problems typically occur during this handoff? When does the individual contributor who will be doing most of the work for the project get involved? How much of the requirements gathering process do they typically need to re-do? Crazy idea: draw a process map and start identifying waste in your requirements gathering process. How can you improve the experience of gathering requirements for both the data team and the stakeholders?

As you may have guessed, I believe the reference interview has something to offer here. First, the individual contributor(s) who will be directly working on the project should be brought into the requirements gathering phase as early as possible. While a project manager or data product manager can direct and guide the process, they should not try to collect all of the requirements in a vacuum and then hand off those requirements to an analyst/engineer. Second, the analysts/engineers/managers involved in the requirements gathering should be conscientious about the emotional energy they are bringing to these stakeholder conversations: adopting an empathetic, non-judgemental, and curious mindset. Third, they should try to follow the key principles of the reference interview in these conversations: that the stakeholder is not likely to start with their true information need, to utilize active listening techniques, and to approach the process as a collaborative exercise in which all parties have special expertise to offer. The data team should iteratively develop a set of requirements and seek consensus on those requirements and the true information need.

If you were to make one change to your typical requirements gathering process to align it closer to a reference interview, what would it be?

For the data team leader

If you’re a data team leader, I hope that you haven’t just skipped ahead to this section - because your whole job is to cultivate an environment that encourages and supports all of the suggestions I made in the previous section for the individual contributor. So let’s review some strategies that will allow your team to start acting more like information professionals, and implement their own version of a reference interview.

Let’s start with the most significant environmental change you can make: creating the data team’s equivalent of a reference desk. When you walk into a library with a question, do you ask the first person you see with a name tag? I mean, maybe - but any trained library page should know to then direct you to the reference desk. But most likely you look for the big desk with signs above it and a reference librarian ready and waiting behind said desk. This librarian isn’t (usually) doing other activities like cataloging or purchasing new books - if they are at the reference desk, their sole job is to help any patron who walks up to the reference desk. How can we replicate this experience for stakeholders reaching out to a (e.g. remote) data team?

First, someone needs to be “at the reference desk”I need to caveat here that this really only applies to medium-to-large data teams, as smaller data teams are unlikely to have capacity for this. - a dedicated analyst/engineer/etc whose only job during that shift is to respond to stakeholder requests. We can borrow from “on-call” best practices for software engineers here as well - while a reference librarian is typically responding to new information requests and an on-call software engineer is typically responding to active issues/outages, the assigned data team member is likely to encounter both of these situations. Regardless, the data team member who is on-call/at-the-desk should not be expected to complete any typical project work during their shift, and it is your job to organize sprints and/or project planning around this.

On the flip side, team members who are not currently on-call/at-the-desk should not be expected to respond to ad-hoc requests. The goal here is to minimize overall context switching, account and plan for all types of work (after all, we all know ad-hoc requests are inevitable), and better serve the data needs of the whole organization without sacrificing the needs of the data team itself. What virtual “signs” can you put up to direct stakeholders to the data team’s “reference desk”, to hopefully minimize the number of redirects that are needed? What culture changes are needed to shift and enforce this new social norm?

Next, you’ll need to consider training: how can you enable data team members to be successful when it’s their turn to serve at the reference desk? Unless you have a librarian on your team (which, honestly, you should consider!), this is going to be a learning experience for everyone. Consider pairing two team members to be at the reference desk together, so they can observe and learn from each other. Collect and encourage feedback, from each other as well as stakeholders.

Here’s a crazy idea: try going to a library with a research topic in mind, and ask a reference librarian for help. Experience the reference interview for yourself as a patron! Your experience may vary depending on the library that you go to, so try visiting a few different libraries (especially if you’re lucky enough to be close to a research university!) - try to pay attention to the overall experience from entering the library with a question to (hopefully) leaving with some resources and/or answers.

If you’re willing to try this data team reference desk idea out, I fully encourage setting it up as an experiment. What are you trying to improve? What metrics can you track? How can you test whether this change is actually improving processes for the data team and your stakeholders? What surveys should you send out before, during, and after the experiment?

A different definition of “service”

For many data teams - and data “thought leaders” on LinkedIn - “service” is a dirty word. You don’t want your data team to be like the typical IT service desk, where data team members are just ticket takers who fulfill stakeholder requests. No, you want your data team to be working on strategic high-value projects that will drive measurable impacts on the business. After all, how can you focus your limited resources on driving the highest impact if your team is churning out dashboards and emailed reports by the dozen?

Let’s reframe the word “service” from an IT context to a librarian context. Going back to “Conducting the Reference Interview”, Chapter 1, the authors state unequivocally that “this book is founded upon the principle that libraries must take a service orientation.”

We are convinced that the institutions that will survive in the twenty-first century are those that serve their clients and give them the help they need. If libraries don’t provide helpful information services, users will turn to other service-providers who are more service-oriented… librarians provide a professional “value added” knowledge component that is lacking in many commercial services. The RUSA (Reference and User Services Association, 2000) “Guidelines for Information Services” state: “Provision of information in the manner most useful to its clients is the ultimate test of all a library does.”

Some useful context for this quote: there was a significant contingent of LIS researchers convinced that search engines would render the profession obsolete. While the role of librarians and the value they provide may have shifted, librarians are still around and more vital than ever. 20 years later, a similar debate is ongoing… this time focused on generative AI. The conclusion to this debate seems even more obvious than the one over search engines.The librarian skillset is only going to become more and more important in this "age of AI"

A side note in the book elaborates on this idea with a further citation from the literature:

Herbert White (1992) says that librarians need to emphasize their strengths. As computers increasingly take over clerical tasks that computers are good at, librarians should focus attention on aspects of service involving human communication that computers can’t do well. Let computers get involved in document identification, document delivery, overdue notices, interlibrary loans and cataloging, White argues, and let librarians take a proactive role in information intermediation.

Reframed: a service-oriented data team is focused on providing information in the manner most useful to its stakeholders, and is a proactive human intermediary between the organization and its data resources. (I’ll posit that this approach is not opposite of a product-orientation, but rather complementary.)

As a data team, you acquire, organize, curate, and retrieve data resources. The team’s collective expertise is vital to interpreting and analyzing these data resources. However, the data team is rarely (if ever) the one making the decisions (or the ones accountable for the results of said decisions). Data teams can be as strategic as possible, working directly with the C-suite on the most important objectives for the organization, but by virtue of their raison d’être, data teams will serve a supporting/helping role.

For some, this can grate - you may want to be the “hero” data scientist, crafting machine learning algorithms that make the decisions for the business, rescuing the organization from heuristics-driven (poor) decisions and instead achieving the lofty goal of data-driven (smart) decisions. Rock, meet hard place - given human nature and organizational realities, your hero aspirations are likely a pipe dream, and the sooner you realize that humans will always be in the loop, the faster you can start making a real impact.

How does your data team currently operate? How do you want your data team to operate in the future? What does your organization currently need out of the data team? What are your organization’s goals for how to use data in the future? How does the data team need to operate in order to help the organization achieve their data goals?

As a data team leader, it’s your responsibility to consider these questions and create a strategic plan to ensure not only that the data team is adding net value, but that the rest of the organization recognizes that value. Do you think that the librarian’s frame for being “service-oriented” should be a part of that strategy? If analysts start new projects or respond to ad-hoc requests with some version of a “reference interview” (vs immediately building out a dashboard based on the description field in a Jira ticket), how do you think that will shift both the actual and perceived value of the data team?

Wrapping Up

At the start of this post, I gave a few examples of scenarios data teams may encounter:

1. A stakeholder you’ve worked with before comes to you with a new request. They saw what kinds of questions could be answered with a dashboard in the last project, and now they have some new questions to try and find answers to
2. You get an email from someone in the organization who wants a dashboard for their data. Right now they do a lot of manual work and one-off visualizations in a spreadsheet, and their friend in another department recently showed them a dashboard that the data team made
3. An executive asks your boss for a set of very specific metrics. Your boss tells you to write some SQL queries to generate numbers for those specific metrics, so they can then email those numbers to the executive

If you made it all the way to the end of this extremely long blog post, then I hope you can start to brainstorm how you might approach these scenarios as a librarian would. The point of this post is not to say that all data teams should adopt the reference interview for all ad-hoc requests and requirements gathering activities for longer-term planned projects. Rather, I hope that you have a new perspective on how your data team’s processes and procedures might be shifted - that you have a new hat you can wear, a new lens through which to see the world, another tool in your toolbox… whatever metaphor floats your boat.

There are 2 key things you should remember about the purpose of the reference interview:

1. A stakeholder’s first stated information need is rarely their true information need
2. The best way to uncover their true information need is by approaching the conversation with empathy, nonjudgement, and genuine curiosity, as a partner who acknowledges everyone’s respective expertise, and by utilizing the techniques of active listening

The idea that stakeholders rarely ask you for what they actually want is hardly a new one or unique to the reference interview. Just ask your product managers, or anyone with a few years of experience interfacing between the people who want things and the people who have to deliver those things. Hopefully this is not a revelation for you either - though if it is, it’s a valuable lesson to learnShachar Meir has given talks and put out content that is very aligned with the ideas in this post, just from his own experience/perspective - I would highly encourage anyone to follow his work!.

Far more important than realizing that stakeholders rarely state outright what information they actually need, is the perspective switch that it is your role to uncover their true information need. You will always be handicapped if you blame your stakeholders for not accurately translating their information/business request into the data team’s language.

The reference interview provides a structured way to accomplish this vital discovery process, and it really doesn’t take too much translation to go from a librarian-performed reference interview to a data analyst-performed reference interview.

To review (in short), this structure is:

1. Listen to the initial request without interruption or judgement
2. Paraphrase the request back to them
3. Ask open ended questions first, followed by clarifying close-ended questions
4. Restate the request yet again, now altered by their responses to your questions. Repeat until the rephrased request is confirmed as accurate
5. Collaborate to find an answer to the rephrased information request, using the interviewer’s expertise in navigating the information system and the interviewee’s guidance and background knowledge. Depending on the need/relationship, either can be the “driver”.
6. Verify that the true information need was met. If not, the process can be repeated, or the requestor may need to be directed to further resources if their information need is beyond the capacity of this process. Finally, you can ask for feedback on the process.

If you have the time, take a moment to think about each of these 3 scenarios. How would your team approach them today? How would a reference librarian approach them? Role play is an extremely effective way to learn and practice new skills - grab a partner and try conducting a reference interview for each of these scenarios!

This post had a lot of questions for you, the reader - many of which were left unanswered, because every reader will have a different answer. I hope that you put some time into considering these questions, and the next time you get one of those annoying ad-hoc requests your memory might be jogged, and something will prompt you to put on your reference librarian hat. (There are certainly worse hats to wear!) I hope you will experiment with the reference interview method - adapt it and make it your own. And I hope that you consider the time that you spent reading this post well spent - thank you for reaching the end!

If after reading this post you are intrigued enough to try any of these strategies for yourself or your team, I would love to hear about it! Please reach out via email, on socials, or by leaving a comment on this post. I would love to hear your story.

Special thanks

Most students in an MSLIS program take a class called “Reference & Information Services”. It’s a core part of the curriculum, if not required. Many MSLIS graduates go on to work in a library and gain experience in conducting the reference interview. Unfortunately, neither applies to me - I veered hard into the “information management” side of my program, and then kept on veering straight into the data field. However, I do have the great fortune of having a group of friends from my cohort who did take Reference, and who did go on to be librarians who conduct reference interviews. When the idea for this blog post was first taking shape, they were the first people I went to for help and resources. They sent me their class syllabus and readings, and patiently answered all of my questions about the reference interview based on their knowledge and experiences. Thank you for the references and the conversations - I wouldn’t have been able to write this without you. To Yvonne Freebourn, Taylor VanTryon, Adam McConville, and Christine Willson, UIUC iSchool Class of 2020.

AlsoThank you to the family, friends, and colleagues who read a draft of this (absurdly long) post in advance and offered feedback/edits/thoughts. I appreciate y'all <3, special shoutout to Amalia Child for citing this postat long last, your footnote can have a link :D long before it even existed, and fighting the good fight with me to spread the wisdom of Library & Information Science to the data world.

Appendix

The data professionals are not okay. We may put on a good face and talk about how excited we are that AI is going to change the world, but behind that mask of professional enthusiasm, we are anxious. Nowhere was this more evident than at the Data Day Texas Town Hall, hosted by Joe Reis and Matt Housley for the 3rd year running. Are executives going to replace us with AI? We knew that AI would likely impact other industries, but will it also reduce job opportunities in the industry responsible for building AI systems? AI is supposed to improve our productivity, to be a tool that we use to do bigger and better things, not a cost-saving mechanism for corporate executives trying to show that this quarter’s numbers are better than expected… right? We talk about how Copilot is just like a super productive intern - but what about the actual interns? If we stop hiring juniors and only hire seniors and supplement them with code generators, what happens in 10 years when suddenly there are no seniors to hire, because they couldn’t get a junior role? Even before that, are students learning the right skills to get hired into an entry-level role? Computer Science programs have been in crisis for a while now, curriculums more and more outdated with every passing year, and students are starting to question the value of a 4-year degree when they can only learn the skills jobs are asking for in bootcamps. Are we, the data professionals who have seized on AI trends as a way to justify our budgets (after all, your AI system is only as good as your data!), in fact building ourselves out of our jobs (not because we should really be replaced, but because the AI is cheaper and seems to be good enough), pulling up the career ladder behind us (why hire juniors if they won’t be productive enough for years), and undercutting the entire higher education system without anything to replace it?

Yeah, we’re anxious - all of these concerns and more were raised in the Town Hall. Of course, there were also the AI optimists, who debated the AI doomers with cheerful counterpoints that a utopia of sky-high productivity and dramatically more free time was just around the corner (though I’m reminded that Keynes thought we’d be working 15 hour work weeks by now), that AI would solve far more problems than it caused (hopefully it starts with the climate crisis, and codes itself into being carbon neutral), and that human ingenuity and adaptability would always prevail, and jobs will just look different rather than going away (though this tends to work across generations, individuals still get left behind). As Ole pointed out the next day during discussions, it’s ironic that the nation with one of the worst social safety nets is charging ahead with AI so determinedly while other developed nations where you can lose your job without worrying you may also lose your life have been dragging their feet on AI - one would think that paradigm should be flipped.

While the Town Hall gave DDTX attendees a chance to give voice to these anxieties in a massive group Data Therapy session (classic Joe Reis), the actual talks offered guidance on how to navigate this next year - the skills data professionals need to focus on, new concepts we need to learn and integrate into our practices, and some old technologies made new again that will be especially relevant in building AI that will actually live up to our hopes (and not our fears). I found the Town Hall comforting - to know that I wasn’t the only one anxious about these things - and the talks inspiring, charting a path forward that I am actually excited for. My experience at Data Day Texas was heavily biased by my interests, given that there were 6 talks happening at once throughout the day (and then 3 discussions at once the next day) and nothing was recorded, but I got a chance to learn about at least some of the talks I couldn’t see live through conversations with the other attendees. (There was one discussion that was recorded due to being live streamed - a town hall style session hosted by Joe Reis and the Super Data Brothers on AI in BI, available here).

There were 3 key themes tying the talks together that I noticed:

1. Learnings from Library & Information Science (and, context is still king)
2. “Human” (soft) skills help us connect with the business (and help them)
3. Data Mesh is a journey, and we’re now in the land of Data Products

Learnings from Library & Information Science (and, context is still king)

If there was one key takeaway that attendees at Data Day Texas walked away with, it was this: there is a lot that data professionals have to learn from Library & Information Science. Don’t believe me? (I am, after all, quite biased on this.) Ask Joe Reis, Matt Housley, Juan Sequeda, and Tony Baer (podcast episode here). In this Age of AI, context is king, and that context must be machine-readable. It’s common knowledge that your AI systems are only as useful as the data they are trained on, but DDTX attendees learned that the data in your data warehouse - no matter how carefully curated, modeled, and quality tested - is not enough. You also need the context - the knowledge - that often exists only in people’s heads. In addition to Data Engineers, you now need Knowledge Engineers. You’ve heard of “moving up the stack”? Time to also move up the DIKW pyramid, and data is at the bottom. What does it mean for context to be machine readable? Think metadata - and then take it one step further and think of metadata in knowledge graphs. Think of text (in all its myriad forms, from books to emails to human speech), the primary method we have for conveying knowledge. We are re-realizing the importance of data modeling in this Age of AI, so how can we model text data? As Bill Inmon explained, with ontologies and taxonomies. Do you know what field has cultivated and expanded the body of knowledge around metadata, knowledge graphs, ontologies, and taxonomies? Library & Information Sciences. As Tony Baer articulated (article here), the Knowledge Engineer (a role Juan Sequeda introduced in his talk) is an AI-driven evolution of the classical Reference Librarian.

Highlighted Talks:

• Meta Grid - metadata management as an understanding of what already is and embracing it, by Ole Olesen-Bagneux (slides)
• We Are All Librarians: Systems for Organizing in the Age of AI, by Jessica Talisman (slides)
• How to Start Investing in Semantics and Knowledge: A Practical Guide, by Juan Sequeda (summary)
• How to become a hero - the journey to text, by Bill Inmon (book)
• Context Engineering: A Framework for Data Intelligence, by Andrew Nguyen (slides)

“Human” (soft) skills help us connect with the business (and help them)

Data teams have spent the last few years struggling to prove ROI and justify their existence on the balance sheet, after costs associated with the “modern data stack” spiraled out of control. We may have fancier tools than ever before, but complaining how the business isn’t “data-driven” enough and throwing data products over the wall for the business to “self-service” isn’t resulting in the ROI we were hoping for. Another takeaway that DDTX attendees walked away with? An acknowledgment that we need to go back to basics and get better at communicating with the business. Yes, talking to people is a skill, and while most people consider it a “soft skill”, it may be among the harder skills you have to learn - but also one of the most important to progressing in your career and growing the impact of your data team. If you don’t want your data team to be treated like an IT service desk - someplace people submit tickets to and expect dashboards in return - but rather a strategic partner that curates the organization’s collective knowledge, then you will need to start valuing “human” skills as much as you do “technical” skills. I’ll note that last year’s opening keynote from Sol Rashidi (“Practitioner turned Executive; what caught me by surprise & lessons I learned about how decisions are really made with data ecosystems”) focused on this takeaway as well, and it is clearly something the data community is still grappling with.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.”

~ Maya Angelou

If you want to communicate and present effectively to executives and key business stakeholders, focus on crafting your story around how you want them to feel, and focus less on specific verbiage. Never underestimate the power of empathy.

Highlighted Talks:

• Bridge Skills: The Hardest Problem Tech Still Can’t Solve, by Eevamaija Virtanen (podcast episode)
• The human side of data: Using technical storytelling to drive action, by Annie Nelson
• Elevating Data in the Business - Bring Data and AI Skills to Life, by Jordan Morrow
• Empowering Change: Building and Sustaining a Data Culture from the Ground Up, by Clair Sullivan (slides)

Data Mesh is a journey, and we’re now in the land of Data Products

Two years ago Zhamak Dehghani gave the opening keynote for Data Day Texas on Data Mesh, a socio-technical approach to building a decentralized architecture for analytical data. The four pillars of Data Mesh (Domain-oriented ownership, Data as a product, Self-serve data platform, Federated computational governance) are meant to work together and reinforce each other, but when building a Data Mesh from scratch you need to start somewhere. Businesses are already roughly divided into domains (and dotted lines can be drawn until more permanent re-orgs fully align business units with domains), and the next step is to create Data Products (or, treat data as a product). Without Data Products, you can’t really have a self-serve data platform, and there is nothing to govern in a federated computational manner. Data Products seem to be the most visible and lasting legacy of the Data Mesh approach, but I think that is because right now we are in Data Product land (so that is all we see around us). Maybe once we all agree on what a Data Product is and what is needed to make Data Products, we can start seeing actual self-serve data platforms and federated computational governance on the horizon. What does it mean for data to be a product? Zhamak Dehghani has a definition, DJ Patil had the original definition, and perhaps we still don’t know, as JGP gathered us all together on Sunday to discuss and try to arrive at a new definition. One thing is clear, though: we all want to build Data Products, and those doing data engineering and data governance need to start learning how to adopt a product mindset. You don’t need to become a product manager, but you should be able to put on a product manager hat or see what you are building through a product management lens. The first stop through the land of Data Products? Data Contracts - the further upstream the better. Data Contracts should serve as the source of truth for our data’s metadata, and they provide the foundation for any computational data governance implemented further on in our journey.

Highlighted Talks:

• Data Mesh is the Grail, Bitol is your Journey, by Jean-Georges Perrin (summary)
• Introduction to Data Contracts, by Mark Freeman
• Data Governance – It’s Time to Start Over, by Malcolm Hawker
• Escape the Data & AI Death Cycle, Enter the Data & AI Product Mindset, by Anne-Claire Baschet and Yoann Benoit (podcast episode)
• Fundamentals of DataOps, by Lisa Cao

Other themes of note

Remember how I said I had to choose just 1 talk of 6 for every time slot? There were a lot of talks I did not get a chance to see. But from what I observed, there are 2 more themes that were prevalent at DDTX: Geeking out over Graphs (of course, this is a tradition at DDTX) and AI in Production (which, not infrequently, overlaps with graphs). A different attendee with different interests may very well say that AI in Production was the most dominant theme, and Geeking out over Graphs was a far more significant theme than those I’ve highlighted, but I can only speak to the talks and discussions I attended, and as much as I may wish I had a time turner (if only for conferences!) … alas, I do not. You can see the full agenda here. I heard a lot of great things about Chip Huyen’s talk, From ML Engineer to AI Engineer, which was heavily influenced by the book she just wrote, AI Engineering. If you missed the talk (like me), you can at least listen to Joe Reis interview Chip about it on his podcast. I also heard some great stories coming out of Michelle Yi’s talk All Your Base Are Belong To Us: Adversarial Attack and Defense and Hala Nelson’s talk Adopting AI in a Large Complex Organization- Aspiration vs Reality. One talk I did catch was Keith Belanger’s on Data Modeling in the Age of AI - the conclusion? LLMs are not going to do your data modeling for you, because data modeling is about understanding business context and thinking strategically. The talk was a spinoff from this article and argued that data modeling (by humans) is more important than ever, and we can take advantage of AI by automating mundane tasks, operating faster and more accurately, allowing us to spend more time focusing on real business value.

Honestly, I could keep going - this is a conference where you could trip and discover a new golden nugget of wisdom. (Side note, if you attended DDTX and have any bootleg recordings of any of the talks, shoot me a DM, because I want to listen to them all). I will say that looking at the agendas for past years, both of these themes show up over and over again, so while the 3 themes I identified may not have been the most dominant (overall), I do think they were the most distinct for this year (compared to previous years).

Diving deeper into my favorite theme: Learnings from Library & Information Science

This year’s keynote (the one talk with no competition for your time) was delivered by Ole Olesen-Bagneux, author of The Enterprise Data Catalog and (upcoming) Fundamentals of Metadata Management. Ole has a BA, MA, and Phd in Library & Information Science, and after listening to his talk the entire audience of over 500 data professionals walked away knowing that Library & Information science, libraries, and reference librarians have an important role to play in how we work with data. Ole presented about an architecture for metadata that he has coined the “Meta Grid” (see the slides here), and learnings from Library & Information Science were central to his conception of the Meta Grid. Ole presented the Meta Grid as the 3rd wave of decentralization - the first being microservices, and the 2nd being data mesh. The Meta Grid acknowledges that an organization has many metadata repositories scattered throughout, and proposes that the best solution is not to centralize the metadata into one monolithic catalog, but rather to connect these silo’d and uniquely shaped metadata repositories while maintaining their inherent decentralization. The result? An IT landscape that runs more smoothly, at reduced cost, and is better positioned to ensure data privacy, security, and innovation. If the connection to libraries seems unclear, consider this: the vast majority of a librarian’s work boils down to managing, curating, and navigating metadata, while operating within an network of decentralized yet connected library systems. Ole has spent the past year figuring out how to best articulate and explain the Meta Grid (and writing a book about it) to an audience that at first tries to understand what it is, and then wonders what he is trying to sell. And as Ole reiterated during the keynote, he is not trying to sell or productize anything - he is simply trying to alleviate a deep and enduring pain every organization at large enough scale experiences with their IT landscape, by applying learnings from Library & Information Sciences to corporate metadata management.

While Ole opened everyone’s mind to the idea that they might have something to learn from Library & Information Science, Jessica Talisman hammered it home with her talk in the afternoon, titled “We Are All Librarians: Systems for Organizing in the Age of AI” (see the slides here). Jessica opened with the argument that if you organize, research, educate, track provenance (lineage), and retrieve information (as most data professionals do), then you are a librarian. And in the age of AI, you would really benefit when performing these librarian activities if you also had some of the tools in a librarian’s toolbox for building a true semantic information system. This is not the first time Jessica has talked about the importance of bringing Library & Information Science skills into AI work - last year she gave a talk titled “What Data Architects and Engineers can learn from Library Science” (recording available here). This year’s talk built on last year’s, and focused in on how to actually approach building an ontology (hint: you don’t start at the ontology step) in a series of steps that Ole suggested afterwards be coined the “Ontology Pipeline”. By starting with a controlled vocabulary, associating those terms in thesauri, then further organizing their relationships in a taxonomy, you will be much better prepared to create the actual final ontology. Jessica highlighted OpenRefine as a tool to mine for themes & terms, and dove into a high-level overview of Linked Data to show the true power of ontologies in an open world (Wikidata being a prime example). She concluded with two pointed citations: (1) that AI systems are cultural technologies (and therefore rely on knowledge management) from Alison Gopnik, and (2) “The Semantic Web is not a separate Web but an extension of the current one, in which information is given well-defined meaning, better enabling computers and people to work in cooperation” from Tim Berners-Lee et al.

In a very similar vein to Jessica’s talk, Juan Sequeda also focused on the “how” for knowlege management and semantic technologies - only briefly reviewing the “why” (which was the focus of his Coalesce 2024 talk). And much like how Jessica opened her talk by listing out activities many data professionals do and arguing that these are foundational activities for librarians (thus, we are all librarians), Juan listed out many such activities (data cleaning, data transformation, standardization, relation extraction, data governance) and argued that this is socio-technical data & knowledge work (thus, we are all knowledge engineers). He defines semantics as a function that gets applied to data, and results in data with knowledge (or, semantics injects knowledge into data). What is the first step to applying semantics? Be more socratic and start asking why - both on a business and personal level (be empathetic & curious… talk to people!). The highest virtue Juan advocates for is reuse: ad-hoc is the enemy of efficiency, and we should focus on incentivizing the quality & reusability of our data products (and the knowledge that is required to use it!). Two acronyms to remember here: DRY (Don’t Repeat Yourself) and FAIR (Findable, Accessible, Interoperable, Reusable). Juan also highlights a methodology that he first started practicing a decade ago around building knowledge graphs, but is more broadly applicable to “knowledge engineering” work. It has three phases, with a business question as the entry point: Start by capturing the knowledge that the business currently has… then implement it through data modeling/transformation and injecting semantics (possibly via knowledge graphs & ontologies)… and finally make that knowledge accessible through data products and catalogs… but focus this entire process around answering a single business question at a time (then do a 2nd, then a 3rd, and maybe even more questions get answered along the way). With reusability comes another virtue Juan emphasizes: that 1+1 should be >2 (aka economies of scale). Why use knowledge graphs? Because everything is connected, and knowledge graphs are the best way to capture all of those relationships, and surfacing these connections in a data catalog is the ultimate way to maximize the reusability of this knowledge. Much like it is the data engineer’s responsibility to steward data through the data engineering lifecycle, organizations should employ knowledge engineers who are responsible for capturing, implementing, and surfacing knowledge.

Featured Content

This post was featured in The Analytics Engineering Roundup weekly newsletter on April 28, 2024.

Thank you Jason Ganz for the shoutout!

March 1st, 2024, was my last day working as a data engineer on the City of Boston Analytics Team. For the past month or so, I’ve been taking some time to reflect, and appreciate just how much I have learned and accomplished during my 2-year tenure at City Hall. This blog post is an attempt to document my growth - and also provide an outlet for the thoughts that have been bouncing around in my brain. So while I do want to let any curious readers know what my journey was like kicking off my career as a mission-driven data engineer, I also reserve the right to meander a bit.

While I did a couple shorter-term independent contracting gigs between earning my degree and working for the City, I consider my role as a data engineer on the City of Boston Analytics Team as my first “real” data job - first time working with a data warehouse, first time working collaboratively with other data engineers (and first time creating & reviewing Pull Requests), first time creating production ETL pipelines with an orchestration platform (and maintaining workflows & pipelines other people created)… all of the firsts that come with working as a part of larger data team rather than completing an entire data project by myself. And besides giving me the opportunity to kickstart my career as a data engineer, working for the City also solidified my passion for civic tech. Knowing that my daily work is directly tied to a mission of improving people’s lives is a significant motivator for me - and that will continue to shape the choices I make in my career.

Looking back, I can divide my tenure into 2 halves:

(1) the first year, during which I learned how to become an effective data engineer on the Analytics team - learning the tools, processes, and standard practices; and

(2) the second year, during which I was able to implement improvements to our tools, processes, and standard practices - and ended up re-architecting our data warehouse & ELT pipelines along the way.

So, let’s meander in (mostly) chronological order.

tl;dr

Maybe you don’t feel like going on a journey with me at the moment, and you just want the highlights - this section is just for you.

What I learned:

1. How to work with data warehouses (ELT, workflow orchestration, flow of transformations, testing the data, etc)
2. Patterns of collaboration & communication with fellow engineers (e.g. PR flow & etiquette, documenting work), analysts (e.g. division of responsibilities, data modeling), and stakeholders (e.g. requirements gathering, advocating for data engineering best practices)
3. The formal process of process improvement - how to develop and communicate innovative projects that will improve daily processes
4. How to start a dbt project from scratch and iteratively grow its architecture & adoption
5. How to present a fairly technical data engineering process improvement project to teammates, leadership, and a wider audience

What I accomplished:

1. Starting & growing the City Analytics Exchange, a casual network & meetup group of data analytics practitioners in local government
2. Teaching workshops (from Software Carpentries) on core technical skills like git, SQL, and python to fellow City employees (part of a larger data culture initiative)
3. Integrating dbt into the data engineering team’s tech stack, and re-organizing the data warehouse along the way
4. Presenting about the dbt migration project at Coalesce 2023
5. Collaborating across many teams & individuals in the department to deliver the team’s first data catalog as an official product supported by the department (SSO, boston.gov URL, data governance work initiated to support continued development)

Links to look at next:

1. Blog post: Analytical Data Warehouses - an introduction
2. Project page: dbt Migration at City of Boston

The first year (aka the year of learning)

When I came into this role, I had no prior experience working with a data warehouse, but a decent amount of experience working with databases. For me, learning to work with an analytical data warehouse was a paradigm shift - I had the core technical skills, but it felt like the conceptual foundation I had built on how to work with databases turned to sand beneath my feet.

As a student, I had learned how to design for and query normalized transactional databases. It had become second nature for me to understand new data by putting it into 3rd Normal Form, and implementing strong constraints on a database was a given. Have messy data? Design a relational model, then clean and wrangle the data into shape using python to output normalized, tidy tables that could then be visualized and analyzed. This was all fine for independent data projects, but it meant that I didn’t even realize I would need a whole new way of conceptualizing data work once I started working on a data team with a data warehouse. The first few months were a shock to the system!

I think there were two pivotal things that helped me adjust and reframe how to work with data in a more collaborative ecosystem: (1) taking the month-long synchronous online course “Analytics Engineering with dbt” from CoRise, taught by Emily Hawkins and Jake Hannan, and (2) staying connected with and learning from the data community online via social media. The combination of actively learning in more of a “classroom” environment, observing and absorbing data professionals talking about their work in a public online forum, and then being able to see how those learnings applied during my daily work at the City, all together had the effect of accelerating and supercharging my learning and re-orienting process. I was able to construct a new conceptual foundation for working with analytical data warehouses within a relatively short time, and then I could continue to iterate and build on that new foundation.

I owe a lot of thanks to Emily and Jake for answering my questions and having conversations with me during the CoRise course as I was building my conceptual foundation - those conversations and the resources they pointed me to were really helpful! I took the CoRise course in February/March, less than a month after starting at the City.

Before taking the course, all I knew was that dbt was popular in the industry and I should probably learn it. It was a very happy coincidence that I had signed up for the dbt CoRise course a while before even starting at the City - I had been hearing about dbt online but didn’t really understand what it was for. It took my brother convincing me over Thanksgiving vacation that if I wanted to work on a data analytics team I really needed to learn dbt, even if I didn’t know what exactly it was used for. So for the sake of “professional development”, I signed up for the CoRise course in hopes that I could at least learn what was even the point of this new tool everyone in the data community was so enamoured of.

After the course, not only did I understand what dbt was, but I understood why it was so valuable when working with a data warehouse… and how it could help improve the data engineering team’s workflows. It took about a year to get to the point of even starting that journey, but the seed was planted pretty early on.

While my knowledge about databases and data modeling has grown a lot over the past couple years, I have still held on to a few biases due to my roots in 3NF. First, I still think that the best thing you can do with messy source data is to normalize it. A normalized relational data model provides a strong and flexible foundation that can be queried and transformed into whatever shape you need it to be next. Second, primary keys are essential. Foreign keys and check constraints may be exceedingly rare in a data warehouse, but primary keys are the one contraint that will always be useful. The idea of primary keys being “unenforceable” and “just for documentation” gives me the heebie-jeebies. I understand why (performance), I just don’t like it.

On the flip side, there are some biases I started with that I have done a complete 180 on. When I first started as a data engineer, I wanted to do everything in python. I was very comfortable with pandas, and you could do much more complicated data transformations in a python script, so I thought that python was the best choice. Now I know that SQL is king - if you can keep it in the database and do the transformation in a SQL query, don’t even think about using python. Also, there’s no need to jump straight from source data to the final dataset in one long, complicated transformation when you can also get there with a few more steps that are simpler and modular - some of those smaller transformation steps may be re-used for other purposes. Finally, there is no single correct way to format/model your data - there are many different possible ways, and the best choice is the one that will actually get used (which is pretty much never the normalized version).

I think my overall point here is this: I was able to go from not really understanding what a data warehouse was to having a good enough understanding of them to re-architect the team’s warehouse and the ELT pipelines powering it inside of 2 years. But that learning journey was rather rocky, and the first few months were especially tough given the lack of formal training. Reflecting on that experience - both how much I had learned by doing & researching on my own, and imagining how that learning process could be smoother for myself and others - was the driving motivator for me writing the introduction to analytical data warehouses blog post (now turning into a series). In fact, that blog post started as a page in the team’s internal documentation that I decided to generalize and expand on, in case other folks might find it useful. I spent a lot of time and brain power on truly grokking the data warehouse from scratch, and that is something that I am both proud of and hope others don’t have to go through alone (though many inevitably will).

One final reflection on this: I realized that the reason I did not initially understand the point of dbt was because I didn’t understand data warehouses and how to work with them. dbt solves a specific problem that every team working with a data warehouse experiences, and it solves it very well, which is why it is very popular within the specific community of data practitioners who work with data warehouses. Once I understood how to work with a data warehouse (compared to a database), understanding the value proposition of dbt and how to use it was much easier. Figuring out how to use dbt within our specific data warehouse… that was something my brain started noodling on endlessly towards the end of year one.

City Analytics Exchange

I mentioned that another important part of my learning process was being connected to the larger data community online, and this perspective was one of the primary motivations behind my City Analytics Exchange initiative.

As a member of the City of Boston Analytics Team, I didn’t just want to learn from my fellow Boston teammates - I also wanted to learn from data analytics practitioners in other city governments. Fortunately I was not alone, and other teammates also wanted to cultivate a network of city analytics professionals! We started by reaching out to former teammates who had moved on to work for other cities, who could then reach out to their new teammates. We also started with a couple of motivating topics - specific tools that we all used or initiatives we were all trying to work on (e.g. knack, open data portals).

I created a free Slack workspace so we had a shared space to communicate, and scheduled time to (remotely) meetup with a loose agenda. We had a goal of meeting roughly once a quarter, and adding 1 new city each time we met. With an existing communication and meeting structure, it was much easier to add new folks each time we talked to other city data teams for some unrelated project - and surprise, suprise, other city analytics folks also wanted to talk to professionals doing the same kind of civic tech work! We got to share our accomplishments, work through problems, and generally collaborate, commiserate, and cross-pollinate in a casual environment of individual contributors doing hands-on-keyboard work.

I’m happy to say that the City Analytics Exchange is still going strong, and other members have stepped up to help facilitate the community moving forward. I hope that it continues to grow and be a resource for city data workers. If you are on a city analytics team and want to join, you can reach out to Amy Hood or Sam Firke.

The process of process improvement

Early in 2023 the Analytics Team collectively took the “Advanced Innovation Training” with Brian Elms from Change Agents Training. We learned about techniques like process mapping & identifying waste, standard work and systems of work, batching vs flow, nudging, and how to calculate the value of innovations/improvements. We teamed up and developed some innovations that we could implement for the Analytics Team (although… most of these never actually happened). Many of the techniques seemed simple or obvious, but I think in large part the specific techniques or toy improvement projects were only the delivery mechanism: the real value was in initiating and enabling a culture shift so individual contributors (rather than managers) could feel empowered to make small (or large) changes to how the team got things done (a.k.a improving processes)… and then have a framework for proposing/communicating these innovations to both teammates and leadership.

In the 2nd half of 2023 two dedicated process improvement coaches (Kayleigh and Nolan) joined the Analytics Team and started to build out training for Boston city workers, and my understanding of the practice was able to advance even further through conversations with them.

The primary reason I wanted to highlight this training and what I learned from it, though, is that you could argue that the dbt migration project I kicked off in year 2 was just one very long process improvement project. I was an individual contributor, not a manager, but I was still able to propose and implement an improvement that would substantially change how the team worked with the data warehouse. I had essentially spent a year constructing an internal process map of how the data warehouse functioned, and I could identify many instances of how implementing dbt would reduce waste in that process. I could use the tools and techniques I learned in the Innovation Training course to communicate the value of spending time implementing dbt to leadership and others who weren’t familiar with the daily work and processes the data engineers and analysts followed.

What a perfect segue into…

The second year (aka the year of doing)

Starting around December 2022, I started to think more seriously about how the data engineering team could start to use dbt. There were three main obstacles I had to overcome: (1) I had to show that dbt would provide a real value-add and would be worth the investment of time and resources so that it could evolve beyond a personal pet project into something the entire team could work on, (2) I needed to get all of the setup figured out so it would be as easy as possible for the second (and third, etc) engineer to start contributing to the project, and (3) I had to decide whether to make dbt fit the team’s current data warehouse architecture, or whether and how to redesign the data warehouse.

I also had a couple of theories in how to overcome these obstacles: (1) the dbt-generated docs site with its clean documentation and pretty lineage graphs would be the biggest selling point of dbt, and (2) the easiest path in getting started would be to work outside of the existing data warehouse in a small separate database using open data with a complex lineage graph. I also hadn’t yet decided on the best path for obstacle #3, and pursuing a toy example that could demonstrate the power of the docs site would buy me some time to think through possible solutions.

Thus began a solid few months of what felt like banging my head against a wall in hopes that answers would fall out. I’m not going to lie - it was rough. It didn’t help that the open data set with a complicated lineage I had chosen, CityScore, hadn’t been maintained and was a much bigger beast than I was anticipating. It was essentially a practice run that never fully materialized - I struggled through figuring out how to set up the infrastructure needed to develop and run a dbt core project locally, and hacked away at how to refactor the CityScore pipeline into a set of dbt models. Even though I never reached my target shareable end product for that toy project, I did learn a lot along the way. It gave me time and the opportunity to clarify my thinking and approach in how to migrate our transformations to dbt in the real data warehouse.

Along the way I decided on how to solve obstacle #3: the best approach would be to redesign the data warehouse with a new set of schemas that were designed with dbt in mind. With the dbt work isolated to an entirely new set of schemas, that new experimental work could proceed without interrupting the existing data & pipelines. We would essentially be slowly duplicating our existing workflows and data, but in a different set of schemas, with all transformations and tests living in dbt, and the import and export tasks re-organized around the dbt transformations.

FYI

If you just want to find the City of Boston dbt project details/links/resources (such as the above presentation), head on over to the project page.

If you want to hear some of the back story on how it all happened, you can keep reading this post.

Redesigning the data warehouse was not a decision I could make in isolation, of course - this was a decision that would effect the entire Analytics team, as well as anybody else who was using the data warehouse. It was also a change that had to stand on its own merits - while the new set of schemas were designed with dbt in mind, simply facilitating a migration to dbt would not be a good enough argument for making such a substantial change. Fortunately, others had been feeling the pain of our existing schema organization as well, and the new set of schemas solved many of these issues: (1) isolating the source/staging layer from “production” tables & views, (2) substantially fewer schemas, (3) eliminating the need to create new schemas every time a new dataset was added that didn’t fit into any of the existing schemas.

The old set of schemas were organized along the 2 dimensions of access level (open, internal, restricted) and data owner (department, team, source system, etc). The new set of schemas preserved the access level distinction, but instead of also organizing by data owner (the reason there were so many schemas), they were organized by data processing stage (staging, development, production). The addition of a “development environment” in the data warehouse was also a huge selling point for the engineers - previously, everything was production.

After getting approval from leadership, I presented this proposal for a new set of schemas to the team in March 2023. I wrote up a long document detailing why the old set of schemas were problematic, why the new set of schemas were organized this way, and how this would change some of our ways of working. I also created a form so that the entire team would have a chance to vote and give input on the new set of schema names - prod vs production vs mart? open_prod vs prod_open? staging vs stg vs source? Overall, the feedback was positive and many people gave their input on the new schema names. And while dbt was mentioned as a next step, the decision to adopt the new schemas was made entirely on its own merits.

With the team’s approval and a new set of empty schemas, I could move forward with the next step: actually getting the real dbt project up and running. Fortunately, I had learned a lot during my experiments over the past few months, so I already had a rough plan of approach. The biggest hurdle would be getting the first model to successfully run - not only being able to run a dbt command on my local computer to query and create a table, but also creating a workflow in Civis that would run a dbt command within a docker container and successfully connect to the database and create that table. And the local development environment needed to be sufficiently documented and easy to setup so that other engineers could do the same thing. Getting both the development environment (VS Code + dbt power user extension, venv, environment variables + profile for database connection) and the production environment (docker container, Civis credentials + environment variables for database connection, Civis workflow) correctly set up took some trial and error, but eventually we figured it out and got it running. By May 2023 the dbt project repo, local development environment configurations, and production civis workflow were all set up and there were a few models added to serve as examples. None of this would have been possible without the support of Albert Lee, our AWS/Docker/all-things-infrastructure wizard.

The next steps were to start populating the staging schemas with the most frequently used core sources (e.g. the 311 database, the permitting/licensing database, the EGIS database) and migrating over transformations that used those data sources, as well as onboarding the engineers most excited to start working on the dbt project. Going from 1 to 2+ people working on the project is always a big but important step, and was a good opportunity to work through any kinks in the process (is the documentation thorough enough? does it work on your machine?). By June 2023 there were 3 of us working on the project, and we were able to ramp up the migration process. I’m so grateful to Jenna Flanagan and David Falta for their strong and early belief in the dbt project (and their continued commitment to it after my departure!).

By July most of the engineering team had been onboarded to the dbt project, and we also had a very good idea of what to focus on migrating in the near term. In addition to the new schemas and dbt migration on the data engineering side, the analysts were starting a migration project of their own - transitioning from Tableau to Power BI. The timing for this double migration worked out perfectly, and we decided that all Power BI dashboards would be built exclusively from the new production schemas (and thus the transformations being handled by dbt). Once the set of dashboards being rebuilt first were determined, the engineering team had a definite list of table dependencies to focus on migrating to dbt and the new schemas. We could also go ahead and document those future dashboards and their dependencies as exposures in the dbt project. And given that the dbt generated documentation site was the initial selling point of dbt (those lineage graphs!), we also focused on getting an internal website (hosted via AWS services) that was updated daily ready for the team to reference.

By September we had successfully migrated all of the high priority Power BI dashboard dependencies, had a hosted dbt-generated documentation site customized with City of Boston branding, and the new set of ELT workflows (the new import workflows and the dbt build workflows) were running daily in production. Within 6 months we had a functioning (if minimal) redesigned data warehouse and dbt-centric ELT workflows… just in time.

Presenting our work in public at Coalesce

I just sketched out the work we were able to do in 6 months, but let me go back in time to the start. In March, before actually doing any of the work but after getting the go-ahead from the team, I submitted a talk proposal to Coalesce, the yearly dbt user conference hosted by dbt Labs. When I looked for other organizations using dbt, I saw a lot of companies - startups and large corporations - but nothing from government or other public sector adjacent organizations. But I knew that there had to be some other public sector data professionals using dbt, and after reaching out to a few folks in my network I found out that the State of California’s Office of Data & Innovation team was also using dbt. Not only that, but Ian Rose, a senior data engineer on the team, would be willing to join me in proposing a talk for Coalesce! I was so happy and grateful that I would not have to submit a talk alone (public speaking is scary! … and I’d never given a conference talk before!), and I knew that with more examples we could present a stronger case to other government data teams that they, too, should consider using dbt.

We submitted the talk proposal in March, not too long after I typed “dbt init” for the official project. Coalesce was happening in October. So I knew that I had 6 months to not only get the dbt project into a reasonably presentable state, learn as much as I could on both the socio- and -technical side of managing this migration, and also figure out what exactly we wanted to say during our 30 minutes on stage. In June we learned that the talk had been accepted, and in July after announcing the talk we were able to add another cospeaker who had been working on a dbt project in the public sector, Laurie Merrell. You can imagine my relief at achieving an MVP (minimum viable project) by September! But while the technical sprint was slowing down, the social sprint was just starting. Presentations have a way of spawning more presentations (not to mention the accompanying practice presentations). So along with working on our official Coalesce talk with Ian and Laurie, I was also crafting presentations for the Analytics Team, the CIO (head of the department), and the department at large. On the one hand, I wanted to make sure everyone at my workplace would be very familiar with the material I would be presenting in public (a presentation that would also be recorded and posted on the internet), and have the chance to provide any feedback/input. On the other hand, I really needed practice presenting this material! The marathon of presentations (and their practice sessions) leading up to Coalesce definitely helped me be more prepared in San Diego.

Attending Coalesce was one of my favorite experiences of 2023. The community of data practitioners that gathers at Coalesce is truly amazing - the online friends I had not yet met in person (including my cospeakers!) and the new friends I was able to meet through the talks and social events, the impressive speakers sharing their work, the professional support networks (Data Angels!), even the dbt Labs folks who had been providing me assistance via slack… the people are the reason why I want to go back and attend Coalesce next year (and the year after that!). I’m not going to into too much detail about my experience at Coalesce, because (1) this post is already too long and (2) I was posting my observations and reflections the week of on LinkedIn, but suffice to say that I was very grateful that our presentation was on the first day so I could devote the rest of my time to just enjoying the conference. Oh, and I also got a cool certification after passing a test within an hour of arriving at the hotel ;)

Fun fact: I had an early flight and had not yet eaten before arriving at the hotel in San Diego, so after finding the exam room I grabbed an acai bowl to eat during the test (the closest place to buy food was a frozen yogurt place and acai bowls are delicious) that may be owed partial credit for me passing and becoming a certified dbt developer. The test itself is mostly a blur but I have vivid memories of that acai bowl.

Product(ion)izing the Data Catalog

Delivering the presentations about dbt to others in the department ended up being extremely useful for the next phase of the dbt migration as well: publishing the dbt-generated documentation site as an officially supported product, with a boston.gov URL and Single Sign-On access. While the engineering team alone was sufficient to migrate the transformations and tests over to the dbt project, many more teams and leaders in the department would need to contribute their time and effort to help turn the documentation site into an officially supported product. This effort was kicked off in November, with the majority of the work allocated for the following quarter (January - March).

Alejandro Jimenez Jaramillo, the department’s inaugural Director of Tech Governance and Policy, immediately saw the utility of an official data catalog in advancing data governance efforts. Aleja and I had worked together on data governance issues when they were a fellow in 2022, and after they rejoined the department in 2023 we were able to continue that data governance work through the data catalog. Without Aleja and others in the department, the data catalog project may have never gotten off the ground. But because of their efforts, and collaboration across many teams in the department, the first phase of the official data catalog was launched and available by my final day. Logging in to datacatalog.boston.gov before losing access to my Boston accounts was the best goodbye gift I could ask for.

Iteration is the root of progress

The first six months of the dbt project (March - September) were devoted to getting a minimum viable project up and running, with the first set of tables needed for high priority Power BI dashboards reliably updating in the new set of schemas. The next six months (October - March) were primarily about getting the rest of the workflows (imports, transformations, tests, exports) migrated, but it was also about iteratively improving our implementation of dbt. With all engineers now comfortable with the essential dbt skills, some were able to dive deeper into specific features in order to improve our usage of them and the overall dbt project.

When I presented about the City of Boston’s dbt project at Coalesce, I also published a skeleton version of our dbt project repository in a public repo. Before my last day, I updated this repo to be in line with our implementation as of March 2024. So, you can easily compare what changes were made over those 6 months. While this does not represent all of our improvements (so, anything that is implemented only in individual model sql/yaml files is not captured), it does capture most of them. I may write another post in the future outlining all of the key features we added that I think should be part of any dbt project (let me know if there’s any interest in that!).

I’m so proud of the progress our team made, and by the time I left in March the migration was about 75% complete and I was fully confident that the team could continue the dbt project work and migration without me - as well as carrying forward the data catalog and data governance work. There were a few key things that I think contributed to the project’s success as a team-wide initiative, rather than just being my personal project (which would not have had a lasting impact!):

1. We set aside time for a weekly dbt sync, where all of the engineers could meet and discuss the status of the migration, propose features and improvements, ask questions to improve their own understanding, and share their own learnings and discoveries. The meetings were half project management and half knowledge sharing, and I was very intentional in making sure they were a collaborative, safe, and open environment. We were all learning and developing this project together. We also traded off facilitation roles so that there was not just one person in charge of the meeting.
2. We made the right things the easiest things. For example, we wanted to make sure that it would be very easy to add documentation to models at a later point - possibly by those less familiar with dbt. So, every model SQL file also had a YAML file with description fields for every column. Through a combination of the codegen dbt package, some custom macros, and bash scripts, it was as easy as specifying the access level and a model name as arguments and running a bash script to create the base YAML and SQL files. It was fast, easy, and accurate.
3. I put a lot of thought into the initial design, and made sure that the database design and dbt project design worked together cohesively. And once that larger design was decided, we stuck to it. This consistency in the overall design and architecture of the project meant we could focus on the smaller implementation details, and be confident that the ground would not move beneath us as we were trying to build on it.
4. We collaboratively developed a “style guide” and standards for our work in the dbt project. Much of this borrowed from the official dbt Labs recommendations (e.g. keeping the source and ref macros at the top in a “model imports” section, having 1 SQL and 1 YAML file per model), and some of it was customized for our project (e.g. using the generate_geom_column macro for all geometry transforms, and making sure every source had the custom table-level test test_table_not_empty). We held each other to these standards in PRs, and could raise any questions in the weekly syncs.
5. We didn’t try to do everything at once, and perfectly for the first time. Instead, we focused on the next thing that needed to happen given the team’s goals and current priorities. That means that there are some things we had not implemented by March that many data teams might consider essential (like CI/CD checks & pipelines), and others things we implemented relatively early that many teams may not consider as important (like custom styling for the dbt docs site). When we had a relatively small number of models, we didn’t focus on optimizing when those models ran. But when we had many sources and were moving more to be updated at night (and the database was overloaded during the morning run) it became important to implement the freshness selector for the nighttime and morning dbt builds. When a migration happens on top of your regular workload, you quickly learn to prioritize only what is absolutely needed next - and having a vision for the future state can help a lot with that more immediate prioritization.

None of this is to say that our migration went perfectly smoothly - we ran into plenty of hiccups and challenges along the way. For over a week our postgres database ran out of memory and crashed every morning at about 9am (right when it was needed for dashboard refreshes). Analysts suddenly lost the permission to query tables they needed. Some models that should be part of the daily run were not included, so some tables had stale data and we didn’t catch it until a stakeholder raised the issue. We were all learning how to build the airplane as we were building the airplane as we were flying it - and because we had a talented and dedicated team of engineers we did eventually diagnose and solve these issues… and we all learned so much in the process!

I’m so excited to see what the team does next with the dbt project. My goal/hope is that the analysts join the engineers in contributing to the dbt project repo. Maybe data owners will start contributing documentation to the data catalog. Analysts may start using dbt metrics (maybe even the semantic layer!) in pursuit of a standardized city-wide metric library. Perhaps in a few years, other data teams throughout the City will develop their own dbt projects, and a multi-project dbt mesh could be established. With the dbt project established, other tools from the modern data stack (those with open source options, of course) could be added to the data engineering team’s toolbox.

In order for analysts and other data professionals to contribute to the dbt project, however, there are 2 key skills that they will need: SQL and git. And that leads into the final accomplishment I would like to talk about: bringing my Software Carpentries instructor experience to the City.

Teaching computational skills workshops for City workers

In grad school, I got involved with the Software Carpentries and became a certified instructor. I learned how to teach core computational skills (bash, git, SQL, python) to beginner learners in the context of also teaching reproducible research skills. I really liked the Carpentries teaching philsophy of live-coding, conscientious language, and getting learners to the point of using the technical skills as fast as possible (workshops average about 3-4 hours). Also, all of the lesson plans and content are open source, so technically anybody can teach a workshop using Carpentries material.

In 2023 I got the chance to teach a couple different Carpentries workshops. When the analysts on the team started expressing interest in having their own github repo for the growing number of projects using python rather than a dashboarding BI tool, I offered to teach the Carpentries workshop on git with some custom content tailored for the team. This ended up being folded into a larger data culture pilot program during the summer and was opened up to other City workers as well. During the latter half of that data culture pilot, I was also able to teach some material on pandas from the Carpentries python workshop.

In December, I also decided to host a SQL workshop and opened it up to any city worker who might be interested. I knew that SQL skills were going to become more important for analysts across the city who may be expected to query the data warehouse in order to rebuild a dashboard in Power BI, and I also just really enjoy teaching the Carpentries SQL workshop.

One amazing outcome of these workshops? Another team at the city who used the Analytics team data warehouse and had specific subject matter expertise used the opportunity to upskill their analysts and started using a GitHub repo to store and collaboratively develop SQL queries to create tables/views that could build on each other. This is a significant and useful step forward from saving long, convoluted SQL in the BI tool, as well as making it much easier on the data engineering team to later incorporate these queries into the dbt project.

Wrapping up

This post is already longer than it needs to be, so I’m not going to spend too much time wrapping up. I will just say this: during my 2+ years at the City of Boston I was able to learn a lot by doing, I accomplished a few key things that I hope will have a lasting impact, and I met some really amazing people whom I hope to stay connected with for the long haul. There are also a lot of projects I worked on that I did not talk about in this blog post - this was primarily about the internal data engineering work I did. However, the vast majority of my time was spent on projects completed for stakeholders, who came to the Analytics team for data project help. It is not really my place to talk in detail about those projects, but I will say that I had a fantastic time contributing my expertise to their work - especially those longer, more complex projects that had a direct impact on constituent lives. I look forward to seeing what all of my former colleagues accomplish in the years to come!

~

Did you scroll directly to the bottom just get to the conclusion, and skip right over the TL;DR? Don’t worry, here it is again:

tl;dr

Maybe you don’t feel like going on a journey with me at the moment, and you just want the highlights - this section is just for you.

What I learned:

1. How to work with data warehouses (ELT, workflow orchestration, flow of transformations, testing the data, etc)
2. Patterns of collaboration & communication with fellow engineers (e.g. PR flow & etiquette, documenting work), analysts (e.g. division of responsibilities, data modeling), and stakeholders (e.g. requirements gathering, advocating for data engineering best practices)
3. The formal process of process improvement - how to develop and communicate innovative projects that will improve daily processes
4. How to start a dbt project from scratch and iteratively grow its architecture & adoption
5. How to present a fairly technical data engineering process improvement project to teammates, leadership, and a wider audience

What I accomplished:

1. Starting & growing the City Analytics Exchange, a casual network & meetup group of data analytics practitioners in local government
2. Teaching workshops (from Software Carpentries) on core technical skills like git, SQL, and python to fellow City employees (part of a larger data culture initiative)
3. Integrating dbt into the data engineering team’s tech stack, and re-organizing the data warehouse along the way
4. Presenting about the dbt migration project at Coalesce 2023
5. Collaborating across many teams & individuals in the department to deliver the team’s first data catalog as an official product supported by the department (SSO, boston.gov URL, data governance work initiated to support continued development)

Links to look at next:

1. Blog post: Analytical Data Warehouses - an introduction
2. Project page: dbt Migration at City of Boston

Many months ago I posted an introductory primer to analytical data warehouses, specifically targeted to folks who were familiar with data work but did not have a lot of experience working with data warehouses. In that post I made reference to a follow-up blog post that would focus more on the transformation flow within a data warehouse:

There may be many, many downstream transformations for a particular data source, and the structure of these can vary widely. There are many different design philosophies around this area of the data warehouse, but I will leave that for another post. Suffice to say, a large portion of the “thinking” work in a data warehouse is in writing these downstream transforms.

Well, this is part 1 of that promised follow-up post (finally!). There is so much to say about the transformation flow that I’ve had to break it up into 3 posts: Part 1 will attempt to demonstrate that you are likely already familiar with the transformation flow in other tools, and relate your experience back to how it would work within data warehouses. Part 2 will go over the major data modeling design paradigms, which provide a common structure for these transformations. And part 3 will focus on how these transformations and data models can be put together and implemented in the data warehouse (this is the one that will focus on dbt).

These posts will continue to be in the same spirit with the same target audience as the initial one: this is written for those of you who work with data but have not yet had a chance (or perhaps have recently begun) to work on a team that uses a data warehouse. I’m going to assume that you’ve started to develop intuitions about how to clean, wrangle, organize, and analyze data, and you want some help jumpstarting your ability to work professionally on a data team that has a data warehouse. And while the previous post introduced some broad principles for what data warehouses are, why you would want to use them, and how they tend to be used, this post will start to dive deeper into the transformation and modeling flow of data within the warehouse - the bread and butter of the “analytics engineer”.

A philosophical aside

Throughout this post, I want you to pay attention to how I talk about the transformation flow. I will use words like “intuition”, “design”, “philosophies”, “choices”, etc. The underlying point is this: data modeling - the thing you are doing when you implement specific transformations - is a creative act. There are no “right” or “wrong” choices (though there are certainly “better” and “worse” choices for your specific context and use case), and with time and practice you can grow your data modeling craft as a creative knowledge worker.

Please note that when I say “creative”, I mean it in the sense of thinking about a concept or solving a problem in a new/different way, not in the imaginative or fantastical sense. Think of data modeling in the same way that you would think of mapmaking: you are creating a simplified and useful representation of the landscape that captures enough detail that it can serve a variety of use cases, while not including so much detail that the map becomes a mess of illegible scribbles. “The map is not the territory” - in other words, the model should not be confused with the reality it is representing. Models are imperfect, limited abstractions that must be interpreted. A model that perfectly and completely describes its reality has no utility at all (besides being impossible) - we simplify and choose what to emphasize because our minds have limits. To go back to maps: would you rather navigate while driving with your map application in “map” view or “satellite” view?

So remember: the work of translating perceived reality into a limited respresentation that can be described in a machine-readable format, and then further transforming, integrating, and analyzing that data into a human-digestable format that can then be used to influence how others perceive reality (and thus their actions, which may effect reality)… is deeply philosophical, creative, and intellectually rewarding work (and hopefully impactful!).

Furthermore, it is important to remember that all of your more abstract data work should be grounded in the reality that is your source truth and the impact that you want your data work to have. So it helps to have both knowledge and interest in that reality you are trying to model and influence. If you are still learning and looking for practice projects, or at the start of your career and looking for jobs, don’t make the mistake of thinking all data work is the same. Focus on the subjects that interest you the most and you have specialized knowledge in - and the parts of the world where you most want to make an impact. For me, that domain is (generally speaking) government services and international development. My favorite data sources (and the focus of my upcoming personal project) are about peace/conflict (a.k.a wars, and preventing/resolving them) and comparing political systems. What domains and data sources are you most interested in?

All of that to say - it is impossible for any one person, blog post, book, or methodology to tell you how to transform and organize your data. What I hope to accomplish in this post is to build upon the intuitions you likely already have, and provide some structure and guiderails for how to go further. In the following posts, I will point to some resources you can use to dive even deeper into the many rabbit holes that exist in the realm of data modeling. But in the end, it is up to you to become the expert on your data and make creative data modeling choices to craft your transformations - and hopefully inform and influence the decisions and actions people take in a positive, productive way.

What’s in a table? (and what’s in a CTE?)

If you read the previous post, then you know that data flows through many different tables and views in the data warehouse, from source to final data/analytical product. However, beyond a few basic rules like making sure your first transform is simple and has a 1:1 relationship with its source table, I did not go into just how these transforms get divided up into tables and organized in the warehouse. Figuring out when a particular transform should be materialized as a table/view, or just broken out into a CTE in your larger query, is often a matter of intuition (intuition founded on learned and internalized guidelines). This can be frustrating for beginners, but in this section I want to show that you probably already have developed some of that intuition - you just developed it in a different context.

Before trying to bridge from intuition you developed either by writing code in notebooks or applying formulas in spreadsheets, let’s define a few key terms and note some caveats.

Transformations. I’ve used this term quite a few times already, and if you’re not sure what that means you’re probably already confused. A transformation simply refers to any type of change you are applying to dataset. Changing the shape of data by joining two tables or aggregating a table is a substantial transformation, while applying functions to columns like trimming whitespace or changing a text field into a datetime are simpler transformations. Adding data by appending rows or updating values is not a transformation. Essentially, any SQL select query that is persisted in the database (the result creates a new table/view) is a transformation.

Tables, views, and CTEs. A table in a database contains physical data structured by rows & columns. A view is a saved query, and doesn’t actually contain data. A “Common Table Expression” (CTE) is similar to a subquery in that it is a self-contained SQL select query inside of a larger SQL query, but while subqueries can exist within a clause of the main query (e.g. the select or from clause), CTEs are pulled out, ordered, and the first CTE is preceded by a with keyword. CTEs are more readable, maintainable, and DRY (a CTE can be referenced multiple times in later queries, while a subquery must be repeated). If CTEs are a new concept for you, here is a good introductory article.

You should already know SQL. If you are unfamiliar with SQL, then I’d suggest you revisit this post (and the previous one) after you have learned the basics of SQL and databases. If there is interest, I can write a separate blog post focused on how to learn SQL and the many free online resources available (as well as excellent books to borrow/buy). For now, if you’re still reading, I’m going to assume you are comfortable with writing SQL select queries.

Bring your own examples. This post does not really have any specific examples to illustrate the transformation flow. Instead, I’m going to be relying on you, the reader, to think back to past projects and fit the data & code you are intimately familiar with to approaches I’m describing. The goal of this post is not to teach you how to do a specific technical thing, but rather to help develop a general mindset and approach to transforming data within a data warehouse. Plus, I’d much rather demonstrate the entire process by blogging my way through a personal project than coming up with a few trite examples - so there may not be any short examples right now, but there will be one very long example in the future. (Please don’t hate - I’m mostly writing these posts for the fun of it!)

Building intuition

Burgeoning data practitioners typically come to data warehouses having already developed some expertise in manipulating data in one of two paradigms: dataframes or spreadsheets. Pick which of the following sections to read depending on which best describes you.

For the pandas (or polars/dplyr/etc) practitioner

I first want to revisit a paragraph from the previous blog post:

If you have worked on enough code-based analytics projects, you have probably learned the value of maintaining a proper data transformation flow. This may show up in how you organize your data folder: you have your “raw” data that is never to be altered, your cleaned/processed data that is an intermediate output, and your final data. You learn the value of using code to transform data - it means all transformations are documented, replicable, and easy to alter. Your notebooks/scripts are a mix of exploration, small cleaning tasks, and big transformations. You may have some functions in there for transformations that need to be performed repeatedly or to encapsulate more complicated code. You are displaying chunks of dataframes in order to visually examine the results and decide what needs to be done next. You are also saving dataframes to CSVs to be used elsewhere. All of these elements show up in the analytical data warehouse as well - they just look a bit different.

Let’s dive deeper into this comparison: the analytical data flow of pandas code in jupyter notebooks and data in files vs the analytical data flow of SQL code in a dbt project and data in database tables.

I’m going to assume that if you chose this section you are already familiar with manipulating data using pandas (or polars, or dplyr) in jupyter (or R markdown) notebooks in a git-tracked repo, and it is how this process works in data warehouses that you are not sure about. If that is not the case, then this section is probably not useful for you, as there is nothing to create a conceptual bridge from. Feel free to skip ahead to the next section or come back later.

It may be useful for you to pause reading and go back to look at your last project. Scan through your notebooks with a critical eye. Don’t focus on the individual lines of code, but rather the overall structure. How did you organize your data files? How did you organize your notebooks? What code did you chunk together into one cell, and what code did you pull out so you could run it separately? When did you display your data, and what were you looking for? What types of transformations get saved to the same dataframe variable name (overwriting the dataframe), and what types of transformations get saved to a new variable name? Why did you choose to use a new variable name, and what naming schema organically developed for your variable names? When did you save your data to a file, and why? What code has been organized into functions, and why did you decide to do that? At what point in the development process did you decide to make that change? What notes/documentation did you include, in code comments or markdown cells? How different is your final completed notebook from how it looked during the development process? Did you pull out any exploratory notebook code and formalize it in a python script? How does the code in the python file compare to the code (that presumably accomplishes the same or a similar goal) in the notebook?

Seriously, take a few minutes to pause reading, pull up GitHub or your local repo, examine your project, and try to answer these questions before continuing. It will help, and this blog post isn’t going to disappear.

Caveats and code review concluded, let’s proceed.

Your notebook likely follows a pattern like this: first, you import the data files you need and save it to a dataframe. Then, you apply some set of transformations, saving it to some new variable name. Then, you might apply a different set of transformations, and save those to a new variable name. Finally, you will save your transformed data to a file - perhaps at multiple points in the process.

It’s useful to save sets of transformations to different dataframes because each time you tweak your code you have to re-run all of the transformations relevant for that dataframe variable name (or risk being unable to re-run your notebook and get the same result). You don’t want a new variable name for every single transformation, because that gets unwieldy, and you don’t want to use the same variable name throughout the notebook, because some blocks of code take longer to run and it becomes tiresome to re-run the whole notebook every time you make a tweak. So, you naturally develop some conventions for when to save a dataframe to a new variable name. Those groups of transformations that get applied to the same variable name - those are the same groups of transformations that correspond to CTEs. CTEs serve two purposes: (1) they allow you to order and chain many transformations, allowing the final overall query to accomplish more than one simple query alone; and (2) they make complex SQL code more readable, by logically grouping together transformations and assigning the result a meaningful name. So, think of your CTE aliases as being analagous to your dataframe variable names, and CTEs as accomplishing the same conceptual chunking that each successive dataframe iteration does.

Why save a dataframe to a CSV file? You may have reached a point where you want to share the resulting transformed data with someone else. Or, you may want to use that data in another notebook. Or, you may be ready to do something else with that data - make a visualization, run a statistical analysis, etc. Regardless, you have identified that the data has been sufficiently transformed that it needs to be preserved in that state in a more permanent and reusable way than just an ephemeral dataframe that will disappear when the notebook is shut down. It is at this same point that you should conclude a transformation being performed by a SQL query, and save the resulting data to the database as a table/view. This will allow others to query the transformed data - for their own subsequent queries, for visualization, for exploratory analysis, for productionized ML pipelines, etc. So, think of saving a dataframe to a CSV as being analagous to saving a query’s output to a table/view.

In short, data goes through many transformations. These transformations tend to be chunked together, due to necessity or conceptual similarity. At some point the transformation is substantial enough that the resulting data has its own value, and that state of the transformed data should be preserved for multiple future uses. This is true both in the dataframe flow of pandas code in jupyter notebooks, and the database flow of SQL queries making tables and views.

For the Excel (or Sheets) Enthusiast

Spreadsheets remain the most common way most people interact with data. They are intuitive, easy to use, and extremely flexible and powerful. For the vast majority of use cases, spreadsheets are sufficient. But if you are reading this, then you have probably already bumped up against some constraints and know that databases can solve your spreadsheet woes.

Have you ever accidently deleted the content of a cell without realizing it? Deleted some cells, only to realize that now the rest of your data has been shifted out of order as well? Realized that your dataset has exactly 1,048,576 rows… and some data is missing? Maybe you have manually, laboriously, transformed your raw data into something analyzable… only to learn that new data is available that needs to go through the exact same process? Or you get handed a spreadsheet to analyze, and the organization of the sheets make you want to question why you decided to work with data in the first place? Do you have a folder full of the exact same data, with some kind of versioning note in the file name (_v2, _FINAL, _FINALFINAL, _2022, _2020-2021, _thisyear, _forCDO, _forJane, etc)? At some point in your data journey, you’re going to look at all of your spreadsheets and think… there has to be a better way.

Congratulations - there is! But if you have worked only with spreadsheets for your whole career, even the word “database” may be intimidating (or, you may be thinking, I already made a database with my spreadsheets!). Big, complex, analytical data warehouses may not initially seem like a viable solution for you, but I promise that if you have already started learning SQL then you are ready to start making that transition. Let’s talk about how the work you are currently doing translates over to the process you will follow when transforming data in the data warehouse, and the advantages this shift will offer.

First, rather than getting emailed a spreadsheet to transform and analyze, that data should already be present in the database as a source table - you just need to know what the table name is. (If the data is not already in the database, that’s probably someone else’s job to get it in there and make sure it gets updated on a regular schedule). Next, you will need to write a SQL query in order to transform it. Think about the transformation process you would follow in your spreadsheet: you may do some data cleaning by making sure that a categorical column has only the set of values that should exist (correcting misspellings, removing whitespace, etc). You may turn a text column into a date column (if Excel hasn’t already tried to do it for you - good luck). You may do some math on your numerical values by creating a new column and using a formula. You may then need to do further math by subtracting that new column from another column. You may need to create a pivot table after these cleaned and transformed columns are created. You may need to create a new sheet with only a subset of the original data, or a new sheet that combines data together from two other sheets. Some of these transformations are small/intermediate - they are needed in order to get the data to its final form, and are typically done by modifying data inplace or creating a new column. Some transformations are more substantial - the result gets its own sheet. Usually transformations have to be done in a specific order. At various points you may save and share the resulting data by emailing the modified spreadsheet to a colleague, or start creating inline visualizations based on this transformed data.

Think about at what point during the transformation process the data reaches a “final” state (which may happen multiple times in the overall process). You want to share out or visualize this transformed data. In a SQL transformation flow, the group of transformations you need to get to that point will compose your overall query, and result in a table or view being created in the database. Rather than emailing the spreadsheet, you can just let your colleague know the new table name, and they can then query that table. You can query that table with a BI tool in order to visualize it. And remember - all of those transformations were done with a SQL query, and the original source table remains unchanged. This means that if the source table gets updated with new data, you can simply re-run the SQL query, all of your transformations will get applied, and your transformed table will have the new data. If you made a mistake or need to modify your query, you can simply update the SQL and re-run the query. This is only possible because the source table remains unchanged.

Now consider all of the transformations that would make up that overall query, and group them into stages. First, this group of transformations must happen. Then, this next group of transformations must happen - but they can only be done after the first group of transformations is done. Maybe you want to break those stages into smaller groups because certain transformations seem to form natural groups, and it makes more narrative sense for them to be separate. These groups of transformations, whether due to the necessity of the order they must be performed or the nature of the transformations, would be equivalent to the CTEs that make up your overall query. CTEs are your query building blocks - each one is complete and sufficient on its own, but stack them together and you can build a more complex overall structure. Chunking transformations together conceptually also helps your overall query to be more readable - after all, you don’t just want the transformation to happen, you also want other people to understand what the transformation is doing.

In short, data goes through many transformations. These transformations tend to be chunked together, due to necessity or conceptual similarity. At some point the transformation is substantial enough that the resulting data has its own value, and that state of the transformed data should be preserved for multiple future uses. This is true both in the spreadsheet flow of functions and pivot tables, and the database flow of SQL queries making tables/views.

Did the conceptual bridges hold?

I hope that these comparisons made sense and you can more easily see how the data transformation work you’ve already been performing translates over to the SQL transformations you will craft to build out your data warehouse. However, so far these bridges are only built out of theoretical foundations - you’ll have to finish the rest of the job by practicing using SQL to transform raw input data into actionable output datasets, and all of the intermediate tables along the way. Most of this will only make sense after you’ve had experience building data models in your own data warehouse. Only with practice can you turn abstract theoretical knowledge into internalized tacit knowledge.

If that still sounds intimidating, and you’re not sure how to get started with your own personal data warehouse project, you are in luck! I have my own personal project that I’ve been meaning to work on for quite a while now, and I’ve decided to write a series of blog posts along the way documenting my process (and motivating my progress). Stay tuned for that series in the coming months.

Purpose-driven design

I want to note that so far, we have been focused on scenarios where we know where we are and where we want to go. We know what the raw source data looks like, and we know what the transformed data needs to look like in order to accomplish a specific purpose (such as creating a visualization, calculating a metric, or sharing summarized data). This is the most fundamental and impactful type of data design. Shifting transformations that you may have performed in a manual, isolated, undocumented way into an data warehouse paradigm where the transformation process can be automated, transparent, and documented is inherently valuable and a necessary first step. You know your data, you are analyzing your data, and you know how to get your data from point A to point B (and point C, D, etc). You know exactly what this data will be used for, what the purposes of the transformations are, and you can see your project through the entire data lifecycle.

However, one drawback of this purpose-driven design paradigm is that it does not scale very well. On large data teams, there may be different people assigned to each task: ingesting, transforming, documenting, visualizing, and analyzing the data. Those tasked with transforming the data may not know all of the many purposes the data may need to be designed for. Furthermore, the data warehouse will need to house many different datasets, for many different purposes, transformed by different people at different times for different projects. If every dataset is only transformed for the initial known purpose, over time the design of the data warehouse will become rather chaotic (this is how you get a spaghetti DAG). Plus, it’s very likely that most of the data will no longer be in use, data transformations will have been duplicated as slightly different use cases arise, and users may not even realize that the data they need is already in the warehouse (though perhaps not in the form they need). This is not necessarily a bad problem to have, as it means that your data warehouse has survived for a few years and has been actively used by many different people. However, at some point it really does become a problem, and someone is going to look at the data spaghetti and think, “there must be a better way.”

And of course, there is. In fact, there are quite a few you can choose from or combine in whatever way suits the larger purpose of your data warehouse. In part 2 (coming soon!) I will cover a few of the most important data modeling design paradigms in broad strokes and link to resources where you can learn more about each one.

When I first started working on the City of Boston Analytics Team, I had never really worked with an analytical data warehouse before. I had experience with databases, and with the data flow in analytics projects (mostly using pandas, in jupyter notebooks), and I had used databases as a starting point for analytics projects, but I had not yet put all of those pieces together. Then I started working as a data engineer whose primary role was to operate within an analytical data warehouse, and I took a 4-week crash course in dbt… and my brain went into overdrive trying to sythesize everything I already knew and was learning in order to come up with a mental model of what an analytical data warehouse was, how it should operate, and why. Now, almost two years later, I’m pretty sure I get it - and I want to share how I conceptualize a data warehouse in hopes that it makes the journey easier for anyone else who may be in the same position I was.

But first, a couple caveats:

1. I’m still learning, and I know my understanding will continue to evolve as I continue to learn
2. This is largely influenced by how my team uses our data warehouse, and it may be different for other teams

Databases vs data warehouses, transactional vs analytical (the what, and why)

An analytical data warehouse is the data source that powers analytical outputs - dashboards, reports, one-off statistical analyses, machine learning models, etc. Having a data warehouse makes sense when you are working in a team - you often need to share data, pick up other people’s projects, and know that an analytical output will continue to function even after a project is wrapped up. Having a reliable, managed, and shared infrastructure is a huge productivity boost. Also, pretty much every BI tool is built to at least connect to a database.

But what makes a data warehouse different from a (relational) database? In large part, this is simply a matter of semantics. Any database can be a data warehouse, it all comes down to how you use it. If you are using a database for analytics, you are likely doing a lot more and more complicated queries to read from the database. If you are using a database to power an application, you are doing a lot more (and need faster) writes to the database. Transactional databases are optimized for writes, and analytical databases are optimized for reads. Some DBMSs are balanced and can be used for both (i.e. PostgreSQL). Others optimize heavily for one use case (Snowflake is pretty much only used for analytics). How you are using a database also dictates how you organize it. If you are using a transactional database to power an application, there is a very good chance your database is normalized (most typically into 3rd or Boyce-Codd Normal Form).

Normalizing your data is the best way to organize your data when you want to reduce the duplication of data, avoid data anomolies for inserts/deletes/updates, and ensure relational integrity. But the normalized structure can make it difficult to answer seemingly simple questions - you often have to write complex queries with many joins. And analytics is all about asking questions of your data.

If you are using your database to answer many different analytical questions, you probably have many different data sources - and many of those data sources may be completely different from each other and never intended to be joined together. This is when a database starts to become a data warehouse - rather than one cohesive place to store data for one purpose (and therefore the data is all related), the data warehouse is more like a data repository. It is stored in a single place because of the convenience, not because the data is necessarily related (though some if it certainly will be).

In a well-designed relational database, it is usually pretty easy to understand how data is organized and linked through primary and foreign keys. You don’t necessarily need to know the data to query it so long as you know the structure. But an analytical data warehouse is not so easy to comprehend. There are almost never foreign keys, because they introduce complications to the ETL processes that update the data. If you are lucky, there are primary keys on some of the tables. To understand how an analytical data warehouse is organized, you instead need to look at the transformation flow.

The transformation flow (the how, and why)

If you have worked on enough code-based analytics projects, you have probably learned the value of maintaining a proper data transformation flow. This may show up in how you organize your data folder: you have your “raw” data that is never to be altered, your cleaned/processed data that is an intermediate output, and your final data. You learn the value of using code to transform data - it means all transformations are documented, replicable, and easy to alter. Your notebooks/scripts are a mix of exploration, small cleaning tasks, and big transformations. You may have some functions in there for transformations that need to be performed repeatedly or to encapsulate more complicated code. You are displaying chunks of dataframes in order to visually examine the results and decide what needs to be done next. You are also saving dataframes to CSVs to be used elsewhere.

All of these elements show up in the analytical data warehouse as well - they just look a bit different. No matter where the data lives, analytics data work should follow a set of core principles that cement data management best practices in the analytical output.

Principles of data management

1. Preserved raw data: raw data should be a direct copy of the source as much as possible, and should not be altered (besides being updated with new data)
2. Replicable transformations: all transformations of the raw data should be replicable/documented (the SQL/python code is saved, preferably in a git-tracked central repository) and automated if needed (transformations performed on a schedule in line with data imports)
3. Tested production data: all data considered “production” (ready for analysis) should be able to pass tests of data quality. These tests should be documented and automated if needed, like the transforms

Preserved raw data

When raw data is loaded into a data warehouse via an ETL process, it is loaded in a “source” table. Different teams have different names for this set of tables - “base”, “staging”, etc - but I will use “source” for the sake of simplicity/consistency. These source tables are often isolated or otherwise identifiable in the data warehouse. Some teams have a separate database only for source data, others have a separate schema or set of schemas for source tables, and some may use a naming convention to identify these tables (e.g. a suffix of “_stg”).

These source tables reflect the actual source data as closely as possible. This means that they may have the wrong datatypes, need to be cleaned (e.g. trimming whitespace), split out into multiple columns (especially if it is JSON text), the column names may need to be changed, or some other transformation is needed before it can be considered “production ready”. But it is important that these raw source tables remain exactly the same as they were when first loaded into the database - you want to avoid alter table statements.

Sometimes, these raw sources may come from a relational database. In this case, you may want to implement primary keys on the source tables - because if the primary key constraint is violated, that means something went wrong in the ETL process. However, implementing foreign keys is generally not recommended, as it can severely complicate the ETL process. Instead, document these foreign keys (in tests, in text, etc) and use them to inform downstream transformations.

Replicable transformations

The next step is to transform these source tables into “production” tables. There may be many transformations before producing the table that will be used for analysis, but there should always be at least one. In a data warehouse, a “transform” usually means a SQL select query that is used to create either a table or a view. For the purpose of this post, “production” refers to tables or views in the database that are ready to be used for any downstream analytical purposes (dashboards, analysis, further downstream views, etc), have been through a development and approval process, and have been designated as being “production” based on its name, schema, database, or some other indication.

The first transform is the simplist - it should select from the source table and apply any basic cleanup that is needed. Let’s call the result of this first transformation the “base production” table. It should correspond to the source table closely, and should have a 1:1 relationship to the source table (so, no joins). It should also always be materalized as a table (not as a view). The base production tables - and all downstream transforms - should live in a separate schema/database from the source tables, or otherwise be easily distinguished from the source tables.

Even if no cleanup or alteration from the base table is needed, this first transformation is still a necessary step. Why? Sometimes there is an error in the ETL process, and the source table may be wiped. If you only have one table (the source table), then you have lost that data, and it will effect any downstream uses on that table. However, if you have a base table (which selects from the source table), and the source table is wiped, then you can stop the transformation before the corresponding base production table is wiped. So, the base production table still has data (even if it is outdated - which is preferable to no data at all). Note: this is only possible if the base production table is materialized as a table, not a view.

Even if a source is static (not being updated by an ETL process), you still want to have this source -> base production table transform, and preserve both this separation and the tranformation applied. Why? You may discover at a later point that the base production table needs to be altered in some way - a column should be renamed, a data type needs to be corrected, whitespace needs to be removed, etc. Or, you may discover that a transform you thought you needed is actually incorrect, and you need to alter the transform. Once you alter the source table, there is no going back (besides dropping the table and re-creating it from the source). Whereas if you have a transformation query, you can simply re-run the query to re-create the altered production table. This is only possible if the raw data in the source table remains the same - and if the SQL used to do the transform is saved (preferably in a git-tracked repo).

Further transformations

While base production tables should be materialized as a table and should have a 1:1 relationship with its source table, all further downstream transformations can be materialized as views and query multiple tables. The primary reason that a downstream transformation would be materialized as a table instead of a view is for performance reasons - if the query is computationally expensive and takes a long time to run. Otherwise, views are preferred.

Note: further downstream transformations should never query a source table - only the production tables/views. These tables/views generally live in the same schemas/databases as the base production tables.

There may be many, many downstream transformations for a particular data source, and the structure of these can vary widely. There are many different design philosophies around this area of the data warehouse, but I will leave that for another post. Suffice to say, a large portion of the “thinking” work in a data warehouse is in writing these downstream transforms.

Tested production data

When new data is imported and transformed on a regular cadence, it is important to have Data Unit Tests (DUTs) to automatically check that previous assumptions about the data still hold true. DUTs can be performed on any table/view in the data warehouse - source, production, or downstream views. For example, you might want to check that the source table contains some minimum number of rows before doing the transform to production - and if the test fails, then the transform will not happen. You may want to check that a field you assume is your primary key is actually unique and has no null values. You may want to check that a categorical field only has a limited list of values - and if it fails the test, you only want it to warn you, but not prevent the transformation. The most important tests (the ones you want to stop downstream transforms if they fail) should be performed as early in the transformation flow as possible (ideally, the source tables), while the nice-to-have tests that should only warn you about failures (and not prevent transformations) can be implemented further downstream.

DUTs are a way to productionize data testing, but can also be useful during the exploratory phase when first designing transformations. You can test your data through a formal DUT structure, or through a series of queries. For example, you may want to select distinct on a field to see all possible values, or select count(distinct __) on a field compared to select count(__) to see whether there are duplicates. Based on this exploratory data testing, you may wish to revise your transform. And while it is important to document and preserve any SQL transform queries so they can be re-run, it is less essential to preserve these ad-hoc data tests - their primary purpose for static data is to inform your SQL query for transforming source data into the production table. However, if the data source may be updated, then formalizing them into DUTs is a good idea.

The implications (or, why dbt)

Imagine you are an analyst, staring at this data warehouse and all of its many tables, trying to figure out which tables are going to have the answer to your question. Or, imagine you are an engineer, informed that something has gone wrong in a dashboard and you need to figure out what (and where) in the data flow something went wrong. What is the one thing that you are really going to want (need!) in order to do your job effectively and efficiently?

Documentation.

You are going to want documentation for your tables, the columns in those tables, the relationships and dependencies between those tables, the tests performed on those tables, and any external dependencies (dashboards relying on those tables, ETL processes powering the source tables). But documentation has a dirty secret - (almost) nobody wants to or has time to write documentation. Unless you make documentation fast and easy to produce (or you pay for it by making documentation someone’s job or primary responsiblity), it isn’t going to happen (at least not consistently).

dbt may be advertised as a tool to transform (and test!) your data, but its real superpower is in the documentation that is produced as a side effect of how those transformations and test are implemented. If you do not currently have an infrastructure to support transformations and data unit tests within your data warehouse, then implementing dbt is a no brainer because it is the easiest out-of-the-box way to accomplish those fundamental tasks. But even if you do have a (likely custom/house built) infrastructure to do those transforms & tests, making the switch to dbt is worth it, because of the documentation that is produced (and the culture of documentation that is encouraged/supported). Once you see a DAG (directed acyclic graph of table nodes) that visualizes every step from source to dashboard, there is no going back.

Why? Because given enough time, your team will accumulate enough data sources (and the people most knowledgable about those sources will leave at some point), and enough complicated transforms and table structures, that working within the data warehouse will become a snakes’ nest of tangled dependencies, table rot, and dangerous assumptions. And if you can’t rely on your data warehouse, then your stakeholders can’t rely on your analytical outputs. Documentation is not just a nice to have - it is essential. Document your data, your transforms, your tests, your analytical outputs, everything - and then put that documentation in one easily accessible and searchable place. dbt makes this easy, and that is the real reason it has become the darling of the data community - and the reason I’ve wanted to implement it for our team ever since I learned enough about the tool and our current data infrastructure.

Which bring us back to our principles of good data management for analytics. There was, in fact, one principle missing from the list, so let’s round it out.

Principles of data management (final version)

1. Preserved raw data: raw data should be a direct copy of the source as much as possible, and should not be altered (besides being updated with new data)
2. Replicable transformations: all transformations of the raw data should be replicable/documented (the SQL/python code is saved, preferably in a git-tracked central repository) and automated if needed (transformations performed on a schedule in line with data imports)
3. Tested production data: all data considered “production” (ready for analysis) should be able to pass tests of data quality. These tests should be documented and automated if needed, like the transforms
4. Documented data: the data (each column, each table) should be documented, as well as the data lineage (how the final data was produced from the raw source data)

Bonus: the development cycle

Now that we’ve brought dbt into the picture, we can talk about one final piece of the (modern) analytical data warehouse: the development cycle. In the principles above (and the blocks of text further above), note the use of the term “production”. The presence of “production” data implies the presence of data that is not “production” - something distinct from the raw source data already described. This missing something is the “development” version of the data - basically, the data/transform that has not yet been finalized.

Tables that are still in development should be distinguished from production tables in the data warehouse. Dev tables may live in a separate database or schema, or be indicated through a table naming convention. Teams may also have less formal (read: social) ways of indicating that a table is in development, though this introduces opportunity for error. A table in development is not finalized - column names, order, and content may change, the table name itself may change, and the logic has not yet been reviewed and approved by the team. Essentially, it’s very risky (and certainly inefficient) to build downstream dependencies on dev tables, though there will always be cases where speedy delivery is a necessity and this rule has to be broken.

One great feature of dbt is that this development phase of tables/views (dbt refers to them all as “models”) is built into how dbt works through the use of profiles. It’s very easy to designate a specific schema or database as a dev schema/database, and to build models agnostic of the dev/prod schema. It’s also why git is a core component of any dbt project - the “main” branch should have models in “production”, while feature branches should have models in “development”. You have to be familiar with the use of version control in a software development cycle in order to use dbt effectively, and the fact that dbt brings these software engineering practices into the modern analytical workflow is considered a major selling point of dbt.

In conclusion

Unless you are already part of a data team that uses a data warehouse, it can be hard to understand what an analytical data warehouse is, how to use it properly, and why it is such an important part of working on a data team. If you’re in school or otherwise studying for a career in data, knowing how to work within a data warehouse will give you a big leg up over other entry level candidates. My hope is that this blog post can (1) convince you that you already know the most important pieces of the puzzle, (2) fill in some of the missing pieces of the puzzle, and (3) help you synthesize and put all of the pieces together to form a cohesive picture.

In future posts I would like to continue to flesh out how to work in a data warehouse, but this concludes the introduction. If you have any questions or future topics you’d like me to focus on, let me know!

This website, and therefore this blog, has fallen by the wayside since I graduated from library school with my MSLIS in the spring of 2020… in the midst of a global pandemic. I wasn’t really sure what to do with the site, beyond letting it exist and adding some projects here and there, but I recently decided that it was time for a revamp (and rebrand/redesign). I wanted my personal website to say “working professional”, not “overly enthusiastic grad student” - though I am keeping all the old posts so the “overly enthusiastic grad student” is not going away. But it has been 3 years (wow!) since I was in grad school, and I have managed to get some professional experience under my belt - though with the pandemic raging as I entered the work force I had a bit of a rocky start.

2020

My last semester of grad school was not the best. There was before spring break, and there was after spring break. Before spring break, everything was continuing on as usual, with some masks starting to appear. After spring break, everything shut down. All classes and work went remote. There was no graduation ceremony. I had no idea that the last time I saw my professors, classmates, coworkers, and firends, it would be the last time I saw them ever. While in undergrad I could not care less about the graduation ceremony, for grad school - a place where I really found my academic home - I was looking forward to the chance to see everyone one last time and say goodbye. Instead, there was just endless isolation.

After graduating into a pandemic with no job offer yet in hand, I did what everyone seemed to be doing - I moved back home. My mom was living alone, I couldn’t afford to pay rent without a job, and after the isolation of the previous months I was looking forward to forming a little family bubble. I put most of my stuff in storage, sure that I would be moving soon to wherever I found a job. And sure enough, within a few months of graduating I got a job offer… with one big catch. I would need to get my security clearance. I was so excited to get a job offer, I ignored the perfectly reasonable advice every entry level professional is given (and ignores): don’t accept the first offer you get. I was told that the process would take about 6-12 months, and I hoped that my previous security clearance (for my internship with the State Department in undergrad) would help speed things up. In the meantime, I could find some short-term contracting gigs to start advancing my data skills and developing experience. So that’s exactly what I did.

Pretty much as soon as I had some income coming in, I kept a promise that I had made to myself years ago at the end of undergrad (when I knew I was going to be doing a lot of traveling). I finally adopted a puppy. I adopted my sweet girl from Saving Grace, a rescue in Raleigh. At the rescue, the name on her tag was “Genuine Risk”. I named her Bella, after the loyal pony Bela in Wheel of Time. She was about 14 weeks old, the last of her litter, and shy as could be, but after 5 minutes of me sitting cross legged in her pen she crawled into my lap and fell asleep. That’s when I knew - and my mom really knew - that she would be coming home with us.

I also took up knitting (I know, such a classic pandemic hobby). The very first thing I knit was a blanket for Bella. My reasoning was: (1) she won’t care what it looks like, so it’s fine if it turns out awful and ugly, and (2) it’s getting cold, she’d probably appreciate a blanket that smells like me. (Actually I lie, the first 4 blanket squares got combined into a blanket for Molly, our much smaller and much older family dog). Sadly, that blanket no longer exists - it got slowly torn apart and consumed because… I gave it to a puppy. C’est la vie. But while the blanket slowly got smaller, I kept on knitting, and eventually started knitting shawls as wedding presents. (6 shawls and 5 weddings later, I’m finally now starting on a shawl for myself)

2021

A year came… and went. Still no word on the security clearance. I had done all of the steps (including going in person for the drug test, polygraph, etc) and I got no updates, no estimated timeline, nothing. I knew that I wanted to kick off my career properly - in a full-time position doing data work as a part of a larger team focused on public service - and the contracting gigs had only been meant as a temporary stopgap until I could join such a team. I had been applying for data librarian jobs and even got to the final round of interviews a couple times, but no dice (actually, my Carpentries lesson was a side product of one of those interviews!).

Finally, in late fall of 2021, I hit gold. A mutual on twitter pointed me towards a posting for a data engineer at the City of Boston’s Analytics Team. It was perfect - data work, in public service, and a genuine way to really start my career. I applied, made a streamlit app for extra credit, and by the end of the year I had accepted a job offer.

I kid you not, a week after accepting that job offer, my security clearance came through. But I knew that the data engineer position in Boston was a better fit for me, so I stuck with it and moved to Boston in January of 2022. Almost a year and a half of waiting on that security clearance… well, sometimes, timing really is everything.

2022

My first year as a Data Engineer on the Analytics Team was a true learning experience - exactly what I was hoping for from my first job to kick off this new career. In grad school, I had learned SQL and how to design databases in 3rd Normal Form. At my job, I learned about analytical data warehouses and how SQL was used in production code, and I leveled up my SQL skills substantially (can you believe we learned about subqueries but not CTEs?). In grad school, I had learned about automated workflows as related to improving data quality. At my job, I learned about data orchestration (hello, YAML files and YAQL) and the pros and cons of different workflow design paradigms. In grad school, I mostly wrote my own python code from scratch to finish. At my job, I learned how to work with a team of engineers and contribute to an existing codebase. I learned how to work and contribute within a larger analytics team - rather than doing every part of the analytics flow myself.

I also made sure to focus on my professional development in other ways. In March I took CoRise’s course “Analytics Engineering with dbt”, taught by Emily Hawkins. In August I took GovEx’s course on Data Governance. Both were valuable and what I learned in them I could immediately apply in my work.

During my first year, I focused on learning how the Analytics Team, and the Engineering Team especially, worked, and how I could best contribute within the existing framework. I learned so much from my coworkers, and I found the experience of having fellow engineers that I could lean on and collaborate with invaluable. But in addition to learning how to contribute to what was already there, I was also starting to identify new ideas that I wanted to add to the mix. Primarily: dbt.

2023

For my second year on the Analytics Team, I wanted to make a substantial impact to improve how the engineering team worked. This involved a lot of thinking and iterating through designs and strategies before I brought my proposals to the team. I wanted to make sure that when it came to the actual implementation of these plans, that it would go as quickly and smoothly as possible. It also meant communicating with my team and building interest and agreement in my proposals. I had been talking about dbt almost since I joined the team, but in a “this is a cool tool” way, not in a “let’s do it” way. If I was going to ask everyone to learn and use this new tool, I wanted to make sure they believed it was worth it too.

I don’t want to go into too much detail about what dbt is and how we are implementing it - I’ll save that for a future blog post. But suffice to say that the project got started early in 2023, and after 6 months we are finally in a sprint of building models and adding to the project, with almost all engineers onboarded and starting to onboard analysts. I’m very excited to see what further progress we can make for the rest of the year - and I’m especially excited to be able to share this experience of implementing dbt for a city analytics team in a talk at this year’s Coalesce (dbt Lab’s annual conference).

Besides the dbt project, this year I’ve also been working on larger and more complex projects that have involved a lot of data architecture/design work. Without going into too much detail, the city has a lot of legacy systems that don’t talk to each other, when city workers really need for information to be passed between those systems in order to do their work well and efficiently. I have been involved in a couple projects this year that have been focused on how to integrate these systems (or at least have all of the data in one place and able to be joined together), particularly for housing and development work. I love building ETL pipelines, but I know a project will be special when I get to do system design and data architecture work before building the pipeline. These projects are always bigger, longer, and trickier, involve working with more stakeholders and teams, and are just so rewarding because you can see the difference your contribution is making and you can build relationships outside of your team.

Another high point for this year on the team is that I’ve had the opportunity to start teaching Carpentries workshops again. I taught a workshop on git because the analysts on the team have been writing more code and wanted to preserve and collaborate on their code in a GitHub repository, and I taught some sessions on python and pandas as a part of a Data Culture pilot program focused on python. Later this year I’m planning on teaching a [SQL workshop](https://swcarpentry.github.io/sql-novice-survey/ and a session on querying APIs with python. Teaching computational skills is something I really enjoy, and it was the one thing I was really missing in this job last year.

Finally, if you think that working on a city analytics team sounds interesting, I will plug the fact that we are currently hiring for several positions - if you live in Boston or want to move to Boston, apply!

My Carpentries teaching journey started out slow - I taught a couple of times that first year, and stuck to the lesson I was most comfortable with (SQL & Databases). However, at the start of the summer in 2021 my work put me in charge (or I volunteered to be in charge) of a group of undergraduate interns who wanted to learn how to use computational methods for open source intelligence analysis. So I put together a curriculum of Carpentries lessons to take my interns from zero-assumed knowledge to the ability to complete a computational analysis in Python. I’m teaching the same curriculum in the Fall (slightly expanded and refined) to another group of undergraduate interns. This allowed me to gain experience in teaching more of the core Carpentries lessons… and also inspired me to develop my own lesson focused on interactive data visualizations!

A semester-long curriculum of Carpentries workshops

Each week I teach for 2 hours, and then learners can practice what they’ve learned and give feedback on the workshop in an assignment (delivered via Google Forms). I’ve included the curriculum schedule below:

Week 0: Self Assessment

Goals:

• Learners assess their current skill level

Assignment:

• Computational Skills Self Assessment form

Week 1: GitHub

Goals:

• Instructor & learners introduce themselves
• Overview of the curriculum
• All learners have a GitHub account and know the basics of creating and modifying a personal repository on GitHub (with GitHub Desktop).

Assignment:

• Assignment 1: GitHub

Resources:

• Getting Started with GitHub - Training by Elizabeth Wickes
• GitHub’s docs on creating a new account
• GitHub’s docs on GitHub Desktop
• GitHub’s docs on creating a new repo
• Writing a good README
• GitHub’s docs on cloning a repo

Week 2: The Shell (part 1)

Lesson Plan:

• Shell workshop lesson plan from Software Carpentries (Lessons 1-5)

Goals:

• become comfortable navigating your computer via the command line
• learn the basic set of Bash commands (pwd, ls, man, cd, mkdir, nano, touch, mv, cp, rm, wc, cat, less, sort, head, tail, echo)
• learn how to chain commands using the pipe, redirect, and append
• learn how to use loops

Assignment:

• Assignment 2: The Shell
• Answer Key for Assignment 2

Further Resources:

• The Linux Command Line - web tutorials
• The Linux Command Line - full free textbook
• Explain Shell

Week 3: The Shell (part 2)

Lesson Plan:

• Shell workshop lesson plan from Software Carpentries (Lessons 6 & 7)
• Shell Extras workshop lesson plan from Software Carpentries (Lesson 2)

Goals:

• Learn how to save a series of commands as a shell script
• Learn how to search files from the command line
  • grep, find
• Learn how to generate and use your own SSH key pair
• Learn how to access a computer remotely
  • ssh, scp

Assignment:

• Assignment 3: The Shell cont
• Answer Key for Assignment 3

Further Resources:

• Setting up and using the SSH config file

Week 4: Git

Lesson Plan:

• Git workshop lesson plan from Software Carpentries

Goals:

• Learn how to use git from the command line
• Set up git on the remote sandbox server
  • git config
• Learn how to create repositories
  • git init
• Learn how to track changes
  • git add, git commit, git status, git log, git diff
• Learn how to use the history
  • HEAD, git show, git checkout, git revert
• Learn how to not track things
  • .gitignore
• Set up a remote repository
  • git remote, git push, git pull
• Learn how to collaborate & resolve conflicts
  • git clone

Assignment:

• Assignment 4: Git
• Answer Key for Assignment 4

Week 5: Python (part 1)

Lesson Plan:

• Plotting & Programming in Python workshop lesson plan from Software Carpentries (Lessons 1-7)

Goals:

• Everyone has Anaconda (and thus Python & Jupyter Lab) installed and is familiar with how to work within Jupyter Lab and use a Jupyter Notebook
• Learn about data types and type conversion in python
• Learn how to use built in functions
• Learn how to get help, read the built-in docs, and read errors
• Learn how to import and use libraries
• Learn how to read data into a dataframe and interact with the dataframe (pandas)

Assignment:

• Assignment 5: Python
• Answer Key for Assignment 5

Further Resources:

• Intro to Anaconda from Software Carpentries

Week 6: Python (part 2)

Lesson Plan:

• Plotting & Programming in Python workshop lesson plan from Software Carpentries (Lessons 8-12)

Goals:

• Learn how to select data from a dataframe (pandas)
• Learn how to plot data (matplotlib.pyplot)
• Learn how to work with lists
• Learn about for loops and the accumulator pattern

Assignment:

• Assignment 6: Python & pandas
• GitHub repo for the pandas practice

Week 7: Python (part 3) + requests & REST APIs

Lesson Plan:

• Plotting & Programming in Python workshop lesson plan from Software Carpentries (Lessons 13-18)
• Storing Information with Dictionaries from Software Carpentries

Goals:

• Learn about conditionals (if, else, elif)
• Looping over files (glob)
• Learn how to write functions
• Learn about the dictionary data type
• Learn how use python’s requests library to query a REST API

Assignment:

• Assignment 7: Python

Further Resources:

• Python & APIs: A Winning Combo for Reading Public Data from Real Python
• Python API Tutorial: Getting Started with APIs from DataQuest

Week 8: Interactive Data Visualizations

Lesson Plan:

• Interactive Data Visualizations in Python workshop lesson plan

Goals

• Learn how to create a new conda environment
• Learn how to wrangle data into a tidy shape
• Learn how to create line plots with plotly
• Learn how to create and run a streamlit app
• Learn how to add widgets to the streamlit app

Week 9: Databases & SQL (part 1)

Lesson Plan:

• SQL workshop lesson plan from Software Carpentries (Lessons 1-5)

Goals:

• Learn how to select data from a table
• Learn how to sort results and remove duplicates
• Learn how to filter, or select subsets of data, based on boolean conditions
• Learn how to calculate new values to be returned in the results
• Learn about missing data, and incorporating NULL into queries

Assignment:

• Assignment 8: SQL

Week 10: Databases & SQL (part 2)

Lesson Plan:

• SQL workshop lesson plan from Software Carpentries (Lessons 6-10)

Goals:

• Learn how to aggregate data in queries (calculate sums, averages, etc)
• Learn how to combine tables in a query
• Learn about data hygiene, and principles of database design like primary & foreign key constraints
• Learn how to create, modify, and delete data
• Learn how to use SQLite databases within a python script

Assignment:

• Assignment 9: SQL
• Review Lesson 10 (Programming with Databases - Python) on your own

Week 11: Regular Expressions

Lesson Plan:

• Regular Expressions workshop lesson plan from Library Carpentries

Goals:

• Learn how to use Regular Expressions (regex)

Assignment:

• Practice using regular expressions by completing some Regex Crosswords

Further Resources:

• Regular Expressions workshop videos from Software Carpentries