Introduction

I admit I’m late to the Large Language Model (LLM) game, as it’s been fueling the current AI boom for a few years now. Modern AI simply hasn’t held my interest up to now. I have to keep somewhat up to date with AI and LLM developments because of my day job as an IT Director, but after that I had no curiosity left to play with it in my spare time. Part of the reason for that is that I’m quite torn on AI in general. It has immense potential to do good, and yet it is undeniable that it is and will continue fueling a massive amount of societal harm as well (misinformation and deepfakes, automated fraud and scams, unapproved use of others’ intellectual property, devaluing artistic output, contributions to addiction/mental illness, amplifying systemic bias, concentrating more power in the hands of the few, cybersecurity risks, and job market upheaval, to name a few). Some of the characteristics of modern AI that concern me the most are privacy, sustainability and cost.

Privacy for obvious reasons: big tech has a 25+ year track record of monetizing and abusing our data every which way possible and has proven to be completely incapable of stopping themselves from acting unethically. I’ve been on a multi-year slow burn crusade to divorce my data from big tech (the primary reason my homelab exists at all), so that concern is core for me. We should retain control of our own data, period. Feeding it into the giant maws of LLM clouds whose corporate overlords have been very squirrelly about telling us what they do with the data we input is definitely not the right direction to be going in terms of data ownership.

As for sustainability, the energy consumption of modern AI is on a completely untenable path. By some estimates it’s now responsible for up to 2% of the world’s consumption. That may not sound like a lot, but that represents a 10x increase in five years, and we’re still in the early years. I fully expect efficiency gains in hardware and software to slow this trend somewhat, but the problem is that our energy infrastructure has basically no room to make space for AI without ripple effects elsewhere. If we’re going to invest in the energy grid, I’d really like it to go towards things like improved reliability, supporting green energy generation, helping the transition to electric vehicles - not appeasing the insatiable energy appetites of AI megacorporations.

We’re already seeing big tech acting in ways that are not in the best interests of society in general to try to secure the needed energy. They are buying up land next to power plants, plugging directly into the feed and getting served first, before the utility’s regular customers. Many communities are all too eager to invite these companies to build datacenters in their city and are offering immense tax breaks on top of prime real estate at a fraction of its actual value, despite the fact that after construction ends, there are relatively few permanent jobs created. Waht’s the benefit for the folks in that community? These big tech corporations are negotiating sweetheart deals that preference them over other power customers, and who gets hurt in the end? You and me, the regular consumer. And these datacenters are resource monsters in other ways too - they consume a ton more electricity to cool the facility, millions of gallons of water per day that receives heat transfer and then gets expelled back out into the environment, affecting the surrounding ecosystem. Prime real estate sucked up by sparsely populated buildings generating little tax base. The list goes on. Perhaps worst of all, some utilities have delayed or canceled planned shutdowns of fossil fuel plants to feed the AI energy demand!

Finally, there’s cost. The brief honeymoon period where anyone with an internet connection could access massively powerful cloud-based LLMs for free or cheap monthly subscriptions is coming to a rapid end. Prices are going up, features are being moved to more expensive tiers, token limits are getting stricter - there’s a whole culture around how to squeeze every last bit of token efficiency via clever prompt authoring. I don’t know about you, but that’s not what I dreamed of spending my time on when I went into this field. I want AI to feel like a helpful tool, not a new source of friction and constraint. Using cloud LLMs is going to be a growing cost going forward and it’s going to price out a lot of folks, segregating us into the haves and have-nots. And the cost issue is not confined to just using LLMs. Have you tried to buy some RAM lately? Pick up a new or even used Mac Mini (mmm, that unified RAM is oh-so-attractive for AI)? Upgrade your GPU? You get my point, AI is overpowering the entire hardware industry right now to meet its insatiable demand.

Put this all together, and you can maybe see why I wasn’t too motivated to really dig into the LLM scene. However, over the last several months what finally forced the issue is that as AI tools get more and more ingrained into the software developer experience, I really couldn’t afford to tiptoe around this space any longer. I manage a team of software developers, and deep LLM and AI tool proficiency is essential at this point. We all need to get better and more experienced at navigating this space, and so we’re diving head first into it at work now. And that finally got me thinking more about how LLMs might be useful in my homelab as well.

My interest quickly went to wondering how I could do this on my terms as much as possible, given that I still have all of the concerns I just mentioned.

Local LLMs - Are They the Answer?

Anyone with a homelab by definition wants to “do it myself!”, so I immediately set out to figure out how viable it would be to add LLMs to my homelab with no cloud interaction. Obviously it COULD be done; people have been doing it for years - as I said, I’m late to the game. My question was more about whether I would find the experience acceptable in terms of both my high expectations for how I want my tools to work for me and my family, and in accordance with my values and the concerns I have.

For some of my concerns, local LLMs are a clear win. Certainly for privacy it’s infinitely preferable to any tool that requires me to upload data to the cloud. For some things in my homelab this means processing medical and financial data, so that’s a hard no for leaving my local environment. It’s also potentially a winner on the cost front, depending on whether I need to invest in new hardware or can use my existing assets. I could easily spend as much money on new hardware as I would on monthly subscriptions. On the other hand, there are no token limits with local LLMs. And cost isn’t the primary driver - both privacy and sustainability would win over cost, meaning I’m willing to spend more if it gets me those other things. So, all in all, the jury is out on cost.

It’s definitely more of a mixed bag for sustainability. In general, datacenters benefit from economy of scale. If we all stopped using cloud LLMs and started using local LLMs, that’s maybe not a net-win for the planet. Maybe we’re all now leaving our computers on 24/7 instead of letting them sleep, so that our AI remains responsive. Maybe distributing that electrical demand across the entire grid is harder to service than having it concentrated in advantageous locations. Lots of ways that local LLMs are or could be less sustainable at scale than cloud LLMs. On the other hand, my house is powered by a solar garden, so my local LLMs consume clean(er) energy. I do turn off my AI systems at certain times of the day or when they are not needed for their particular task, and a lot of my AI footprint is running on systems that are running anyway for other purposes, so the incremental add is pretty small. I’m also not asking a full-scale hundreds-of-billions-of-parameters cloud model questions that can be answered effectively by a 10b parameter local model at a much lower cost.

Finally, will it deliver an acceptable experience? Certainly local LLMs are far more limited in capability than modern cloud-based commercial LLMs. But do I need that level of capability, or could I meet my goals effectively within the limits of a homelab? There’s really only one way to find out - I’d need to experiment and get some real data.

Local LLMs are definitely not perfect, and let’s also acknowledge that there are use cases of AI that are currently difficult or impossible to replicate at local scale, so it’s definitely not a panacea. Having said that, they have enough advantages over cloud LLMs to be worth further exploration.

What this Blog Series Will Cover, and Why

This series will be a running journal of my experiments with local LLMs. I’ll share not just what I’m building technically, but also what I’m learning conceptually as I go. And I’ll keep coming back to my core concerns with AI and assess how my local LLMs are addressing them.

I’ve got many potential use cases to explore, from chatbots to document processing, home automation to coding assistance. I hope to discover which are feasible and which will be difficult to satisfy locally. I may even get into building some of my own tools.

Why? I’m well aware that this is not a high-traffic blog. :-) My writing here is first and foremost for me - it helps me solidify what I am learning, and I enjoy the process of writing. Of course I do hope that I can help others in the process - otherwise, why put this online at all? I suspect there are a lot of others like me, reluctantly beginning to engage more deeply with LLMs as they wrestle with the larger, more general concerns of modern AI and not wanting to feel like they’re contributing to the problem. We can’t ignore AI, it’s going to be everywhere in our lives if it isn’t already. We’ll all have to come to terms with how we will choose to engage with it. Maybe what I’m doing will help others to keep going with their own process. I don’t really know where this will take me, only that I finally have the curiosity and motivation to explore and I want to capitalize on that. I’m sure I’ll learn something interesting, no matter the outcome!