The Generalist's Moment

I love the idea of the Generalist - especially as described by David Epstein in his excellent book "Range - Why Generalists Succeed in a Specialist World". I like to think of myself as a generalist. I love trivia, I love problem solving, and I love borrowing ideas from fields where I am not an expert, and applying it to problems in my world. And I think this is actually the huge unlock that generalists have with AI today. AI is an incredible technology, it knows and "understands" concepts and topics at very deep level, likely getting into PhD territory, especially in the sciences. AI models are incredible specialists. They also are incredible at finding patterns in data, and relating ideas to others. What else is true, (at least right now) is they need YOU to ask the questions. So in my view of the world, broadly split into two types of people. 1. the Specialist and 2. The Generalist, i think the Generalist has an unfair advantage when using AI systems. The specialist has a narrow focus, they love to go deep on problems, that's great, and a specialist using an AI can unlock incredible depth of understanding. We do not have great data showing that contemporary AI models can unlock new / novel science though, so there is a limit to how "deep" a specialist with AI can go. If anything the rate of advancement is likely limited by the real world constraints of experiments, of real work - pairing human and AI to be able to posit new ideas, and then work to prove them. Now I do believe this is coming, and it will be truly transformational, but I think this pace will be slower by default.

Where i think the true unlock of today is, is unleashing the Generalist. The Generalist is the Jack of All Trades, the sampler, the dabbler. They still have domain expertise, they still need to know enough to solve problems, but they have a more curious mind, looking for ways to bring together disparate ideas. And this is where today's AI's are absolute beasts. They allow a generalist to pull at threads of tangential or orthogonal ideas, finding problems with similar shapes but different domains, and then use AI to tie it together. The generalist can now go as deep as they need with AI's help to synthesize new ideas, by combining old. This bypasses the constraint of needing to create a new concept, and then prove it in the real world. This borrows proven concepts from other fields and helps you drill down with them. If a couple of years ago, a Generalist was limited by his depth of knowledge across multiple domains, it was quite likely that they were looking for answers to a problem they had, in domains they had a top level understanding of. This was still very useful, but today, the generalist with ai, can find these relationships much more easily. Instead of looking for related ideas at the top level, you can find related ideas at a deeper level.

As a simple example, a current problem I'm working on is how to use remote sensing techniques to help assess / monitor vegetation health - particularly on reclaimed oil and gas leases on annual/perennial cropland. Now I have a decade of field experience at looking at these sites, understanding the benefits, limitations and constraints of doing this the traditional way. But as I started learning about multispectral imaging with drones and satellites, I could see obvious improvements and new opportunities. I could see the density of data available, and how the power of statistics could start to help understand the variation of fields and crop health. Of comparing one area to another, but I didn't know the statistical frameworks for what I wanted to do.

I started doing this in earnest in 2021, with google searches. Like many people, I don't have much of a background in statistics. I took two stats classes in university, I did not enjoy them, I did not do well in them. They were abstract ideas that I had no application for. And I'll be blunt, the subject of "statistics" is huge. Its enormous. and it's nuanced. My google searchers were quite fruitless, because I couldn't find examples that fit to my niche, and I didn't know enough about stats, to search for equivalent ideas in adjacent fields of work. But with the advent of ChatGPT, i began to describe the problem I was having, and what I was trying to do. I was having a hard time figuring out how to compare the vegetation health trends of a field over two years, because the crop type was different (canola one year, wheat the next), also one year was drier than the other, so the entire field was less healthy.

In my head i knew that it was not important to know what the NDVI values were for each pixel, what I cared about was which areas were better or worse than the rest of the field. Which areas were better or worse than the average? What if I could do that in terms of standard deviation, because on a somewhat normal distribution like NDVI, I knew that standard deviation would be a pretty good way of determining how "important" a change was. Well, with some iteration and back and forth, i figured out that this is a very common and very simple statistical technique called z-scores. I just needed to do a z-score transformation of the NDVI raster, and i would have a way to look at crop / vegetation health data which was data defined, comparable over time, and accounted for crop type, temp and precip changes etc.

I'm guessing that, depending on who is reading this, you're either thinking, what the hell is an NDVI raster, what the hell is a z-score, or who doesn't know that z-scores are the easiest way to standardize data. And that is exactly my point. If you're standing outside of these domains, it sounds like you're reading Greek. If you're standing inside these domains, what i'm describing as a breakthrough is actually what first year students are taught. But together these two ideas can combine and solve a real problem that I have which is worth solving. And now with current AI reasoning models, this generalist skill is supercharged even more. What a gorydamn great time to be alive.

The last point that i want to try get across today is that many ideas sound good at a high level, but can be hard to see if the actually work. The ideation can be the easiest part of the process. And for years I was full of bright ideas that piled up in notebooks and sit there to this day. So what I think is perhaps even a bigger unlock, is being a generalist, and letting AI unlock coding for you. At this moment in time, AI is amazingly capable in many subjects, but is arguably the best at coding. Coding can be tested, software is written to do A THING on a computer, and then it can be ran to see if THE THING worked. If it did THE THING, the code = worked, if it did not do THE THING, the code != worked. AI can propose a soluiton, run it, see if it worked or not, and iterate until THE THING worked. Do not take this super power for granted. Software applies to so many domains - it can super charge you because it can help you bring your ideas into the world and see if they work. My z-score example from above, it sounds great - but instead of jotting it down in my notebook, I just told ChatGPT to explain the process to me, to write some code to do a z-score transformation on an NDVI raster. And then load it in QGIS and see if the results match my interpretation and my field experience.

The software meets domain expert is such a good example, that I couldn't stop thinking of this Paul Graham essay I'd read. Paul Graham is the co-founder of Y Combinator — a Silicon Valley startup incubator — and in the essay "How to Get Startup Ideas", he brings up a lot of great points, but he makes them from the perspective of Silicon Valley, where software engineers and computer science grads are looking to try come up with new ideas. One key idea is that if you can "hack", if you can write code, you can go from

Merely thinking 'that's an interesting idea' to instead thinking 'that's an interesting idea. I'll try building an initial version tonight.

This is essentially what I was trying to say above, just in a much more concise sentence. But I think the next paragraph in his essay is even more poignant, especially if we flip the perspective. Paul says: >

The clash of domains is a particularly fruitful source of ideas. If you know a lot about programming and you start learning some other field, you'll probably see problems that software could solve. In fact, you're doubly likely to find good problems in another domain: (a) the inhabitants of that domain are not as likely as software people to have already solved their problems with software, and (b) since you come into the new domain totally ignorant you don't even know what the status quo is to take it for granted.

But let's take that idea and flip it on its head. Software used to require incredible domain expertise in order to leverage it, so much so that Paul suggests going out to dabble in the world to find problems that software can solve. But the situation we are facing is now reversed. AI can write code very, very well. So as domain experts, the best thing you can do is try to figure out what problems you have in your niche that can be solved with software, but never have been — because it's too niche, or because people with software experience didn't even know there was a problem worth solving. This is the future. There are huge productivity, efficiency, and new domain unlocks that will be unleashed because experts can now write their own code. If you are reading this, I encourage you to use your AI model of choice to try to give shape to the problems you're trying to solve. Use AI to try to crystallize those problems into things that can be solved by adjacent domains. Play with ideas. Play with AI. Get AI to write your first piece of software. Get Claude ([claude.ai](https://claude.ai)) to create you an artifact. Go to [replit.com](https://replit.com) and ask the Replit Agent to build you something. Pick something simple and see what it can do. Keep pushing the bar, and I think you'll be surprised how easy it is to be a Generalist, and how amazing it feels to bring an idea to life with software. You Can Just Do Things.

Jesse Lawrence