Why are Boltzmann brains a problem again?

I'm positive I'm about to display an embarrassing level of ignorance, but this objection to us living in a slowly dissolving random fluctuation in an infinite random universe never made much sense to me, and not because I think I am a brain floating in the vacuum of space with all of my memories implanted. Nevertheless, I'll be as careful with my wording as I can.

If I understand it correctly, the objection is there would be far more small fluctuations in an infinite random universe than there are large fluctuations, and so almost all brains in that universe would be aimlessly floating in a vacuum, the chance result of atoms coming together into that configuration. This means that if you do live in an infinite random universe, you would basically have to be a Boltzmann brain, which if nothing else sounds a bit bleak.

I accept that small fluctuations the size of brains are far more numerous than large universe sized fluctuations, but aren't we forgetting that brains in our 'fluctuation' didn't spontaneously form by chance? It was an iterative, non-random process, and you would need a fluctuation large enough to produce stars, planets & complex chemistry for that process to even happen. Once you had that situation, the evolution of brains is basically inevitable. That can't be said about small fluctuations, which could be just about anything (e.g. a dining chair).

It seems to me like this argument is conflating low entropy with complexity. Sure, brains are good examples of low entropy, but how is that being calculated exactly? Aren't crystals also examples of extremely low entropy? If so, that tells me the calculations of entropy ignore how each part of the brain relates to other parts of the brain, as well as the processes involved in the development of a brain or a crystal. In other words, it is far too reductionist to really capture the full complexities involved in what it 'means to be a brain'.

I admit I'm not qualified to form an opinion either way. Maybe low entropy vs complexity is a distinction without a difference here, or I'm playing far too fast & loose with these concepts. Hopefully I'll take the time to read Sean Carroll's paper on the matter when I get the chance, but as it stands my intuition is firmly unsatisfied with the 'Boltzmann brain' argument.

For the record I don't think we do live in a infinite random universe, but I would have thought that, in that scenario, almost all brains would appear within large fluctuations that are at least the size of a single galaxy, formed through a Darwinian process rather than random chance. There's nothing 'obviously wrong' with that idea anyway.