I'm struggling to know how it's possible to really block something like a profiler on things like public fedi profiles.

It's public so it really doesn't matter. All the above would do is say that my page is visitable by search engines which abide by rules in the 'robots.txt' file. Its likely that many won't or won't even declare that they are search engines/bots etc.

The problem is I don't have a big online presence like you might think, so things like this would help a lot.