I feel like we're going though an evolution of the web, and no matter what we do, it's going to happen.
The web is going to change (for better or worse) or die, and there's nothing we can do about it.
The web killed printed media to a large degree and AI will do the same, Resistance is futile!
You may be right that resistance is futile. But I don't think adaptation is. And there's no reason our adaptation can't be rooted in resistance.
For example, maybe smaller local forums make a comeback and their communities decide to hide all threads behind auth. (I don't necessarily see that happening, just an example.)
And honestly given the stronger feelings people are developing against what I'll just call "creative use of generative AI", I'm starting to think maybe resistance isn't futile... Poisoning original digital art so it's less useful for image generators...social shaming of AI generated music being laundered on platforms...those things do feel like meaningful steps towards resistance.
Without authors being paid to create new content AI is nothing/it's irrelevant!
Either some startup needs to come up with compensation for authors or the big players need to set up a system that still gets authors paid as Im guessing in five to ten years we are not visiting websites. Our soon to be AI friend (Facetime the "friend," or just talk or text it) seen on our lockscreens or in a hologram is visiting all sites to create visuals of the info and displaying/discussing it with us immediately upon request.
To be honest, is it really that bad that the web is dead? I understand the value of a forum, but as far as the content is concerned if we were to go back to the days of physical magazines I wouldn't be upset.
It is not the medium that is dying, rather it is the content that is dying. People are saying the web is dying because there's no longer financial incentive for actual humans to create content (there may be other incentives, and those types of forms may flourish). That goes equally for other media as well: print, digital, it doesn't matter.
> Nothing that I publish here has come from AI or answer engines. Every word that is written comes from this human.
I think we're rapidly approaching the point where no one will be able to make this claim anymore. AI summaries and answers are ubiquitous and our knowledge or beliefs are directly or indirectly informed by them. We can avoid 1st order AI use, but it is impossible to avoid 2nd order and further exposure.
The water supply has been poisoned and everyone needs to drink.
Set your site to 401 unauthorized with a basic challenge if an auth header isn’t sent, and set the auth description to “Enter anything to proceed. All human access is authorized. Unauthorized non-human access is prohibited.”. Crawlers can’t parse the instructions and will deadstop on them, while people will shrug and enter any password, which will work.
Anubis is also viable and popular, but it lacks the legal threat to AI of being able to file a federal hacking claim against a scraper’s unauthorized intrusion if they code their scraper to transmit an empty/invalid/valid authentication header.
What decade do you think it is? :) Depending on who you ask, captcha bots have become better at solving them than humans...
There's almost nothing you can do that "AI" can't while keeping it easy enough for your average joe that wants to login. Especially considering "the grandma test"...
Very feasible solution I guess the only issue is that we now have to add friction for legitimate users which will only accelerate their migration to AI summaries at the top of the page.
Haven’t for a while. There’s a bunch of open source projects that provide a Captcha and Cloudflare bypass proxy where you just point your scraping through the proxy and it takes care of the challenges. It’s rather trivial to handle nowadays.
A bunch of the torrent trackers are now behind Cloudflare so the pirate community has been maintaining many of these projects in order to enable their autodownloaders like Sonarr, Radarr, Lidarr, etc.
>Note the sponsor ads on the right confirming this theft was performed for profit, and in flagrant breach of copyright. Yet Google’s answers are perfectly comfortable using stolen content and even providing links to it.
cry me a river, boo hoo i have to visit your ad ridden website to read something and you're mad that google is fixing that somewhat
I feel like we're going though an evolution of the web, and no matter what we do, it's going to happen. The web is going to change (for better or worse) or die, and there's nothing we can do about it. The web killed printed media to a large degree and AI will do the same, Resistance is futile!
You may be right that resistance is futile. But I don't think adaptation is. And there's no reason our adaptation can't be rooted in resistance.
For example, maybe smaller local forums make a comeback and their communities decide to hide all threads behind auth. (I don't necessarily see that happening, just an example.)
And honestly given the stronger feelings people are developing against what I'll just call "creative use of generative AI", I'm starting to think maybe resistance isn't futile... Poisoning original digital art so it's less useful for image generators...social shaming of AI generated music being laundered on platforms...those things do feel like meaningful steps towards resistance.
[delayed]
Without authors being paid to create new content AI is nothing/it's irrelevant!
Either some startup needs to come up with compensation for authors or the big players need to set up a system that still gets authors paid as Im guessing in five to ten years we are not visiting websites. Our soon to be AI friend (Facetime the "friend," or just talk or text it) seen on our lockscreens or in a hologram is visiting all sites to create visuals of the info and displaying/discussing it with us immediately upon request.
To be honest, is it really that bad that the web is dead? I understand the value of a forum, but as far as the content is concerned if we were to go back to the days of physical magazines I wouldn't be upset.
It is not the medium that is dying, rather it is the content that is dying. People are saying the web is dying because there's no longer financial incentive for actual humans to create content (there may be other incentives, and those types of forms may flourish). That goes equally for other media as well: print, digital, it doesn't matter.
There's an incentive to create content, but it's skewed towards video (or towards creating content of any sort when you already have a fan base).
Trying to create a fan base with the written word in 2025 is probably a bad idea.
Only that there is no way back. Those physical magazines would contain the same AI generated content.
> Nothing that I publish here has come from AI or answer engines. Every word that is written comes from this human.
I think we're rapidly approaching the point where no one will be able to make this claim anymore. AI summaries and answers are ubiquitous and our knowledge or beliefs are directly or indirectly informed by them. We can avoid 1st order AI use, but it is impossible to avoid 2nd order and further exposure.
The water supply has been poisoned and everyone needs to drink.
I do agree that no AI content is primordial and I’d personally accompany it with a “no AI crawlers allowed”.
Feeding AI with new high quality content only add to the problem. We must stop feeding it.
Set your site to 401 unauthorized with a basic challenge if an auth header isn’t sent, and set the auth description to “Enter anything to proceed. All human access is authorized. Unauthorized non-human access is prohibited.”. Crawlers can’t parse the instructions and will deadstop on them, while people will shrug and enter any password, which will work.
Anubis is also viable and popular, but it lacks the legal threat to AI of being able to file a federal hacking claim against a scraper’s unauthorized intrusion if they code their scraper to transmit an empty/invalid/valid authentication header.
> Crawlers can’t parse the instructions
What decade do you think it is? :) Depending on who you ask, captcha bots have become better at solving them than humans...
There's almost nothing you can do that "AI" can't while keeping it easy enough for your average joe that wants to login. Especially considering "the grandma test"...
“while people will shrug and enter any password, which will work.”
I think you might be overestiating how much patience humans have when browsing a site
Can you say more about why a crawler couldn't parse an auth deacription and also what is an auth description?
> what is an auth description?
They are probably referring to the text in the basic auth "pop-up" which is usually set like
WWW-Authenticate: Basic realm="my text"
Very feasible solution I guess the only issue is that we now have to add friction for legitimate users which will only accelerate their migration to AI summaries at the top of the page.
[dead]
So... a reasonably stupid question - captchas and bot protection (e.g. cloudflare stuff) do not deter bots anymore?
Haven’t for a while. There’s a bunch of open source projects that provide a Captcha and Cloudflare bypass proxy where you just point your scraping through the proxy and it takes care of the challenges. It’s rather trivial to handle nowadays.
A bunch of the torrent trackers are now behind Cloudflare so the pirate community has been maintaining many of these projects in order to enable their autodownloaders like Sonarr, Radarr, Lidarr, etc.
They're more of a speed bump. Deter drive-by scraping and whatever they call script kiddies these days.
>Note the sponsor ads on the right confirming this theft was performed for profit, and in flagrant breach of copyright. Yet Google’s answers are perfectly comfortable using stolen content and even providing links to it.
cry me a river, boo hoo i have to visit your ad ridden website to read something and you're mad that google is fixing that somewhat
What ads?
So AI writers will now include an owl emoji to indicate that they are human writers.
This is so useless