To play the devil's advocate. If you were running a large public forum, and you knew that many companies had started to scrape all data off your site, and were going to cumulatively make billions off that data, and some of those billions will come from polluting your forum with crap content, would you continue running your site in the open?
What is the game theory here? Twitter cooperates and OpenAI defects, and we call that a win?
Alternatively, build a private invite only dataset of specific communities. Scale horizontally instead of having one single central dataset! That's still a huge win for user ownership.
It doesn't have to be one company controlling who gets access, the users can decide this
The problem is that the users and their capital are not organized enough to suggest or demand for this. On the other side of the table we have the "establishment" which may not be formally organized but has enough shared incentives among itself where the outcomes aren't any different as if it were indeed formally organized and pooling resources.
In this sense we have this intractable push and pull going on with just about any community of sufficient size. We have the users who might want or should want some privacy and respect and other such benefits, and then we have the people who actually invest and build these platforms, who are incentivized to deliver other things than what the users best interest might be. Either you empower users to have more money to be able to roll their own solutions, or you try and set up a world where the incentives of capital perfectly match the needs of the individual or collective of individuals, which is probably impossible to do.
It's tragic that the LLM craze OpenAI kicked off is threatening to ruin one of the greatest common goods ever invented in the open internet. But hey, at least a handful of giant corporations and investors are making money, so I guess that counts as a win.
Practically speaking, 'some' of the money is the same or just as good as 'all' of the money in terms of a functioning economy and not pure capitalism.
People pirate software, yet people still develop and sell it.
Yes, it's not the most profitable thing done alone. Only for those who find the nice combination or feedback loop of 'fit', demand, improvement, and expansion.
If you can make billions off the thing, you can presumably handle some GeoIP/rate limiting... or simply, not caring. Anything that falls through the cracks is categorically insignificant to your grand nature.
To justify it, if one must, consider it a trial. As your friendly neighborhood dealer would say: "The first taste is free".
Fact Check: Reddit didn't hike API prices by an absurd amount.
That was was the story spread far and wide by the Apollo app developer that was believed by the gullible and angry Reddit masses.
But Reddit actually set a reasonable API price, as evidenced by the fact that a year and a half later there are still five 3rd party apps running on reasonable subscriptions:
1. Experts can comment, but I think the value of multi person conversational data from forums is uniquely valuable and in short supply relative to just blogposts/news stories on the internet.
2. The absolute economic value of the entire corpus of Twitter is much more valuable than any single boatforum.com like website. So Twitter has a much more incentive to lock itself down than boatforum.com.
Twitter was "special" because they actually had an API.
No other social media site (besides Reddit) had one. Facebook kind of tried, although theirs was always a lot more limited, and got shut down when it turned out Cambridge Analytica used it for widespread election fraud[1].
Both Twitter and Reddit did basically the same thing at roughly the same time, Twitter just had the misfortune to be under the control of Elon Musk, so the move was perceived as ideological.
[1] As it later turned out, Cambridge Analytica was basically a "nothing burger", they claimed to be able to accomplish a lot while, in reality, accomplishing very little. The damage to interoperability was already done, though.
Indeed. CA's impact on election outcomes was likely negligible; Antonio Martínez's "Chaos Monkeys" makes the case that Trump using CA was more indicative of the overall notion that Trump's team was spending money to try a bit of everything (online and offline) and the Clinton campaign wasn't. Their tactical failure was believing they could redirect the money to down-ticket races because Clinton / Trump was such an obvious matchup that they didn't need to spend to win.
What CA did show was that Facebook's statements about protecting user privacy were fundamentally incompatible with the way their API worked, so they had to shut it down because the alternative would have been to just sort of... Let it hang in the air that it wasn't hard for a third-party to build a system to completely bypass user intent in scoping their information.
(I had the misfortune of trying to write a Facebook app about fifteen years prior, and that was my takeaway at the time also... "Do people, like, realize that their whole process for protecting scraping the social network via third-party app integration is the honor system?" Turns out people didn't).
What Mastodon is doing seems suitably ironic in this situation.
For those unaware, Mastodon's APIs are extremely open and it's very easy to scrape, to the point of providing you with a "firehose" of all public posts that an instance sees, both local and federated, with no authentication required. They also have an extreme anti-scraping culture, anybody who admits to running any kind of scraper which is not strictly opt-in, even for benign / scientific purposes, is very quickly shunned and blocked. Most instances also have a "disallow scraping via robots.txt" policy by default.
The results? I posted a canary token[1] link on a medium-sized, well-federated, well-protected instance which disallows scraping, and it got hit by some shady social media crawler in a fraction of a second. It started getting hit by many other strange crawlers later on, and it still keeps getting visits (mostly from Google now).
It's an ecosystem of people that seem insistent on the idea that you can put stuff online behind no authentication wall and expect to stay exclusively on servers you believe it should be on.
And I don't know what to tell them, because that's not how the internet has ever worked.
Here's a thought: someone "trustworthy" should maintain a Chrome extension or Tapermonkey script that automatically scrapes data from various social media sites in a fully anonymized fashion. As people browse Twitter, Reddit, or XYZ, the posts/comments are sent to some aggregation system. It might be against TOS, but certainly far less than scraping, and you couldn't tell, as it's the user driving what gets scraped.
I don't use Twitter often, but I'd run something like that if there were strong anonymity guarantees. Seems like a win-win for everyone.
I was thinking of that exact strategy for two scraping problems I have. It would be a good way to gently scrape Berlin.de for appointments, and Immobilienscout24 for new flats.
From what I can tell, it would work fine so long as the user is actively looking at those pages.
Yeah, something generic to work for any use case would be nice, but privacy becomes more difficult as you need to tailor the situation to each site to maintain privacy (i.e. only pull the information on the apartment/flat listing, or public tweet or reddit comment, etc...)
To me, Twitter is by far the most toxic social network. It has the power to turn the most interesting and smartest people into bitter trolls. Maybe some people get value from it, but it requires serious self discipline to not get dragged into stupid arguments.
I've tried to "get" Twitter since the early days. I've created a couple of Twitter accounts over the years to support various creative endeavours but I've never found myself getting much value out of it as a user.
I heard that it's original use-case was that back 2008 there were no good ways to do group messaging over SMS on a cell phone. That's a problem and solution that I can understand.
But as a broader social network? I don't get it.
Things that bother me to the point that they are deal breakers:
- The character limits (there is nothing worse in life than reading through a x/N self-reply to read something long ... I'd rather file a tax return)
- Showing me posts from pages I don't follow in the feed (in 99% of the cases I'm aware of that page/profile already and have chosen not to follow it because I don't like them)
- Ads in the news feed
Some have said that they like Twitter for getting news. I have way better options for that.
I know that I'm not representative of the typical person, generally speaking, but Twitter is one of those things where I really can't understand why anyone likes it and uses it, let alone why it is so popular. And it's not that I'm hating on something I don't know anything about ... I've honestly tried to use it and get value out of it, but I've never found anything of value on offer.
> but Twitter is one of those things where I really can't understand why anyone likes it and uses it, let alone why it is so popular.
I think it's a carryover of general nostalgia for how the Internet "used to be", i.e. before every site with a few million users decided to start experimenting with algorithms to max out user engagement and ad impressions, leading to the hell we see everywhere, from IG to Tiktok.
Yes, there was a time when Twitter wasn't a toxic mess. It's the last social network that was popular before smartphones took off.
Interesting. That might explain why I dislike it so much. As a middle aged person who grew up with an early iteration of the world wide web for a decade to a decade and a half before Twitter even existed, both smart phones and social media mark a turning point for me from what the Internet "used to be."
Twitter, in my mind (and maybe this is perception and not reality), ushered in infinite scroll and short bites of information. Twitter is to forums what TikTok is to documentaries. I see Twitter and the "mobile revolution" going hand in hand (something that left me behind because I still dislike using a smart phone, generally, and rarely do compared to most other people).
But I guess if you're a great deal younger than me, and you grew up with an Internet where Twitter just always existed, then it might represent some earlier version of the Internet that is drastically different from what you consider to be "contemporary" (though, putting the TikTok mention aside, I'm still not sure what that view of the contemporary Internet is if Twitter is what we're comparing it to).
I guess I'm just so old that I still see Twitter as a relatively new phenomenon. Very different from the nostalgia that I feel for what the world wide web used to be when I was young.
It's gotten dramatically worse since premium started rewarding payouts for engagement/views, alongside zero moderation. Now there's an actual incentive to be a jerk, a spammer, and make repeated bad faith arguments / "just asking questions" -- people reply, your tweet is seen by more people, you get paid a share of ad revenue.
Then you have the regurgitated messages in replies (AI or otherwise) endlessly copied and distorted. Recently it's been 5+ pages until I see a real reply, while I have 3000+ accounts blocked by now. It's getting harder and harder to find the real person who wrote that witty reply, and not the endless blue check accounts who copy-pasted it to farm engagement on their visibility-boosted tweets.
Tangential but there's my disappointment to AI in there: they haven't found a way to build convincingly human bots, let alone a procedural content creation system.
Both AI believers and doomers insist generative AI could surpass humans by every single metric that matters in social media. All we've got is first Nigerian, then Indian, and now increasingly Pakistani spammers desperately reply bombing trending posts made by 10 years alpha users, as if whoever behind it completely failed to do it and has been wasting gullible human spammer candidates faster than Aperture Science.
> they haven't found a way to build convincingly human bots, let alone a procedural content creation system.
The 'they' being the corporations that hired thousands of people from the very countries you're associating with spammers , to review, correct and train AI models?
I have a Mastodon account, a Bluesky account, and apps for both. Mastodon doesn’t replace the comedy part of Twitter, and Bluesky is great but 99% of local social media is still on Twitter. So I’m “trapped” on Twitter until they move to Bluesky, but they won’t because a part of that crowd is addicted to the revenue share off their political tweets.
Not parent, but I did use Mastodon for about a year. I ended up moving back to Twitter because:
1. I just couldn't vibe with the culture there. From my POV, Mastodon is made out of pearl-clutchers and politics.
2. So much drama. The FediSearch drama. The Raspberry Pi incident. It's just so tiring and you feel like you need to walk on eggshells all the time.
3. So much drama. You would just pray that your admin didn't get into a spat with another admin and get defederated. You could get a solo server, but that costs money and you might get blocked by a large server's admin anyway.
4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere. Solo, yes, but costs money.
At least with Twitter, the rules are sort of well known, and you can follow anyone there unless they block you personally.
I heard BlueSky is good, though. Haven't tried it yet. Nostr was also another one, to get around the admin drama issue, but it doesn't seem very popular.
>4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere.
I would have thought a well-designed decentralized system would allow you to, for instance, download/export all your posts, and then import them all to a new server, in case you have to move servers for some reason.
The server-oriented aspect of Mastodon was the second-worst decision ever (the first one was the name). Mainstream social media doesn't ask you to pick a server (nor deal with the consequences of it going down), its replacement shouldn't either.
As for the server-oriented aspect, what's the alternative? There's only two ways I see to run a social media service like this: centralized and decentralized.
The mainstream ones are centralized, of course: FB, Xitter, etc. You just connect to their web server and do everything there. Any centralized alternative would have be a well-financed corporation able to set up a huge IT infrastructure to accomplish the same task. But the whole idea of Mastodon is to not be a big, for-profit corporation, and to be in the hands of the users instead, which I don't think is possible with a centralized service.
So the only alternative is a decentralized service. But this means you have to have multiple servers, leading to the whole "federated" approach, so users can run their own servers and control things themselves, or use accounts on servers of their choice. This of course leads to all the consequences you named, and more: servers going down, servers defederating other servers, etc.
But again, what's the alternative? If the requirement is "not run by a single big for-profit company", I don't see one. If you think that having a decentralized service actually be competitive with a centralized one like Xitter simply isn't feasible, that's fair.
My prior complaint still stands though: if you want users to have freedom (of which server to put their account on, etc.), then why would you not have a way of migrating their account/data to another server, and force them to lose everything if their server disappears?
This obviously varies person to person, but I don't find it to be that. I carefully maintain who I am following (about 600 people and companies, mainly in the infosec / natsec space) and when I start to see bad behavior/content I'm quick to unfollow people. Also, I only use the Following tab, never the For You tab.
I never did much with Twitter until I took over the Carolina Code Conference, but it's been by far the easiest place to engage with different tech communities. Just the "Follow All" button that appears after I follow something like Gleam or Roc is hugely helpful.
I probably spend more time on Twitter now than I ever did before.
> It has the power to turn the most interesting and smartest people into bitter trolls.
There is tremendous value in having access to large quantities of data demonstrating the various failure modes of relatively intelligent minds. Nothing is being (publicly) done with this data so far, but this will not last forever.
This is something that has always bothered me about the /r/SSC community (and others): the "no culture war topics" constraint + moderators locking threads when the Rationalists start collectively behaving irrationally.
A lot of users/communities are stuck because of network effects & the long history of posts (they still reference them, like living wikis)
These pipeline tools can help people migrate if they want without losing that history & network (my prediction is once people see what's possible with an open API, that'll further motivate user migration, or for twitter to open up again)
> the long history of posts (they still reference them, like living wikis)
As someone with a multitude of cat photo threads, 100x this.
While I'm around on Twitter, Bluesky and Mastodon, the "post history" aspect is my most pressing issue with Mastodon. Like, when you move your account, you cannot take your old posts to the new server - you're straight out of luck if your instance decides to call it quits for whatever reason. And even if the instance doesn't give up, being unable to view old(er) photos is something I very commonly notice on a bunch of Mastodon servers.
People should do a lot of things. They shouldn't smoke and they should work out, but here we are in 2024 where the phillip morris stock has outperformed ibm over the last 5 years and the obesity crisis shows no signs of letting up. Knowing something is bad is the first step of course but clearly not enough to drive a real behavioral change, and we aren't even fully in agreement as a society that social media is harmful like how cigarettes or obesity are known to be harmful.
>we aren't even fully in agreement as a society that social media is harmful like how cigarettes or obesity are known to be harmful.
There's more than one society on Earth, so it's hard to say that "society" "knows" that smoking is bad without specifying the society. As an American who lives outside the US, one striking place where the US clearly outperforms peer nations, in my daily experience, is with smoking: Americans just don't smoke any more unless they're poor and uneducated. There's no smoking inside (except bars), there's no smoking within XX feet of business entrances, etc. In the US, smokers have really become pariahs since the 80s and early 90s. Some of that has been undone with the rise of "vaping", but still, if you hang out with college-educated people, you probably won't meet any smokers. It's just not like that in other countries: people of all social classes still smoke. Here in Japan, it's quite common to see some salaryman riding a bicycle while puffing on a cigarette, and many restaurants have smoking areas (usually sealed off with separate ventilation in modern buildings). There's even some old locals-only eateries that have (non-separate) smoking inside. Europe isn't any better.
As for obesity, only truly delusional Americans try to claim that "fat is healthy". All the rest know it isn't, even if they're obese, but the society makes it very hard to not be overweight, mainly because everyone drives cars and doesn't walk.
Social media used to be about content sharing, but it's clear that it's far more profitable to keep your users on your site at all times and keep the content they create in house rather than linked somewhere else.
HN is probably the last true aggregator site left that I know of.
It's partly why the web is so much worse. There's really no reason to create content outside of a walled platform is it is getting increasingly difficult to find an audience for it. Even blogging is an uphill battle since more and more social media sites penalize link sharing (they want you to create the content on their platform and leave it there).
That's why it's not surprising that these APIs are disappearing since the fundamental model has changed.
But that's too sensible and user-friendly. It would be obviously weird and wrong if they inserted ads and Musk's tweets into someone's RSS feed. Yet showing those unwanted insertions is the whole purpose of the company now.
I know. Because it wasn’t compatible with their aim of showing ads.
And nothing has changed after Twitter went private under Musk, except that the reasons for stuffing unwanted content into users’ timelines are slightly different.
I don't think that's fair - getting back the chronological timeline itself made the purchase a good one in my eyes, alongside the For You feed, which seems to work wonderfully (for me). Not to forget you can literally turn off ads if you want. I seem to be a real minority, but I do genuinely believe Musk has done a great job with Twitter (or X).
You could also for a long time send SMS (if you registered your number) to 40404 to tweet to Twitter. It's still in my address book from the era when I was tweeting T9 from dumb phones or using voice-to-SMS to tweet on early smart phones.
It has been difficult to rescue data from Twitter even before purchase. On our case it was relevant because this is online digital history for the people in my country.
The only thing we can is motivate more people to use open platforms like NOSTR where API or data/identity handling is completely different.
Funny, just now I've been playing around with the various tweet deleters and trying to get something working; presently I think I'm about to settle on something involving a basic screen macro recorder thing, like one of the iterations of AHK.
I'm somewhat surprised that this space feels relatively dormant compared to the more complex stuff out there.
Twitter was originally a microblogging service that had rss feeds to syndicate things or monitor the microblogs of people/companies that were interesting, it has gone way far off into the fields.
Same journey reddit is making, starting after it prepared to go public.
The only useful thing on Twitter that I ever saw was the lovely and tender Dog Rates https://x.com/dog_rates. You could read anonymously and be all aww and schucks about all those good dogs. They've thankfully stopped engaging with this cesspool that it became and moved somewhere else, Instagram perhaps? Somewhere where I can't read without an account, so I don't read it anymore.
The whole point of the Internet up until about 2019 was that it was so cheap to host information it was basically free. If your site scaled up, you covered the hosting costs with ads or donations or something. The expensive part was finding content, so "user generated content" sites had to entice users to post stuff. That resulted in an implicit social contract with the users; these sites lived in fear of the users taking their content elsewhere.
Now the Web is expensive for some reason, and users have become so dependent on a small number of sites as a communications medium that the megacorporations running them feel like they have infinite leverage, and the social contract of user-generated content is completely forgotten.
Yep. Short-sighted companies don't realize that API was a truce, to save money and increase security (e.g. giving user/pass to third-party clients -> OAuth) on both sides. But anything that can be displayed, can be scraped... https://www.eff.org/issues/analog-hole
I honestly don't know what to think of this time. On one hand it's sad in principle to see Reddit and Twitter lock down to the point they have.
But on the other hand they'd both already become cesspools by that time, and I was still visiting them daily. And now I've quit them both which is a good thing.
> What if I could ask patio’s archive: “what are some good books to read about [topic]” or “what advice would you give to someone trying to get a job at Stripe”
Or what if I could ask: "Given Omer Shehata's Twitter history, formulate a phishing scam that he would be likely vulnerable to".
The problem I see with here is that there are far more bad actor use cases for identifiable user data than good. In my opinion the main reason most social networks have stopped doing public by default and now do private by default is because not doing so opens them up to Cambridge Analytica type scandals where people don't realise what they're signing up for.
Personally if you do this, I would be very clear with your users that by submitting their data it will be made available publicly in an identifiable form. And that even if they revoke their data from your service it's possible for their data will continue to be archived by others, possibly for malicious reasons.
Cognitive security vulnerabilities like this are the thing I'm most concerned about. I think it's right to be very upfront about risks like these, and I'm even considering if we want to walk back the fully public thing and make it private / invite-only instead.
Cheapest is to stay away from anything related to the big clouds, especially Amazon with their "premium bandwidth" charges.
I'd probably spin up a storage box from Hetzner (or similar service for dedicated servers) as a first step, as you get unmetered connection. It should last you a pretty long while as most content will be fetched from caches.
While I have noticed an issue with certain voices seemingly being amplified a bit more.
It is hard to argue that Twitter is still where many of these conversations are happening. People already have their followers there and unless you have a big name that can easily get people to move over to something, it is going to be pretty hard to move.
Personally I have tried to rebuild on other platforms but the engagement keeps happening on twitter even when I make the same posts on the new alternatives.
Twitter is excellent if you're looking for a social network where you can consume content and interact with domain experts in various fields.
In my experience it's really only those who go there for politics that get annoyed by it – perviously it was the right-wingers who were angry because it was too left-wing, and now it's left-wingers who think it's too right-wing.
Agree. I don't even use the 'for you' feed, and even when I check occasionally I've been able to cater my algorithm so it's purely related to the domain I'm interested in which is rust, startups, and tech news.
But my personal follow feed has all the interesting topics that I follow so it's easy to just stick to that.
While Twitter has absolutely degraded in quality over the past year or so, it is one of the last places were experts still interact with people.
I remember in the early days of Reddit it was pretty common to interact with all sorts of relatively well known people in their field. Now it's just soccer moms and bot farms.
While there are still a few experts lurking around HN, I've noticed this has declined as well over time. In part I suspect because the HN crowd has increasingly become surprisingly less technically expert on average.
I don't notice a political change -- in part because I'm not political at all on Twitter. But there's a LOT of noise on everything. Half the time I can't tell if I'm in the replies to a given post or somewhere else, because so many of the replies are non-sensical.
TikTok has become my "random browsing" feed. But I will go to Twitter and YouTube to get full-length unedited content when I need specific content.
You can't escape politics because most domain experts can't resist to engaging in sterile debates. And if you manage to find those who don't, you can't tune your feed to not show you some garbage.
IME, there's two things that were a bad problem pre-takeover and exploded afterwards: porn spam bots (like bots, follow bots, reply bots) that just acted on keywords in your tweets and Russian disinformation bots that also act on keywords. It has gotten better the last few weeks with the various leaks and a Verfassungsschutz investigation in Germany that uncovered the "Doppelgänger" botnet, but it's still an absurd nuisance.
> In my experience it's really only those who go there for politics that get annoyed by it – perviously it was the right-wingers who were angry because it was too left-wing, and now it's left-wingers who think it's too right-wing.
Well... and brands. Brands don't want their advertisements to appear next to someone showing the Nazi salute, calling for executions or whatever garbage they're pushing, and it is measurable both in financial data as well as in the quality of the ads - it used to be household brands, now it's mostly dropshipping, fake mobile game and cryptocurrency scams.
To play the devil's advocate. If you were running a large public forum, and you knew that many companies had started to scrape all data off your site, and were going to cumulatively make billions off that data, and some of those billions will come from polluting your forum with crap content, would you continue running your site in the open?
What is the game theory here? Twitter cooperates and OpenAI defects, and we call that a win?
Alternatively, build a private invite only dataset of specific communities. Scale horizontally instead of having one single central dataset! That's still a huge win for user ownership.
It doesn't have to be one company controlling who gets access, the users can decide this
The problem is that the users and their capital are not organized enough to suggest or demand for this. On the other side of the table we have the "establishment" which may not be formally organized but has enough shared incentives among itself where the outcomes aren't any different as if it were indeed formally organized and pooling resources.
In this sense we have this intractable push and pull going on with just about any community of sufficient size. We have the users who might want or should want some privacy and respect and other such benefits, and then we have the people who actually invest and build these platforms, who are incentivized to deliver other things than what the users best interest might be. Either you empower users to have more money to be able to roll their own solutions, or you try and set up a world where the incentives of capital perfectly match the needs of the individual or collective of individuals, which is probably impossible to do.
You having a problem with companies making billions? Why not just regulate that, instead of the latter "...and cumulatively make billions" part?
If companies making billions isn't the part that's problematic, the billions part can be left out, and the real problem can be discussed instead.
It's tragic that the LLM craze OpenAI kicked off is threatening to ruin one of the greatest common goods ever invented in the open internet. But hey, at least a handful of giant corporations and investors are making money, so I guess that counts as a win.
Practically speaking, 'some' of the money is the same or just as good as 'all' of the money in terms of a functioning economy and not pure capitalism.
People pirate software, yet people still develop and sell it.
Yes, it's not the most profitable thing done alone. Only for those who find the nice combination or feedback loop of 'fit', demand, improvement, and expansion.
If you can make billions off the thing, you can presumably handle some GeoIP/rate limiting... or simply, not caring. Anything that falls through the cracks is categorically insignificant to your grand nature.
To justify it, if one must, consider it a trial. As your friendly neighborhood dealer would say: "The first taste is free".
This is a problem for literally every website that isn't completely paywalled. Twitter is not special in this regard.
Yes, and Twitter isn't reacting specially in this regard. See Reddit hiking API prices by an absurd amount.
Fact Check: Reddit didn't hike API prices by an absurd amount.
That was was the story spread far and wide by the Apollo app developer that was believed by the gullible and angry Reddit masses.
But Reddit actually set a reasonable API price, as evidenced by the fact that a year and a half later there are still five 3rd party apps running on reasonable subscriptions:
Infinity For Reddit
Nara For Reddit
Narwhal 2
Now for Reddit
Relay For Reddit
To continue playing the devil's advocate:
1. Experts can comment, but I think the value of multi person conversational data from forums is uniquely valuable and in short supply relative to just blogposts/news stories on the internet.
2. The absolute economic value of the entire corpus of Twitter is much more valuable than any single boatforum.com like website. So Twitter has a much more incentive to lock itself down than boatforum.com.
Twitter was "special" because they actually had an API.
No other social media site (besides Reddit) had one. Facebook kind of tried, although theirs was always a lot more limited, and got shut down when it turned out Cambridge Analytica used it for widespread election fraud[1].
Both Twitter and Reddit did basically the same thing at roughly the same time, Twitter just had the misfortune to be under the control of Elon Musk, so the move was perceived as ideological.
[1] As it later turned out, Cambridge Analytica was basically a "nothing burger", they claimed to be able to accomplish a lot while, in reality, accomplishing very little. The damage to interoperability was already done, though.
Indeed. CA's impact on election outcomes was likely negligible; Antonio Martínez's "Chaos Monkeys" makes the case that Trump using CA was more indicative of the overall notion that Trump's team was spending money to try a bit of everything (online and offline) and the Clinton campaign wasn't. Their tactical failure was believing they could redirect the money to down-ticket races because Clinton / Trump was such an obvious matchup that they didn't need to spend to win.
What CA did show was that Facebook's statements about protecting user privacy were fundamentally incompatible with the way their API worked, so they had to shut it down because the alternative would have been to just sort of... Let it hang in the air that it wasn't hard for a third-party to build a system to completely bypass user intent in scoping their information.
(I had the misfortune of trying to write a Facebook app about fifteen years prior, and that was my takeaway at the time also... "Do people, like, realize that their whole process for protecting scraping the social network via third-party app integration is the honor system?" Turns out people didn't).
> Do people, like, realize that their whole process for protecting scraping the social network via third-party app integration is the honor system?
That sentiment goes back pretty far in Facebook's history
[1] https://www.yahoo.com/news/mark-zuckerberg-branded-early-fac...
What Mastodon is doing seems suitably ironic in this situation.
For those unaware, Mastodon's APIs are extremely open and it's very easy to scrape, to the point of providing you with a "firehose" of all public posts that an instance sees, both local and federated, with no authentication required. They also have an extreme anti-scraping culture, anybody who admits to running any kind of scraper which is not strictly opt-in, even for benign / scientific purposes, is very quickly shunned and blocked. Most instances also have a "disallow scraping via robots.txt" policy by default.
The results? I posted a canary token[1] link on a medium-sized, well-federated, well-protected instance which disallows scraping, and it got hit by some shady social media crawler in a fraction of a second. It started getting hit by many other strange crawlers later on, and it still keeps getting visits (mostly from Google now).
It's an ecosystem of people that seem insistent on the idea that you can put stuff online behind no authentication wall and expect to stay exclusively on servers you believe it should be on.
And I don't know what to tell them, because that's not how the internet has ever worked.
Here's a thought: someone "trustworthy" should maintain a Chrome extension or Tapermonkey script that automatically scrapes data from various social media sites in a fully anonymized fashion. As people browse Twitter, Reddit, or XYZ, the posts/comments are sent to some aggregation system. It might be against TOS, but certainly far less than scraping, and you couldn't tell, as it's the user driving what gets scraped.
I don't use Twitter often, but I'd run something like that if there were strong anonymity guarantees. Seems like a win-win for everyone.
Does anything like this exist today?
Reminds me a little of RECAP (https://free.law/recap), an automated scraper/saver/sharer for PACER (the US court electronic records system).
Obviously the content is very different, but the technology is basically doing what you’re talking about, minus anonymizing the data.
I use RECAP constantly and try to always upload everything I grab from PACER.
I think this administration finally passed a law to make PACER free, which is good.
https://www.reuters.com/legal/government/fed-judiciary-says-... | https://archive.today/2wddY
https://fingfx.thomsonreuters.com/gfx/legaldocs/egvbkwemjpq/...
Good find. Yeah, pretty much exactly like this.
Not quite the same, but https://returnyoutubedislike.com/ is in a similar vein of an extension crowd-sourcing a return of data.
I was thinking of that exact strategy for two scraping problems I have. It would be a good way to gently scrape Berlin.de for appointments, and Immobilienscout24 for new flats.
From what I can tell, it would work fine so long as the user is actively looking at those pages.
Yeah, something generic to work for any use case would be nice, but privacy becomes more difficult as you need to tailor the situation to each site to maintain privacy (i.e. only pull the information on the apartment/flat listing, or public tweet or reddit comment, etc...)
I wonder if those websites as a response might start adding fingerprint code in the source to indicate whose user session it was.
The city of Berlin wouldn't care. The housing website would probably start with lawyers.
Users should just get off this continued tragedy and API access wouldn't be an issue
To me, Twitter is by far the most toxic social network. It has the power to turn the most interesting and smartest people into bitter trolls. Maybe some people get value from it, but it requires serious self discipline to not get dragged into stupid arguments.
I've tried to "get" Twitter since the early days. I've created a couple of Twitter accounts over the years to support various creative endeavours but I've never found myself getting much value out of it as a user.
I heard that it's original use-case was that back 2008 there were no good ways to do group messaging over SMS on a cell phone. That's a problem and solution that I can understand.
But as a broader social network? I don't get it.
Things that bother me to the point that they are deal breakers:
- The character limits (there is nothing worse in life than reading through a x/N self-reply to read something long ... I'd rather file a tax return)
- Showing me posts from pages I don't follow in the feed (in 99% of the cases I'm aware of that page/profile already and have chosen not to follow it because I don't like them)
- Ads in the news feed
Some have said that they like Twitter for getting news. I have way better options for that.
I know that I'm not representative of the typical person, generally speaking, but Twitter is one of those things where I really can't understand why anyone likes it and uses it, let alone why it is so popular. And it's not that I'm hating on something I don't know anything about ... I've honestly tried to use it and get value out of it, but I've never found anything of value on offer.
> but Twitter is one of those things where I really can't understand why anyone likes it and uses it, let alone why it is so popular.
I think it's a carryover of general nostalgia for how the Internet "used to be", i.e. before every site with a few million users decided to start experimenting with algorithms to max out user engagement and ad impressions, leading to the hell we see everywhere, from IG to Tiktok.
Yes, there was a time when Twitter wasn't a toxic mess. It's the last social network that was popular before smartphones took off.
Interesting. That might explain why I dislike it so much. As a middle aged person who grew up with an early iteration of the world wide web for a decade to a decade and a half before Twitter even existed, both smart phones and social media mark a turning point for me from what the Internet "used to be."
Twitter, in my mind (and maybe this is perception and not reality), ushered in infinite scroll and short bites of information. Twitter is to forums what TikTok is to documentaries. I see Twitter and the "mobile revolution" going hand in hand (something that left me behind because I still dislike using a smart phone, generally, and rarely do compared to most other people).
But I guess if you're a great deal younger than me, and you grew up with an Internet where Twitter just always existed, then it might represent some earlier version of the Internet that is drastically different from what you consider to be "contemporary" (though, putting the TikTok mention aside, I'm still not sure what that view of the contemporary Internet is if Twitter is what we're comparing it to).
I guess I'm just so old that I still see Twitter as a relatively new phenomenon. Very different from the nostalgia that I feel for what the world wide web used to be when I was young.
It's gotten dramatically worse since premium started rewarding payouts for engagement/views, alongside zero moderation. Now there's an actual incentive to be a jerk, a spammer, and make repeated bad faith arguments / "just asking questions" -- people reply, your tweet is seen by more people, you get paid a share of ad revenue.
Then you have the regurgitated messages in replies (AI or otherwise) endlessly copied and distorted. Recently it's been 5+ pages until I see a real reply, while I have 3000+ accounts blocked by now. It's getting harder and harder to find the real person who wrote that witty reply, and not the endless blue check accounts who copy-pasted it to farm engagement on their visibility-boosted tweets.
Genuinely getting more and more unusable.
Tangential but there's my disappointment to AI in there: they haven't found a way to build convincingly human bots, let alone a procedural content creation system.
Both AI believers and doomers insist generative AI could surpass humans by every single metric that matters in social media. All we've got is first Nigerian, then Indian, and now increasingly Pakistani spammers desperately reply bombing trending posts made by 10 years alpha users, as if whoever behind it completely failed to do it and has been wasting gullible human spammer candidates faster than Aperture Science.
> they haven't found a way to build convincingly human bots, let alone a procedural content creation system.
The 'they' being the corporations that hired thousands of people from the very countries you're associating with spammers , to review, correct and train AI models?
Why not, um... stop using it?
Mastodon has a lot of great people, and a lot less of all this stuff.
I have a Mastodon account, a Bluesky account, and apps for both. Mastodon doesn’t replace the comedy part of Twitter, and Bluesky is great but 99% of local social media is still on Twitter. So I’m “trapped” on Twitter until they move to Bluesky, but they won’t because a part of that crowd is addicted to the revenue share off their political tweets.
Not parent, but I did use Mastodon for about a year. I ended up moving back to Twitter because:
1. I just couldn't vibe with the culture there. From my POV, Mastodon is made out of pearl-clutchers and politics.
2. So much drama. The FediSearch drama. The Raspberry Pi incident. It's just so tiring and you feel like you need to walk on eggshells all the time.
3. So much drama. You would just pray that your admin didn't get into a spat with another admin and get defederated. You could get a solo server, but that costs money and you might get blocked by a large server's admin anyway.
4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere. Solo, yes, but costs money.
At least with Twitter, the rules are sort of well known, and you can follow anyone there unless they block you personally.
I heard BlueSky is good, though. Haven't tried it yet. Nostr was also another one, to get around the admin drama issue, but it doesn't seem very popular.
>4. So much drama. Pray that your server doesn't shut down, because you can't import your posts elsewhere.
I would have thought a well-designed decentralized system would allow you to, for instance, download/export all your posts, and then import them all to a new server, in case you have to move servers for some reason.
The server-oriented aspect of Mastodon was the second-worst decision ever (the first one was the name). Mainstream social media doesn't ask you to pick a server (nor deal with the consequences of it going down), its replacement shouldn't either.
As for the server-oriented aspect, what's the alternative? There's only two ways I see to run a social media service like this: centralized and decentralized.
The mainstream ones are centralized, of course: FB, Xitter, etc. You just connect to their web server and do everything there. Any centralized alternative would have be a well-financed corporation able to set up a huge IT infrastructure to accomplish the same task. But the whole idea of Mastodon is to not be a big, for-profit corporation, and to be in the hands of the users instead, which I don't think is possible with a centralized service.
So the only alternative is a decentralized service. But this means you have to have multiple servers, leading to the whole "federated" approach, so users can run their own servers and control things themselves, or use accounts on servers of their choice. This of course leads to all the consequences you named, and more: servers going down, servers defederating other servers, etc.
But again, what's the alternative? If the requirement is "not run by a single big for-profit company", I don't see one. If you think that having a decentralized service actually be competitive with a centralized one like Xitter simply isn't feasible, that's fair.
My prior complaint still stands though: if you want users to have freedom (of which server to put their account on, etc.), then why would you not have a way of migrating their account/data to another server, and force them to lose everything if their server disappears?
I agree about the name: they shouldn't have stolen the name of a great metal band.
This obviously varies person to person, but I don't find it to be that. I carefully maintain who I am following (about 600 people and companies, mainly in the infosec / natsec space) and when I start to see bad behavior/content I'm quick to unfollow people. Also, I only use the Following tab, never the For You tab.
I never did much with Twitter until I took over the Carolina Code Conference, but it's been by far the easiest place to engage with different tech communities. Just the "Follow All" button that appears after I follow something like Gleam or Roc is hugely helpful.
I probably spend more time on Twitter now than I ever did before.
It's intentionally engineered to be as emotionally provocative as possible, to drive engagement, to drive and revenue.
My For You feed kept turning into videos of public fights and Karen's calling the cops, etc. no matter what feedback I gave or what I clicked on.
Deleted it and haven't looked back. Won't click on a link from x.com either.
> It has the power to turn the most interesting and smartest people into bitter trolls.
There is tremendous value in having access to large quantities of data demonstrating the various failure modes of relatively intelligent minds. Nothing is being (publicly) done with this data so far, but this will not last forever.
This is something that has always bothered me about the /r/SSC community (and others): the "no culture war topics" constraint + moderators locking threads when the Rationalists start collectively behaving irrationally.
A lot of users/communities are stuck because of network effects & the long history of posts (they still reference them, like living wikis)
These pipeline tools can help people migrate if they want without losing that history & network (my prediction is once people see what's possible with an open API, that'll further motivate user migration, or for twitter to open up again)
> the long history of posts (they still reference them, like living wikis)
As someone with a multitude of cat photo threads, 100x this.
While I'm around on Twitter, Bluesky and Mastodon, the "post history" aspect is my most pressing issue with Mastodon. Like, when you move your account, you cannot take your old posts to the new server - you're straight out of luck if your instance decides to call it quits for whatever reason. And even if the instance doesn't give up, being unable to view old(er) photos is something I very commonly notice on a bunch of Mastodon servers.
People should do a lot of things. They shouldn't smoke and they should work out, but here we are in 2024 where the phillip morris stock has outperformed ibm over the last 5 years and the obesity crisis shows no signs of letting up. Knowing something is bad is the first step of course but clearly not enough to drive a real behavioral change, and we aren't even fully in agreement as a society that social media is harmful like how cigarettes or obesity are known to be harmful.
>we aren't even fully in agreement as a society that social media is harmful like how cigarettes or obesity are known to be harmful.
There's more than one society on Earth, so it's hard to say that "society" "knows" that smoking is bad without specifying the society. As an American who lives outside the US, one striking place where the US clearly outperforms peer nations, in my daily experience, is with smoking: Americans just don't smoke any more unless they're poor and uneducated. There's no smoking inside (except bars), there's no smoking within XX feet of business entrances, etc. In the US, smokers have really become pariahs since the 80s and early 90s. Some of that has been undone with the rise of "vaping", but still, if you hang out with college-educated people, you probably won't meet any smokers. It's just not like that in other countries: people of all social classes still smoke. Here in Japan, it's quite common to see some salaryman riding a bicycle while puffing on a cigarette, and many restaurants have smoking areas (usually sealed off with separate ventilation in modern buildings). There's even some old locals-only eateries that have (non-separate) smoking inside. Europe isn't any better.
As for obesity, only truly delusional Americans try to claim that "fat is healthy". All the rest know it isn't, even if they're obese, but the society makes it very hard to not be overweight, mainly because everyone drives cars and doesn't walk.
Many moons ago Twitter used to have RSS (Atom?) feeds for each user so you could use any old news aggregator to keep up to date.
Social media used to be about content sharing, but it's clear that it's far more profitable to keep your users on your site at all times and keep the content they create in house rather than linked somewhere else.
HN is probably the last true aggregator site left that I know of.
It's partly why the web is so much worse. There's really no reason to create content outside of a walled platform is it is getting increasingly difficult to find an audience for it. Even blogging is an uphill battle since more and more social media sites penalize link sharing (they want you to create the content on their platform and leave it there).
That's why it's not surprising that these APIs are disappearing since the fundamental model has changed.
But that's too sensible and user-friendly. It would be obviously weird and wrong if they inserted ads and Musk's tweets into someone's RSS feed. Yet showing those unwanted insertions is the whole purpose of the company now.
Twitter shut off RSS access over 10 years ago along with their popular 1.1 API.
I know. Because it wasn’t compatible with their aim of showing ads.
And nothing has changed after Twitter went private under Musk, except that the reasons for stuffing unwanted content into users’ timelines are slightly different.
I don't think that's fair - getting back the chronological timeline itself made the purchase a good one in my eyes, alongside the For You feed, which seems to work wonderfully (for me). Not to forget you can literally turn off ads if you want. I seem to be a real minority, but I do genuinely believe Musk has done a great job with Twitter (or X).
You could get SMS notifications whenever a user tweeted, too. Imagine signing up for that today!
You could also for a long time send SMS (if you registered your number) to 40404 to tweet to Twitter. It's still in my address book from the era when I was tweeting T9 from dumb phones or using voice-to-SMS to tweet on early smart phones.
That makes more sense to disable as SMS is expensive. This is still fairly easy to set up yourself using a tool like IFTTT[1] or similar.
[1] https://ifttt.com/search/query/twitter
Here is the washington post doing this with TikTok users to reverse engineer the algorithm!! https://thewashingtonpost.formstack.com/forms/help_investiga...
they've got data from 800 users so far, with watch data on 55 million videos
It has been difficult to rescue data from Twitter even before purchase. On our case it was relevant because this is online digital history for the people in my country.
The only thing we can is motivate more people to use open platforms like NOSTR where API or data/identity handling is completely different.
https://www.community-archive.org/
https://github.com/TheExGenesis/community-archive
I hate to be the one that says this, but what's to stop someone poisoning this and uploading a file of someone else's fake tweets?
The system should require multiple identical copies uploaded independently.
The "independently" part is hard, could require some kind of web of trust.
Funny, just now I've been playing around with the various tweet deleters and trying to get something working; presently I think I'm about to settle on something involving a basic screen macro recorder thing, like one of the iterations of AHK.
I'm somewhat surprised that this space feels relatively dormant compared to the more complex stuff out there.
(APIs suck)
Twitter was originally a microblogging service that had rss feeds to syndicate things or monitor the microblogs of people/companies that were interesting, it has gone way far off into the fields.
Same journey reddit is making, starting after it prepared to go public.
At this point, just nope out of it and use something like Bluesky
The only useful thing on Twitter that I ever saw was the lovely and tender Dog Rates https://x.com/dog_rates. You could read anonymously and be all aww and schucks about all those good dogs. They've thankfully stopped engaging with this cesspool that it became and moved somewhere else, Instagram perhaps? Somewhere where I can't read without an account, so I don't read it anymore.
Here's a few more in my feed (no affiliation)
- https://x.com/contextdogs (out of context dogs)
- https://x.com/_B___S (B&S)
- https://x.com/buitengebieden (Buitengebieden)
> They've thankfully stopped engaging with this cesspool that it became and moved somewhere else
No, they haven't. They're still actively posting on Twitter.
It's just that if you are viewing a profile logged out you get the most popular tweets instead of the most recent ones.
The trend of shutting down / charging steeply for API access has fundamentally changed the internet.
The whole point of the Internet up until about 2019 was that it was so cheap to host information it was basically free. If your site scaled up, you covered the hosting costs with ads or donations or something. The expensive part was finding content, so "user generated content" sites had to entice users to post stuff. That resulted in an implicit social contract with the users; these sites lived in fear of the users taking their content elsewhere.
Now the Web is expensive for some reason, and users have become so dependent on a small number of sites as a communications medium that the megacorporations running them feel like they have infinite leverage, and the social contract of user-generated content is completely forgotten.
Hosting information isn't expensive, trying to justify your billion dollar valuation is.
Back to the old web scrapping you go.
Yep. Short-sighted companies don't realize that API was a truce, to save money and increase security (e.g. giving user/pass to third-party clients -> OAuth) on both sides. But anything that can be displayed, can be scraped... https://www.eff.org/issues/analog-hole
I honestly don't know what to think of this time. On one hand it's sad in principle to see Reddit and Twitter lock down to the point they have.
But on the other hand they'd both already become cesspools by that time, and I was still visiting them daily. And now I've quit them both which is a good thing.
Neat boundaries on the Town Square
Where you need to pay in order for your most freedom-y of free speech to even be visible to others.
Freedumb of speech, not reach!
See also: Simpsons, "elected to lead, not read"
switched to mastodon, bluesky long ago.
I've been very happy with Mastodon.
> What if I could ask patio’s archive: “what are some good books to read about [topic]” or “what advice would you give to someone trying to get a job at Stripe”
Or what if I could ask: "Given Omer Shehata's Twitter history, formulate a phishing scam that he would be likely vulnerable to".
The problem I see with here is that there are far more bad actor use cases for identifiable user data than good. In my opinion the main reason most social networks have stopped doing public by default and now do private by default is because not doing so opens them up to Cambridge Analytica type scandals where people don't realise what they're signing up for.
Personally if you do this, I would be very clear with your users that by submitting their data it will be made available publicly in an identifiable form. And that even if they revoke their data from your service it's possible for their data will continue to be archived by others, possibly for malicious reasons.
Dev here.
Cognitive security vulnerabilities like this are the thing I'm most concerned about. I think it's right to be very upfront about risks like these, and I'm even considering if we want to walk back the fully public thing and make it private / invite-only instead.
Elon is trying really hard to destroy Twitter, isn't he?
> cheaply as possible (put everything in S3
yikes.
To be charitable I hope the author refers to the S3 API here, of which there are several competitors and open source implementations.
Many companies offer "object storage" which is compatible with the S3 API
And there are some open source projects such as Minio which offer the same (and I forget what are the others?)
feedback is very much needed
For reference, this is the GitHub issue: https://github.com/TheExGenesis/community-archive/issues/72
Cheapest is to stay away from anything related to the big clouds, especially Amazon with their "premium bandwidth" charges.
I'd probably spin up a storage box from Hetzner (or similar service for dedicated servers) as a first step, as you get unmetered connection. It should last you a pretty long while as most content will be fetched from caches.
[flagged]
[flagged]
While I have noticed an issue with certain voices seemingly being amplified a bit more.
It is hard to argue that Twitter is still where many of these conversations are happening. People already have their followers there and unless you have a big name that can easily get people to move over to something, it is going to be pretty hard to move.
Personally I have tried to rebuild on other platforms but the engagement keeps happening on twitter even when I make the same posts on the new alternatives.
The beauty/horror of network effects
Twitter is excellent if you're looking for a social network where you can consume content and interact with domain experts in various fields.
In my experience it's really only those who go there for politics that get annoyed by it – perviously it was the right-wingers who were angry because it was too left-wing, and now it's left-wingers who think it's too right-wing.
Personally I've noticed practically no change.
Agree. I don't even use the 'for you' feed, and even when I check occasionally I've been able to cater my algorithm so it's purely related to the domain I'm interested in which is rust, startups, and tech news.
But my personal follow feed has all the interesting topics that I follow so it's easy to just stick to that.
While Twitter has absolutely degraded in quality over the past year or so, it is one of the last places were experts still interact with people.
I remember in the early days of Reddit it was pretty common to interact with all sorts of relatively well known people in their field. Now it's just soccer moms and bot farms.
While there are still a few experts lurking around HN, I've noticed this has declined as well over time. In part I suspect because the HN crowd has increasingly become surprisingly less technically expert on average.
I don't notice a political change -- in part because I'm not political at all on Twitter. But there's a LOT of noise on everything. Half the time I can't tell if I'm in the replies to a given post or somewhere else, because so many of the replies are non-sensical.
TikTok has become my "random browsing" feed. But I will go to Twitter and YouTube to get full-length unedited content when I need specific content.
You can't escape politics because most domain experts can't resist to engaging in sterile debates. And if you manage to find those who don't, you can't tune your feed to not show you some garbage.
Helping to prop up the site's racist, neo-nazi owner doesn't bother you at all?
> Personally I've noticed practically no change.
IME, there's two things that were a bad problem pre-takeover and exploded afterwards: porn spam bots (like bots, follow bots, reply bots) that just acted on keywords in your tweets and Russian disinformation bots that also act on keywords. It has gotten better the last few weeks with the various leaks and a Verfassungsschutz investigation in Germany that uncovered the "Doppelgänger" botnet, but it's still an absurd nuisance.
> In my experience it's really only those who go there for politics that get annoyed by it – perviously it was the right-wingers who were angry because it was too left-wing, and now it's left-wingers who think it's too right-wing.
Well... and brands. Brands don't want their advertisements to appear next to someone showing the Nazi salute, calling for executions or whatever garbage they're pushing, and it is measurable both in financial data as well as in the quality of the ads - it used to be household brands, now it's mostly dropshipping, fake mobile game and cryptocurrency scams.
[1] https://qz.com/twitter-x-ad-revenue-tumbles-elon-musk-grok-1...
[flagged]