Citation needed? I'm not a fan of residential proxies but these are pretty wild claims that aren't substantiated. Calling them a "national security threat" implies it is a threat to e.g., the continued existence of the United States. Is it really that? All of the activities described are already violations of the Computer Fraud and Abuse Act.
> At the very least, major American ISPs (Comcast, AT&T) should detect clearly suspicious activity coming from customer IPs and warn them to scan their computers, check their TV apps, and find whatever is turning their internet into a proxy.
What are we saying? Really, what are we saying? We should turn domestic ISPs into domestic surveillance apparatuses to "detect clearly suspicious activity"? What is "clearly suspicious activity"? Section 230 is still law.
The router could also be a good place to display live TCP usage and last 30 days usage and other network activity. There should also be a service that can identify whether specific IP addresses belong to botnets or other malicious infrastructure. If there are any abnormal activity mobile app can notify the user.
Governments could even purchase residential proxies themselves and check whether they're being used for malicious activity.
It isn't that difficult, but I guess governments just aren't that interested.
> The router could also be a good place to display live TCP usage and last 30 days usage and other network activity. There should also be a service that can identify whether specific IP addresses belong to botnets or other malicious infrastructure.
> It isn't that difficult, but I guess governments just aren't that interested.
I recently switched from my previous ISP because they randomly broke my ability to use my own router, and during the time when I had to default to using their own modem/router hardware before I could get the new ISP to come and set things up, I could barely even get a signal in my office upstairs (which had previously been connected via a mesh endpoint) because their router didn't expose any way for me to split 5 GHz and 2.4 GHz, and the router absolutely refused to let my devices connect via 2.4 GHz despite them having more than 90% packet loss due to the weak 5 GHz signal.
Regardless of how "easy" it is, I don't trust ISPs not to screw it up somehow and probably cause a lot more concrete damage (even if the individual issues they cause are smaller in magnitude) than the theoretical concerns of "national security" that, as far as I can tell from reading this thread, have caused a total of like a few hours of downtime one day in a couple decades.
So it's okay for organized crime to do it, but not normal plebs? The former is obviously a lot more harmful and in reality this is just a wakeup call to get their shit together.
People were dismissive of security far too long. I think it's actually good that AI instills some fear into people causing it to take a lot more seriously than they have previously done so.
They're also a wholly necessary endeavor until the surveillance industry stops discriminating against IP ranges with endless captcha nagwalls. That, including its likely next development of remote attestation, is a much deeper problem for individual liberty. Individual liberty is itself more important than "national security" (which is more about protecting the government rather than the People) and thus needs to be addressed first.
I'm a bit on the fence about this, but leaning towards the "bad luck". I'm sure there's a large swathe of nuance that I'm missing, but my simplistic view is: If they don't want to be scraped then they don't get included in "the thing" which, at minimum, is a data point for consumers to make consumer decisions about.
It strongly depends on how scraped data is being used.
If your 'cat' hasn't been able to catch their 'mouse' then the cat needs to get smarter, or look for alternative sources of mice, or the cat should be considered 'unviable'.
Have you approached them to get access to their data? Have you explained to them how your service can benefit their business?
(I generally come from a position of suspicion as to why someone wants to scrape data that the owner goes to certain lengths to protect, but then I'm also an 'information wants to be free' kinda person, but the Internet is increasingly an untrustworthy place, so security is overruling narrative).
Government website data should be openly available one way or another (at least within the country, and with reasonable security provision against obvious maliciousness).
If it has to be scraped then there may be other problems (which include lack of resources to make the data API accessible).
That's cutting the tether from a _lot_ (there's no way to overstate this) of useful information, but that's potentially one of the great things about the open AI/LLM models, is that all that info is baked in there.
Disclaimer: As far as I understand it. Please educate me if I'm way off the mark.
Also, how much of the content of reddit is in Common Crawl? (same disclaimer applies to this comment)
I also understand 'the archival mindset', I hoard a bunch of data. But I also understand the logarithmic graph of futility.
> I generally come from a position of suspicion as to why someone wants to scrape data that the owner goes to certain lengths to protect
Let's try this for example. Suppose you want to create a price comparison site.
A lot of the major retailers don't want these, because they want the customer going to their site when they want to buy something, not to the price comparison site that tells the customer which retailer has the best price and then might not always be them.
If the comparison site doesn't have the prices from retailers who sometimes have the best price then it can't serve its function -- customers still have to check sites manually. And comparison sites are pro-consumer whereas blocking them is anti-competitive, so the ones doing the scraping are the good guys.
Suppose someone wants to create a price comparison site, but is unable to access all the data that would serve their customers in finding the lowest price.
The I guess the site is either useless, or can be used in combination with customer's own research that includes sites not included in the comparison. The site is still useful, it's just not exhaustive. There's nothing much in the world that's exhaustive, it's all a range of percentages.
Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Everyone seems to want _everything_ despite _everything_ not being necessary to provide a service. The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Just because residential proxies are somewhat of an 'easy answer' doesn't mean they're right. Use more creativity and imagination! Or follow a new idea. It's not like price comparison sites are a revolutionary idea.
(I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic).
> so the ones doing the scraping are the good guys.
Everyone thinks they're the good guys. If there's profit involved then there's always an element of delusion.
> The I guess the site is either useless, or can be used in combination with customer's own research that includes sites not included in the comparison. The site is still useful, it's just not exhaustive.
There is a significant difference between having the prices for 85% of retailers because you haven't figured out how to get pricing from 15% of them and having the prices for 5% of retailers because the biggest ones block you from getting their prices. The amount of work you can save the consumer, and thereby the usefulness of the service to them, changes by a dramatic amount.
> Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Now consider the possibility that they sometimes have the best price and sometimes don't.
> The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Suppose you go to the price comparison site to look for the best price, the best price in its data set is $22, then you go to Amazon and they have it for $19 because it doesn't have their prices. The customer then starts checking Amazon in addition to the price comparison site. Their price is only actually lower 15% of the time, but another 75% of the time it's exactly the same, so the customers who don't want to keep checking two sites start checking only Amazon instead of only the price comparison site, at which point Amazon gets to charge them a higher price on 10% of stuff, or a higher price on more than 10% once people have been trained not to check. And prevent them from patronizing a random competitor when their price is exactly the same
Causing that to happen is the reason they don't want their prices in the comparison site. If not being listed there hurt them then they wouldn't be trying to prevent scraping.
> I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic
Consider also that you may be able to justify doing this when buying electronics, or choosing which brand of something to buy on a recurring basis, but if you just want the best price on the product you already know you want, it's crazy to spend hours to save cents. But completely sensible to switch to a search box that would save you 10 cents on every $2 purchase by giving you price-sorted results from multiple retailers who all sell the same products, if it's allowed to exist.
Apologies for the double post, but I've got an alternate perspective:
Do you allow the data you've scraped to be scraped? Do you share it as freely as you desire the 'companies that want to hide it' would? Or do you consider the scraped data is 'hard earned reward for effort' and therefore has value that others should subscribe to your service for?
If anonymous currencies can be considered a national security threat despite making up less than 0.001% of the world's total value, then why not anonymous packets?
Once the establishment has understood how to profit from anonymous currencies, then they'll be taken off the 'bad' list (no matter how many scams continue to be foisted upon the unwashed masses).
Ad networks like meta ads, google ads use residential IPs + small LLM to check landing page of your advertisers for malware and scams. They are already paying millions for this service.
Sometimes their system malfunctions (cough cough) and you get charged for those clicks but you fail to report them as fraudulent as you've no proof as the IPs and useragent all appear normal to ad agencies and advertisers.
Residential proxies are necessary to access services that try to ban VPNs without compromising anonymity. If they’re banned, it’s easier for websites to require information that can be used to track you (IP address), since most people don’t use VPNs they won’t care.
More importantly, a residential proxy can be mutual. If a group of people agree to forward traffic to each others’ IP, how is it not their business? It obfuscates their identities, but this is the Internet, not e.g. a government office or test center where you obviously can’t walk in with someone else’s ID.
Banning non-consensual proxies makes sense since those are effectively malware, even though they make anonymity harder, so does banning theft and here you’re stealing someone’s internet. I’m sure we can convince enough laymen to knowingly install proxies by paying them.
I think the fact that they're laymen means that there's still a pocket of non-consensuality. Taking advantage of someone's ignorance of potential consequences is as bad as malware in my opinion. If you're outlining all the potential bad outcomes before signing up a 'mark', then that's slightly more OK.
Myself being "not a layman" would definitely not allow anonymous usage of Internet that's tied to my home/residence/identity to be used by some internet rando just because they pay me money.
I would support, however, small family/friends groups who know and trust each other well enough to not do anything that'll cause a police raid on each others homes (having endured a police raid and the ensuing 8 months of not being told anything about the progress of whatever they're doing, it's not something I'd wish on a stranger, never mind a friend/family member).
> Taking advantage of someone's ignorance of potential consequences is as bad as malware in my opinion. If you're outlining all the potential bad outcomes before signing up a 'mark', then that's slightly more OK.
Sure, the service should be upfront, although most laymen probably won't read. But if people had to really understand consequences before they accepted anything, most would miss out on most overall mutual offers. Importantly, the expected consequences are low (as long as the service isn't greedy), and the ratio of tech people agreeing would suggest that if most laymen did understand they would still.
If so many sites and services didn't go out of there way to block or flag VPN users as suspicious then I would have no need to use residential / mobile proxies.
Separately, would you let your home/personal connection be used as a residential proxy?
Personally, I would not, except if it was for a friend / family member, but I would still ask them, pointedly, why my internet connection is required rather than their own.
Disagree fairly strongly. IP addresses known to be malicious are unequal and should be treated as such.
Just because the idea of net neutrality exists doesn't mean it's true. I'm sure there's a specific context for it, and 'security' is not that context. It was about data/packet prioritisation wasn't it? Unrelated to security.
Yeah, mail server IP reputation is a good example. I wouldn't want to be flooded by emails from spammy mailservers IPs and I'm glad that my provider de-priortizes them.
The reason that works for mail servers is that they first require the IP address to have a valid reverse DNS entry, which is not just compromising a random machine with a random IP address but having administrative control over that IP block, which basically limits you to data centers and business internet providers, and then blocks IP addresses that send spam, i.e. that are owned by providers who give reverse DNS entries to spammers. The premise is that most people aren't trying to run a mail server so you require something that takes unusual bureaucratic interaction in order to get.
Trying to restrict things to end-users is the total opposite of that. The common addresses everybody gets automatically are the thing you're trying to allow. Address reputation is pointless because residential customers get dynamic IPs and the reputation you're trying to record for some IP address can get swapped with a different customer at any time.
Sounds like a good reason that residential proxies are a bad thing, or at least require better regulation, so as to minimise damage to unwitting end-users.
It also feels like a description of the perfect camouflage to facilitate doing bad things: "don't block them because you might block an innocent bystander". Putting innocent bystanders in harms way sounds like someone else is the bad guy, not the person doing the blocking.
> Sounds like a good reason that residential proxies are a bad thing, or at least require better regulation, so as to minimise damage to unwitting end-user
The problem being that attack traffic is disproportionately coming from devices that are compromised, which is already illegal, and you can't fix that by making it harder to use residential proxies for things that are legitimate, like sharing IP addresses between real users so they can't be used as a personal tracking ID.
> It also feels like a description of the perfect camouflage to facilitate doing bad things: "don't block them because you might block an innocent bystander".
Cloudflare promotes putting your site behind Cloudflare to inhibit censorship, because then the censors have to block their entire service (which is half the internet) to block anything. It's not always a bad thing.
> Putting innocent bystanders in harms way sounds like someone else is the bad guy, not the person doing the blocking.
If there is an alleged thief on the subway and you respond by lobbing a grenade into the subway, there is more than one bad guy.
Anything connected to network can become a national security threat. IoT devices, internet connected smart appliances and all. Computers that run some OS is least of the worry because of the various layers of protection that can be added to them. But it is not the case with the IoT devices or SMART(!) devices where, if the firmware or any app ecosystem gets hacked / cracked / modified or sideloaded without any user intervention, could turn into a large zombie C2C botnet that can cause havoc.
Proxying is least of the worries!
When manufacturers bring out the smart features that require constant data collection, monitoring (obviously in the name of service quality, troubleshooting, firmware / app updates and subscription of services (!!)), it is oblivious to the fact that security and privacy at a personal / state / national level is never considered (for various reasons such as the design, profit interests and overheads / compliance perspectives and what not!) to be part of the ecosystem. Bad actors with sufficient knowledge and tools could exploit and profit from it.
It has become more of a planned obsolescence / profit / greed based industrial / corporate culture in rolling out unwanted stuff / monitoring and controlling as a feature that gets exploited to the core.
Honesntly, I don't have an answer for that, but have some thoughts that I wanted to share.
I do not want a cleaning robot or a water purifier or a printer or a TV or a refrigerator or an oven or a smart light bulb or a smart lock or even my car to have undisclosed, unwarranted connectivity to internet for whatever reasons.
I do not care if it is the manufacture or the service provider or whomsoever it may be!
I want all of the network connectivity options to be explained with all the controlling options and boundaries and data collection that happens on any of the connected medium to be as much transparent with the option of preventing or restricting what one does not want to go out of the device.
Will anyone do that?
It ill never happen! Sadly!
This problem will balloon further without adequate controls and killing the networking at a ground level would only be the only solution!
But, in the case of a connected device having an inbuilt connectivity option (like the m2m based options) in a smart vehicle, even that is not possible!
No, your paranoia-outage is the true "national security threat" - a threat to freedom. Stop fanning the flames and calling for more government intervention.
After decades of watching people use these three words, I've come to the conclusion that 'national security threat' is a right-wing dog-whistle for 'we have no actual proof, but we dislike it, so let's ban it'.
They are basically 'won't someone please think of the children?', but with higher stakes.
it's a general politician dogwhistle, don't underestimate "the left" (especially in countries like the US, where the democrats are really not that left compared to other left parties in e.g. europe; leaving you with a pointless right-wing vs right-wing fight where nobody wins)
Its worse than that. Its "think of the children" but with slightly more truth to the statement than " think of the children". Because it is definitely a national security threatwhile primarily actually being said to |imit freedoms and increase market power abuse possibilities for big tech and/or media.
This is just the largest residential proxy provider BrightData(previously Luminati that used a free VPN to source residential bandwidth)
The overall residential proxy market is too large and often undetected by intelligence tools. BrightData is actually much more compliant and malicious actors wouldn't be allowed access. They have an extensive KYC and use-case vetting process.
Actually they have several different products based on IP type and quality. The credit-card only access you're talking about is for shared and flagged residential proxies. ISPs legally sell them to businesses and they can work for some use cases, but you need a video call and KYC to get access to a high quality IP pool, such as real mobile IPs they source from p2p services like VPNs.
There is nothing in that explanation that makes it sound anything like an above board trustworthy business that doesn't make one feel dirty for interacting with.
> you need a video call and KYC to get access to a high quality IP pool
They need to know who to blame (cough come after cough) if their top-shelf stock gets tainted by some script kiddie.
If someone with informed consent wants to run a residential proxy, that's their right and unlikely to be a national-security problem.
When it comes to involunary proxies, those are really just one of many possible symptoms stemming from a real problem: Shitty security. (Edit: And shitty contract/privacy laws.)
Shitty security is tolerated by our markets, is is protected from fixes due to copyright law, and it is even encouraged by parts of our government that want to exploit the flaws. We will gain far more from fundamental quality-improvements than we will from whack-a-mole-ing on this symptom.
I'd be interested in the stats showing what percentage of residential proxies have 'informed consent', and the scale of what 'informed' means beyond "included in terms and conditions".
> We will gain far more from fundamental quality-improvements than we will from whack-a-mole-ing on this symptom.
100%, but I feel like that's both true and rarely executed upon in most governments for most topics.
If I own a computer, I should be able to run whatever damn software I want on it, including software that provides a network proxy to others. (N.B. a smart TV is a computer I own.)
Yes, yes, we should fight deceptive software that doesn't disclose it has a proxy function. Fine. But there's no way to square 1) fighting all residential proxies, including ones run with informed consent and 2) preserving the public's access to general-purpose compute at home.
I'd rather live with the proxies than for someone to tell me that if I run squid, I'm damaging national security.
Very well: your ISPs, telecomm providers, and cloud services also own computers, and they should be able to run whatever they want on the devices they own, and they choose to block all your traffic, block all traffic destined for your premises, and flag your hardware IDs in certain databases, which is software they choose to run because they own the servers.
The "solution" is to shoot CGNAT and to finally force the ISPs to adopt IPv6 so everybody can drop the addresses of a detected residential proxy into a ban list.
Continuing to allow CGNAT is what allows the residential proxies to hide because it tumbles the IP identity of bad actors with normal people.
No, they are correct, residential proxies behind a big ISP's CGNAT is much more valuable for miscreants because you can't block the individual who installed a free android flashlight with a residential proxy SDK embedded.
My personal opinion is they're not hard to detect at all. Latency is a huge giveaway, If the client can't respond in a time you'd expect a client to be able to based on its geographic location, that's a pretty big tell they're using proxies. In fact, if you did latency sensitive stuff, you'd likely find out whether you wanted to or not.
That's assuming they're tunneling the entire connection and not terminating it locally and then forwarding the data after it's downloaded.
And also assuming that the only reason for higher latency is physical distance rather than crappy WiFi or corporate nanny filters or the client device swapping because the user can't afford more RAM.
This is not always the case, although plausible, as residential proxies exit on a real residential IP geographically near the victim/target, so RTT looks normal most times and latency won't reveal them. The actual tells are elsewhere: ASN/IP reputation, TLS (JA3/JA4) fingerprint vs. claimed client, and session-behavior inconsistencies.
But the overall latency will be longer than expected because you add the latency from the proxy connection as well. If you're able to ping back to the connecting IP in combination with some clientside js to help measure total latency I think you should be able to reliably detect proxy users in general?
I don’t understand this being a “national security” threat.
Besides, mobile proxies exist which work differently. All you need is a mobile data plan that you run a proxy server on. It allows for easy IP rotation and since it’s a pool shared with other customers you cannot easily block it. This is because of things like CGNAT.
IP rotation is easy because reconnecting to the network gives you a new IP address.
Static residential proxies also are a thing, even if they are less effective sometimes.
If anyone is curious where this leads to, I will happily point to China, Kazakhstan and Russia as prime examples.
Can’t wait to have our very own Roskomnadzor MITMing everything and national security laws mandating Kazakhstan style TLS interception on every single device.
Just a small over the air update via your friendly neighbourhood corporations Microsoft, Apple and Google.
jayturley | 17 hours ago
macintux | 17 hours ago
Shank | 17 hours ago
> At the very least, major American ISPs (Comcast, AT&T) should detect clearly suspicious activity coming from customer IPs and warn them to scan their computers, check their TV apps, and find whatever is turning their internet into a proxy.
What are we saying? Really, what are we saying? We should turn domestic ISPs into domestic surveillance apparatuses to "detect clearly suspicious activity"? What is "clearly suspicious activity"? Section 230 is still law.
cute_boi | 17 hours ago
Governments could even purchase residential proxies themselves and check whether they're being used for malicious activity.
It isn't that difficult, but I guess governments just aren't that interested.
saghm | 16 hours ago
> It isn't that difficult, but I guess governments just aren't that interested.
I recently switched from my previous ISP because they randomly broke my ability to use my own router, and during the time when I had to default to using their own modem/router hardware before I could get the new ISP to come and set things up, I could barely even get a signal in my office upstairs (which had previously been connected via a mesh endpoint) because their router didn't expose any way for me to split 5 GHz and 2.4 GHz, and the router absolutely refused to let my devices connect via 2.4 GHz despite them having more than 90% packet loss due to the weak 5 GHz signal.
Regardless of how "easy" it is, I don't trust ISPs not to screw it up somehow and probably cause a lot more concrete damage (even if the individual issues they cause are smaller in magnitude) than the theoretical concerns of "national security" that, as far as I can tell from reading this thread, have caused a total of like a few hours of downtime one day in a couple decades.
advisedwang | 17 hours ago
The interviewee seems pretty inept yet still managed to uncover something really interesting and nefarious.
yellow_lead | 17 hours ago
> Actual data and identity loss of American citizens.
> Malware that can record video and audio from infected devices.
> Infrastructure for foreign covert influence campaigns.
> Botnets used in hacking and DoS attacks
These things have existed for 20+ years. Bad but not exactly a national security threat.
cute_boi | 17 hours ago
himata4113 | 17 hours ago
People were dismissive of security far too long. I think it's actually good that AI instills some fear into people causing it to take a lot more seriously than they have previously done so.
squigz | 16 hours ago
basilikum | 17 hours ago
anabis | 17 hours ago
DANmode | 17 hours ago
BLKNSLVR | 15 hours ago
The nation's security infra is a reflection of the nation's security legislation and regulation.
Aside: TheSInIoT is my DMZ's wifi password.
mindslight | 17 hours ago
bofadeez | 17 hours ago
GolfPopper | 15 hours ago
brikym | 16 hours ago
BLKNSLVR | 16 hours ago
It strongly depends on how scraped data is being used.
If your 'cat' hasn't been able to catch their 'mouse' then the cat needs to get smarter, or look for alternative sources of mice, or the cat should be considered 'unviable'.
Have you approached them to get access to their data? Have you explained to them how your service can benefit their business?
(I generally come from a position of suspicion as to why someone wants to scrape data that the owner goes to certain lengths to protect, but then I'm also an 'information wants to be free' kinda person, but the Internet is increasingly an untrustworthy place, so security is overruling narrative).
harimau777 | 16 hours ago
BLKNSLVR | 16 hours ago
If it has to be scraped then there may be other problems (which include lack of resources to make the data API accessible).
alightsoul | 16 hours ago
BLKNSLVR | 16 hours ago
That's cutting the tether from a _lot_ (there's no way to overstate this) of useful information, but that's potentially one of the great things about the open AI/LLM models, is that all that info is baked in there.
Disclaimer: As far as I understand it. Please educate me if I'm way off the mark.
Also, how much of the content of reddit is in Common Crawl? (same disclaimer applies to this comment)
I also understand 'the archival mindset', I hoard a bunch of data. But I also understand the logarithmic graph of futility.
AnthonyMouse | 15 hours ago
Let's try this for example. Suppose you want to create a price comparison site.
A lot of the major retailers don't want these, because they want the customer going to their site when they want to buy something, not to the price comparison site that tells the customer which retailer has the best price and then might not always be them.
If the comparison site doesn't have the prices from retailers who sometimes have the best price then it can't serve its function -- customers still have to check sites manually. And comparison sites are pro-consumer whereas blocking them is anti-competitive, so the ones doing the scraping are the good guys.
BLKNSLVR | 15 hours ago
The I guess the site is either useless, or can be used in combination with customer's own research that includes sites not included in the comparison. The site is still useful, it's just not exhaustive. There's nothing much in the world that's exhaustive, it's all a range of percentages.
Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Everyone seems to want _everything_ despite _everything_ not being necessary to provide a service. The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Just because residential proxies are somewhat of an 'easy answer' doesn't mean they're right. Use more creativity and imagination! Or follow a new idea. It's not like price comparison sites are a revolutionary idea.
(I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic).
> so the ones doing the scraping are the good guys.
Everyone thinks they're the good guys. If there's profit involved then there's always an element of delusion.
AnthonyMouse | 13 hours ago
There is a significant difference between having the prices for 85% of retailers because you haven't figured out how to get pricing from 15% of them and having the prices for 5% of retailers because the biggest ones block you from getting their prices. The amount of work you can save the consumer, and thereby the usefulness of the service to them, changes by a dramatic amount.
> Also, maybe the source that isn't scrape-able becomes less popular as a result of not being included in the price comparison list. If they're always more expensive, then neither you nor they have any advantage in listing on your site. If they're always cheaper, then your site may have no purpose to serve.
Now consider the possibility that they sometimes have the best price and sometimes don't.
> The lack of a certain set of data may itself be a data point, and should be used as marketing material for or against the company resisting the scraping.
Suppose you go to the price comparison site to look for the best price, the best price in its data set is $22, then you go to Amazon and they have it for $19 because it doesn't have their prices. The customer then starts checking Amazon in addition to the price comparison site. Their price is only actually lower 15% of the time, but another 75% of the time it's exactly the same, so the customers who don't want to keep checking two sites start checking only Amazon instead of only the price comparison site, at which point Amazon gets to charge them a higher price on 10% of stuff, or a higher price on more than 10% once people have been trained not to check. And prevent them from patronizing a random competitor when their price is exactly the same
Causing that to happen is the reason they don't want their prices in the comparison site. If not being listed there hurt them then they wouldn't be trying to prevent scraping.
> I'm a consumer who happily does significant research before a buying decision, going to various sites and checking prices, warranties, model numbers, reviews, availability, delivery times, and all the shit. And prefer doing the research myself than trusting a price comparison site, so I'm not the target market, thus my bias in my commentary and opinion on that topic
Consider also that you may be able to justify doing this when buying electronics, or choosing which brand of something to buy on a recurring basis, but if you just want the best price on the product you already know you want, it's crazy to spend hours to save cents. But completely sensible to switch to a search box that would save you 10 cents on every $2 purchase by giving you price-sorted results from multiple retailers who all sell the same products, if it's allowed to exist.
indianmouse | 16 hours ago
BLKNSLVR | 16 hours ago
Do you allow the data you've scraped to be scraped? Do you share it as freely as you desire the 'companies that want to hide it' would? Or do you consider the scraped data is 'hard earned reward for effort' and therefore has value that others should subscribe to your service for?
fnord77 | 15 hours ago
akoboldfrying | 15 hours ago
fnord77 | 15 hours ago
alexspring | 16 hours ago
iammrpayments | 16 hours ago
GolfPopper | 15 hours ago
abofh | 16 hours ago
bharatsuthar | 16 hours ago
BLKNSLVR | 16 hours ago
ietcd | 16 hours ago
bharatsuthar | 16 hours ago
faangguyindia | 16 hours ago
Sometimes their system malfunctions (cough cough) and you get charged for those clicks but you fail to report them as fraudulent as you've no proof as the IPs and useragent all appear normal to ad agencies and advertisers.
armchairhacker | 16 hours ago
More importantly, a residential proxy can be mutual. If a group of people agree to forward traffic to each others’ IP, how is it not their business? It obfuscates their identities, but this is the Internet, not e.g. a government office or test center where you obviously can’t walk in with someone else’s ID.
Banning non-consensual proxies makes sense since those are effectively malware, even though they make anonymity harder, so does banning theft and here you’re stealing someone’s internet. I’m sure we can convince enough laymen to knowingly install proxies by paying them.
BLKNSLVR | 16 hours ago
Myself being "not a layman" would definitely not allow anonymous usage of Internet that's tied to my home/residence/identity to be used by some internet rando just because they pay me money.
I would support, however, small family/friends groups who know and trust each other well enough to not do anything that'll cause a police raid on each others homes (having endured a police raid and the ensuing 8 months of not being told anything about the progress of whatever they're doing, it's not something I'd wish on a stranger, never mind a friend/family member).
armchairhacker | 13 hours ago
Sure, the service should be upfront, although most laymen probably won't read. But if people had to really understand consequences before they accepted anything, most would miss out on most overall mutual offers. Importantly, the expected consequences are low (as long as the service isn't greedy), and the ratio of tech people agreeing would suggest that if most laymen did understand they would still.
BLKNSLVR | 12 hours ago
Do we have solid numbers for that?
charcircuit | 16 hours ago
LastTrain | 16 hours ago
charcircuit | 16 hours ago
No one is being hurt by someone sharing their internet with me a few times a week.
LastTrain | 7 hours ago
BLKNSLVR | 14 hours ago
Separately, would you let your home/personal connection be used as a residential proxy?
Personally, I would not, except if it was for a friend / family member, but I would still ask them, pointedly, why my internet connection is required rather than their own.
charcircuit | 12 hours ago
BLKNSLVR | 12 hours ago
fragmede | 11 hours ago
zdc1 | 16 hours ago
BLKNSLVR | 16 hours ago
Just because the idea of net neutrality exists doesn't mean it's true. I'm sure there's a specific context for it, and 'security' is not that context. It was about data/packet prioritisation wasn't it? Unrelated to security.
bharatsuthar | 15 hours ago
AnthonyMouse | 10 hours ago
Trying to restrict things to end-users is the total opposite of that. The common addresses everybody gets automatically are the thing you're trying to allow. Address reputation is pointless because residential customers get dynamic IPs and the reputation you're trying to record for some IP address can get swapped with a different customer at any time.
BLKNSLVR | 9 hours ago
It also feels like a description of the perfect camouflage to facilitate doing bad things: "don't block them because you might block an innocent bystander". Putting innocent bystanders in harms way sounds like someone else is the bad guy, not the person doing the blocking.
AnthonyMouse | 2 hours ago
The problem being that attack traffic is disproportionately coming from devices that are compromised, which is already illegal, and you can't fix that by making it harder to use residential proxies for things that are legitimate, like sharing IP addresses between real users so they can't be used as a personal tracking ID.
> It also feels like a description of the perfect camouflage to facilitate doing bad things: "don't block them because you might block an innocent bystander".
Cloudflare promotes putting your site behind Cloudflare to inhibit censorship, because then the censors have to block their entire service (which is half the internet) to block anything. It's not always a bad thing.
> Putting innocent bystanders in harms way sounds like someone else is the bad guy, not the person doing the blocking.
If there is an alleged thief on the subway and you respond by lobbing a grenade into the subway, there is more than one bad guy.
indianmouse | 16 hours ago
Proxying is least of the worries!
When manufacturers bring out the smart features that require constant data collection, monitoring (obviously in the name of service quality, troubleshooting, firmware / app updates and subscription of services (!!)), it is oblivious to the fact that security and privacy at a personal / state / national level is never considered (for various reasons such as the design, profit interests and overheads / compliance perspectives and what not!) to be part of the ecosystem. Bad actors with sufficient knowledge and tools could exploit and profit from it.
It has become more of a planned obsolescence / profit / greed based industrial / corporate culture in rolling out unwanted stuff / monitoring and controlling as a feature that gets exploited to the core.
Honesntly, I don't have an answer for that, but have some thoughts that I wanted to share.
I do not want a cleaning robot or a water purifier or a printer or a TV or a refrigerator or an oven or a smart light bulb or a smart lock or even my car to have undisclosed, unwarranted connectivity to internet for whatever reasons.
I do not care if it is the manufacture or the service provider or whomsoever it may be!
I want all of the network connectivity options to be explained with all the controlling options and boundaries and data collection that happens on any of the connected medium to be as much transparent with the option of preventing or restricting what one does not want to go out of the device.
Will anyone do that?
It ill never happen! Sadly!
This problem will balloon further without adequate controls and killing the networking at a ground level would only be the only solution!
But, in the case of a connected device having an inbuilt connectivity option (like the m2m based options) in a smart vehicle, even that is not possible!
userbinator | 16 hours ago
DocTomoe | 15 hours ago
They are basically 'won't someone please think of the children?', but with higher stakes.
rf15 | 14 hours ago
snapplebobapple | 13 hours ago
nephihaha | 8 hours ago
BLKNSLVR | 15 hours ago
Specific Domains:
Wildcard domains: Source: https://blog.includesecurity.com/2026/06/the-smart-tv-in-you...bharatsuthar | 15 hours ago
The overall residential proxy market is too large and often undetected by intelligence tools. BrightData is actually much more compliant and malicious actors wouldn't be allowed access. They have an extensive KYC and use-case vetting process.
chhxdjsj | 14 hours ago
bharatsuthar | 13 hours ago
chhxdjsj | 12 hours ago
BLKNSLVR | 8 hours ago
> you need a video call and KYC to get access to a high quality IP pool
They need to know who to blame (cough come after cough) if their top-shelf stock gets tainted by some script kiddie.
BLKNSLVR | 12 hours ago
So a pretty good first step in blocking then.
Terr_ | 15 hours ago
When it comes to involunary proxies, those are really just one of many possible symptoms stemming from a real problem: Shitty security. (Edit: And shitty contract/privacy laws.)
Shitty security is tolerated by our markets, is is protected from fixes due to copyright law, and it is even encouraged by parts of our government that want to exploit the flaws. We will gain far more from fundamental quality-improvements than we will from whack-a-mole-ing on this symptom.
BLKNSLVR | 14 hours ago
> We will gain far more from fundamental quality-improvements than we will from whack-a-mole-ing on this symptom.
100%, but I feel like that's both true and rarely executed upon in most governments for most topics.
quotemstr | 15 hours ago
Yes, yes, we should fight deceptive software that doesn't disclose it has a proxy function. Fine. But there's no way to square 1) fighting all residential proxies, including ones run with informed consent and 2) preserving the public's access to general-purpose compute at home.
I'd rather live with the proxies than for someone to tell me that if I run squid, I'm damaging national security.
ButlerianJihad | 15 hours ago
quotemstr | 15 hours ago
bsder | 15 hours ago
Continuing to allow CGNAT is what allows the residential proxies to hide because it tumbles the IP identity of bad actors with normal people.
AnthonyMouse | 11 hours ago
ButlerianJihad | 8 hours ago
Chu4eeno | 2 hours ago
torginus | 15 hours ago
AnthonyMouse | 11 hours ago
And also assuming that the only reason for higher latency is physical distance rather than crappy WiFi or corporate nanny filters or the client device swapping because the user can't afford more RAM.
simoneree | 11 hours ago
Chu4eeno | 2 hours ago
j027 | 14 hours ago
Besides, mobile proxies exist which work differently. All you need is a mobile data plan that you run a proxy server on. It allows for easy IP rotation and since it’s a pool shared with other customers you cannot easily block it. This is because of things like CGNAT.
IP rotation is easy because reconnecting to the network gives you a new IP address.
Static residential proxies also are a thing, even if they are less effective sometimes.
cloudie78 | 14 hours ago
Can’t wait to have our very own Roskomnadzor MITMing everything and national security laws mandating Kazakhstan style TLS interception on every single device.
Just a small over the air update via your friendly neighbourhood corporations Microsoft, Apple and Google.
You know. To keep you safe.
grigio | 13 hours ago
blini-kot | 8 hours ago
lets just hope some day there will be enough threats for nations to crumble