You are completely mad if you are running any kind of valuable workload on Runpod community or Vast. Hosts are absolutely peeking inside the VM's (you can see it openly discussed in their Discords).
There's got to be a ton of valuable workloads that don't care much about secrecy.
Who cares if someones spying on the process of fine tuning a model to help us pick strawberries? Even if they could somehow find a competitor to sell it to - the competitor wouldn't be interested because the risk of violating trade secret law is surely greater than whatever they can glean from our training run...
That's a common misunderstanding. If you learn of a trade secret through improper means, you can be liable federally under the DTSA or similar state statutes like CUTSA in California.
These were the statutes cited in the Anthony Levandowski lawsuit 9yrs back, where Uber settled for a 0.34% equity stake in their company.
If you give away trade secrets by intentionally giving them to another party without an explicit agreement of secrecy no statute in the world will protect it.
You have to take “reasonable” measures to protect trade secrets. Uploading them to a service without any confidentiality agreement or worse, where the provider officially has the right to use that data, defeats the definition of a reasonable measure.
Sure, but if you have the option of training your strawberry picking model for 1/5th of the price knowing that you will be spied upon, that's probably still a good deal.
Just like you can find all of my 3D models online because not being able to make them private is the price for the free license of Onshape, and I'm happy to let you look at those models rather than pay whatever a commercial license would cost.
Exactly. I've recently been training a number of models, and the value to any potential competitor would be approximately zero.
It's so integrated with the specific inputs of my product, and the outputs that it needs, that the model just has no value to anyone else.
If a competitor needed such a model, they need to train their own that matches the form of their inputs and outputs, which is just going to be different.
As long as your training data itself isn't sensitive. If that's e.g. photos of products and you're training a quality assurance photo recognition software, that's probably fine, if it's unredacted support tickets, not fine.
I used to work at a company who used to do this, but I'd argue that it almost never does what you want re: concealing the input space. There are so many ways to recover the input space from the latents, and often they are constructed expressly to contain exactly enough information to do precisely that.
But how much effort would that require? ... is what I was going to say but in this age of "claude reverse engineer this doohickey" maybe the better question is how much compute and how many tokens it would require.
> a competitor to sell it to - the competitor wouldn't be interested because the risk of violating trade secret law is surely greater than whatever they can glean from our training run
IANAL, but it seems to me that to have something as a trade secret usually requires showing that the secret keeper put in significant effort to keep the thing a secret, usually there being some contract in place that defines the domain of knowledge that is considered a secret if the knowledge (i.e. models) need to be shared.
Uploading the models to an unscrupulous hosting provider where there was only some boilerplate terms of services agreed to.. would not seem to be that.
Being careless in how you handle your corporate data will be evidence against your trade secret claims. You don't just get to claim anything as a trade secret, you need to actively go about trying to keep it a secret. Going with vendors who you know or had reason to believe were going to mishandle your "trade secrets" likely makes them no longer trade secrets.
Coca-Cola can't go dropping leaflets around with the recipe for Coca-Cola Classic and then claim the recipe is a trade secret.
IANAL. Talk to your own lawyer before you mishandle your data.
GPUs set to lower power states and CPUs with frequency scaling turned down. That's the biggest hit and miss for what I'm doing. Honestly don't care if people are looking at my workloads as it's just processing public data. But do care if it's someone's laptop that's actively using 1/2 the vram.
each machine's power limit and performance are displayed and filterable on vast. not sure about other marketplaces but vast is pretty transparent about the measured specs of their machines
It might have been fixed, but I had a problem a few months ago where I would get 2x5090s, but they were limited at 80% usual clock. Nothing was different on the rental card.
I haven't seen any Discord discussions, but the nature of the platform guarantees there will be leakage third parties. RunPod consists of random people renting out their own GPUs for use. They aren't owned by RunPod.
In any system where large numbers of people are renting out their homelab equipment, It's to be expected that there is no consistent data privacy policy. I can't even imagine trying to enforce some sort of system that would guarantee privacy of workloads.
I hold the inverse risk assessment- I think acting on a peek risk is much higher with larger operations such as OpenAi or eg Amazon -as they've allegedly done the same on their product marketplace and deciding on generic products, in addition to open source-alike service offerings.
Larger entities have the resources, the inclination and the protective legal teams to act much differently than a resource tight n-tier platform operator
Large firms have a much harder time hiding that kind of behavior, and they have much more to lose.
Large firms require processes and controls around everything. If Amazon is peeking into corporate customers' data, then there have to be mechanisms designed to allow that, a whole bunch of people know, it's part of engineering designs, and a whistleblower is going to leak it to the press.
And then Amazon is going to lose all of its business customers once its privacy guarantees were shown to be intentionally circumvented.
It is simply not economically rational for a company like Amazon or OpenAI to look at the data of its corporate customers.
Regular consumers are a different story, because you need to respond to abuse, illegal activity, etc. So they don't have the same privacy guarantees.
But corporate customers, that stuff is watertight. No cloud provider is going to risk losing their entire business because they violated their privacy guarantees.
> If Amazon is peeking into corporate customers' data, then there have to be mechanisms designed to allow that, a whole bunch of people know, it's part of engineering designs, and a whistleblower is going to leak it to the press
An awful lot of this sort of thing in the corporate world is based on pinky-promises and legal threats, more than on robust technical controls.
Every production system I have ever seen has at least one set of dev-accessible keys that has access to the production DB. Mature systems at least had audit logging around who retrieved those keys, less mature systems often didn't have any kind of internal access logs.
Is "Amazon the corporate entity" snooping on your data? Absolutely not. Is someone with Amazon internal access snooping on your data? You'll never know either way.
Given that even non-Amazon vendors can use Amazon.com search features to discover bestselling products from third-party sellers in various categories, I doubt there's anything contractually prohibiting Amazon from using the same information to drive their own product mix.
Same goes for selling prices. Published retail prices are not trade secrets.
UpCloud's [0] spot instances sometimes match the pricing of Vast.ai instances. I mostly use Vast.ai if my dual 3090s can't handle it anymore for non private things and move to UpCloud spot instances if it needs more.
You are ignoring the real usecases here. Not everything is large scale, secretly, b2b enterprise applications. Instead, many consumer applications are fully fine with being out in the open.
As an example, midjourney use to just publish ever generated image to their website.
If you are just doing inference things are public/open from the start, you dont have to worry much about the stuff you are talking about here.
> As an example, midjourney use to just publish ever generated image to their website.
Which is kind of genius, if you think about it. Users can't get away with generating smut, CSAM, or copyrighted material, because it goes straight to the public record.
Why would anyone in the not un-nice parts of the world aged past the teens feel like others seeing the smut they've generated being something to get away with? The people not into discussing their sexual lives with you are highly unlikely to be comfortable bringing up even the existence of yours anyways, and the people you have shared interests with now just have a new springboard for curiosity and conversation. Sure, you have the Swedish municipality director that a couple of weeks ago chose to resign after it came to light that he didn't quit his onlyfans work after all. But even that was more about lack of openness, breach of trust in a trusted position, than the nature of the activities in and of themselves.
I don't know man. I agree that people probably shouldn't care, but the US figures sort of highly in discussions about the internet, and it's a place with pretty regressive attitudes towards that kind of thing.
There's a lot of use cases that are not "commercially sensitive" that would require data center level privacy. E.g. a good percentage of academic research and experimentation, long running "boilerplate processing" for monitoring logs for e.g. a bird watching system or large scale web crawling where the individual website itself is mostly public. There are also use cases where the data itself is useless without other data to correlate. E.g. many log analytics use cases, if you are missing the main database data itself then the logs have zero value when intercepted. There are also use cases where you are doing general industrial level hotdog/not hotdog classification where unit economics is the most important and there's no real value to privacy e.g. on a factory floor (though sabotage is a bigger risk here).
definitely, it prevents providers from interfering with your workloads as a renter. afaik only datacenter GPUs (Hopper/Blackwell) support CC and not every gpu platform has that enabled for their GPUs. One of the reasons why AWS/Azure/GCP etc can charge a premium compared to other GPU cloud platforms where trust and privacy are not always a given.
Honestly I think every one is peaking at everything, I don't want to be slanderous but I think even the more trusted cloud providers peak, though they do so more quietly and carefully.
I work at a big cloud provider and this is absolutely not happening on the regular. More and more safeguards have been put in place over the years to prevent that.
Vast.ai seems like a scam to me. I funded my account with a little bit of money ($25) and tried repeatedly to reserve an instance, but it would just hang: no response, just waiting to provision it.
Gave up after 4 attempts and wrote the $25 off as a cost of the learning experience.
Speaking of Runpod.io GPU availability, it is definitely increased in the past 6 months, getting 80Gb+ vram on demand takes few mins (not 5-6 hours like it used to be) at 1-3 usd an hour rate ... which made me thought a week ago - finally GPU bubble peak arrived
Another thought - i want to audit the actual usage of GPU like actual efficient inference time not billed one without relying on "trust me bro" billing. Especially on AWS. Had no trust issue with Runpod.io
Any effective inference audit ideas are appreciated
I tried the TensorDock, which now offers H100 "from $2.25/hr", two years ago. The SSD IO was terrible, maybe oversold. My training workers was stuck at loading the dataset and cannot fully utilize the GPU.
One thing that I'm not seeing mentioned very frequently is that it's really hard to actually get the GPUs that hyperscalers advertise depending on your region, use of cloud credits etc.
On Azure if you have credits (even hundreds of thousands of USD of credits through programmes like MSFT for startups) you just can't get any GPU quota in any region, even if you beg and plead with support reps. Even if you have cash it's often difficult to get any of the cheaper cards. You can forget about spot instances etc because the regions are essentially all at 100% capacity.
I've heard on other hyperscalers that it's also challenging.
This is super chilling for anything other than GenAI inference/training because if you have some more traditional workload that requires GPUs (traditional ML, other kinds of modelling) you're SOL.
And not just hyperscalers... even the neoclouds who profess to have fantastic availability and GPUS NOW just tell you to jog on. Though of course they do that after they've signed you up and promised preemptible instances :)
Oh god I dealt with this recently while training my own 1B model.
Runpod - mostly okay, except sometimes instances get dropped on smaller ones. Prices float around 3.20 mostly for H100s. Cheaper ones exist but are PCIe. Best DX out of the bunch honestly (well, AX, as my agent was the one dealing with it). Didn't end up doing the 1B run on it, but did 100M+ runs and some minor experimental ones. Great for agentic exploration, "RSI", minor training loops and research.
Vast.ai - a bit pricier for the same H100s and it didn't have any clusters. So I could at best split my workload across 2-3 different H100s and spend time syncing. Not worth it.
Akash - since I'm trying to stay in a reasonable budget, I decided to try Akash, which was the worst experience out of all. Cards would be "available" but would never respond to your request. After trying to contact support via Discord, I got dozens of fake support accounts adding me to scam discords and asking for my wallet key. Support answered after a while, but without a real answer.
Mithril.ai - Ended up using this one, while it seems quite startupy, the actual offer is nice, it was easy to get going and they offer quite cheap H100s, offering singles and clusters.
Tbh while Skypilot is great, the marketplace fragmentation is leaving room open for a openrouter-like compute provider.
Looking at the report, I didn't even know one could use CoreWeave without an enterprise account, seems it could be cheaper per hour than the same cards elsewhere. Might have saved me a few hundred bucks, thanks!
One interesting thing going is device fraud on the 3rd party market places - i.e changing the clock speed, dividing devices up and renting same one at same time and hoping rentees won’t notice, everything you can think of someone is trying.
lots of interesting dynamics in the preemptible (spot) markets as well. If you’re smart i’d say there is a >10x cost advantage available over hyperscalar OD pricing with basically no downside. All easy with agents now as well, so it will likely soon be competed away (you already see this with spot rates increasing significantly in last 3 months and converging to just below the od prices)
> dividing devices up and renting same one at same time and hoping rentees won’t notice
I don’t think it’s possible to overcommit a GPU rental, even if idle people are going to notice their weights being kicked out of VRAM since it takes minutes to load them back in.
jdc0589 | 23 hours ago
last time I was trying to run some workloads there the pricing of an h100 was largely irrelevant.
alienbaby | 23 hours ago
codemk8 | 23 hours ago
dpflan | 23 hours ago
covi | 21 hours ago
plantain | 23 hours ago
kirubakaran | 23 hours ago
smallmancontrov | 23 hours ago
tarpitt | 23 hours ago
gpm | 23 hours ago
Who cares if someones spying on the process of fine tuning a model to help us pick strawberries? Even if they could somehow find a competitor to sell it to - the competitor wouldn't be interested because the risk of violating trade secret law is surely greater than whatever they can glean from our training run...
elevation | 23 hours ago
There is law for patents. Trade secrets exist outside the realm of enforcement.
gpm | 22 hours ago
crazygringo | 22 hours ago
https://en.wikipedia.org/wiki/Trade_secret#United_States
polishTar | 22 hours ago
These were the statutes cited in the Anthony Levandowski lawsuit 9yrs back, where Uber settled for a 0.34% equity stake in their company.
close04 | 21 hours ago
You have to take “reasonable” measures to protect trade secrets. Uploading them to a service without any confidentiality agreement or worse, where the provider officially has the right to use that data, defeats the definition of a reasonable measure.
Aurornis | 22 hours ago
Privacy should be a default. It doesn’t matter what you’re doing.
It’s also unlikely they know what’s in the VMs until after they look at it.
tgsovlerkhgsel | 22 hours ago
Just like you can find all of my 3D models online because not being able to make them private is the price for the free license of Onshape, and I'm happy to let you look at those models rather than pay whatever a commercial license would cost.
crazygringo | 22 hours ago
It's so integrated with the specific inputs of my product, and the outputs that it needs, that the model just has no value to anyone else.
If a competitor needed such a model, they need to train their own that matches the form of their inputs and outputs, which is just going to be different.
tgsovlerkhgsel | 22 hours ago
woodson | 22 hours ago
jszymborski | 22 hours ago
fc417fc802 | 20 hours ago
behnamoh | 22 hours ago
Unless the competitor is in a country like China.
nyeah | 22 hours ago
But how likely is that?
Phemist | 22 hours ago
Uploading the models to an unscrupulous hosting provider where there was only some boilerplate terms of services agreed to.. would not seem to be that.
vel0city | 21 hours ago
Coca-Cola can't go dropping leaflets around with the recipe for Coca-Cola Classic and then claim the recipe is a trade secret.
IANAL. Talk to your own lawyer before you mishandle your data.
m00x | 23 hours ago
It's fine for comfyui workloads or simple LLM inference, but you lose both privacy and stability.
aunty_helen | 23 hours ago
guppy | 21 hours ago
m00x | 19 hours ago
Aurornis | 23 hours ago
Can you explain? I haven’t used Runpod and don’t have time to search through a Discord to figure out what someone is talking about.
epistasis | 22 hours ago
In any system where large numbers of people are renting out their homelab equipment, It's to be expected that there is no consistent data privacy policy. I can't even imagine trying to enforce some sort of system that would guarantee privacy of workloads.
dv_dt | 23 hours ago
Larger entities have the resources, the inclination and the protective legal teams to act much differently than a resource tight n-tier platform operator
crazygringo | 22 hours ago
Large firms have a much harder time hiding that kind of behavior, and they have much more to lose.
Large firms require processes and controls around everything. If Amazon is peeking into corporate customers' data, then there have to be mechanisms designed to allow that, a whole bunch of people know, it's part of engineering designs, and a whistleblower is going to leak it to the press.
And then Amazon is going to lose all of its business customers once its privacy guarantees were shown to be intentionally circumvented.
It is simply not economically rational for a company like Amazon or OpenAI to look at the data of its corporate customers.
Regular consumers are a different story, because you need to respond to abuse, illegal activity, etc. So they don't have the same privacy guarantees.
But corporate customers, that stuff is watertight. No cloud provider is going to risk losing their entire business because they violated their privacy guarantees.
swiftcoder | 22 hours ago
An awful lot of this sort of thing in the corporate world is based on pinky-promises and legal threats, more than on robust technical controls.
Every production system I have ever seen has at least one set of dev-accessible keys that has access to the production DB. Mature systems at least had audit logging around who retrieved those keys, less mature systems often didn't have any kind of internal access logs.
Is "Amazon the corporate entity" snooping on your data? Absolutely not. Is someone with Amazon internal access snooping on your data? You'll never know either way.
winddude | 22 hours ago
AWS has also strip mined open-source software, elasticsearch, redis, mongo, browser rendering tools, etc.
jasomill | 17 hours ago
Same goes for selling prices. Published retail prices are not trade secrets.
joshuaissac | 21 hours ago
According to this comment, they did have a mechanism and a process for that: https://news.ycombinator.com/item?id=23929959
Yet it seems to have had negligible impact on their ability to obtain and retain customers.
mmnfrdmcx | 23 hours ago
WildGreenLeave | 22 hours ago
[0]: https://upcloud.com/
giwook | 22 hours ago
stale2002 | 22 hours ago
As an example, midjourney use to just publish ever generated image to their website.
If you are just doing inference things are public/open from the start, you dont have to worry much about the stuff you are talking about here.
swiftcoder | 22 hours ago
Which is kind of genius, if you think about it. Users can't get away with generating smut, CSAM, or copyrighted material, because it goes straight to the public record.
simonra | 21 hours ago
swiftcoder | 21 hours ago
Onavo | 22 hours ago
bojangleslover | 22 hours ago
jam6099 | 7 hours ago
sixtyj | 22 hours ago
epistasis | 22 hours ago
I've used them a bit to prototype stuff where the data didn't matter.
nxobject | 20 hours ago
kodoman | 22 hours ago
falsemyrmidon | 22 hours ago
giwook | 21 hours ago
But it definitely does happen.
janalsncm | 22 hours ago
ProfessorZoom | 21 hours ago
zizzigy | 23 hours ago
qoez | 23 hours ago
uneekname | 23 hours ago
> What the 4.57× is not
> - InfiniBand
> - A datacentre SLA
> - Enterprise compliance, committed capacity...
ok
InsidiousWombat | 23 hours ago
gizmodo59 | 22 hours ago
refulgentis | 22 hours ago
fg137 | 23 hours ago
mlmonkey | 21 hours ago
Gave up after 4 attempts and wrote the $25 off as a cost of the learning experience.
ronsor | 9 hours ago
For example, ones in China simply never can pull images. I don't know how they got approved!
nico | 23 hours ago
I’ve run into many hosted GPUs with processes stuck, with outdated drivers/libraries, or poor Internet connections
buredoranna | 23 hours ago
[1] "Silicon Index". Built against pricing observations, including listed prices and transaction data.
[2] "Ornn Compute Price Index (OCPI)". Claiming to use only executed transactions.
Your mileage may vary, but I find these new indices to be quite compelling.
[1] https://www.silicondata.com/products/silicon-index
[2] https://data.ornn.com/
And yup, its not just how much these resources cost, but how much people are willing to pay for them.
(edit: formatting)
blobbers | 20 hours ago
ablooja | 19 hours ago
williamgators | 23 hours ago
tokepi | 23 hours ago
happyPersonR | 23 hours ago
matt3210 | 22 hours ago
pwython | 22 hours ago
adeptima | 22 hours ago
Another thought - i want to audit the actual usage of GPU like actual efficient inference time not billed one without relying on "trust me bro" billing. Especially on AWS. Had no trust issue with Runpod.io
Any effective inference audit ideas are appreciated
dyauspitr | 21 hours ago
covi | 21 hours ago
Lowest $1.90/hr Average $7.08/hr Median $7.91/hr
https://gpus.skypilot.co/?gpu=H100&count=8
gigatexal | 21 hours ago
Havoc | 21 hours ago
cvhc | 21 hours ago
noodlesUK | 20 hours ago
On Azure if you have credits (even hundreds of thousands of USD of credits through programmes like MSFT for startups) you just can't get any GPU quota in any region, even if you beg and plead with support reps. Even if you have cash it's often difficult to get any of the cheaper cards. You can forget about spot instances etc because the regions are essentially all at 100% capacity.
I've heard on other hyperscalers that it's also challenging.
This is super chilling for anything other than GenAI inference/training because if you have some more traditional workload that requires GPUs (traditional ML, other kinds of modelling) you're SOL.
imimayj1337 | 20 hours ago
hypendev | 20 hours ago
Runpod - mostly okay, except sometimes instances get dropped on smaller ones. Prices float around 3.20 mostly for H100s. Cheaper ones exist but are PCIe. Best DX out of the bunch honestly (well, AX, as my agent was the one dealing with it). Didn't end up doing the 1B run on it, but did 100M+ runs and some minor experimental ones. Great for agentic exploration, "RSI", minor training loops and research.
Vast.ai - a bit pricier for the same H100s and it didn't have any clusters. So I could at best split my workload across 2-3 different H100s and spend time syncing. Not worth it.
Akash - since I'm trying to stay in a reasonable budget, I decided to try Akash, which was the worst experience out of all. Cards would be "available" but would never respond to your request. After trying to contact support via Discord, I got dozens of fake support accounts adding me to scam discords and asking for my wallet key. Support answered after a while, but without a real answer.
Mithril.ai - Ended up using this one, while it seems quite startupy, the actual offer is nice, it was easy to get going and they offer quite cheap H100s, offering singles and clusters.
Tbh while Skypilot is great, the marketplace fragmentation is leaving room open for a openrouter-like compute provider.
eitally | 20 hours ago
https://newsletter.semianalysis.com/p/clustermax-30-the-indu...
hypendev | 20 hours ago
Looking at the report, I didn't even know one could use CoreWeave without an enterprise account, seems it could be cheaper per hour than the same cards elsewhere. Might have saved me a few hundred bucks, thanks!
itkovian_ | 18 hours ago
lots of interesting dynamics in the preemptible (spot) markets as well. If you’re smart i’d say there is a >10x cost advantage available over hyperscalar OD pricing with basically no downside. All easy with agents now as well, so it will likely soon be competed away (you already see this with spot rates increasing significantly in last 3 months and converging to just below the od prices)
nojs | 11 hours ago
I don’t think it’s possible to overcommit a GPU rental, even if idle people are going to notice their weights being kicked out of VRAM since it takes minutes to load them back in.