First the Google Cloud outage [0], then the CA private key leak [1]...
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
I am reading this as a leak of every possible piece of metadata but not an actual leak of customer data. They have SQL logs which do contain customer data and there is a mention of some snapshots/imports.
(POV when storing 1PB in Oracle) when I saw the 10 TB in another post, I was like "wait, isn't this sold to scale out, we have 100 times the data here".
AWS private rates gives PingCAP ~47% off list price. Wonder which services got the biggest discounts? Unsurprisingly for a db company the cross-az traffic is 15% of their bill.
If true, this is next level incompetence from PingCap. Apparently the attackers set up a replication pipeline for a month?!
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
Man I recently executed a migration from Aurora to Planetscale. I seriously considered PingCap but decided against it because I thought they cloud pricing was too complex. I wonder if my personal information is in the leak.
From personal experience having worked with multiple "enterprise-grade" pentest vendors (including NCC), I think the entire pentest industry is overdue for a big shake-up.
> TiDB, the open-source distributed SQL database powered by PingCAP, today published the results of a rigorous source-code security assessment by NCC Group, one of the world’s leading cybersecurity consultancies.
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
That confused me too since I remember reading they are Rust proponents. Apparently their storage engine TiKV is written in Rust, and TiDB is the database built on top.
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
It sounds like their jira instance was truly the keys to the kingdom, given the creds _and_ tickets describing all the flaws needing fixing. A golden README.MD to escalate to their hearts content
With support from outside counsel and third-party forensic specialists, there is no evidence that data stored in TiDB was impacted. We keep on investigating and will provide updates on any new findings.
They have the fucking data in a tarball and Pingcap claims no data was impacted. This sounds like the bad blood story part 2 all over again.
rvz | a day ago
...and now this...
> Today we are making the PingCAP breach public. We possess the complete TiDB Cloud control plane, its billing and finance estate, its GCP storage estate and its AWS cost and billing estate, its Jira, its CRM, its analytics, its source code, and a four-year log of every connection every customer ever made to its product, from the IP addresses of their application servers.
This is an entire leak of ALL PingCap TiDB's customers AND data.
All of this in a single day. It is over.
[0] https://news.ycombinator.com/item?id=50163961
[1] https://news.ycombinator.com/item?id=50166579
gregwebs | 23 hours ago
hdjdnxjzhzhs | 23 hours ago
binsquare | 23 hours ago
Size is not a good indicator for content quality
gregwebs | 22 hours ago
fock | 11 hours ago
berenddeboer | 6 hours ago
qingcharles | 23 hours ago
arpinum | a day ago
perpetualpear | an hour ago
dangoodmanUT | a day ago
and how are people not talking about this more
dilyevsky | 21 hours ago
dherls | 23 hours ago
> We even stood up a replication feed of their own control-plane database using credentials committed in their own repositories — GCP service-account keys and an Azure service principal — and let their infrastructure deliver it to us. Every six hours, for the last twenty-two days of the campaign, from 8 September to 30 September, that pipeline faithfully re-exported every table of the operational heart of TiDB Cloud — tenants, users, API keys, cluster credentials, changefeeds, migrations — and pushed it out.
nemothekid | 23 hours ago
esafak | 22 hours ago
xKingfisher | 21 hours ago
[1]https://www.pingcap.com/press-release/tidb-ncc-group-securit...
kobieps | 11 hours ago
hintymad | 9 hours ago
It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.
somat | 31 minutes ago
"good opsec will save you from bad crypto. but the converse is not true, good crypto will not save you from bad opsec"
jacquesm | 3 hours ago
wannabe44 | 21 hours ago
But what matters is that it was all memory safe and had no data races.
boorbat | 14 hours ago
wannabe44 | 13 hours ago
jedisct1 | 17 hours ago
boorbat | 14 hours ago
squirrellous | 14 hours ago
thenthenthen | 2 hours ago
llama052 | 2 hours ago
I find it really baffling how much they are downplaying this incident on their statement.
”An external incident response firm reviewed PingCAP’s production systems and found no evidence that the attacker ran commands or maintained persistent access in the systems reviewed”
That does not appear to be true.
bombcar | 2 hours ago
zergrush | an hour ago
im not an expert but if I was an investor I'd try to find a buyer for my shares and run
aliasxneo | 2 hours ago
Holy hell, I've heard of this happening, but I've never personally seen it in my 10 years as an SRE. That's absolutely wild.
adamparsons | 2 hours ago
neya | an hour ago