Hey HN
We’re the core engineering team at Caudena (which is used globally by investigative and intelligence agencies, including: Europol, Interpol, BKA, DHS, IRS-CI, FBI, NPA and others), and we just released the technical details behind Prism - our real-time, in-memory C++ database for blockchain analysis.
To tackle the massive scale and complexity of blockchain data, we had to get creative with low-level engineering:
- We utilize barebone servers with 2TB RAM and 48 Cores.
- Implemented lock-free concurrent data structures
- Developed a custom memory management system
- Leveraging CPU-level vectorization
- Built a custom in-memory columnar/graph database from scratch
We’d love to AMA about:
- the engineering choices we made
- crazy optimizations that paid off
- pitfalls we hit
Ask us anything about scaling, memory trade-offs, building real-time analytics on immutable data, or the crypto-forensics space.
Looking forward to a great convo!
When implementing the lock-free stuff, was portability (across processors) a goal? If yes, did you have to deal with anything specific? Do you notice any difference in behavior of correct implementations when ran on different processors? How do you test for correctness of lock-free stuff?
EDIT: Oh and did you implement from scratch? Why not use eg. the RCU implementation from folly?
We built something very similar back in 2016, in the jvm with unsafe memory and garbage-free data structures to avoid GC pauses. The dynamic clustering is not too hard, are you able to dynamically undo a cluster when new information shows up?
Are you running separate instances per customer to separate the information they have access to?
Curious why you chose C++? Were there aspects of other languages/ecosystems like Rust that were lacking? Would choosing Rust be advantageous for blockchains that natively support it (like Solana)?
To be clear: I don't mean to imply you should have done it any other way. I'm interested mainly in gaps in existing ecosystems and whether popular suggestions to "deprecate C++ for memory safe languages" (like one made by Azure CTO years ago) are realistic.
> "Built a custom in-memory columnar/graph database from scratch"
This seems like an odd place to spend your resources. What do Prism's benchmarks look like vs Memgraph, KX kdb+, Apache Ignite, TigerGraph, etc.?
> barebone servers
You mean bare-metal servers?
If the FBI tells you wallet A and wallet B belong to the same actor, how do you use that information, so that they can see it on their view, without leaking it to Europol?
I didn't even know there were companies doing work in the "blockchain services" space. Kinda cool, tech begets tech, begets tech.
Love the C++ work, btw
is it true that XMR / monero is untraceable?
will you open source? very interesting.
Thank you for the AMA. A few initial questions:
- Would it be possible to open source your DB in the future? I think there are challenges in blockchain analysis (e.g. internal transactions) that goes beyond the specific DB.
- Having used Chainalysis and others, your product seems superior based on your presentation. Which blockchains do you support?
- Is there a "HN Code" to test Prism?
why you spyin on folks
This feels like griftware?
This should be a Show HN?
[flagged]
You really had to call it Prism (PRISM), didn't you?
It's great to see C++ resulting in orders of magnitude cost reduction anyway. Do you have more details on the various C++ tricks done for optimization?