I don't understand the "Triadic Modulation Loop" block, does anyone else?
Also
> Competing interests: AA has a provisional patent application for the algorithm used in this paper.
Results in this paper look way too good, I guess we'll have to wait for peer reviews and replications to see if it's true.
When you stack transformers, don't you get meta-attention and higher mental states?
Can the anthropomorphic scam continue unchecked ? Apparently yes.
I did a chat with Gemini about the paper, and tldr is... * They introduce a loop at the beginning between Q, K, and V vectors (theoretically representing "question", "clues" and "hypothesis" of thinking) * This loop contains a non linearity (ReLU) * The loop is used to "pre select" relevant info * They then feed that into a light weight attention mechanism.
They claim OOM faster learning, and robustness acro domains. There's enough detail to probably do your own PuTorch implementation, though they haven't released code. The paper has been accepted into AMLDS2025. So peer reviewed.
At first blush, this sounds really exciting and if results hold up and are replicated, it could be huge.
This is, intuitively, a really exciting title. Looking forward to reading / seeing similar work.
I don't want to dismiss this outright but I'm skimming this paper and pretty skeptical of something that's from a single guy that doesn't appear peer reviewed, spends most of its time talking about actual biology, comes up with a "RELU6" (RELU but minimum value 6), and then pushes detailed review to a future paper.