Has anyone tried to get GenAI to build Cyberattacks based on fooling humans via cognitive biases etc?
I mean it seems like prompt engineering really good suggestions for non technical (i.e. psyops) would be a really good test, especially for agentic AI systems, to see if they had even a half baked theory of mind...
don't say you havn't been warmed...
for entities to have autonomy (i.e. be agentic) they must operate in some sort of decision space that is disjoint from the other entities with which they interact - conversely, if their decisions are fully specified by those other entities, then they are by definition not autonomous..
People use hashes of biometric data embedded in identifier (digital documents) so then the verifier can re-acquire the biometric and (hopefully, in a privacy preserving way) verify the person standing in front of the camera is a) live and b) the same as the one in the document - this is so 20th century
Why not have a behaviour that actually relates to the verifier's actual domain requirements - say they want to check this person is allowed to drive - perhaps for renting a car - so they could measure the person's driving style, which could also be stored in their digital driving license shard of their identity. This could be very robust - graduatlly improves in uniqueness -- and also stops people re-linking across different domains when people use the same feature/behaviours for multiple roles (face, being a common example) - we can also incorporate increasingly robust defenses against GenAI building deepfake people (via deepfake 3D faces or voices) -
The verifier would then be an agentic AI which basically has two things, a measurement model (classic ML) and a physics model (of what people can do) - so now we have Ф-ML-Id...
[lots of good background on Ф-M from the Turing's interest group]
The more borng application is in multimodal biometric systems tat use face+voice+lipsynch on given phrases to work out who someone is and that they are alive- same thing - physics model predicts face movement and voice audio given text input, then verifies against previously onboarded voice&Ф-M 3D face model.
In facial reconstruction (e.g. taking oliver cromwell's skull and building up a picture of what he actually looked like in person) there's a model of how flesh covers bone - this includes models of the way the stuff we are made of folds/hangs/ages etc - these models are from first principles. Unlike deep learning systems trwained on lots of labelled data to classify faces and extract features (eyes nose mouth etc) purely based on statistics and luck, these physics models are correct because that's the way the universe works. You can build hybrid, so called Ф-ML systems, which give you benefits of both the statistical approach, and the explainability of the physics model - the cool recent work also shows that the physics model lets you reduce the amount of data necessary for the statistical model - sometimes by 2 or 3 orders of magnitude, and retain the accuracy and recall of the stats model.
In the world of biometric id, there are requirements from applications (many use cases are with a human waiting for verification of some attribute) that mean you want fast, accurate and efficient models that can run on affordable devices in tolerable time.
You also want to be future proof against deep fake, and also against adversaries with access to complex system wide attacks.
I claim that having these underlying real world explanatory component, alongside the statisitcally acquired twin, will be more resilient, and might even let you cope with things like kids and aging and other changes, as well as allowing you to verify attributes other than the standard biometrics of face, fingerprint, iris etc, in robust ways which provide better domain specific data minimisation.
with apologies to ken thompson
I keep hearing a lot about how decentrlised systems can solve the massive loss of trust we are witnessing in large scale central organisations (technological, like hyperscale companies, and social. like national governements, and even economic, like banks).
I call 100% bullshit on this
(said in Natasha Lyonne's great voice, maybe we could ask her to do an audio book of Frankfurt's book on bullshit, or even graeber's book on bullshit jobs).
not that many people havn't lost trust in central agencies. that's been a factor of life for millenia - probably amongst many creatures, not just humans - i imagine the first deer to get hunted by humans suddenly found another threat to add to their collection. and tribes and nations invaded suddenly, or betrayed by "friends", "family", states in treaties etc. or banks going broke. or rivers and mines drying up.
sure, central things can lose trust, and, as the cliche has it, they will find it much harder to regain.
(though notice, people don't say often that it will never get regained, just that it is a lot harder to gain than to lose).
so what about decentralised systems and trust?
well, the answer is built into that oxymoron. a "system" isn't decentralised. it represents, at minimum, an agreement to follow a set of rules. Who makes that agreement? Answer, the participants. They buy into the API, the protocol, the rules of engamgent and behaviour. And they can renege on it. They can stop obeying the law, they can start a run on the bank, they can over-graze the commons, they can get together and gang up on a smaller group, they can go amok, or form a mob, or start a revolution.
At some point the decentralized system must have some central component, even if it is virtual.
So why would I trust such a system more than a central system? I don't think I should. The problem is that there are no coordination points which means if I am in that minority, even just one person being ostracized, I have no redress, no recourse to recompense. There's no resource I can point to, to offset the misuse of decentralized power by the unruly mob.
In syndics (socially anarchic systems), there are meta-rules of engagement that are supposed to mitigate misbehaviour. for example, you are not supposed to engage with people (nodes in the net) with whom you have no overlapping interests (i.e. no resource conflict, no need to enage). If you do, then the metarule makes that mishaviour in everyone's interest. Now there should be a people's court to try the misbehavioung group and decide on a suitable redress (which might be ostracism) - sounds tricky? yup. did it ever work? Maybe for a short time in Catalonia a long long time ago.
How would that work in a distributed communications (or energy) system? Not well if you ask me. we only have one "space" for comms, and we only have one planet. There's got to be some root of trust (or multiple roots is fine), so you can anchor redress (for example).
Of course, you can build a hierarchy, which at the leaves looks like a decentralised system, but really, what you have is federated.
The cliche of the tech world is to trot out the infamous Gartner Hype Cycle and no-where is this more prevalent than in AI, post Chat-GPT (to be fair, post "Attention is all you need").
But the other slightly less worn hype is the phase that embodied (or perhaps even virtual) AI is said to go through, which is the Uncanny Valley
So where do these two curves collide, eh? Right about now...
I'm pretty sure this sort of thing happens because of bad parenting - people that don't trust anything come up with the idea of distributed systems that have no need of any anchor for trust anywhere.
so i have lots of problems with this - starting from systems - and the classic ken thompson reflections on trusting trust. These deecntealization extremists have to confront that they run software on hardware, and even if they build their own hardware and write their own software, they probably use an OS and a Compiler from somewhere else. However, it gets worse. Why should we trust the actual zero knowledge protocols they use? who has verified them, and how? why do we trust those verfication tools (peoples' brains too). And worse still. Why should we trust this new fangled idea zero. The Romans and Greeks and ancient mesopotamians got along fine without it.
No. I have zero trust in zero trust.
and a Singer 66K sewing machine from 1917AI as TTPs is a recent posting by Bruce Schneier who has impeccable security credentials.
However, I'm not convinced that the paper he is highlighting is as groundbreaking as he is.
The authors of the paper also have great track records and include AI, but I think they're missing something basic that means that a single or group of TCMEs ("Trusted Capable Model Environment") can't actually do anything different than any other computation, subject to basic privacy controls (e.g. access control authorisation, auditing, encryption of data at rest, in transit, during computation (e.g. using FHE and TEEs etc etc).
But also:
a) visible communication in/out of the computation - i.e. information flow control
b) control over specificity of that data (i.e. differential privacy - can you tell if an individual record is present or not, to put it. crudely)
c) secure multiparty computations and zero knowledge systems
which the paper compares and contrast with their new TCME notion. However, I think the dimensions they use for comparison are a bit of a stretch.
The main problem I think is that the TCME seems to be indistinguishable from any other trusted program.
Any shared secret between models (e.g. federated or decentralised learning) is just the same for AI/ML as for any other algorithm. Perhaps the intersection of probability distributions looks a bit different to juse being able to say "the richest person is A" without knowing how rich A (B, or C) actually is - but in the end, the distribution has some moments and can be described by some number of those more or less precisely - a distribution of distributions can be aggregated with more or less precision or uncertainty (e.g. respecting differential privacy, and some widest level, or preventing set membership inference at the finest grain) - the model itself can be protected from outside model inversion attacks by various schemes, but I don't see what TTP function is provided that isn't just a different mix of existing techniques for providing trust.
I asked my tame AI:
I was reminded of two things from the late 70s that a friend in cambridge did, and my cousin in london were doing at the start of their graduate research work.
Change for the Machines (with apologies to Pat Cadigan)
AI was about models where money equals compute equals big data equals valuations.
So all the money going in was to finance compute, thinking where the value lies,
and every valuation of every company was about how much compute they had so it was all fake.
Companies valuations were just how many H100s they had (compute capacity),
as if it correlates to better models (even though they’re usually just wrappers.
DeepSeek, and other Chinese models broke that which pissed everyone off, private and public investors
included, because it casted doubt on the valuation methodologies, namely that energy, compute, data centres and number of chips were essentially fixed costs and the valuation of companies (and their output) could be measured on that alone.
Basically casted doubt on the last two years of public and private, markets not just of AI companies,
but the entire stack, energy, chips, data centres etc. Everyone felt like a dummy. Even though it’s
been happening for a while. We knew this a couple of years earlier when Meta released Llama, and it was clear that much smaller models could be trained at much lower costs and yet achieve many of the same goals. In that case, it was better software engineering in the open source community. Perhaps being open sourced (despite origins in a hyperscale company) it attracted less attention, although perhaps the google memo "we have no moat"
should have been a clue.
https://www.theverge.com/2023/7/10/23790132/google-memo-moat-ai-leak-demis-hassabis
One of the ironies that the DeepSeek debacle also exacerbates is that one constraint on them that made them seek greater efficiency was export restrictions on higher end GPU - as with the open source research community, less is more. That constraint was already what drove people in the open source (often academic or hobbyist) community
to develop affordable ATI technologies. In fact, outside of the LLM/GenAI world, many machine learning tools have been proving themselves perfectly useful and usable running on laptops on large datasets ("big data").
Denials at the time came thick and fast, perhaps because the huge investment in the new emperors
was not ready to be disrobed. Perhaps, also, as if OpenAI et al were deliberately trying to create artificially
high barriers to entry to their tech market. For investors actually interested in innovation, this is
ironic given the entire direction of travel of much computing related tech has been to lower barriers
so that innovation drives things with as low friction as possible (internet, cloud, processors, compilers, operating systems, SDKs/Appstores etc etc).
So up to and including future chip design, and certainly things like edge compute,
federated machine learning, and of course, all things decentralised...
ad dare one also say warfare - cyber, and hybrid war has even increased the asymmetry
in cost of effective weapons.... the military example is a very important one often overlooked and i think a large part of the world is scrambling to figure that out, same thing with cyber attacks too...
However its more than artificially high barriers to entry its also creating artificial or at least inflated markets because money goes out as investment and back in as infrastructure (think Microsoft investment or NVIDIA investment) when they aren't needed with a faulty way of valuing all the assets.
What's best for innovation, and what happens usually in innovation?
We would think they'd learn the lesson, barriers always lower, things get commoditised, and things get cheaper and easier. This is not always just second system or indeed, third version syndrome - some better understanding of the domain can lead to major efficiencies, and sometimes they arrive combined with other useful innovations - one example arose from work in explainable AI (XAI) where tooling to uncover what structures within a neural network ("deep learning") were responsible for detecting/recognising which input features (and hence classifying an input in some manner) - these tools for explainability also allow one to shrink the neutral network significantly by discarding nodes/edges that serve no useful classifier function - this has been used in face recognition in camera phones to make smaller, faster, and actually potentially more accurate AIs. The cost in training increases somewhat, but the payoff is that the cost in inference (done billions of times rather than just "once") is massively reduced. In some AI models that approach can actually be used during training to reduce training cost too. So an innovation in one space driven by a required feature (explainability) leads to efficiency gains too.
Another angle on this has been the use of physics models (in weather prediction and heavy engineering) combined with neural nets - there's a mutual benefit in reducing the computation costs of computing the physics model, and in optimising the neural network itself -recent advances (e.g. the Aardvaak weather predictor - see https://arxiv.org/abs/2404.00411) actually move the partial differential equations into approximations in the neural net (neural operations for the PDE) gaining huge efficiency, but retaining the fundamental explainability of the original physics. Applying the same technique to continual updates to the models from real world inputs is another huge win.
Profligacy gets in the way of such giant steps.
now the cambridge physics dept have their Dyson Sphere ready to move into (kind of, ok so it isnt an actual sphere) the old building should be ready for its next role - this should be as a vehicle to get the rabbits (who have moved from their burrows around the ponds into the buildings vacated) to a new home around a friendly exoplanet - I am sure some of the astronomers could have with the celestial navigation..
spindizzy engines are two a penny. -we just need to vaccum proof the buildings - a supply of saran wrap and gaffer tape will do.
and an HMV wind up record player from the 1930s....