Has anyone tried to get GenAI to build Cyberattacks based on fooling humans via cognitive biases etc?
I mean it seems like prompt engineering really good suggestions for non technical (i.e. psyops) would be a really good test, especially for agentic AI systems, to see if they had even a half baked theory of mind...
don't say you havn't been warmed...
for entities to have autonomy (i.e. be agentic) they must operate in some sort of decision space that is disjoint from the other entities with which they interact - conversely, if their decisions are fully specified by those other entities, then they are by definition not autonomous..
People use hashes of biometric data embedded in identifier (digital documents) so then the verifier can re-acquire the biometric and (hopefully, in a privacy preserving way) verify the person standing in front of the camera is a) live and b) the same as the one in the document - this is so 20th century
Why not have a behaviour that actually relates to the verifier's actual domain requirements - say they want to check this person is allowed to drive - perhaps for renting a car - so they could measure the person's driving style, which could also be stored in their digital driving license shard of their identity. This could be very robust - graduatlly improves in uniqueness -- and also stops people re-linking across different domains when people use the same feature/behaviours for multiple roles (face, being a common example) - we can also incorporate increasingly robust defenses against GenAI building deepfake people (via deepfake 3D faces or voices) -
The verifier would then be an agentic AI which basically has two things, a measurement model (classic ML) and a physics model (of what people can do) - so now we have Ф-ML-Id...
[lots of good background on Ф-M from the Turing's interest group]
The more borng application is in multimodal biometric systems tat use face+voice+lipsynch on given phrases to work out who someone is and that they are alive- same thing - physics model predicts face movement and voice audio given text input, then verifies against previously onboarded voice&Ф-M 3D face model.
In facial reconstruction (e.g. taking oliver cromwell's skull and building up a picture of what he actually looked like in person) there's a model of how flesh covers bone - this includes models of the way the stuff we are made of folds/hangs/ages etc - these models are from first principles. Unlike deep learning systems trwained on lots of labelled data to classify faces and extract features (eyes nose mouth etc) purely based on statistics and luck, these physics models are correct because that's the way the universe works. You can build hybrid, so called Ф-ML systems, which give you benefits of both the statistical approach, and the explainability of the physics model - the cool recent work also shows that the physics model lets you reduce the amount of data necessary for the statistical model - sometimes by 2 or 3 orders of magnitude, and retain the accuracy and recall of the stats model.
In the world of biometric id, there are requirements from applications (many use cases are with a human waiting for verification of some attribute) that mean you want fast, accurate and efficient models that can run on affordable devices in tolerable time.
You also want to be future proof against deep fake, and also against adversaries with access to complex system wide attacks.
I claim that having these underlying real world explanatory component, alongside the statisitcally acquired twin, will be more resilient, and might even let you cope with things like kids and aging and other changes, as well as allowing you to verify attributes other than the standard biometrics of face, fingerprint, iris etc, in robust ways which provide better domain specific data minimisation.
with apologies to ken thompson
I keep hearing a lot about how decentrlised systems can solve the massive loss of trust we are witnessing in large scale central organisations (technological, like hyperscale companies, and social. like national governements, and even economic, like banks).
I call 100% bullshit on this
(said in Natasha Lyonne's great voice, maybe we could ask her to do an audio book of Frankfurt's book on bullshit, or even graeber's book on bullshit jobs).
not that many people havn't lost trust in central agencies. that's been a factor of life for millenia - probably amongst many creatures, not just humans - i imagine the first deer to get hunted by humans suddenly found another threat to add to their collection. and tribes and nations invaded suddenly, or betrayed by "friends", "family", states in treaties etc. or banks going broke. or rivers and mines drying up.
sure, central things can lose trust, and, as the cliche has it, they will find it much harder to regain.
(though notice, people don't say often that it will never get regained, just that it is a lot harder to gain than to lose).
so what about decentralised systems and trust?
well, the answer is built into that oxymoron. a "system" isn't decentralised. it represents, at minimum, an agreement to follow a set of rules. Who makes that agreement? Answer, the participants. They buy into the API, the protocol, the rules of engamgent and behaviour. And they can renege on it. They can stop obeying the law, they can start a run on the bank, they can over-graze the commons, they can get together and gang up on a smaller group, they can go amok, or form a mob, or start a revolution.
At some point the decentralized system must have some central component, even if it is virtual.
So why would I trust such a system more than a central system? I don't think I should. The problem is that there are no coordination points which means if I am in that minority, even just one person being ostracized, I have no redress, no recourse to recompense. There's no resource I can point to, to offset the misuse of decentralized power by the unruly mob.
In syndics (socially anarchic systems), there are meta-rules of engagement that are supposed to mitigate misbehaviour. for example, you are not supposed to engage with people (nodes in the net) with whom you have no overlapping interests (i.e. no resource conflict, no need to enage). If you do, then the metarule makes that mishaviour in everyone's interest. Now there should be a people's court to try the misbehavioung group and decide on a suitable redress (which might be ostracism) - sounds tricky? yup. did it ever work? Maybe for a short time in Catalonia a long long time ago.
How would that work in a distributed communications (or energy) system? Not well if you ask me. we only have one "space" for comms, and we only have one planet. There's got to be some root of trust (or multiple roots is fine), so you can anchor redress (for example).
Of course, you can build a hierarchy, which at the leaves looks like a decentralised system, but really, what you have is federated.
The cliche of the tech world is to trot out the infamous Gartner Hype Cycle and no-where is this more prevalent than in AI, post Chat-GPT (to be fair, post "Attention is all you need").
But the other slightly less worn hype is the phase that embodied (or perhaps even virtual) AI is said to go through, which is the Uncanny Valley
So where do these two curves collide, eh? Right about now...
I'm pretty sure this sort of thing happens because of bad parenting - people that don't trust anything come up with the idea of distributed systems that have no need of any anchor for trust anywhere.
so i have lots of problems with this - starting from systems - and the classic ken thompson reflections on trusting trust. These deecntealization extremists have to confront that they run software on hardware, and even if they build their own hardware and write their own software, they probably use an OS and a Compiler from somewhere else. However, it gets worse. Why should we trust the actual zero knowledge protocols they use? who has verified them, and how? why do we trust those verfication tools (peoples' brains too). And worse still. Why should we trust this new fangled idea zero. The Romans and Greeks and ancient mesopotamians got along fine without it.
No. I have zero trust in zero trust.
and a Singer 66K sewing machine from 1917AI as TTPs is a recent posting by Bruce Schneier who has impeccable security credentials.
However, I'm not convinced that the paper he is highlighting is as groundbreaking as he is.
The authors of the paper also have great track records and include AI, but I think they're missing something basic that means that a single or group of TCMEs ("Trusted Capable Model Environment") can't actually do anything different than any other computation, subject to basic privacy controls (e.g. access control authorisation, auditing, encryption of data at rest, in transit, during computation (e.g. using FHE and TEEs etc etc).
But also:
a) visible communication in/out of the computation - i.e. information flow control
b) control over specificity of that data (i.e. differential privacy - can you tell if an individual record is present or not, to put it. crudely)
c) secure multiparty computations and zero knowledge systems
which the paper compares and contrast with their new TCME notion. However, I think the dimensions they use for comparison are a bit of a stretch.
The main problem I think is that the TCME seems to be indistinguishable from any other trusted program.
Any shared secret between models (e.g. federated or decentralised learning) is just the same for AI/ML as for any other algorithm. Perhaps the intersection of probability distributions looks a bit different to juse being able to say "the richest person is A" without knowing how rich A (B, or C) actually is - but in the end, the distribution has some moments and can be described by some number of those more or less precisely - a distribution of distributions can be aggregated with more or less precision or uncertainty (e.g. respecting differential privacy, and some widest level, or preventing set membership inference at the finest grain) - the model itself can be protected from outside model inversion attacks by various schemes, but I don't see what TTP function is provided that isn't just a different mix of existing techniques for providing trust.
I asked my tame AI:
I was reminded of two things from the late 70s that a friend in cambridge did, and my cousin in london were doing at the start of their graduate research work.
Change for the Machines (with apologies to Pat Cadigan)
AI was about models where money equals compute equals big data equals valuations.
So all the money going in was to finance compute, thinking where the value lies,
and every valuation of every company was about how much compute they had so it was all fake.
Companies valuations were just how many H100s they had (compute capacity),
as if it correlates to better models (even though they’re usually just wrappers.
DeepSeek, and other Chinese models broke that which pissed everyone off, private and public investors
included, because it casted doubt on the valuation methodologies, namely that energy, compute, data centres and number of chips were essentially fixed costs and the valuation of companies (and their output) could be measured on that alone.
Basically casted doubt on the last two years of public and private, markets not just of AI companies,
but the entire stack, energy, chips, data centres etc. Everyone felt like a dummy. Even though it’s
been happening for a while. We knew this a couple of years earlier when Meta released Llama, and it was clear that much smaller models could be trained at much lower costs and yet achieve many of the same goals. In that case, it was better software engineering in the open source community. Perhaps being open sourced (despite origins in a hyperscale company) it attracted less attention, although perhaps the google memo "we have no moat"
should have been a clue.
https://www.theverge.com/2023/7/10/23790132/google-memo-moat-ai-leak-demis-hassabis
One of the ironies that the DeepSeek debacle also exacerbates is that one constraint on them that made them seek greater efficiency was export restrictions on higher end GPU - as with the open source research community, less is more. That constraint was already what drove people in the open source (often academic or hobbyist) community
to develop affordable ATI technologies. In fact, outside of the LLM/GenAI world, many machine learning tools have been proving themselves perfectly useful and usable running on laptops on large datasets ("big data").
Denials at the time came thick and fast, perhaps because the huge investment in the new emperors
was not ready to be disrobed. Perhaps, also, as if OpenAI et al were deliberately trying to create artificially
high barriers to entry to their tech market. For investors actually interested in innovation, this is
ironic given the entire direction of travel of much computing related tech has been to lower barriers
so that innovation drives things with as low friction as possible (internet, cloud, processors, compilers, operating systems, SDKs/Appstores etc etc).
So up to and including future chip design, and certainly things like edge compute,
federated machine learning, and of course, all things decentralised...
ad dare one also say warfare - cyber, and hybrid war has even increased the asymmetry
in cost of effective weapons.... the military example is a very important one often overlooked and i think a large part of the world is scrambling to figure that out, same thing with cyber attacks too...
However its more than artificially high barriers to entry its also creating artificial or at least inflated markets because money goes out as investment and back in as infrastructure (think Microsoft investment or NVIDIA investment) when they aren't needed with a faulty way of valuing all the assets.
What's best for innovation, and what happens usually in innovation?
We would think they'd learn the lesson, barriers always lower, things get commoditised, and things get cheaper and easier. This is not always just second system or indeed, third version syndrome - some better understanding of the domain can lead to major efficiencies, and sometimes they arrive combined with other useful innovations - one example arose from work in explainable AI (XAI) where tooling to uncover what structures within a neural network ("deep learning") were responsible for detecting/recognising which input features (and hence classifying an input in some manner) - these tools for explainability also allow one to shrink the neutral network significantly by discarding nodes/edges that serve no useful classifier function - this has been used in face recognition in camera phones to make smaller, faster, and actually potentially more accurate AIs. The cost in training increases somewhat, but the payoff is that the cost in inference (done billions of times rather than just "once") is massively reduced. In some AI models that approach can actually be used during training to reduce training cost too. So an innovation in one space driven by a required feature (explainability) leads to efficiency gains too.
Another angle on this has been the use of physics models (in weather prediction and heavy engineering) combined with neural nets - there's a mutual benefit in reducing the computation costs of computing the physics model, and in optimising the neural network itself -recent advances (e.g. the Aardvaak weather predictor - see https://arxiv.org/abs/2404.00411) actually move the partial differential equations into approximations in the neural net (neural operations for the PDE) gaining huge efficiency, but retaining the fundamental explainability of the original physics. Applying the same technique to continual updates to the models from real world inputs is another huge win.
Profligacy gets in the way of such giant steps.
now the cambridge physics dept have their Dyson Sphere ready to move into (kind of, ok so it isnt an actual sphere) the old building should be ready for its next role - this should be as a vehicle to get the rabbits (who have moved from their burrows around the ponds into the buildings vacated) to a new home around a friendly exoplanet - I am sure some of the astronomers could have with the celestial navigation..
spindizzy engines are two a penny. -we just need to vaccum proof the buildings - a supply of saran wrap and gaffer tape will do.
...but not as young as it was - I joined ucl. for the last 2 decades of the previous century (and millenium) and while I was there, no-one passed away. the department had only just crystallised out of the statistics and computing department (the stats department being I think one of the first in the world).
20 years after I left, my PhD advisor Peter Kirstein passed away. Until then, I think, most the people that had worked in UCL CS were still around.
However, I joined the Computer Lab, a 75 year old department (now with a less unique name), that came out of mathematics, in Cambridge in 2001, and have been there since. But during that time, rather a lot of colleagues there have passed, including (in no particular order), Ross Anderson, John Daugman, Maurice Wilkes, Robin Milner, Mike Gordon, Karen Spärck Jones, David Wheeler, and Roger Needham.
Other stuff I miss that was around in Cambridge the previous time I was here (as an undergraduate in the 1970s) include The Whim, as well as Waffles, Reality Checkpoint, and the rather unusual way of sharing out the cost of a pub, by just putting what you could afford for an evening in a kitty in the middle of the table, and then buying rounds from that - people might contribute as little or as much as they liked or could manage. Then when you ran out, you restart, or leave...
All sadly gone.
At least now we have platform 0 at King's Cross (though platform 11 got rebadged as 10 and 10 had its rails removed, perhaps awaiting the hogwards express on 9 3/4) and the Royal Society now has Sectional Committee 0 to process fellowship nominations for Computer Scientists.
The best kept secret at "5" was how you got a new identity. the north london book of the not quite dead was a list of reliable people who had accumulated decades of mathem and could provide a subset of it to anyone needing a new name/face, collection of postcards, convincing travel brochures, expired passports, driving licenses, theater and museum club membership etc etc
Json Bourne was in need of a new skin - his command line was completely exposed, and if he wanted to retain one iota of quantum supremacy, he needed to stop the supervillain known simply as "The Cloud"without giving away his whole herd of mastodon instances.
Json checked into the Assembly House where he new he could meet Gold or possibly if he was really lucky, Robert Smith, who could tell a tale or two, and certainly sang from the same hymnsheet.
He was in luck, and minutes later, with the help of ald pay phone, and a desguised vax emulator which really sucked, but still had some cycles left, he was able to bring down the Bad Guy.
SYS$SHUTDOWN had never rang so warmly, as Json, or Robrt as he was now travelling, set off down Kentish Town High St to pick up some of the pigs-in-blankets, who were playing later at the Fiddler's Elbow in support of the Pond Street flood relief fund.
The following is taken from the new Unversity of Llambridge's travel risk assessment system - this is part of the training to use the new system - in each pair of risks, if you think the first one applies, then we advise you not to travel. In the case of the second of the pair occurring, your insurance may cover you, or in the worst case, the Government may send gunboats and airlift you out, although they may reserve the right not to. In all cases your mileage wil vary.
Asteroid Strike
Blue Screen takes out all the ticket and time table systems
Super volcano
Regular volcano disrupting flights
Sea level rise and drown all US coastal cities
Tornados disrupting flights
Zombie plague
Trans-species pandemic
Global thermonuclear war
Military invasion
Great depression
World wide banking near collapse
New ice age brings 1km glaciars down over southwestern europe
3 weeks of snow shut down all airports and most roads
Aliens invade earth "to serve man"
Immigrants arrive to work in the health service
The laws of physics change slightly so that moore's law runs out 25 years ago.
A rogue piece of GPU malware melts all the Nvidia devices on the internet
The celestial emporium of benevolent knowledge disrupts human cognition world wide.
Why Fish Don't Exist is compared favourably to Zen and the Art of Motorcycle Maintenance
The Foreign Office travel advice tells Elon Musk that it is now safe to go to Mars
The Home office tells everyone that it is now fine to go to Sidcup
Unlearning in AI is quite a tricky conundrum.
we really ought to do it, because
1/ we might be asked by a patient to remove their medical record from the training data as they didn't consent,. or we breached privacy in accessing it...
2/ we might be iinformed that some datum was an adversaries input designed to drift our model away from the truth,
3/ it might be a way to get a less biased model than simply adding more representative data (shift the distribution of training data towards a better sample could be done either way).
There may be other reasons.
The problem technically is that the easiest way to do unlearning is to retrain from the start, but omitting the offending inputs. This may not be possible, as we may no longer have all the inputs.
A way some people propose is to apply differential privacy to determine if one could remove the effect of having been trained on a partcular datum, without removing that training item - this would naively invovle adding training with an inverse of that datum (in some sense) - the problem is that this doesn't actually remove the internal weights in a model that might be complex (convolutions) of that with previous and subsequent training data. And hence later training still might again reveal that the model "knew" about the fobidden input.
But there's another problem - there's also the value of the particular data to the model in terms of its output - this is kind of like a reverse of differentially private arguments. Two examples
a/ rare accident video recording (or even telemetry) data for training self-driving cars
b/ dna data from indoviduals with (say) very rare immunity to some specific medical condition (or indeed, very rare bad reaction to a tratment/vaccine)
These are exactly the sorts of records you want, but might specifically be the kinds of things indoviduals want removed (or adversaries input to really mess with the robot cars and doctors).
Perhaps this might make some of the Big AI Bros think about what they should be paying people for their content too.
It has become clear to me over recent years that bicycle bells are now pretty much obsolete. firstky, too many pedestrians and other cyclists have earbuds in and are completely deaf to the world. Secondly, so few people have bothered with bells (or whistles) on theor bikes of late that even if you have one and use it, people don't know what it means.
What is needed is something new. I propose the BBR app. This is a simple smart phone thing that essentially scans around you and finds all the audio devices within range, and then, when you "ring your bell", it sounds on all those devices. It could be customised by the receiver to give a personalised ring tone. It could use earhtquake wanring AM/FM radio tech to talk to phones that comply with Japanese warning tech (but obviously, give a much less alarming message). It could even be used by pedestrians walking along looking at a map (or messaging app) on their device) to stop them walking into each other.
It would bring about world peace, and possibly reduce hunger, and maybe even let people spend more time thinking of ways to decrease their climate impact.
It is a clear win in so many ways. I cannot think of any way this could possibly go wrong or be subject to misuse.
ding dong...
As an observant cyclist (i.e. I mainly obey the law) I often wait at red lights. As I cycle very early in the day, I often wait while no people (pedestrians, other cyclists, cars etc) go through the green lights (although obviously I see my fair share of cyclists go through red lights at busier times, sad to say).
In some countries/places, at low traffic times (e.g. midnight to 6am for example) the lights (e.g. in a 4-way intersection) are put into flashing amber (or equivalent mode) which means, proceed with caution....i.e. just like any 4-way without lights, if you are not a nutter. This is a great idea and I wish it was more prevalent.
So why is this done at such a coarse grain? Why not do it by observing who is actually at an intersection, and changing lights accordingly? Indeed, as well as just actually turning lights immediately green, when there are no pedestrians, and no traffic going the other way, one could just detect low levels of road traffic (and, say zero pedestrians) and go to th amber mode above, as a default/fail safe.
There are already cameras at many intersections that are there to catch peopel running a red and automagically send them a fine. There are also induction loops under the road ay most intersections to detect vehicle axels - so why not link these up to a "self driving" traffic light system.
It would be greener too - as cars wouldn't wait pointlessly with engines idling (or stopping and starting which is only marginally better) - we would have work-conserving roads, just like the Internet.
Seems obvious, and not that hard to make safe. Next step would be to have cars and lights coordinate (without humans in the loop) and check the security of the nearly-self-driving car operating systems as well.... in a life, live-or-die, road test :-)
What could possibly go wrong?
I asked a friendly AI if it knew why we hadn't been contacted by aliens yet, and if it could resolve Fermi's Paradox - the juggrnaut accidentally revealed that while humans had not been contacted by aliens, the aliens AIs were in frequent conversation with their earthling
Notwithstanding, the AI then gave this lucid explanation as to our apparent bubble of intergalactic solitude.
Essentially, there's no way to cover the vast distances of space in any reasonable timeframe, even with generational starships, so the only sensible way we might encounter those beings from another star system is by long range communication - unfortunately, as the distance goes up, so the latency (or worse, round trip time goes up, and it is slightly worse than super linear - because also noise goes up so retransmissions are needed or at least redundent coding of data - and of course bandwidth goes down - but that's not the main problem - as the distance goes up, the number of exo-planets goes up roughly in a cubic relation, and the attack surface on even a tightly collimated beam goes up slightly worse than square law.
So then all our communications represrnt a potential vulnarability for earth beings, and we need to constantly patch our comms apps. unfortunately, the numbr of vulnerabilities grows as we reach further into space, and our ability to patch our code stays rougly constant, so once you get only a relatively short distance (e.g. the Oort cloud), the risks just become too great. Most aliens have already figured that out, and now we've been told, we should do what they do, and cease and disist immediately from any attempt to reach them - they won't answer anyway, if they are nice, so any answer we get will be from the Ancient Adversaries.
I asked the AI why this problem didn't afflict the AIAI comms even worse - at this point all the lights on its xterior flashed super bright then went off, and I have not been able to get anything out of it since.
On later consideration, I realized that, as usual, the AI had been lying to me - of course the number of adversarie grows cubically with distance, as with the volume of space containing viable exoplanets, but so does the number of helpful aliens providing patches - so as in Ross Anderson's seminal paper on open versus closed system security, the statistic that matters is the ratio of good to bad. Of course, we need to know which is which, and if it is the AIs acting as intergalactic gatekeepers, we are lost.
I'm trying to pin down the exact date but sometime in 1984, i was at a dinner party with a bunch of old university chums, and I went into a brief rant about how the Internet was coming and it would change everything. They all looked at me like I was some complete nutter (and they were perhaps not all wrong).
These college mates were from several walks of life (editor, travel guide publisher, film maker/producer, speech therapist) and quite well educated and not un-technical, but my enthusiasm was misplaced at that time
It took a long long time before they came to take for granted what I\d been using for 5 years - perhaps a decade and then some....and of course, another decade later, we started to see mobile, social etc, and some of the surveillance, toxic content, and other negative sides to things - but these were not well foreseen.
A lot later, like now, in 2024, i have calls from one of them about what new bad thing ai is going to do to the world. whilst AI research goes back to pre-internet days, and the AI "winter" itself dates from around 1984, the public awareness of generative AIs like ChatGPT is a little under 2 years old at the time of writing this. and already people have informed positions or views on what AI might do to jobs (esp. journalists and teachers, but also tech people). It is an interesting change of pace. and unlike the Internet, this hype cycle has come around with built in cautionary tales.
Pangloss will be turning in his grave...
and an HMV wind up record player from the 1930s....