Wednesday, May 18, 2016

Every Day Data Science Challenges

1 Guitar Strings

Different strings break at different rates. You can buy them singly or in sets of 6 (occasionally with a spare top E) - really what you want is just in time delivery of a new set wtih a distribution of strings (EBGDAE) that matches the wear/tear rate for you, your guitar (classical, flamenco, acoustic, electric etc) and tone/newnewss you like - this could be crowdsources by instrumenting tuning apps on phones which would notice when you tune from way below (e.g more than a 5th below the right note for that string, probably indicates a new string being put on) -

The statistics could be aggregated, and classes of users found, and then companies (like my fave ) could build orders for you -

2 Bicycle Wheel Spokes

I've lost 4 spokes over the last 5 months cycling in Cambridge - probably, they went on the appalling potholes on station road, or the tree roots across burrel's walk - wouldn't it be nice to know where these occcurred so I could report them to the council (and get money:-)

This could easily be done with accelerometers in smart phones....and GPS - look for rapid up/down movement - then afterwards (when a spoke has gone) you should be able to find the periodic wave of the bike as the wheel is now eliptical....

Thursday, April 07, 2016

Miro's Law


To his excellency, High Admiral, 1st Fleet, Third Arm Galaxy 74434
From Captain Moore, First Contact Team, outer quadrant 2,
planet three, yellow star 17.

Your Excellency,

I am writing to report on the results of the strange mission we have undertaken exploring the deserted planet full of wonders known by its former inhabitants, as far as we can tell, as Silt or perhaps Clay - our translators are working flat out to improve on our understanding as there is so much potentially to be gained from their technology and civilization.

We have uncovered a most exciting cache of documents which may finally explain the departure of the beings who constructed these wonderful buildings and devices. Everywhere we look there are suitable perches on tall polls, connected by long nesting strings. Marching across the countryside between what we believe to be their latrine sites, are long flat landing strips where competition for suitability for breeding stock amongst our warriors would be idea. Enough of the marvels. Back to the documents, which appear to be from a savant, desperate to solve some crisis that has struck their ecosystem....he writes

"I am sending you this letter to ask for your advice. As I travel often to far away lands, and present my work at conferences, I am inevitably showered with gifts, and amongst these is always at least one ball point pen. Since we developed the fishnet" (we're not sure if that is the right term) " I have had almost no use for these gadgets, and have steadily been accumulating them in killing jars" (again, not clear, but it seems that the 'jar' is something to preserve things in at least we can tell that). "Now, I am out of jars, and am unable to store sufficient foods for the winter" (now you see why we are fairly sure of our translation).

"Hence I am writing to all my fellow savants, to ask if they have any idea how we can solve this problem. The number of ball points is growing hyper-exponentially, and threatens the whole of the Fellowship of the Royal Society, and so we realise that we have to reach out to our cousins across the seas for help

I remain yrs, etc etc

p.s. I enclose a pen for your use in response"


Your excellency, there are many responses, but most echo the concern, and dare I say despair at the situation. Finally, however, after much work, it seems that one amongst these giants of intellect proposes a possible solution, for it is the last letter, and in it are many strange symbols which resemble our own formalism for hyperspace drive, and yet appear to arrive at a simpler solution....the letter concludes

"and so we suddenly understood that if we could figure out how to channel the gravity waves just right, the spaghettification phenomenon will allow even the largest of our fellow humans to fit down the inter-dimensional tunnel formed from the tubular casing of the ballpoint through the frames, and into the landscapes that miro draw, it seems from true life, rather than, as critics of the day said, his dreams. the rest can easily be worked out from this sketch...

I leave now, as you know, however, I enclose your last pen, which you will find sufficiently charged to allow you to join us in Blue II should you wish, although as you are of hungarian origin, you may resonate with the richer Mikrokosmos..."

So, your excellency,
It seems that the "humen" of "mud" were finally able to lose enough weight through some fantastic new plan, to soar to other dimensions.....we have yet to complete the proof from the sketch mentioned above, but I hope that we can report to you on this soon,

I remain, as ever, your nest-issue of the fourth degree, captain Moore...

captain etc

p.s. I enclose a pristine unused biro pen for your perusal, and who knows, perhaps use....

Sunday, February 28, 2016

Opening up the Billion Sided Market for our IoT data.

In the HAT project, we came up with the idea for  starting a data exchange for all of us to exploit our data for fun and profit.

There are several important innovations we are bringing to the IoT world:

  • Multi-sided market - we are all now used to the two-sided markets of the smart phone & the cloud - we get apps and services for "free", in reality, trading data about our selves (wishes from searches, preferences from likes,  places from location checkins, etc etc). However, the market is heavily tipped in favour of the large cloud providers, and the user has little knowledge or control over her data, and in particular, very little view of its use and value. The HAT changes all that by providing a hub for each user with storage processing and interfaces for access by other parties, but with visibility, control and above all, valuation for the data.
  • Democratised data - HAT providers store the data and provide access, so we need a marketplace for the valuation - an exchange, where bidders can establish openly a price. This could be at a fine or coarse grain - for example, usage of utilities (power, water etc) typically is interesting for service providers, but typically, aside from billing, fine grain use is only really interesting to the actual consumer in their home or office. Alternatively, monetizing usage information about retail goods could be traded directly with retailers or even wholesalers for discounts, loyalty points, or money, and can include preferences for really accurately targeted advertisements in exchange for further discounts or e-cash. 
  • Freedom - freedom to switch hub, to choose aggregators who have a better deal, or provide stronger service guarantees, is a given  - the large number of HATs is trivially deployed and scaled out in today's cloud based world. This encourages innovation in HAT technology itself. The symmetry of the business relationships allows this dynamic, in contrast to the asymmetric power wielded by the centralised services of the last fifteen years.
  • Silo Busting - the IoT world is notoriously not an Internet of Things, but a hodge podge of many different services, overlayed on the internet and the cloud, but not in any way connected to each other. The HAT changes that by creating a collection of places where data from multiple worlds can be integrated by new applications and new customers from any of the millions of sides of the new market. We are strongly technology agnostic when it comes to IoT at the "lower level" - of course there are good reasons for different systems to work in different ways. We break open the silos by allowing user-centered integration of data. Its about you, so you control it, whatever it is. Cosmetics, entertainment, clothes, energy, well-being, you name it. Think of the value being missed by existing isolated systems when they cannot put 1+1+1 together, but can only see how single values (kilowat hours, litres, meters) increase over time, instead of being able to combine together information with meaning! A space for a million apps for combining your data - more innovation, driven by new value made out of new joins across the seams of the legacy disjointed IoT world.
  • Privacy Protection - We really care about our privacy. The legacy cloud systems today (your social media, web mail, search, travel portal usage) currently do a half baked job on this. When we take far more personal information into the HAT, it is essential we offer much much stronger assurances, applying the very best practice in technology and also written in to the terms & conditions in plain language. If one HAT doesn't get it right, it is easy to move to another. This enables an eco-system with constantly improving, transparent, control over data visibility - once again, another dimension on which to innovate.

Monday, February 15, 2016

Zika App idea

Back in the day, during the H1/N1 epidemic, we did this Flu Phone App to track people's encounters (via phone proximity using say bluetooth (could also use GPS tracking on phone, or even call data records with cell phone company cooperation, if you want less accuracy). The idea was to extract events when people self reported with symptoms, and then (in a privacy preserving manner) extracet the encounters between that individual and others (infected or not) in the population, and then to work out from this various epidemic parameters (susceptibility of different members of population, infectiousness, recovery rate, asymptomatic carriers/herd immunity levels in segments of population, etc etc), as well as possibly nailing elements of the vector....

So with the current Zika virus, it is pretty clear that it is spread by a particular mosquito type (the same as spreads Dengue Fever).

So we could take the app described above (and its reporting infrastructure) and
add one very simple thing - if the phone app turns on the mike, you can tell from sound whether there is one of these little beasties near you- wing sounds have characteristic frequency which is in audio range and sensitivity of human ear and certainly of the (usually better) microphone/audio system on a phone - more info about the Aedis Agypti sound of female mosquito which is the one you care about being not bitten by in terms of Zika.

If there were several people running such an app in the same location, you might even tll roughly where the mosquito was and avoid it (though that's a bit fanciful).

At least, however, you'd be able to look at the incidents of people being co-locaed with mosquitos of the right type, and the infection rate. ANd possibly (over time) look at the spread caused by an uninfected mosquito biting an infected person....thus
mosquito -> person -> mosquito ->person

of course, the same app might possibly also tell you of cases of person->person where there's no mosquito detected...which would also be useful data for epidemiologists

A thought.

Tuesday, February 09, 2016

panic, moi?

So there's this great new report from the Berkman about the worries various governments have that the technology we are starting finally to make use of to protect our privacy may also mean that "bad guys" can get away without being caught.

It is deeply ironic that there's precious little evidence that having untramelled access to everyone's Internet data for the last 20 years has done a single thing to prevent one terrorist death. It is also ironic that when there was access to encrypted data, during WWII, from Station X (Bletchley, breaking the code, the Enigma and its variations etc etc), it was not used to prevent Atlantic shipping from being sunk by U-boats as that would have given away the fact the allies knew where the subs were (i.e. had likely broken all the codes). It was finally "used" to know that the germans did not know about where the D-Day landings were to be. This was to prove useful (although not necessarily decisive) in winning/ending the second world war.

However, note interestingly that spotter planes could often see U-Boats surface, and it was the location of the sub when it sent an encrypted report (aka "meta-data") that let the Turing folks break the code the 2nd time. There's no evidence that the NSA have known about Al Quaeda before 9/11 or that the Spanish, UK and French had any idea about the Madrid, London or Paris terrorists ahead of time. If they did, and didn't say because it would "reveal" their capability, in a post Snowden era, this is just plain stupid, actually criminal. Given several events have happened after Snowden, and there's precious little evidence the bad guys used much more than basic comms (SMS, instant messaging) then, it is evidence that the security apparatus is not fit-for-purpose.

Thus, the report above is right about meta-data (what's sometimes called communications data, as opposed to content, or "control" as opposed to "data").

Interestingly, was talking to some lay folks recently about what the police do if they find someone unconscious (or worse) with no id, but a smart phone, and that smart phone is locked (and, in modern iphone or android, encrypted). So
1/ If you have an ICE ("In Case of Emergency") configured, it can be called from a locked screen on an iPhone, and you can configure android the same if you want.
2/ The phone company can workout what the IMEI and number of the phone is from the location, and from that, could give the police a list of caller and callee IDs so they could try a few til they get someone...plus the account information would likely give name/address/bank info.
3/ If the phone is backed up in the iCloud, its quite likely the back up isn't encrypted

All of this could also be done with someone "of interest" who is perfectly conscious, but unaware:-)

So there. Fire the NSA and GCHQ and get someone in who has a clue.

Monday, January 25, 2016

blockchain for gun control

so distributed ledger technology is a new technology that is all the rage in some government circles. while Bitcoin as the exemplar of the use of the technology for an electronic replacement for cash and credit cards, has its detractors (and they are mostly not wrong), the underlying system allows one to track transaction history associated with a physical object  - one of the UK government's use cases in the report linked above, is the idea of being able to avoid buying "blood diamonds".

so how  about we propose using this for arms control (everything from nukes, to hand guns, and ammo) ? there are ways even without putting "smarts" in the gun (ballistics can often match gun/ammo to each other in any case, and one can move to more careful signatures easily)...

then one could start to look at liability. i.e. people that own weappns would have to take responsibility for a change.

Thursday, January 07, 2016

investigatory ploughsharing bill - srambling for safety

for a thorough report on today's scrambling for safety 2016 debate, its hard to beat George Danezis blog - one thing I was going to ask about was the really broken part of the bill, which prevents any discussion between a service provider and the agency that serves a warrant on them for intercepoton (whether a standard surveillance or a bulk one, or interference on a device or a broad spectrum of devices).

I realize that some level of stealth is, by definition, needed during the surveilance - however the world is rapidly evolving, and it is clear that operators and service providers are at the bleading edge and are able to offer (and do, in practice under today's laws in the UK)  on a request  (e.g. no, you don't want that IP address, you want this URL prefix, as that's a load balancer/VM, NATed device that changes etc etc) - in my example question (no., you don't want to run interference on that device as it isn't just a routine users ipad, its their tesla dashboard, and if you weaken the random number generator in the OS on that device, you open it up to hackers who will crash the car), not only is it obvious the security and police agencies don't have expertise yet in the area, we need to have a cooperatively evolveable law - latching the law (the first in 500 years to admit that agencies need these powers, but under legal controls) we need to make sure it isn't the last law made in the area either - just as the "Internet Connection Record" is meaningless in the world today, so the interference model is extremely dangerous in the IoT space, where there are currently more devices that are not end-users comms gadget (==phone/skype) than are - pretty soon, there will be 100s or 1000s of devices - monitoring these is mostly a waste of resources (more haystacks to not find needles in) - interfering with these devices (e.g. pacemakers, car brakes, traffic lights) is incredibly dangerous - [footnote...]

proportionality requires risk assessment - "collateral damage" that is a death because of interference on a device which causes a car crash or a heart failure, is not assessable today. it may be one day, but I posit that it is not an acceptable risk level for gleaning a little bit more sigint, that probably wont be acted on anyhow. Basically, this blows out of the water any fig leaf of proportionality, unless there is a wholly different way to manage (transparently) the codes of practice, in a way that future proofs (actually makes fit for purpose for today's internet) this dodgy draft bill.

footnote - let not forget algorithmic lawyers - when the music biz wanted to chill the p2p file sharing world, they started getting s.w that generated letters to threaten disconnecting users from their ISP - one fabulous case ended up with a tech guy defending himself in court, because the IP address the lawyers s/w detected allegedly uploading music in breach of copyright from. was his HP laster printer. doh. if they can get that wrong, then the spooks software can and will confuse a crims phone with an innocent ("collateral damage") bystander's  auto-defibrillator or internet enabled insulin pump.

Blog Archive

About Me

My Photo
misery me, there is a floccipaucinihilipilification (*) of chronsynclastic infundibuli in these parts and I must therefore refer you to frank zappa instead, and go home