multi-sensory tagging

cjmconnors — Tue, 03/24/2009 - 15:06

[Note: making a private thread public.]

I want to be able to tag things by virtual representations of sensory inputs: colors, textures, sounds, tastes, smells, emotions. Need to be able to do on mobile as well to incorporate real-world experiences and tag them this way at the time of their happening to capture the full essence/context.

Would also be cool to add things like the lat/long (GPS data), weather-data for where/when a pic was taken.

Could be neat to add info on "what's playing" to a tagged item. (Shazam? Verizon or Blackberry song ID tools?)

cjmconnors's blog

Ok, I have no idea what you

Dining Tables (not verified) — Sun, 05/31/2009 - 11:27

Ok, I have no idea what you guys are talking about but it sounds really cool! Can computers actually do anything like this yet?

Tom
Dining Tables

The beauty of brainstorming!

christine — Mon, 06/01/2009 - 12:05

Yes and no - computers can do lots of these things already; but don't because there isn't yet a demand, or the right people haven't put the right puzzle pieces together. Some of these things are in development and not ready for prime-time, and some are just pure fancy right now! That was the point of the exercise for Kevin and I - to get our creative juices flowing. It may be decades before we see some of these ideas in our every day lives, but it's good to have a goal! Cheers!

Mobile Device Control

christine — Tue, 03/24/2009 - 15:22

So, I know developers can use the actuator in the iPhone, but can they control the vibration tool in mobile devices? That could be interesting in terms of sensory input/output for multi-sensory tagging.

mobile apps

christine — Fri, 08/01/2008 - 13:54

Already possible, but ought to include ability to take a pic with my camera-phone or Eye-Fi enabled digital camera and Multi-Sensory tag it. The market for apps seems to be reasonable:

http://www.macrumors.com/2008/08/01/iphone-app-store-numbers-reveal-larg...

moving through multisensory N-space

Kevin — Mon, 07/21/2008 - 00:11

Hmmm... maybe we can extend this idea. Before we talked a little bit about getting the context from prior and post posts [sic]. (Or heck, even those posts of your friends in a FriendFeed sort of way.) What if we developed a way to interpolate between semantic contexts? So, we've got multiple points in time (themselves in N-space), and we try to 'move' from one of those (sets of) points to another. The nice thing is we wouldn't even try to do it mathematically (unless TM decides he's got an extra few hours), we could move through/across ontological branches to get from one point (in time) to another. If we could 'show' the movement from a set of contextual points to another set - that was meaningful of course - Cowabunga! Sure this is lots easier to conceptualize than to do, but hey!

Physical location and time are the obvious ones. Moving semantically from one concept to another is harder, but we could even look for the closest matches in the user's own history. So, a shortest path approach through an existing ontology is easy enough, although my suggestion here is to only show those points in the path that have user-specific instantiations. If we can represent movement across multiple ontologies simultaneously (in some unified? way)... So, what we would really be doing is moving from one point in one sensory context to another, and trying to coordinate in a meaningful way.

wearables as context

Kevin — Mon, 07/21/2008 - 00:05

So all the dang RFID tags are being put into clothes, so the answer to "What was I wearing?" becomes possible!!! And, heck, if it's RFID, and we're in a social situation, "What were *they* wearing?"

Add relative position, over time, and all of a sudden you've got a virtual representation of movement. It would be like when a camera is circling around you. You sample at certain points, and then interpolate to get the movement. Put a person's picture in the clothes, and then move through the recreated context a la Photosynth.

a little creepy, but

christine — Tue, 08/05/2008 - 11:52

i's not just wearables... think about a dog that's been chipped. We could associate a smooth-coated dog with the tag smooth, or soft or fluffy. A wiry-haired dog could be "scratchy" or rough, etc.

People are getting chipped; could a person be associated with tags of the users emotional reactions to that person?

Could a tangible be associated with the users reaction? Traffic accidents waiting to happen if someone were driving down the road and had random shipments trigger emotion tags on their personal feed...

elaborate, please - emotion tags, tangible -> reaction

Kevin — Sat, 08/09/2008 - 00:45

"...with tags of the users emotional reactions" - who's being tagged and who's doing the tagging here?

"...a tangible...with the users reaction" - same

from our user's perspective

cjmconnors — Sat, 08/09/2008 - 20:00

OK, so, let's say you use BrightKite or Loopt or some kind of location-based or RFID-like tool. And let's say you have a Chumby-like app on your mobile which is sync'd to your FriendFeed. You walk into one of your favorite restaurants, and your tag cloud starts displaying things like "asshole" "anger" "resent" as well as "sinful" "yummy" and "best friend" "dancing" "shopping." Why get all those various things? Because you're with one of your best pals, with whom you like to shop and go dancing, and you tagged your friend with these labels when you added her to your social network somewhere (say, via OpenSocial.) Not 5 feet from you is the dessert cart, with a molten chocolate cake, which while posting a great recipe you found for it to your del.icio.us feed you tagged as "sinful" and "yummy." Then, you look around the dining room, and see your ex, the lying, cheating bastard who broke your heart a year back. Got the idea?

Now, the long term would be the ability to have your own chip trigger a physical/emotional reaction, but that would be a bad idea; it makes me think of the Star Trek movie when Data first got his emotion chip and became paralyzed. Don't think I'd want something triggering any more emotions in me than I already have!

Or to take it further, if we add the trust bit, you can be notified if there are people in the room who are in your friends trust networks - let's say you're at a party where you don't know a lot of people. It could be like LinkedIn for the party - and, depending on your mood that night, you can gravitate towards the people your intellectual friends hang with, or your work friends, or folks like the crazy people you hung with in college. You also now have a way to start a conversation more meaningful than "Hey, I see you're on BrightKite too."

Again, social implications here for non-professional environments, but no worse than anything that already goes on.

taking it yet further, and out of my sphere of knowledge

cjmconnors — Sun, 08/10/2008 - 09:53

Connect this to a human-interface device - a monitor for heart rate, blood pressure etc - in a bracelet or watch (been done before) to capture the physical manifestation of an emotional or intellectual reaction, to include that with the time and geospatial data. Think of the applications for medical research; rather than forcing the user to stop and journal certain metrics, or to lug around equipment, have it all added WITH the bonus of having it in a fuller context or their physical and emotional states - with a link to the history of the interactions -- how we behave in certain situations or around certain people. Could be as simple as tracking down an allergy or as complex as identifying a trigger for multiple personality disorders.

Yikes, I'm out of my league here... Want to chime in?

subvocalization as a capture and retrieval command

Kevin — Sun, 07/20/2008 - 23:51

Ok, so, if we're *there* then we might as well be able to capture subvocalizations (did I mention MIT?) as both the capture and retrieval mechanism. It may be the closest we can come to a command language that is not our brains. So, we literally utter, "Capture" and our set of contexts is captured. Utter "Recall " and we go out and get our (now-patented) representations of related contexts.

If we don't get "there" anytime soon, it won't matter if we've gotten the basis for the cross-sensory mappings. We will be taking all the digital context (what's on the screen), and the physical context (what's outside the screen, including what we're feeling/sensing). Provide the mappings, and someone else can put on the front and back ends.

Jott

christine — Mon, 07/21/2008 - 09:35

Similar to how simple it is to Jott - I think there was a project once upon a time where we were talking about voice blogging in the field with speech-to-text transformation and then analysis. But it would be interesting to combine all of this on something like Swurl's timeline: http://cjmconnors.swurl.com/timeline; or take Swurl one further and plot a map. Could be a fun "Tufte" project!

Brings up a host of socio-political implications.

adding emotional context

Kevin — Sun, 07/20/2008 - 23:43

And emotions?!? Are you kidding me? Ok, so let's go *way* out here: if we're going to be wired in, then why not get the polygraph-equivalent for the person and use that as part of our index? We could represent this, but the key would be the user recognizing the representation for the memory. (So, "My heart rate was elevated, I was sweating." - How do we 'say' this?) This could evolve...and far. If we can just show the cross-sensory indexing approach works, then we could go out to an MIT, get their wearable whatevers, and plug their stuff in. Then we play it back through their little monocle (praying that the visual representation captures the cross-sensory stuff), and... omg.

adding context(s)

Kevin — Sun, 07/20/2008 - 23:41

Add to geo location: speed, direction, mapping to be able to suggest, "I was going south on I-10 that Friday afternoon" and be able to put that itself in context. For instance, "It was Friday afternoon, and just 45 minutes after I twittered, 'insert-twitter-text-here'." And omg I love the adding "what's playing" to the tagging. This is so unbelievably simple, and oh-so-smart multi-sensory indexing! (I love the weather idea too.)

Then we could consider semantic proximity, and if that seems overwhelming (or we can learn to tell when it's just plain wrong), we could do TF/IDF. We could consider time/location, thesauri or taxonomy or ontology branches.

I want to capture texture (CC made me think of this in one of her synesthesia descriptions).

Remix data

christine — Mon, 07/21/2008 - 11:58

The obvious thing - which I'm posting here just to capture and avoid a DUH moment - is to pull in open data from sources like Wikipedia, Flickr, and user-authenticated sources that are close semantic matches. An ontology won't be enough - an index itself is interesting only to a select group of infosci uber-geeks. See Vicinit (European-focused iPhone 2.0 app).

Excellent, now I have a reason to upgrade to the 3G... ;) Get your phone Lynch!