Motor adaptation during a sound oriented task (NCM 2017, poster publication)

This poster will be presented in Dublin during the 2017 annual meeting of the Society for the Neural Control of Movement (NCM 2017).

Motor adaptation during a sound oriented task

Eric O. Boyer1,2, Frederic Bevilacqua2, Sylvain Hanneton3, Agnes Roby-Brami1,
1 ISIR – CNRS UMR 7222, UPMC, 2: IRCAM – STMS-CNRS, UPMC, 3: LPP – CNRS UMR 8242, University Paris Descartes, Paris, France

Introduction. Movement sonification systems appear promising for sensori-motor learning in providing users with auditory feedback of their own movements [1]. However, research on sonification for sensori-motor learning has been mainly directed toward “movement oriented tasks” where the instruction and the attention is put on the movement itself. In contrast, the aim of the present study was to test a situation were the instruction and attention is given to the sound, that we call a “sound-oriented task”.
The sonification mapping relied on the metaphor of friction sounds produced by drawing movements. We focused on the drawing of ellipses which is characterized by a well known invariant velocity-curvature relationship [2,3]. The replay of friction sounds (registered or synthesized) can evoke the shape of the drawings [4] and induce a sensori-motor perceptive bias on the reproduction of visual motion [5] .
Elaborating further on these bases, we tested the effect of on-line sonification with friction-like sounds on the kinematics and the shape of elliptical drawing movement. The mapping was implemented as a band pass filter whose center frequency varied linearly with the velocity of the movement [3,4]. We analyzed the motor adaptation of the drawing movements when the subject had the instruction to maintain a constant sonification pattern while the frequency of the filter changed without his/her knowledge.
Our hypothesis was that the alteration of the sonification mapping would induce temporal and/or spatial adaptation of the movement.

(see the poster for methods and results)

Discussion. The motor adaptation tended to compensate for the changes in sound feedback induced by the changes in the sound-movement mapping. This demonstrate that the participant could adapt their movement to the “sound oriented” task. The adaptation was manifested by modifications of the kinematics. There was a tendency for increase of frequency and decreased size of the drawing in the control situation. In addition, the movement was faster with larger movements when the gain of the mapping was increased and slower with smaller movements when it decreased. The global shape and orientation of the ellipse was not modified in 2D.
This demonstrates that the participant privileged the stability of the geometrical shape and adapted their velocity in order to satisfy the instruction to keep the sonification pattern constant. The increase in velocity was more due to a change in frequency while the decrease was more due to a shrinking of the shape, suggesting different movement regimen [e.g. alternative versus discrete, 7]. The modification of the angle of the ellipse during the experiment in 3D but not in 2D was probably due to greater inertial constraints as shown by Pfann et al [8]. Participants who were instructed to draw circles with shoulder-elbow movements made ellipses with increasing eccentricity when the velocity increased (order of magnitude: 1m/s). In addition, the elongation of the ellipse was in the direction of least inertia. A similar effect was also observed for handwriting-like movements (similar to our 2D task) by Dounskaia et al. [9] but for a much higher velocity regimen (instruction level “as fast as possible”, 0.34m/s), when the velocity we used corresponds to their “self paced” level).

Conclusion. This study demonstrates that movement sonification can be used i) to induce implicit motor adaptation in both planar and 3D movements and ii) to control the direction and magnitude of this adaptation through mapping parameters modification.

 

Nouvelle publication : Investigating three types of continuous auditory feedback in visuo‑manual tracking

Investigating three types of continuous auditory feedback  in visuo‑manual tracking

Éric O. Boyer, Frédéric Bevilacqua, Patrick Susini et Sylvain Hanneton
Exp Brain Res – DOI 10.1007/s00221-016-4827-x
(electronic publication, paper soon)

Abstract

The use of continuous auditory feedback for motor control and learning is still understudied and deserves more attention regarding fundamental mechanisms and applications. This paper presents the results of three experiments studying the contribution of task-error, and user-related sonification to visuo-manual tracking and assessing its benefits on sensorimotor learning. First results show that sonification can help decreasing the tracking error, as well as increasing the energy in participant’s movement. In the second experiment, when alternating feedback presence, the user-related sonification did not show feedback dependency effects, contrary to the error and task-related feedback. In the third experiment, a reduced exposure of 50% diminished the positive effect of sonification on performance, whereas the increase of the average energy with sound was still significant. In a retention test performed on the next day without auditory feedback, movement energy was still superior for the groups previously trained with the feedback. Although performance was not affected by sound, a learning effect was measurable in both sessions and the user-related group improved its performance also in the retention test. These results confirm that a continuous auditory feedback can be beneficial for movement training and also show an interesting effect of sonification on movement energy. User-related sonification can prevent feedback dependency and increase retention. Consequently, sonification of the user’s own motion appears as a promising solution to support movement learning with interactive feedback.

Keywords: Tracking · Auditory feedback · Sensorimotor learning · Sound · Interaction

Nouvelle publication : Sensori-Motor Learning with Movement Sonification: Perspectives from Recent Interdisciplinary Studies

 Sensori-Motor Learning with Movement Sonification: Perspectives from Recent Interdisciplinary Studies

Access to the full text HERE

This article reports on an interdisciplinary research project on movement sonification for sensori-motor learning. First, we describe different research fields which have contributed to movement sonification, from music technology including gesture-controlled sound synthesis, sonic interaction design, to research on sensori-motor learning with auditory-feedback. In particular, we propose to distinguish between sound-oriented tasks and movement-oriented tasks in experiments involving interactive sound feedback. We describe several research questions and recently published results on movement control, learning and perception. In particular, we studied the effect of the auditory feedback on movements considering several cases: from experiments on pointing and visuo-motor tracking to more complex tasks where interactive sound feedback can guide movements, or cases of sensory substitution where the auditory feedback can inform on object shapes. We also developed specific methodologies and technologies for designing the sonic feedback and movement sonification. We conclude with a discussion on key future research challenges in sensori-motor learning with movement sonification. We also point out toward promising applications such as rehabilitation, sport training or product design.

Keywords: sonification, movement, learning, sensori-motor, sound design, interactive systems

Citation: Bevilacqua F, Boyer EO, Françoise J, Houix O, Susini P, Roby-Brami A and Hanneton S (2016) Sensori-Motor Learning with Movement Sonification: Perspectives from Recent Interdisciplinary Studies. Front. Neurosci. 10:385. doi: 10.3389/fnins.2016.00385

Focus sur la notion de proprioception

Le mot « proprioception » définit un 6ème sens assez méconnu. Pour tout arranger, il existe des confusions et des débats sur sa définition. Cet article est l’occasion de faire un point le plus rigoureux possible.

Etymologie et origine du terme

La proprioception (formé de proprio-, tiré du latin proprius, « propre », et de [ré]ception) ou sensibilité profonde désigne la perception, consciente ou non, de la position des différentes parties du corps. Elle fonctionne grâce à de nombreux récepteurs musculaires et ligamentaires, et aux voies et centres nerveux impliqués (définition wikipédia)

La proprioception s’oppose à l’ « extéroception » car les signaux nerveux sont générés par des événement produits à l’intérieur du corps, contrairement aux signaux extéroceptifs qui ont pour origine des modifications de l’environnement (toucher, vision, audition).

Les récepteurs sensoriels impliqués dans la proprioception

Il s’agit de récepteurs placés dans les os, les muscles (fuseaux neuromusculaires, organes tendineux de Golgi), dans les viscères et dans les articulations (capsules articulaires). Les fibres nerveuses issues de ces récepteurs font synapse avec la moelle épinière et les signaux remontent ainsi vers les centres supérieurs (cortex somatosensoriel primaire, cervelet par exemple). Certains incluent dans la proprioception le sens de l’équilibre, c’est à dire les signaux issus du système vestibulaire (dit vulgairement « oreille interne »). Mais cela peut être discuté car cet organe réagit à l’orientation de la tête par rapport à la gravité terrestre qui est un signal externe. Le système vestibulaire est-il un organe proprioceptif ou extéroceptif ? En tout cas il donne des informations sur les mouvements de la tête dans l’espace.

Confusion avec d’autres termes voisins

Il faut faire attention car d’autres termes définissent des aspects de la perception des mouvements et des déformations du corps. Il en résulte des débats sans fin sur la définition de ces termes, débats qui n’ont pas beaucoup d’intérêt. Essayons quand même d’y voir plus clair.

Le terme le plus proche est celui de kinesthésie (du grec kinesis signifiant ‘mouvement’ et aisthesis : ‘sensibilité’). La kinesthésie est une perception consciente de la position et des mouvements des différentes parties du corps. Certains différencient les sens kinesthésiques de la proprioception donnant à celle-ci un sens plus général et à la kinesthésie un sens plus spécifique, en excluant par exemple le sens de l’équilibre de la kinesthésie. Ainsi l’oreille interne ferait partie des organes proprioceptifs mais pas de la kinesthésie.

Mais attentions, certains distinguent proprioception et intéroception. L’ « intéroception » ne concernerait que les signaux issus des capteurs viscéraux.

Il existe aussi un autre terme qui interfère avec la notion de proprioception : le sens « haptique ». L’haptique, du grec ἅπτομαι (haptomai) qui signifie « je touche », désigne la science du toucher, par analogie avec l’acoustique ou l’optique. Ce terme a été introduit en psychologie par Revesz (1934, 1950). Au sens strict, l’haptique englobe le toucher et les phénomènes kinesthésiques, c’est-à-dire la perception du corps dans l’environnement. En effet, pour saisir la forme d’un objet, son volume ou même son poids, il faut le toucher et le manipuler. La manipulation de l’objet va stimuler à la fois les récepteurs tactiles de la peau et les récepteurs proprioceptifs présents dans les muscles, les tendons et les articulations.

On trouve en effet des définitions du mot kinesthésie qui interfèrent fortement avec la définition du sens haptique. Certaines viennent de Henri Piéron même !

Kinesthésie : Sens du mouvement; forme de sensibilité qui, indépendamment de la vue et du toucher, renseigne d’une manière spécifique sur la position et les déplacements des différentes parties du corps. « La kinesthésie exploratrice avec les modifications synchrones des impressions cutanées, renseignant sur la forme, l’état des surfaces, le volume, le poids, etc. (…) [intervient] pour permettre l’identification » (Piéron, La Sensation,1945, p. 41)

Dans la citation de Piéron ci-dessus, on a l’impression qu’il parle plutôt du sens haptique.

Déafférentiation (interruption du fonctionnement des circuits proprioceptifs)

Différentes affections des nerfs, de la moelle et de l’encéphale peuvent atteindre la proprioception : traumatisme, compression par une tumeur, inflammation, accident vasculaire, trouble métabolique (carence en vitamine B12). Une atteinte de la proprioception entraîne une altération des sensibilités profondes élémentaires : le patient ne peut pas, les yeux fermés, reconnaître la position de ses différents segments de membre. Elle se traduit également par une ataxie (absence de coordination des mouvements), avec une instabilité en position debout, accentuée lorsque les yeux sont clos (signe de Romberg). La marche est également perturbée. L’anesthésie osseuse se traduit, à l’examen clinique, par l’absence de perception de la vibration provoquée par un diapason appliqué sur les os superficiels.

Références

  • P.M. Gagey, B. Weber (2004) Posturologie ; Régulation et dérèglements de la station debout. Troisième édition, préface du professeur Henrique Martins da Cunha, Elsevier Masson, Paris

  • J. Paillard (1976) Tonus, posture et mouvements. In Kayser C. (Ed) Physiologie, tome II, Flammarion (Paris): 521-728.

  • H. Piéron. La Sensation, guide de vie, Paris, Gallimard, 1945.

  • G. Revesz, System der optischen und haptischen Raumtäuschungen, Zeitschrift für Physiologie, 131, 296-375, 1934

  • G. Revesz, Psychology and Art of the Blind, London, Longmans Green, 1950

  • C. S. Sherrington (1906) The integrative action of the nervous system. New Haven, Yale University Press

  • Charles S. Sherrington (1900) the musclar sense. In Edward A. shafer Ed. Textbook of physiology (Edimbourg-London, 1900) t. II, 1006.

Nouvelle publication : SoundGuides, Adapting Continuous Auditory Feedback to Users

SoundGuides: Adapting Continuous Auditory Feedback to Users

Jules Françoise Ircam, Paris, France
Olivier Chapuis Univ Paris Sud, CNRS & INRIA, Orsay, France
Sylvain Hanneton Université Paris-Descartes, PARIS, France
Frédéric Bevilacqua IRCAM, Paris, France

We introduce SoundGuides, a user adaptable tool for auditory feedback on movement. The system is based on a interactive machine learning approach, where both gestures and sounds are first conjointly designed and conjointly learned by the system. The system can then automatically adapt the auditory feedback to any new user, taking into account the particular way each user performs a given gesture. SoundGuides is suitable for the design of continuous auditory feedback aimed at guiding users’ movements and helping them to perform a specific movement consistently over time. Applications span from movement-based interaction techniques to auditory-guided rehabilitation. We first describe our system and report a study that demonstrates a ‘stabilizing effect’ of our adaptive auditory feedback method.

Proceeding
CHI EA ’16 Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems
Pages 2829-2836
ACM New York, NY, USA ©2016
table of contents ISBN: 978-1-4503-4082-3 doi>10.1145/2851581.2892420

Le lien vers la page de l’article ici : http://dl.acm.org/citation.cfm?id=2892420&CFID=621448898&CFTOKEN=84867349

Rapport technique : interactions avec un écran tactile

Ce document décrit comment acquérir les données issues d’un écran tactile et les transmettre par le protocole OSC. La méthode utilise le module evdev de Python et la librairie liblo. Cette méthode est sans doute utilisable pour d’autres dispositifs de pointage.

Interactions avec un écran tactile (tactile display testing) by Sylvain Hanneton

Nouvelle publication : Touching Sounds: Audio Virtual Surfaces

Abstract : This prospective study concerning the perception of audio virtual surfaces (AVSs) was inspired by two different research fields: sensory substitution and haptic and touch perception. We define Audio Virtual Surfaces as regions of space that trigger sounds when the user touches it or moves into it. First, we describe an example of interactive setup using an AVS to simulate a sonic interaction with a virtual water tank. Then, we present an experiment designed to investigate the ability of blindfolded adults to discriminate between concave and convex auditory virtual surfaces using only the gesture-sound interaction. Two groups received different sound feedback, a static one indicating presence in the AVS, and a static+dynamic one (related to the component of the hand velocity tangential to the surface). In order to demonstrate that curvature direction was correctly perceived, we estimated their discrimination thresholds with a psychophysical staircase procedure. Results show that most of the participants were able to learn the task. The best results were obtained with the additional dynamic feedback. Gestural patterns emerged from the interaction, suggesting the use of auditory representation of the virtual object. This work proposes a contribution to the introduction in virtual reality of sonic interactions with auditory virtual objects. The setups we present raise new questions, at both experimental (sensory substitution) and application levels (design of gesture-sound interaction for virtual reality).

Published in the proceedings of the SIVE 2015 satellite conference of VR 2015

Touching Sounds: Audio Virtual Surfaces by Sylvain Hanneton

Hacking the kyto heart rate monitor 2901

Image

The  kyto heart rate monitor 2901 is a small and cheap heart rate monitor that send data to the computer via an USB port. It is delivered with a convenient PC software that give elementary data (heart beat and heart rate). The sensor is an optical sensor fixed to the ear. This device give a stable measure of the heart rate, is non-invasive, and does not require electrodes and complex installation.

 

The Kyoto HRM 2901

The Kyoto HRM 2901

The very simple Kyto software does not run with Linux and we propose here to try to capture the data sent by the device under Linux.

Serial port

When the device is plugged it is listed as « /dev/hidraw0 » (or « /dev/hidraw1 ») in the « /dev » directory. Let’s imagine that the name of the device is « /dev/hidraw0 ». We have first to allow the user to access the serial port :

sudo chown username /dev/hidraw0

Then incoming serial port data can be read by any terminal (tty) software.

The data

The device send a packet of 5 octets (c1 to c5) for each heart beat. We built a small application to extract the decimal values (short int) corresponding to each octet and computed also the heart rate. The following table gives an example of the data transmitted by the device :

Time (ms) HR (beat/min) Data number c1 c2 c3 c4 c5
289 207 5 65499 3 0 22 3
1073 76 5 65500 3 0 20 3
1849 77 5 65501 3 0 22 3
2633 76 5 65502 3 0 22 3
3417 76 5 65503 3 0 29 3
4225 74 5 65504 3 0 42 3
5025 75 5 65505 3 0 43 3
5833 74 5 65506 3 0 49 3
6633 75 5 65507 3 0 41 3
7433 75 5 65508 3 0 41 3
8217 76 5 65509 3 0 26 3
8985 78 5 65510 3 0 6 3
9729 80 5 65511 3 0 65522 2
10473 80 5 65512 3 0 65520 2
11217 80 5 65513 3 0 65523 2
11961 80 5 65514 3 0 65521 2
12705 80 5 65515 3 0 65520 2
13457 79 5 65516 3 0 65530 2
14225 78 5 65517 3 0 8 3
15001 77 5 65518 3 0 10 3
15776 77 5 65519 3 0 23 3
16561 76 5 65520 3 0 24 3
17345 76 5 65521 3 0 19 3
18120 77 5 65522 3 0 21 3
18904 76 5 65523 3 0 23 3
19688 76 5 65524 3 0 23 3
20472 76 5 65525 3 0 25 3
21248 77 5 65526 3 0 17 3
22032 76 5 65527 3 0 26 3
22816 76 5 65528 3 0 24 3
23600 76 5 65529 3 0 26 3
24384 76 5 65530 3 0 24 3
25176 75 5 65531 3 0 34 3
25960 76 5 65532 3 0 29 3
26752 75 5 65533 3 0 31 3
27536 76 5 65534 3 0 26 3
28328 75 5 65535 3 0 25 3
29104 77 5 0 4 0 20 3
29888 76 5 1 4 0 23 3
30664 77 5 2 4 0 22 3
31440 77 5 3 4 0 12 3
32224 76 5 4 4 0 23 3
33000 77 5 5 4 0 22 3
33784 76 5 6 4 0 26 3
34576 75 5 7 4 0 28 3
35360 76 5 8 4 0 25 3
36136 77 5 9 4 0 20 3
36912 77 5 10 4 0 15 3
37696 76 5 11 4 0 26 3
38488 75 5 12 4 0 32 3
39280 75 5 13 4 0 33 3
40080 75 5 14 4 0 37 3
40864 76 5 15 4 0 29 3
41656 75 5 16 4 0 27 3
42448 75 5 17 4 0 35 3
43248 75 5 18 4 0 39 3
44048 75 5 19 4 0 45 3
44856 74 5 20 4 0 47 3
45656 75 5 21 4 0 40 3
46447 75 5 22 4 0 36 3
47239 75 5 23 4 0 30 3
48007 78 5 24 4 0 7 3

It is clear that the two first octets are related to the count of heart beats. The two last octets are related to the heart rate. The third octet is always 0.