Handwriting movement analysis

Handwriting movement analysis is the study and analysis of the movements involved in handwriting and drawing. It forms an important part of graphonomics, which became established after the "International Workshop on Handwriting Movement Analysis" in 1982 in Nijmegen, The Netherlands. It would become the first of a continuing series of International Graphonomics Conferences. The first graphonomics milestone was Thomassen, Keuss, Van Galen, Grootveld (1983).

Handwriting is historically considered the widest taught motor skill. It is also one of the first, and often the only motor skill that children will learn at elementary school. It takes years of practice and maturing before a person has mastered the adult handwriting skill. Handwriting is not considered only as a movement that leaves a visible trace of ink on paper (product) but it can also be considered as a movement (process). Understanding of the handwriting product will not be complete until the handwriting process is understood. Therefore, handwriting movement has been researched since measurement techniques became available.

However, before recording and processing handwriting movements were within reach for those interested in studying handwriting movements, three components were required: Devices to capture handwriting movements, laboratory computers to store and process the movement data, and computer software which enables the researcher to do this under specific experimental paradigms without the need to program untested custom software. Handwriting movement analysis software is also used for studying drawing, eye–hand coordination, or any other situation where the researcher wishes to record movements using a pen.

Elisha Gray's "Telautograph", US Patent 386,815 (1888), followed by four more similar patents (1891–93). See the "Annotated Bibliography in Pen Computing and Handwriting Recognition" by Jean Renard Ward (http://users.erols.com/rwservices/biblio.html). Handwriting could only be transmitted by wire and reproduced elsewhere in real-time. Scripture (1895) developed a writing apparatus that enabled storage of pen positions on paper at 100 Hz. This apparatus permitted measurement of durations of individual handwriting strokes (McAllister, 1900).

Handwriting movements are fast, non-repetitive with a primary frequency around 5 Hz and a bandwidth of about 10 Hz. While sampling rates of 20 Hz would theoretically suffice, up-sampling will be needed to properly visualize the Lissajous-like handwriting and drawing strokes. Higher-than-necessary sampling rates such as 100 Hz are preferred as this would also allow low-pass filtering or smoothed data with reduced equipment and quantization noise by factor √100/20 = √5. A laboratory computer will be needed to store, process, and visualize massive numbers of samples. It took more than 50 years for computers to be available in laboratories. Electronic analog computers were used until digital computers came within reach for research: Wang Laboratories, Digital Equipment Corporation (DEC), Apple Inc., IBM PC (Personal Computer), Norsk Data, Atari, Osborne Computer Corporation, and Data General. Most of these innovative mini and microcomputer companies have discontinued their operation.

The first devices to accurately record handwriting that could be connected to computers were graphics tablet, or digitizer, x-y-tablet, graphics pad, with electronic pens as we know them today. Among the earliest tablets are the Styalator electronic tablet with pen for computer input and handwriting recognition in 1957 ^[1] and the commercial products by Vector General. The Vector General products reported the position of the stylus at 100 Hz and claimed an accuracy of 0.1 cm. Typical tablets sense the position of the pen electromagnetically. Touch-sensitive tablets cannot be used when the hand is resting on the tablet unless they implement some form of hand rejection. Tablets can have a display built in (e.g., as in a tablet PC). Still today, handwriting tablets are the gold standard to record handwriting. Sampling rates used to be 100 Hz until it was decided that the minimum rate for Human Input Devices HID should be at least 133 Hz, bumping the sampling rates up to 133 – 200 Hz. The advantage is a 15%–40% reduction of device noise and quantization noise. The digitizer technology belongs to the most accurate and cost-effective pointing devices. Dynamic accuracy of 0.01 cm at constant frequency is achievable. Opaque tablets are produced by Wacom who also produces display digitizers, Euronovate SA, Hanvon,^[2] VisTablet,^[3] Adesso,^[4] Genius.^[5]

Pen-based handwriting capturing devices ^[6] have been developed but never achieved the accuracy of tablets. Currently many novel systems appear on the market that "record" handwriting, such as those by Anoto, and also EMG-based systems.

Many pen movement recording systems capture, not only the x and y coordinates of the pen top, but also axial pen pressure, an x an y tilt or altitude and azimuth of the pen barrel.

Handwriting movement measurement systems can capture:

• x = Horizontal coordinates; parallel to the baseline in Western script
• y = Vertical coordinates; perpendicular to the horizontal base line and in the writing surface.
• z = Axial pen pressure; as pressure data are often non-linearly related to actual pressure, pen orientation will be needed to estimate normal pressure.

Optionally, digitizers can deliver the orientation of the pen barrel relative to the tablet:

• Altitude = How steep the pen is held; this angle can be used to estimate pen pressure perpendicular to the paper from the axial pen pressure.
• Azimuth = Direction of the pen barrel projected on the x-y plane.

Ideally, each set of coordinates should be sampled simultaneously and at a fixed frequency, and include times stamps per coordinate to correct non-isochronous sampling. Additional features that can be measured by the digitizers (mostly in past models) include pen height, pen barrel rotation, and grip forces (e.g., at the 3 finger grip areas).

Handwriting movements are being studied from many disciplines including kinesiology, human movement science, biomechanics of the hand, fine motor control, handedness, human-computer interaction, visuomotor control, visual feedback, goal-directed movements, drawing, experimental psychology, psychiatry, extrapyramidal symptoms (EPS) or movement side effects due to medication, neurology, movement disorders, Parkinson's disease, dystonia, writer's cramp, physiotherapy, remedial handwriting instruction, occupational therapy, child development, developmental disorders, education, elementary education, home schooling, reeducation, linguistics, language, communication, stuttering, forensic document examination, document analysis, forensic document examination or questioned document examination, signature verification and identification, handwriting image analysis, computer science, artificial intelligence, handwriting recognition, etc.

The next wave consisted of packaged software that could be made available to record handwriting at many locations. Most initial software systems were developed by university researchers who, often, were the only ones capable of using it. Even today, it is a major accomplishment to make software available as a package that can be installed on an unknown computer and can be used after a brief familiarization time by other users who have not been involved.

Over the last several years, software packages have appeared on the market that can be used by many other researchers interested in the field of handwriting movement analysis.

The first handwriting movement analysis offered for sale (around 1980) was CSWin by Science And Motion^[7] and developed by Christian Marquardt and Norbert May in Munich, Germany. CSWin was marketed in Germany and is being used in many German hospitals. It was used for treating 500 writer’s cramp patients. The present company focuses on golf training and was established in 2003 and is run by the owner, Christian Marquardt. Their oldest publication is by Marquardt and Mai (1994).

Another early system was Oasis by KikoSoft, The Netherlands, which was established in 1995 by Peter De Jong. Oasis can be customized by its flexible macro language. In 1998, this system was used to develop an automated test-battery for psychopharmacological research: Orgabat. One of the oldest references to Oasis is De Jong, Hulstijn, Kosterman, and Smits-Engelsman (1996).

At Columbia university, Prof. Dr. Seth Pullman developed Pullman Spiral Acquisition and Analysis.^[8] It is used to test tremor in Parkinson patients. In 2002, Pullman received US Patent 6,454,706: "System and method for clinically assessing motor function". Patients with motor disorders such as Parkinson's disease draw spirals. The software calculates first-order smoothness, second-order smoothness, tightness of the spiral, zero-crossing rate, second-order zero-crossing rate, and derives from these score a degree of severity score. Their system enables objectively assessing motor function by physicians who are not skilled or experienced in evaluating motor disorders, for example general practitioners or pediatricians who are not certified in the practice of neurology. An early publication about his spiral analysis is by Pullman (1998).

Another Handwriting Analysis system is Neuroskill by Verifax, Boulder, Colorado, USA, which was founded in 1990 by Dr. Ruth Shrairman and Alex Landau. Neuroskill was designed for biometric measurement, security purposes, and Parkinson medication effects and has many applications in movement disorders. Verifax began operations with the aim of developing a biometric tool for the verification of signatures from a distance (VeriFax Autograph Technology). Verifax developed two more applications using application-specific modifications of their customized Neuroskill software: Applications for substance abuse screening and detection, monitoring for toxic inhalants and environmental distress, and accurate signature identification for security/privacy protection and forgery detection. Target markets could include neuromuscular disease centers, drug and alcohol abuse clinics, occupational health centers and the security industry. In the process, they applied their technology to biometric measurements as a clinical monitoring tool for physicians investigating neuromuscular diseases.

In 2003, NeuroSkill received a US patent 6,546,134: "System for assessment of fine motor control in humans". Their method estimates stability, smoothness and synchronization of the writer's motion as quantifying measures of the neurological function using their Correlation Function Analysis (CFA) of behavioral signals. CFA returns numerical scores and charts expressing stability of the handwriting strokes and the characteristics of the phase distortions in reproducing cursive samples.

Another application was to evaluate persons with critical skills (e.g., airline pilots, bus drivers) for physical and mental performance impairments caused by stress, physiological disorders, and alcohol and drug abuse using their proprietary VeriFax Impairoscope writing instrument. This last application raised the possibility of using a space-qualified Impairoscope variant to evaluate astronaut performance with respect to the impacts of stress, fatigue, excessive workload, build-up of toxic chemicals within the space habitat, etc.

In 2009, the iNeuroskill web portal was established under a new business entity: iNeuroskill. The website enables Parkinson patients to upload their signatures that were recorded using a digitizing tablet. They receive immediate feedback regarding their fine motor function in the form of a chart analysis.^[9] Their oldest article is by Morgenthaler, Shrairman, and Landau (1998).

MovAlyzeR was developed by NeuroScript,^[10] Tempe, AZ, USA. NeuroScript was founded in 1997 by Prof. Dr. George Stelmach, who has since retired, and Dr. Hans-Leo Teulings. In 1999 Gregory M. Baker joined as MovAlyzeR’s designer and implementer. This handwriting movement analysis software is the first to demonstrate that it can discern movement side-effects due to schizophrenia medication better than with any conventional evaluation method used in psychiatry today (international patent pending) (Caligiuri et al., 2009a, b). MovAlyzeR is currently the only handwriting movement analysis software that is certified for Microsoft Windows XP and Vista. It can be integrated with MATLAB and perform image processing on scanned handwriting exemplars. It is used in fields ranging from research in human movement sciences, kinesiology, psychology, education, aging research, psychiatry, neurology, occupational therapy, forensic document examination, computer science (handwriting recognition, signature verification), to educational demonstrations or student projects in these fields. The oldest references to MovAlyzeR are Teulings and Romero (2003), Teulings and Van Gemmert, (2003), Romero and Teulings (2003).

At the University of Haifa, Dr. Sara Rosenblum and Patricia L (Tamar) Weiss and colleagues developed a computerized handwriting evaluation system called POET: Penmanship Objective Evaluation Tool using MATLAB. It was used to administer visual stimuli and to record and analyze handwriting movements. They researched the Air Phenomenon: Pen movements above the paper (air strokes). The oldest mention of POET is in Rosenblum, Parush, Epstain, and Weiss (2003).

Soon afterward, POET was developed further and renamed to ComPET: Computerized Penmanship Evaluation Tool. It is used to study children with dysgraphia, Developmental Coordination Disorders (DCD) and adults with several pathologies as Multiple Sclerosis (MS) Depression, Alzheimer, Parkinson as well as aging effects.^[11] It is also used with verbal-based lie detection technology such as the polygraph.

Eye and Pen ^[12] is a software developed at the University of Poitiers (France) to synchronously record handwriting and eye mouvements. Its first version aims at studying periods of pause and writing in text production (G-Studio; 1994). After a complete rewriting to support MS-Windows, eye mouvement recording was added to get insight into reading during writing and information retrieval. Its evolution is the fruit of collaborations with handwriting researchers, mostly of the psychology lab of the University of Poitiers (Denis Alamargot, Eric lambert, Cyril Perret, Thierry Olive). Now it integrates audio recording, synchronization signals (from/to other computers, software or devices), data tagging tools as well as various data filtering and extraction modules, opening ways to multimodal researches. After a commercial distribution (2004-2018), it is now freely distributed. The oldest references to Eye and Pen are Chesnet and Alamargot (2005), Alamargot, Chesnet, Dansac, Ros, C. (2006).

MedDraw is a computer-based drawing-task diagnosis and rehabilitation system project between the University of Kent, UK and the University of Rouen, France, coordinated by Dr. Richard Guest and started in 2003. The project aims to develop a robust, state-of-the art, yet easy to use clinical system producing objective diagnostic recommendations across a range of clinical conditions. Their focus is to detect spatial neglect in the visual field and organization of movement disorder. They will remain focused on drawing-based diagnosis of these disorders. The first research publications that mention MedDraw are by Kaplani, Guest, and Fairhurst (2005), and by Glenat, Heutte, Paquet, and Mellier (2005). The programme goes offline since 2006.

The Extended Drawing Test is a computerized graphonomic assessment for arm and hand function. The EDT measures the ability of the subject to draw vertical lines, with both the left and right hands. To compare performance between gross arm movements and fine finger control, the subjects draw lines holding either the tablet's pen (held by the fingers) or a mouse (held by the whole hand). The latter movements do not include finger movements. Norms have been established for 3- to 70-year-old healthy persons. Deviations from the healthy norms will reflect different pathologies for different patient groups, e.g., hemiplegic stroke patients. The first paper mentioning EDT is Vuillermot, Pescatore, Holper, Kiper, and Eng (2009).

• Questioned document examination
• Calligraphy
• Doodle
• Graphology
• Graphonomics
• MovAlyzeR handwriting movement analysis software

• Alamargot, D., Chesnet, D., Dansac, C. & Ros, C. (2006). Eye and Pen: a new device to study reading during writing. Behaviour Research Methods, Instruments and Computers, 38(2), 287-299. Eye and Pen: A new device for studying reading during writing
• Caligiuri, M.P., Teulings, H.L., Dean, C.E., Niculescu, A.B., Lohr, J.B. (2009a). Handwriting Movement Kinematics for Quantifying EPS in Patients Treated with Atypical Antipsychotics. Psychiatry Research (in press).
• Caligiuri, M.P., Teulings, H.L., Dean, C.E., Niculescu, A.B., Lohr, J.B. (2009b). Handwriting Movement Analyses for Monitoring Drug-Induced Motor Side Effects in Schizophrenia Patients Treated with Risperidone. Human Movement Sciences (in press).
• Chesnet, D. & Alamargot, D. (2005). Analyse en temps réel des activités oculaires et grapho-motrices du scripteur : intérêt du dispositif « Eye and Pen ». L'Année Psychologique, 105(3), 477-520. Analyse en temps réel des activités oculaires et grapho-motrices du scripteur : intérêt du dispositif « Eye and Pen »
• Chesnet D., Guillabert F., & Espéret E.(1994). G-STUDIO : un logiciel pour l'étude en temps réel des paramètres temporels de la production écrite. L'Année Psychologique, 94(2),283-293. G-STUDIO : un logiciel pour l'étude en temps réel des paramètres temporels de la production écrite
• De Jong, W.P., Hulstijn, W., Kosterman, B.J.M., & Smits-Engelsman, B.C.M. (1996). OASIS software and its application in experimental handwriting research. In M.L. Simner, C.G. Leedham, & A.J.W.M. Thomassen (Eds.), Handwriting and drawing research: Basic and applied issues (pp. 429–440). Amsterdam: IOS Press.
• Glenat, S., Heutte, L., Paquet, T., Mellier, D. (2005). Computer-Based Diagnosis of Dyspraxia: the MEDDRAW project, to appear in 12th Conference of the International Graphonomics Society, IGS 2005, Salerno, Italy, June 2005.
• Kaplani, E., Guest, R.M., Fairhurst, M.C. (2005). Feature-based Assessment of Visuospatial Neglect Severity in a Computer-based Line Cancellation Task, 12th Conference of the International Graphonomics Society, IGS 2005, Salerno, Italy, June 2005.
• Marquardt, C., Mai, N. (1994). A computational procedure for movement analysis in handwriting. J Neurosci Methods. Apr;52(1):39–45.
• McAllister, C.N. (1900). Research: A movement used in handwriting. Yale Psychological Laboratory, 8, 21–63.
• Morgenthaler, G.W., Shrairman, R., Landau, A. (1998). Space technology and applications international forum – 1998. AIP Conference Proceedings, Volume 420, pp. 736–742 (1998).
• Pullman, SL. (1998). Spiral analysis: a new technique for measuring tremor with a digitizing tablet. Mov Disord. 13 Suppl 3:85–9.
• Romero, D.H., & Teulings, H.L. (2003). Submovement analysis in goal-directed movements. H.L. Teulings, Van Gemmert, A.W.A. (Eds.), Proceedings of the 11th Conference of the International Graphonomics Society (IGS2003), 2–3 November 2003, Scottsdale, Arizona, USA. ISBN 0-9746365-0-9. (p. 103–106).
• Rosenblum, S., Parush, S., Epstain, L., Weiss, P.L. (2003). Process Versus Product Evaluation of Poor Handwriting among Children with Developmental Dysgraphia and ADHD. In H.L. Teulings & A.W.A. Van Gemmert. Proceedings of the 11th Conference of the International Graphonomics Society. USA; Scottsdale Arizona. p. 169–173.
• Scripture, E.W. (1895). New apparatus. Studies from the Yale Psychological Laboratory, III, 106.
• Teulings, H.L., & Maarse, F.J. (1984). Digital recording and processing of handwriting movements. Human Movement Science, 3, 193–217.
• Teulings, H.L. & Romero, D.H. (2003). Submovement analysis in learning cursive handwriting or block print. In H.L. Teulings, A.W.A. Van Gemmert (Eds.), Proceedings of the 11th Conference of the International Graphonomics Society (IGS2003), 2–3 November 2003, Scottsdale, Arizona, USA. ISBN 0-9746365-0-9. (p. 107–110).
• Teulings, H.L. & Van Gemmert, W.A. (2003). Goal-directed movements in menu selection in computer-user-interfaces. In H.L. Teulings, Van Gemmert, A.W.A. (Eds.), Proceedings of the 11th Conference of the International Graphonomics Society (IGS2003), 2–3 November 2003, Scottsdale, Arizona, USA. ISBN 0-9746365-0-9. (p. 99–102).
• Thomassen, A.J. W.M., Keuss, P.J.G., Van Galen, G.P., Grootveld, C. (Eds.) (1983). Motor Aspects of Handwriting. Acta Psychologica, Vol. 54.
• Vuillermot, S, Pescatore, A, Holper, L, Kiper, D and Eng, K. (2009). An Extended Drawing Test for the Assessment of Arm and Hand Function with a Performance Invariant for Healthy Subjects. Journal of Neuroscience Methods 177(2): 452–460.

• Annotated bibliography of references to handwriting recognition and pen computing
• Notes on the History of Pen-based Computing (Youtube)